NSVQ for 3DGS Compression

• NSVQ is a differentiable quantization framework that compresses 3D Gaussian Splatting scenes by jointly learning discrete attribute codebooks using a noise substitution mechanism.
• It selectively preserves high-precision attributes while compressing others via separate codebooks, achieving up to a 45× reduction in memory with minimal rendering quality loss.
• NSVQ enables efficient gradient flow, integrates seamlessly with standard 3DGS pipelines, and boosts rendering speed for bandwidth- and latency-sensitive applications.

Noise-Substituted Vector Quantization (NSVQ) is a differentiable quantization framework introduced for compressing 3D Gaussian Splatting (3DGS) scene representations. 3DGS relies on millions of “splats” (anisotropic 3D Gaussians) parameterized by high-dimensional float attributes, which results in prohibitive storage requirements—typically around 1 GB per scene. NSVQ addresses this limitation by jointly learning discrete attribute codebooks and attribute assignments while preserving end-to-end differentiability using a noise-injection mechanism. This permits substantial memory reduction with minimal loss in rendering quality and guarantees compatibility with standard 3DGS pipelines (Wang et al., 3 Apr 2025).

1. Model Framework and Attribute Factorization

A standard 3DGS model represents a scene as $N$ splats, each $G_i$ defined by a vector of 59 real-valued attributes:

• $x_i \in \mathbb{R}^3$: 3D position
• $o_i \in \mathbb{R}$: opacity
• $s_i \in \mathbb{R}^3$: scaling parameters
• $r_i \in \mathbb{R}^4$: rotation (covariance)
• $c_i \in \mathbb{R}^3$: color
• $sh_i \in \mathbb{R}^{45}$: spherical-harmonic coefficients

NSVQ-GS preserves $x$ and $o$ in full precision, while the attributes $(s, r, c, sh)$ are compressed via four separate codebooks. Each codebook $C_*\in\mathbb{R}^{2^{K_*} \times D_*}$ discretizes its respective attribute using $2^{K_*}$ codes, where $K_*$ is the bitwidth:

Attribute	Codebook ($C_*$)	Dimensionality ($D_*$)	Bitwidth ($K_*$)
$s$	$C_s$	3	$K_s$
$r$	$C_r$	4	$K_r$
$c$	$C_c$	3	$K_c$
$sh$	$C_{sh}$	45	$K_{sh}$

Each splat stores the code indices $(k_{si}, k_{ri}, k_{ci}, k_{shi})$, i.e., only $K_s + K_r + K_c + K_{sh}$ bits per splat for these attributes (Wang et al., 3 Apr 2025).

2. Differentiable Quantization via Noise Substitution

Hard vector quantization by $\arg\min_{e\in C}\|z-e\|^2$ is non-differentiable due to the discrete assignment. To enable backpropagation, NSVQ replaces the attribute vector by a noisy substitute:

$$\tilde z^q = z + \|z - e_i\|_2 \cdot \frac{n}{\|n\|_2}, \quad n \sim \mathcal{N}(0, I_D)$$

Here $z$ is the current attribute vector, $e_i$ is its closest codebook element, and $n$ is a random vector. Both $\|z - e_i\|_2$ and $n/\|n\|_2$ are differentiable with respect to $z$ and $e_i$, thus gradients flow from the loss to both the encoder and the codebook entries. This circumvents the need for a straight-through estimator (Wang et al., 3 Apr 2025).

During training, this mechanism is applied independently to each attribute ($s, r, c, sh$) via their respective codebooks.

3. Training Objective and Optimization Schedule

The joint optimization combines:

• Reconstruction loss: $L_{\mathrm{recon}}$ (per-pixel $L_2$ error between rendered and ground-truth images)
• Opacity regularization: $L_{\mathrm{opacity}} = \sum_{i=1}^N o_i$ (used for pruning low-opacity splats)
• (Optional) VQ commitment loss: $$L_{VQ} = L_{VQ,s} + L_{VQ,r} + L_{VQ,c} + L_{VQ,sh}$$ (as in VQ-VAE, to encourage codebook utilization)

The combined loss:

$$L = L_{\mathrm{recon}} + \lambda_{\mathrm{opacity}}\,L_{\mathrm{opacity}} + \beta (L_{VQ,s} + L_{VQ,r} + L_{VQ,c} + L_{VQ,sh})$$

where $\lambda_{\mathrm{opacity}}$ and $\beta$ control regularization during pruning and codebook stabilization, respectively. In fine-tuning, assignments are frozen and $\beta$ is set to zero (Wang et al., 3 Apr 2025).

The four-phase training schedule is:

1. Warm-up: Full precision rendering and latent optimization
2. Pruning: Remove low-opacity splats
3. Vector quantization: Train with NSVQ and update codebooks
4. Fine-tuning: Freeze quantization, optimize only model parameters

4. Pseudocode for NSVQ Training Loop

The training procedure is as follows:

Initialize 3DGS model; initialize codebooks C_s, C_r, C_c, C_sh via K-means. for iter = 1 to 45_000: if iter <= 15_000: # Warm-up render with full-precision Gaussians L ← L_recon elif iter <= 20_000: # Pruning render full-precision L ← L_recon + λ_opacity ⋅ sum(o_i) prune low-opacity splats elif iter <= 43_000: # Vector quantization for each splat i: compute z_s = s_i; nearest code e_s = C_s[k_si] tilde_s_i = NSVQ(z_s, e_s) # Repeat for r, c, sh render with quantized attributes L ← L_recon + β⋅L_{VQ} backpropagate L; update model and codebooks every M batches: replace unused codes else: # Fine-tuning fix code indices; tilde_s_i = e_s render, L ← L_recon update only model parameters

(Wang et al., 3 Apr 2025)

5. Compression Ratio, Reconstruction Fidelity, and Rendering Speed

NSVQ-GS achieves significant storage savings by storing only code indices and codebooks:

• Original memory: $N$ splats × 59 floats × 32 bits
• Compressed: $$N \times (3 \times 32 + 1 \times 32 + K_s + K_r + K_c + K_{sh})$$ bits for splats, plus codebooks

The compression ratio is defined as:

$$\text{Compression Ratio} = \frac{\text{Original Memory}}{\text{Compressed Memory}}$$

For NSVQ-GS(16k) ($K_s = 14$), on Mip-NeRF360:

Model	PSNR	SSIM	LPIPS	Size	Compression Ratio	FPS (rendering)
NSVQ-GS(16k)	27.28	0.807	0.239	16.4 MB	$\approx 45\times$	103
CompGS(16k)	27.03	0.804	0.243	18 MB
Baseline 3DGS	—	—	—	$\sim$1 GB ($\sim$734 MB float)	—	43

Rendering throughput approximately doubles after compression, attributed to reduced per-splat data transfer and cache-friendly codebook access (Wang et al., 3 Apr 2025).

6. Codebook Utilization and Gradient Flow

NSVQ’s differentiable formulation enables gradient flow w.r.t. both attributes and codebook vectors. Only active codes receive gradients; hence, to prevent codebook collapse, rarely used codes are periodically replaced by randomly perturbed copies of active codes during training.

This mechanism negates the need for straight-through estimators and ensures joint optimization stability. In fine-tuning, code assignments become fixed and the noise-injection is removed, yielding deterministic attribute decoding at inference (Wang et al., 3 Apr 2025).

7. Compatibility, Deployment, and Practical Implications

The final NSVQ-GS model is a standard list of Gaussians with associated codebooks and per-splat code indices. All inference-time operations—codebook lookup, attribute decoding, and $\alpha$-blending—are compatible with existing 3DGS viewers (CPU or GPU) and do not require auxiliary neural decoders. This design ensures seamless integration with web-based viewers, 3D editors, and SLAM systems. The memory and speed improvements are preconditions for practical deployment in bandwidth- or latency-sensitive environments.

A plausible implication is that NSVQ-GS enables large-scale 3D scene distribution and complex scene rendering with commodity hardware, aligning 3DGS compression performance with industry application requirements (Wang et al., 3 Apr 2025).