NSVQ for 3DGS Compression

• NSVQ is a differentiable quantization framework that compresses 3D Gaussian Splatting scenes by jointly learning discrete attribute codebooks using a noise substitution mechanism.
• It selectively preserves high-precision attributes while compressing others via separate codebooks, achieving up to a 45× reduction in memory with minimal rendering quality loss.
• NSVQ enables efficient gradient flow, integrates seamlessly with standard 3DGS pipelines, and boosts rendering speed for bandwidth- and latency-sensitive applications.

Noise-Substituted Vector Quantization (NSVQ) is a differentiable quantization framework introduced for compressing 3D Gaussian Splatting (3DGS) scene representations. 3DGS relies on millions of “splats” (anisotropic 3D Gaussians) parameterized by high-dimensional float attributes, which results in prohibitive storage requirements—typically around 1 GB per scene. NSVQ addresses this limitation by jointly learning discrete attribute codebooks and attribute assignments while preserving end-to-end differentiability using a noise-injection mechanism. This permits substantial memory reduction with minimal loss in rendering quality and guarantees compatibility with standard 3DGS pipelines (Wang et al., 3 Apr 2025).

1. Model Framework and Attribute Factorization

A standard 3DGS model represents a scene as $N$ splats, each $G_i$ defined by a vector of 59 real-valued attributes:

• $x_i \in \mathbb{R}^3$: 3D position
• $o_i \in \mathbb{R}$: opacity
• $s_i \in \mathbb{R}^3$: scaling parameters
• $r_i \in \mathbb{R}^4$: rotation (covariance)
• $c_i \in \mathbb{R}^3$: color
• $sh_i \in \mathbb{R}^{45}$: spherical-harmonic coefficients

NSVQ-GS preserves $x$ and $o$ in full precision, while the attributes $G_i$0 are compressed via four separate codebooks. Each codebook $G_i$1 discretizes its respective attribute using $G_i$2 codes, where $G_i$3 is the bitwidth:

Attribute	Codebook ($G_i$4)	Dimensionality ($G_i$5)	Bitwidth ($G_i$6)
$G_i$7	$G_i$8	3	$G_i$9
$x_i \in \mathbb{R}^3$0	$x_i \in \mathbb{R}^3$1	4	$x_i \in \mathbb{R}^3$2
$x_i \in \mathbb{R}^3$3	$x_i \in \mathbb{R}^3$4	3	$x_i \in \mathbb{R}^3$5
$x_i \in \mathbb{R}^3$6	$x_i \in \mathbb{R}^3$7	45	$x_i \in \mathbb{R}^3$8

Each splat stores the code indices $x_i \in \mathbb{R}^3$9, i.e., only $o_i \in \mathbb{R}$0 bits per splat for these attributes (Wang et al., 3 Apr 2025).

2. Differentiable Quantization via Noise Substitution

Hard vector quantization by $o_i \in \mathbb{R}$1 is non-differentiable due to the discrete assignment. To enable backpropagation, NSVQ replaces the attribute vector by a noisy substitute:

$o_i \in \mathbb{R}$2

Here $o_i \in \mathbb{R}$3 is the current attribute vector, $o_i \in \mathbb{R}$4 is its closest codebook element, and $o_i \in \mathbb{R}$5 is a random vector. Both $o_i \in \mathbb{R}$6 and $o_i \in \mathbb{R}$7 are differentiable with respect to $o_i \in \mathbb{R}$8 and $o_i \in \mathbb{R}$9, thus gradients flow from the loss to both the encoder and the codebook entries. This circumvents the need for a straight-through estimator (Wang et al., 3 Apr 2025).

During training, this mechanism is applied independently to each attribute ($s_i \in \mathbb{R}^3$0) via their respective codebooks.

3. Training Objective and Optimization Schedule

The joint optimization combines:

• Reconstruction loss: $s_i \in \mathbb{R}^3$1 (per-pixel $s_i \in \mathbb{R}^3$2 error between rendered and ground-truth images)
• Opacity regularization: $s_i \in \mathbb{R}^3$3 (used for pruning low-opacity splats)
• (Optional) VQ commitment loss: $s_i \in \mathbb{R}^3$4 (as in VQ-VAE, to encourage codebook utilization)

The combined loss:

$s_i \in \mathbb{R}^3$5

where $s_i \in \mathbb{R}^3$6 and $s_i \in \mathbb{R}^3$7 control regularization during pruning and codebook stabilization, respectively. In fine-tuning, assignments are frozen and $s_i \in \mathbb{R}^3$8 is set to zero (Wang et al., 3 Apr 2025).

The four-phase training schedule is:

1. Warm-up: Full precision rendering and latent optimization
2. Pruning: Remove low-opacity splats
3. Vector quantization: Train with NSVQ and update codebooks
4. Fine-tuning: Freeze quantization, optimize only model parameters

4. Pseudocode for NSVQ Training Loop

The training procedure is as follows:

$r_i \in \mathbb{R}^4$7 (Wang et al., 3 Apr 2025)

5. Compression Ratio, Reconstruction Fidelity, and Rendering Speed

NSVQ-GS achieves significant storage savings by storing only code indices and codebooks:

• Original memory: $s_i \in \mathbb{R}^3$9 splats × 59 floats × 32 bits
• Compressed: $r_i \in \mathbb{R}^4$0 bits for splats, plus codebooks

The compression ratio is defined as:

$r_i \in \mathbb{R}^4$1

For NSVQ-GS(16k) ($r_i \in \mathbb{R}^4$2), on Mip-NeRF360:

Model	PSNR	SSIM	LPIPS	Size	Compression Ratio	FPS (rendering)
NSVQ-GS(16k)	27.28	0.807	0.239	16.4 MB	$r_i \in \mathbb{R}^4$3	103
CompGS(16k)	27.03	0.804	0.243	18 MB
Baseline 3DGS	—	—	—	$r_i \in \mathbb{R}^4$41 GB ($r_i \in \mathbb{R}^4$5734 MB float)	—	43

Rendering throughput approximately doubles after compression, attributed to reduced per-splat data transfer and cache-friendly codebook access (Wang et al., 3 Apr 2025).

6. Codebook Utilization and Gradient Flow

NSVQ’s differentiable formulation enables gradient flow w.r.t. both attributes and codebook vectors. Only active codes receive gradients; hence, to prevent codebook collapse, rarely used codes are periodically replaced by randomly perturbed copies of active codes during training.

This mechanism negates the need for straight-through estimators and ensures joint optimization stability. In fine-tuning, code assignments become fixed and the noise-injection is removed, yielding deterministic attribute decoding at inference (Wang et al., 3 Apr 2025).

7. Compatibility, Deployment, and Practical Implications

The final NSVQ-GS model is a standard list of Gaussians with associated codebooks and per-splat code indices. All inference-time operations—codebook lookup, attribute decoding, and $r_i \in \mathbb{R}^4$6-blending—are compatible with existing 3DGS viewers (CPU or GPU) and do not require auxiliary neural decoders. This design ensures seamless integration with web-based viewers, 3D editors, and SLAM systems. The memory and speed improvements are preconditions for practical deployment in bandwidth- or latency-sensitive environments.

A plausible implication is that NSVQ-GS enables large-scale 3D scene distribution and complex scene rendering with commodity hardware, aligning 3DGS compression performance with industry application requirements (Wang et al., 3 Apr 2025).