---
title: NSVQ for 3DGS Compression
url: https://www.emergentmind.com/topics/noise-substituted-vector-quantization-nsvq
type: topic
---

# NSVQ for 3DGS Compression

Noise-Substituted Vector Quantization (NSVQ) is a differentiable quantization framework introduced for compressing 3D Gaussian Splatting (3DGS) scene representations. 3DGS relies on millions of “splats” (anisotropic 3D Gaussians) parameterized by high-dimensional float attributes, which results in prohibitive storage requirements—typically around 1 GB per scene. NSVQ addresses this limitation by jointly learning discrete attribute codebooks and attribute assignments while preserving end-to-end differentiability using a noise-injection mechanism. This permits substantial memory reduction with minimal loss in rendering quality and guarantees compatibility with standard 3DGS pipelines [2504.03059].

## 1. Model Framework and Attribute Factorization

A standard 3DGS model represents a scene as $N$ splats, each $G_i$ defined by a vector of 59 real-valued attributes:

- $x_i \in \mathbb{R}^3$: 3D position
- $o_i \in \mathbb{R}$: opacity
- $s_i \in \mathbb{R}^3$: scaling parameters
- $r_i \in \mathbb{R}^4$: rotation (covariance)
- $c_i \in \mathbb{R}^3$: color
- $sh_i \in \mathbb{R}^{45}$: spherical-harmonic coefficients

NSVQ-GS preserves $x$ and $o$ in full precision, while the attributes $(s, r, c, sh)$ are compressed via four separate codebooks. Each codebook $C_*\in\mathbb{R}^{2^{K_*} \times D_*}$ discretizes its respective attribute using $2^{K_*}$ codes, where $K_*$ is the bitwidth:

| Attribute | Codebook ($C_*$)  | Dimensionality ($D_*$) | Bitwidth ($K_*$) |
|-----------|-------------------|------------------------|------------------|
| $s$       | $C_s$             | 3                      | $K_s$            |
| $r$       | $C_r$             | 4                      | $K_r$            |
| $c$       | $C_c$             | 3                      | $K_c$            |
| $sh$      | $C_{sh}$          | 45                     | $K_{sh}$         |

Each splat stores the code indices $(k_{si}, k_{ri}, k_{ci}, k_{shi})$, i.e., only $K_s + K_r + K_c + K_{sh}$ bits per splat for these attributes [2504.03059].

## 2. Differentiable Quantization via Noise Substitution

Hard vector quantization by $\arg\min_{e\in C}\|z-e\|^2$ is non-differentiable due to the discrete assignment. To enable backpropagation, NSVQ replaces the attribute vector by a noisy substitute:

$$\tilde z^q = z + \|z - e_i\|_2 \cdot \frac{n}{\|n\|_2}, \quad n \sim \mathcal{N}(0, I_D)$$

Here $z$ is the current attribute vector, $e_i$ is its closest codebook element, and $n$ is a random vector. Both $\|z - e_i\|_2$ and $n/\|n\|_2$ are differentiable with respect to $z$ and $e_i$, thus gradients flow from the loss to both the encoder and the codebook entries. This circumvents the need for a straight-through estimator [2504.03059].

During training, this mechanism is applied independently to each attribute ($s, r, c, sh$) via their respective codebooks.

## 3. Training Objective and Optimization Schedule

The joint optimization combines:

- Reconstruction loss: $L_{\mathrm{recon}}$ (per-pixel $L_2$ error between rendered and ground-truth images)
- Opacity regularization: $L_{\mathrm{opacity}} = \sum_{i=1}^N o_i$ (used for pruning low-opacity splats)
- (Optional) VQ commitment loss: $L_{VQ} = L_{VQ,s} + L_{VQ,r} + L_{VQ,c} + L_{VQ,sh}$ (as in VQ-VAE, to encourage codebook utilization)

The combined loss:

\[
L = L_{\mathrm{recon}} 
 + \lambda_{\mathrm{opacity}}\,L_{\mathrm{opacity}} 
 + \beta (L_{VQ,s} + L_{VQ,r} + L_{VQ,c} + L_{VQ,sh})
\]

where $\lambda_{\mathrm{opacity}}$ and $\beta$ control regularization during pruning and codebook stabilization, respectively. In fine-tuning, assignments are frozen and $\beta$ is set to zero [2504.03059].

The four-phase training schedule is:

1. Warm-up: Full precision rendering and latent optimization
2. Pruning: Remove low-opacity splats
3. Vector quantization: Train with NSVQ and update codebooks
4. Fine-tuning: Freeze quantization, optimize only model parameters

## 4. Pseudocode for NSVQ Training Loop

The training procedure is as follows:

```python
# Pseudocode for NSVQ-GS training
Initialize 3DGS model; initialize codebooks C_s, C_r, C_c, C_sh via K-means.

for iter = 1 to 45_000:
    if iter <= 15_000:  # Warm-up
        render with full-precision Gaussians
        L ← L_recon
    elif iter <= 20_000:  # Pruning
        render full-precision
        L ← L_recon + λ_opacity ⋅ sum(o_i)
        prune low-opacity splats
    elif iter <= 43_000:  # Vector quantization
        for each splat i:
            compute z_s = s_i; nearest code e_s = C_s[k_si]
            tilde_s_i = NSVQ(z_s, e_s)
            # Repeat for r, c, sh
        render with quantized attributes
        L ← L_recon + β⋅L_{VQ}
        backpropagate L; update model and codebooks
        every M batches: replace unused codes
    else:  # Fine-tuning
        fix code indices; tilde_s_i = e_s
        render, L ← L_recon
        update only model parameters

# Save final: (x_i, o_i, k_si, k_ri, k_ci, k_shi) per splat, all codebooks
```
[2504.03059]

## 5. Compression Ratio, Reconstruction Fidelity, and Rendering Speed

NSVQ-GS achieves significant storage savings by storing only code indices and codebooks:

- Original memory: $N$ splats × 59 floats × 32 bits
- Compressed: $N \times (3 \times 32 + 1 \times 32 + K_s + K_r + K_c + K_{sh})$ bits for splats, plus codebooks

The compression ratio is defined as:

\[
\text{Compression Ratio} = \frac{\text{Original Memory}}{\text{Compressed Memory}}
\]

For NSVQ-GS(16k) ($K_s = 14$), on Mip-NeRF360:

| Model             | PSNR  | SSIM   | LPIPS | Size    | Compression Ratio | FPS (rendering) |
|-------------------|-------|--------|-------|---------|------------------|-----------------|
| NSVQ-GS(16k)      | 27.28 | 0.807  | 0.239 | 16.4 MB |   $\approx 45\times$  | 103             |
| CompGS(16k)       | 27.03 | 0.804  | 0.243 | 18 MB   |                  |                 |
| Baseline 3DGS     |  —    |   —    |   —   | $\sim$1 GB ($\sim$734 MB float) |  — | 43              |

Rendering throughput approximately doubles after compression, attributed to reduced per-splat data transfer and cache-friendly codebook access [2504.03059].

## 6. Codebook Utilization and Gradient Flow

NSVQ’s differentiable formulation enables gradient flow w.r.t. both attributes and codebook vectors. Only active codes receive gradients; hence, to prevent codebook collapse, rarely used codes are periodically replaced by randomly perturbed copies of active codes during training.

This mechanism negates the need for straight-through estimators and ensures joint optimization stability. In fine-tuning, code assignments become fixed and the noise-injection is removed, yielding deterministic attribute decoding at inference [2504.03059]. 

## 7. Compatibility, Deployment, and Practical Implications

The final NSVQ-GS model is a standard list of Gaussians with associated codebooks and per-splat code indices. All inference-time operations—codebook lookup, attribute decoding, and $\alpha$-blending—are compatible with existing 3DGS viewers (CPU or GPU) and do not require auxiliary neural decoders. This design ensures seamless integration with web-based viewers, 3D editors, and SLAM systems. The memory and speed improvements are preconditions for practical deployment in bandwidth- or latency-sensitive environments.

*A plausible implication is that NSVQ-GS enables large-scale 3D scene distribution and complex scene rendering with commodity hardware, aligning 3DGS compression performance with industry application requirements* [2504.03059].

Source: https://www.emergentmind.com/topics/noise-substituted-vector-quantization-nsvq