---
title: Sparse Neural Radiance Grids (SNeRG)
url: https://www.emergentmind.com/topics/sparse-neural-radiance-grids-snerg
type: topic
---

# Sparse Neural Radiance Grids (SNeRG)

Sparse Neural Radiance Grids (SNeRG) are a hybrid volumetric scene representation and rendering architecture that enables real-time, photorealistic view synthesis of complex 3D scenes using compact, block-sparse voxel grids augmented with learned feature vectors. SNeRG transforms a trained Neural Radiance Field (NeRF)—which typically requires hundreds of multilayer perceptron (MLP) evaluations per-ray for image synthesis—into a sparse, quantized 3D grid structure suitable for rapid memory access and efficient GPU-based rendering. This approach achieves high-quality view-dependent effects and fine geometric detail, while significantly reducing memory footprint and computation for commodity hardware deployment [2103.14645]. The SNeRG design has also inspired related work in fast, 3D-aware generative modeling using sparse voxel grids [2206.07695].

## 1. Architectural Reformulation and Core Principles

SNeRG introduces a deferred NeRF architecture, structurally departing from the canonical NeRF MLP. Standard NeRF takes as input a 3D point $\mathbf{x} = \mathbf{r}(t)$ and a viewing direction $\mathbf{d}$, outputting density $\sigma(\mathbf{x})$ and color $c(\mathbf{x}, \mathbf{d})$. The rendered color along a ray is computed via the standard volume rendering integral:
\[
\hat{C}(\mathbf{r}) = \int_0^{\infty} T(t) \sigma(\mathbf{r}(t)) c(\mathbf{r}(t),\mathbf{d})\,dt,\quad
T(t) = \exp\left(-\int_0^t \sigma(\mathbf{r}(s))\,ds\right)
\]
which is then typically approximated with discrete quadrature [2103.14645].

The deferred SNeRG architecture splits the NeRF MLP into two stages:

- **MLP₁**: $\mathbb{R}^3 \rightarrow (\sigma, \mathbf{c}_d, \mathbf{v})$
  - Outputs per-voxel density $\sigma$, diffuse color $\mathbf{c}_d \in [0,1]^3$, and a low-dimensional feature vector $\mathbf{v} \in [0,1]^4$.
- **MLP₂**: $(\mathbf{V}, \mathbf{d}) \rightarrow \Delta \mathbf{c}_{spec}$
  - Small, per-pixel MLP (2 layers, 16 channels) that computes the view-dependent color residual from the opacity-weighted, ray-integrated feature $\mathbf{V}$ and viewing direction.

Rendering proceeds by accumulating the (opacity-weighted) diffuse color and features along each ray and performing a deferred shading pass for view-dependent appearance:
\[
\hat{\mathbf{C}}_d(\mathbf{r}) = \sum_k T(t_k) \sigma(t_k) \delta_k \mathbf{c}_d(t_k), \quad
\mathbf{V}(\mathbf{r}) = \sum_k T(t_k)\sigma(t_k)\delta_k \mathbf{v}(t_k), \quad
\hat{\mathbf{C}}(\mathbf{r}) = \hat{\mathbf{C}}_d(\mathbf{r}) + \mathrm{MLP}_2(\mathbf{V}(\mathbf{r}),\,\mathbf{d})
\]
Opacity regularization is imposed to promote sparsity around surfaces:
\[
L_s = \lambda_s \sum_{i,k} \log(1 + \sigma(\mathbf{r}_i(t_k))^2/c)
\]
where $\lambda_s=10^{-4}$ and $c=0.5$, applied to coarse samples only.

## 2. Sparse Voxel Grid Data Structure

SNeRG "bakes" a dense sampling of the trained NeRF MLP₁ onto a sparse, block-based 3D voxel grid aligned with the scene’s axis-aligned bounding box. For grid resolution $N \in [1000, 1300]$, the structure is as follows:

- **Macroblocks**: Voxels are grouped into macroblocks of size $B^3$ ($B=32$).
- **Indirection Grid**: A grid of size $(N/B)^3$ storing either a null entry (“empty”) or an index into a compact 3D texture atlas.
- **Texture Atlas**: Dense 3D arrays for all occupied macroblocks, each storing tuples $(\alpha, \mathbf{c}_d, \mathbf{v})$ per-voxel, quantized to 8 bits per channel.
- **Voxel Quantization**: Convert density to alpha via $\alpha = 1 - \exp(-\sigma v)$, with $v=1/N$ the voxel width; $\mathbf{c}_d$ and $\mathbf{v}$ are constrained to $[0,1]$ for efficient quantization.

Block sparsity is addressed via visibility culling, discarding macroblocks with max $\alpha < \tau_\alpha = 0.005$ or max view transmittance $T < \tau_{vis} = 0.01$ across all training cameras [2103.14645].

## 3. Baking Pipeline and Data Compression

The SNeRG pipeline replaces per-frame NeRF evaluation with the following off-line scene baking procedure:

1. **Dense Evaluation**: Sample MLP₁ densely at each voxel center in the $N^3$ grid; output raw $(\sigma, \mathbf{c}_d, \mathbf{v})$.
2. **Density-to-Alpha Conversion**: Apply $\alpha = 1-\exp(-\sigma v)$.
3. **Visibility Culling**: Discard macroblocks with near-zero occupancy or visibility.
4. **Anti-Aliasing**: For each surviving voxel, average outputs of MLP₁ at 16 Gaussian-distributed spatial offsets per voxel to compute anti-aliased $(\alpha, \mathbf{c}_d, \mathbf{v})$.
5. **Quantization**: Store $(\alpha, \mathbf{c}_d, \mathbf{v})$ as 8-bit integers.
6. **Compression**:
   - Indirection grid: Losslessly encoded as PNG slices.
   - Atlas: Can be compressed via PNG, JPEG, or lossy video codecs (e.g., H.264) at macroblock granularity.
7. **Optional Fine-Tuning**: Freeze baked values, retrain MLP₂ for ~100 epochs (Adam, lr=$3\times10^{-4}$) to recover quantization-induced fidelity loss.

Per-scene baking has $O(N^3)$ cost due to the dense grid evaluation, but this is amortized over inference and typically takes $\approx$1 minute on a GPU [2103.14645].

## 4. Real-Time Rendering Algorithm

View synthesis with SNeRG replaces the costly per-ray MLP stack with accelerated texture lookups and a single residual MLP₂ evaluation per pixel:

1. **Coarse Ray Marching**: Rays are marched through the indirection grid in steps of $B/N$. Ray-box intersection tests efficiently skip empty macroblocks.
2. **Fine Marching in Occupied Blocks**: Within non-empty blocks, step size $\Delta=1/N$; for each sample $t$:
   - Lookup block index and fetch $\alpha$.
   - If $\alpha=0$, continue. Otherwise, trilinearly interpolate $(\alpha, \mathbf{c}_d, \mathbf{v})$ from adjacent voxels.
   - Accumulate
     \[
     T \leftarrow T \cdot (1 - \alpha), \quad
     \hat{\mathbf{C}}_d \leftarrow \hat{\mathbf{C}}_d + T_{old} \cdot \alpha \cdot \mathbf{c}_d, \quad
     \mathbf{V} \leftarrow \mathbf{V} + T_{old} \cdot \alpha \cdot \mathbf{v}
     \]
   - Terminate on reaching $\geq 0.95$ opacity.
3. **Deferred Shading**: Evaluate MLP₂ once per pixel, $(\mathbf{V},\mathbf{d}) \mapsto \Delta \mathbf{c}_{spec}$; output
   \[
   \hat{\mathbf{C}} = \hat{\mathbf{C}}_d + \Delta \mathbf{c}_{spec}
   \]
GPU implementation uses 3 $\times$ 8-bit 3D textures (for $\alpha$, RGB, features) and GLSL for MLP₂ evaluation [2103.14645].

## 5. Memory Footprint and Computational Performance

The SNeRG pipeline achieves compact, scalable memory usage and real-time rendering rates:

- **Compressed Memory**: $\approx$86 MB per scene (synthetic 360°), $\approx$50 MB for real scans; stored as compressed PNG or JPEG.
- **Uncompressed GPU Memory**: $\approx$1.7 GB per scene (for $N = 1300$).
- **Performance**:
   - Synthetic scenes ($800 \times 800$): $\approx$84 fps (AMD Radeon 5500M, 85 W, 0.99 fps/W).
   - Real 360° scenes ($\sim1000 \times 770$): 40–55 fps.
   - Forward-facing (real): 27–60 fps.
- **Scene Baking**: $\approx$1 minute on GPU with $O(N^3)$ MLP evaluations.

Quantization and block-level sparsity enable practical deployment but may cause slight blurring and potential “floating alpha” artifacts, which fine-tuning of MLP₂ can address [2103.14645].

## 6. Extensions and Related Sparse Voxel Grid Approaches

Sparse block-based voxel grids for radiance field parameterization have influenced broader 3D-aware generative modeling trends. Notably, the VoxGRAF architecture ("Fast 3D-Aware Image Synthesis with Sparse Voxel Grids") implements a SNeRG-style grid with progressive grid growing, free-space pruning, and 3D convolutional scene generators, omitting per-ray MLP evaluations at inference time [2206.07695]. The core representation retains two fields per voxel: density $\sigma_v$ and RGB color $\mathbf{c}_v \in [0,1]^3$, supported by active-voxel bitmasks/compressed coordinate lists and modular 3D CNN inference.

VoxGRAF demonstrates that (1) sparse 3D CNNs can replace MLPs when trained with appropriate regularization, (2) real-time novel-view rendering (up to 167 FPS amortized) and competitive image fidelity (FID = 9.6 on FFHQ $256^2$) are achievable; and (3) memory requirements (e.g., $\sim$0.9 GB at $128^3$ with 95% sparsity) are substantially lower than dense radiance field models [2206.07695].

## 7. Advantages, Limitations, and Practical Considerations

**Advantages**:

- Real-time view synthesis ($>30$ Hz) on commodity hardware via block-skipping and atomic trilinear lookups.
- Compact encoding (tens of MB), suitable for storage- and bandwidth-constrained contexts.
- Preservation of view-dependent and geometric detail with high rendering quality.
- Implementation with standard GPU 3D texture primitives and a small residual MLP.

**Limitations**:

- Quantization and block-based sparsity can induce minor visual artifacts (blurring, floating alpha holes).
- Fixed voxel grid resolution incurs a trade-off between speed and rendering quality; higher $N$ yields better fidelity at increased memory and bake time costs.
- Scene-specific preprocessing (scene bounding box, dense training coverage) is required for effective visibility culling and block sparsity.
- Baking is an offline step with $O(N^3)$ MLP₁ invocations.
- The approach is not suited to highly dynamic or deformable scenes without full re-baking.

In summary, Sparse Neural Radiance Grids provide an efficient, compact, and hardware-friendly method for 3D scene representation and real-time novel view synthesis, with technical features and limitations rigorously characterized in the foundational study [2103.14645] and extended in 3D-aware synthesis methods such as VoxGRAF [2206.07695].

Source: https://www.emergentmind.com/topics/sparse-neural-radiance-grids-snerg