PlenOctrees: Efficient Real-Time NeRF Rendering

• PlenOctrees are sparse, hierarchical octree structures that bake NeRF outputs into voxel grids, enabling real-time volumetric rendering with significant speed-ups.
• They sample trained NeRF-SH networks over dense grids to capture scalar densities and spherical harmonics, which are then optimized for efficiency and quality.
• Extensions like Fourier PlenOctrees use truncated Fourier series to encode temporal dynamics, achieving interactive rendering of dynamic scenes with high fidelity.

PlenOctrees are sparse, hierarchical octree data structures designed for real-time volumetric rendering of Neural Radiance Fields (NeRFs), encompassing both static and dynamic scenes. Each leaf node within a PlenOctree encodes not only a scalar volume density $\sigma(\mathbf{x})$ at a spatial location $\mathbf{x}$ but also a compact, view-dependent radiance representation, typically parameterized by spherical harmonics (SH) coefficients. The innovation of PlenOctrees lies in baking the outputs of a trained NeRF—traditionally a computationally intensive, multi-layer perceptron—into an efficient, sparse voxel grid that allows for high-throughput ray traversal and image synthesis at interactive frame rates. Subsequent advancements such as Fourier PlenOctrees further extend the concept to 4D radiance fields by encoding temporal variation in density and color via truncated Fourier series, making real-time, high-fidelity rendering of dynamic scenes feasible with modest GPU resources (Yu et al., 2021, Wang et al., 2022, Rabich et al., 2023).

1. Structural Basis and Rendering in PlenOctrees

PlenOctrees (Yu et al., 2021) are constructed by spatially subdividing the scene into a multi-scale, sparse octree. Each leaf stores:

• A scalar density $\sigma(\mathbf{x})$
• A set of SH coefficients $$\mathbf{z}(\mathbf{x}) \in \mathbb{R}^{L^2 \times 3}$$

During rendering, a ray is traversed through the sparse octree, sampling intersected voxels. For each sample:

• The view-dependent color is reconstructed by evaluating the SH-function:

$$\mathbf{c}(\mathbf{d}; \mathbf{z}) = S\left( \sum_{\ell=0}^{\ell_{\max}} \sum_{m=-\ell}^\ell z_{m, \ell} Y_{m, \ell}(\mathbf{d}) \right)$$

where $S$ is a sigmoid function to bound color to $[0, 1]$.

• The composite pixel color is determined via volume rendering:

$$\hat{C}(\mathbf{r}) = \sum_{i=1}^{N} T_{i} (1 - e^{-\sigma_i \delta_i}) \, \mathbf{c}_i$$

with transmittance $T_{i} = \exp(-\sum_{j< i} \sigma_j \delta_j)$ and sampling intervals $\delta_i$.

By baking the necessary scene information into a sparse voxel structure, PlenOctrees achieve dramatic acceleration: real-time rendering at resolutions such as $\mathbf{x}$0 pixels at over 150 FPS, exceeding baseline NeRF MLPs by several thousand times (Yu et al., 2021).

2. Extraction and Optimization from NeRF

To build a PlenOctree, a pretrained NeRF-SH network is queried on a dense 3D grid. A visibility filter is applied to discard low-opacity voxels:

1. For each grid cell, the NeRF is evaluated to obtain $\mathbf{x}$1 and SH coefficients.
2. Voxels with high alpha contributions across any training ray are retained; others are omitted to maintain sparsity.
3. The surviving voxels form the leaf set of the octree.

Average values of $\mathbf{x}$2 and SH coefficients are computed within each voxel, which are then stored in the octree leaves. This process—tabulation plus filtering—requires about 10–15 minutes for a typical synthetic scene (Yu et al., 2021).

Additionally, direct fine-tuning of leaf values is performed via random ray sampling and backpropagation through the volume renderer, independent of the original neural network. This further closes any quality gap with the original NeRF and can be completed in a few epochs, leveraging analytical gradients for efficient optimization.

3. Temporal Extension: Fourier PlenOctrees

Fourier PlenOctrees (FPOs) (Wang et al., 2022, Rabich et al., 2023) introduce a principled approach to representing dynamic, time-varying scenes. The density and SH color attributes at each leaf are modeled as truncated Fourier series over time:

• Density:

$\mathbf{x}$3

• Color:

$\mathbf{x}$4

where $\mathbf{x}$5, $\mathbf{x}$6 is the number of frames, and $\mathbf{x}$7 are the Fourier coefficients for density and color, respectively.

Union PlenOctrees are created by merging the spatial structures from all per-frame PlenOctrees, then performing a discrete Fourier transform (DFT) on the per-frame signals at each leaf to obtain the per-leaf temporal basis coefficients. This enables compact and efficient encoding of dynamic content, with reconstruction performed via inverse DFT during rendering.

The resulting approach allows real-time rendering of dynamic radiance fields at 100 FPS and memory footprints on the order of a few GB, with image quality exceeding other state-of-the-art dynamic NeRF approaches (e.g., PSNR 35.21, SSIM 0.9910, LPIPS 0.0217) (Wang et al., 2022).

4. Compression Artifacts and Fourier PlenOctree Enhancements

Truncated Fourier expansion induces artifacts in dynamic scenes:

• Ringing and ghosting: High-frequency content removal leads to “bleeding” of geometry or color from temporally distant frames.
• Blurring of fine motion: Inadequate basis support for fast or small-scale temporal changes leads to oversmoothing.
• Transfer-function saturation issues: The mapping $\mathbf{x}$8 saturates for large densities, making the DFT-based compression especially error-prone near surfaces (where small density differences matter).

To mitigate these issues, FPO++ (Rabich et al., 2023) applies:

• Logarithmic encoding of densities to compress their dynamic range before DFT, denoted $\mathbf{x}$9, improving basis efficiency near surfaces.
• Component-dependent scaling to zero-center densities, which enhances DFT approximability of free-space transitions.
• Temporal augmentation by appending copies of start/end frames, mitigating the periodicity assumption of vanilla DFT and thus suppressing temporal boundary artifacts.

These steps yield baseline-free PSNR improvements of 6+ dB and qualitative elimination of “ghost” after-images, especially in scenes with rapid motion (Rabich et al., 2023).

5. Dynamic Structure Adaptation: Dynamic PlenOctrees

Dynamic PlenOctrees (DOT) (Bai et al., 2023) address the limitations of fixed-tree structures by allowing the octree to evolve during learning. Observing that static octrees are suboptimal for scenes with varying spatiotemporal complexity, DOT periodically merges or refines nodes based on rendering signal statistics:

• Pruning nodes whose cumulative ray-contribution weights fall below a set threshold (indicative of low visual significance).
• Refining nodes with disproportionately high contribution, allocating detail adaptively.

Hierarchical feature fusion is employed: during pruning, features from child leaves are averaged into the parent; during splitting, parent features are copied into new children, maintaining radiance and density consistency.

DOT reduces memory by over 55% on NeRF-synthetic and by nearly 69% on Tanks & Temples data, speeds rendering by up to 2.5x (250 FPS to 474 FPS at 800×800), and slightly increases PSNR over static PlenOctree baselines (Bai et al., 2023).

6. Quantitative Evaluation and Comparative Table

Architecture	PSNR	SSIM	LPIPS	FPS	Memory
PlenOctree (static, tuned)	31.71	0.958	0.053	250	1.93 GB
DOT (dynamic)	32.11	0.959	0.053	452	0.87 GB
Fourier PlenOctree (FPO)	35.21	0.9910	0.0217	100	7.25 GB
FPO++ (improved)	29.90	0.948	0.094	373	2.4 GB

These numbers reflect benchmarking on NeRF-synthetic and real dataset scenes (Yu et al., 2021, Wang et al., 2022, Bai et al., 2023, Rabich et al., 2023).

7. Limitations and Future Directions

PlenOctree architectures, particularly when extended for 4D radiance fields, face several practical constraints:

• High capture cost due to dependence on dense, calibrated multi-camera setups or accurate silhouettes.
• Significant memory requirements compared to vanilla NeRFs, though much more efficient than frame-wise MLPs for dynamics.
• Degradation in rapidly changing scenes unless Fourier basis truncation is appropriately chosen, which increases storage.
• Potential over-expansion of the bounding volume in dynamic scenes with large global motions.

Proposed advancements include adaptive temporal basis pruning, hybrid representation combining tree and deformation models, sparse-view/monocular scene capture, and alternative spatiotemporal encoding bases beyond Fourier, such as wavelets (Wang et al., 2022, Rabich et al., 2023).

PlenOctrees and their dynamic and temporal extensions now represent a leading real-time solution for neural radiance field rendering in both static and dynamic environments, underpinning scalable AR/VR visualization, 6-DOF navigation, and interactive visualizations at previously unattainable frame rates (Yu et al., 2021, Wang et al., 2022).