4D Neural Voxel Splatting: Dynamic Scene Rendering with Voxelized Guassian Splatting (2511.00560v1)

Abstract: Although 3D Gaussian Splatting (3D-GS) achieves efficient rendering for novel view synthesis, extending it to dynamic scenes still results in substantial memory overhead from replicating Gaussians across frames. To address this challenge, we propose 4D Neural Voxel Splatting (4D-NVS), which combines voxel-based representations with neural Gaussian splatting for efficient dynamic scene modeling. Instead of generating separate Gaussian sets per timestamp, our method employs a compact set of neural voxels with learned deformation fields to model temporal dynamics. The design greatly reduces memory consumption and accelerates training while preserving high image quality. We further introduce a novel view refinement stage that selectively improves challenging viewpoints through targeted optimization, maintaining global efficiency while enhancing rendering quality for difficult viewing angles. Experiments demonstrate that our method outperforms state-of-the-art approaches with significant memory reduction and faster training, enabling real-time rendering with superior visual fidelity.

• The paper presents 4D Neural Voxel Splatting, integrating neural Gaussian splatting with voxelized representations to achieve efficient dynamic scene rendering.
• It introduces a unified 4D voxel architecture and selective deformation strategy that effectively model spatial-temporal dynamics while reducing memory and computational costs.
• Quantitative results on HyperNeRF and Neu3D datasets demonstrate significant improvements in PSNR, speed, and memory usage, enabling real-time applications.

4D Neural Voxel Splatting: Dynamic Scene Rendering with Voxelized Gaussian Splatting

Introduction

The paper presents 4D Neural Voxel Splatting (4D-NVS), a dynamic scene rendering framework that integrates voxel-based representations with neural Gaussian splatting, targeting efficient and high-fidelity novel view synthesis in dynamic environments. The method addresses the memory and computational bottlenecks inherent in extending 3D Gaussian Splatting (3D-GS) to dynamic scenes, where naive approaches incur memory overhead proportional to the number of frames. 4D-NVS leverages a compact set of neural voxels and learned deformation fields to model temporal dynamics, achieving $\mathcal{O}(fV+F)$ memory complexity, where $V$ is the number of voxels, $f$ is the feature dimension, and $F$ is the deformation network parameter count. The framework is designed for real-time rendering and rapid training, making it suitable for resource-constrained applications such as embodied AI and edge robotics.

Methodology

Unified 4D Voxel Architecture

4D-NVS extends 3D voxel grids into the temporal domain by treating time as an additional dimension in the voxel feature space. Each voxel encodes both spatial and temporal features, enabling the generation of time-varying neural Gaussians. Unlike Scaffold-GS, which produces static Gaussians, 4D-NVS voxels generate dynamic Gaussians via learned temporal features and deformation fields.

On-Demand Neural Gaussian Generation

At each timestamp, visibility culling identifies voxels within the camera frustum, reducing computational complexity. For each visible voxel, $k$ neural Gaussians are generated with positions $\mu_i = x_v + O_i \cdot l_v$, where $x_v$ is the voxel center, $O_i$ are learnable offsets, and $l_v$ is the scale. Gaussian attributes (opacity, color, scale, rotation) are decoded from anchor features using dedicated MLPs conditioned on viewing direction and distance.

HexPlane Temporal Deformation

Temporal dynamics are modeled using HexPlane decomposition, which factorizes the 4D space-time volume into six 2D planes (XY, XZ, YZ, XT, YT, ZT). Features are extracted via bilinear interpolation and aggregated, then processed by a shallow MLP to produce deformation features. Separate decoders predict position, rotation, and scale changes, which are applied to the Gaussians. Notably, only geometric properties (position, scale, rotation) are deformed; appearance properties (color, opacity) remain fixed to ensure training stability.

Selective Deformation Strategy

Empirical analysis revealed that deforming all Gaussian properties leads to instability and error accumulation. The selective deformation strategy—restricting deformation to geometric attributes—maintains stable gradients and convergence, while still capturing complex motion patterns.

View-Adaptive Refinement

A novel refinement stage identifies underperforming viewpoints using PSNR-based and gradient-based detection. Crude views are prioritized for adaptive densification and focused training, improving rendering quality for challenging regions without incurring global overhead.

Figure 1: Visual comparisons on the HyperNeRF dataset demonstrate superior rendering quality of 4D-NVS over prior methods.

Experimental Results

Quantitative Evaluation

On the HyperNeRF dataset, 4D-NVS achieves 28.5 dB PSNR and 0.872 MS-SSIM, outperforming state-of-the-art methods while using only 3,050 MiB of memory and requiring 13 minutes of training. On the Neu3D dataset, the method attains 33.12 dB PSNR and 43 FPS, with a significant reduction in memory and training time compared to competitors.

Figure 2: Neu3D dataset visualization; 4D-NVS balances detail across complex dynamic regions, outperforming alternatives.

Figure 3: Continuous frames on HyperNeRF; 4D-NVS (top) preserves more details than 4DGS (bottom), especially in dynamic regions.

Ablation Studies

Ablation experiments confirm the efficacy of volume regularization and total variation loss, with both contributing to improved PSNR and MS-SSIM. The selective deformation strategy yields the best balance between quality and training stability, as deforming only geometric properties avoids error propagation.

Figure 4: Left: with $L_{vol}$, Right: without $L_{vol}$; volume regularization prevents oversized Gaussians and blurring.

Figure 5: Left: without view refinement, Right: with view refinement; adaptive refinement enhances consistency and detail.

Memory and Speed Analysis

4D-NVS achieves an 86% reduction in memory usage compared to TiNeuVox-B and a 94% speedup over HyperNeRF. The method is deployable on consumer GPUs and suitable for real-time interactive environments.

Limitations

While 4D-NVS demonstrates rapid convergence and real-time rendering, challenges remain in handling large motions and noisy camera poses. Popping artifacts persist in some scenarios, indicating the need for further improvements in voxel representation and rasterization.

Implications and Future Directions

4D-NVS advances the state-of-the-art in dynamic scene rendering by decoupling spatial and temporal modeling, enabling efficient deployment in VR/AR, robotics, and digital content creation. The selective deformation and view refinement strategies are particularly impactful for resource-constrained applications. Future research should focus on enhancing motion robustness, eliminating popping artifacts, and exploring hybrid explicit-implicit representations for scalability.

Conclusion

4D Neural Voxel Splatting introduces a memory- and computation-efficient framework for dynamic scene rendering, leveraging unified 4D voxel architectures, selective deformation, and adaptive view refinement. The method achieves superior visual fidelity, rapid training, and real-time performance, establishing a new benchmark for dynamic neural rendering systems.