4C4D: 4 Camera 4D Gaussian Splatting

Abstract: This paper tackles the challenge of recovering 4D dynamic scenes from videos captured by as few as four portable cameras. Learning to model scene dynamics for temporally consistent novel-view rendering is a foundational task in computer graphics, where previous works often require dense multi-view captures using camera arrays of dozens or even hundreds of views. We propose \textbf{4C4D}, a novel framework that enables high-fidelity 4D Gaussian Splatting from video captures of extremely sparse cameras. Our key insight lies that the geometric learning under sparse settings is substantially more difficult than modeling appearance. Driven by this observation, we introduce a Neural Decaying Function on Gaussian opacities for enhancing the geometric modeling capability of 4D Gaussians. This design mitigates the inherent imbalance between geometry and appearance modeling in 4DGS by encouraging the 4DGS gradients to focus more on geometric learning. Extensive experiments across sparse-view datasets with varying camera overlaps show that 4C4D achieves superior performance over prior art. Project page at: https://junshengzhou.github.io/4C4D.

• The paper introduces a Neural Decaying Function that adaptively balances geometric and appearance optimization during 4D reconstruction.
• It leverages a visibility-driven masking module to maintain stable learning under extreme sparse-view conditions.
• Experimental results show significant improvements in PSNR and perceptual metrics compared to previous methods on diverse benchmarks.

4C4D: 4 Camera 4D Gaussian Splatting

Introduction and Motivation

Sparse-view dynamic scene reconstruction is a core problem for enabling practical 4D content creation, with applications ranging from VR, film production, to interactive virtual environments. Existing neural scene representations, including NeRFs and their real-time variants, have demonstrated success primarily in densely captured multi-view settings, yet face severe geometric and temporal failures under highly sparse captures. "4C4D: 4 Camera 4D Gaussian Splatting" (2604.04063) directly addresses this by proposing a framework capable of reconstructing high-fidelity 4D dynamic scenes from as few as four portable cameras, leveraging an adaptive neural decaying mechanism on 4D Gaussian opacities to resolve the intrinsic imbalance between appearance and geometry supervision in sparse multi-view optimization.

Methodology

The 4C4D framework builds upon the 4D Gaussian Splatting (4DGS) paradigm, which extends 3DGS with temporally evolving Gaussian primitives parameterized by position, rotation, scale, opacity, temporal parameters, and higher-order spherical harmonics for view-dependent appearance encoding. The observation underlying 4C4D is that, under sparse-view supervision, 4DGS easily overfits appearance while failing to reconstruct valid scene geometry, as the optimization procedure naturally focuses on regions and attributes that most directly reduce photometric error, even when this leads to geometric inconsistency.

To decouple and re-balance the optimization pressure, 4C4D introduces a Neural Decaying Function, $f_\theta$, realized as a lightweight network controlling per-Gaussian opacity decay dynamically throughout training. Given each Gaussian's position, rotation, and opacity, $f_\theta$ provides a learned decay factor which determines the rate of opacity attenuation over time, thereby adaptively allocating gradients towards geometric or appearance aspects as required by the optimization phase.

Figure 1: Overview of 4C4D, where a Neural Decaying Function $f_\theta$ is used to control the opacity decay of 4D Gaussians, focusing learning gradients on geometry modeling during joint optimization.

Critically, applying this neural decay indiscriminately leads to instability: gradients are only available for regions visible in the current views and timesteps. To avoid corrupting the background or occluded regions, 4C4D incorporates a spatial-temporal visibility detection module, which segregates the set of Gaussians into visible and non-visible, and only modulates the former via $f_\theta$, while imposing a constant minimal decay on the latter. This visibility-driven assignment enforces stable learning of both geometry and appearance, even under extreme sparsity.

Experimental Evaluation

4C4D is rigorously tested on multiple benchmarks, including Neural3DV, ENeRF-Outdoor, Mobile-Stage, and the newly introduced self-captured Dyn4Cam dataset. Each scene is reconstructed with only four camera views, with remaining datasets held out for generalization testing. Evaluation metrics include PSNR, DSSIM, and LPIPS, emphasizing both per-pixel and perceptual image quality as well as geometric consistency across time.

Strong numerical results are consistently observed:

• On Neural3DV, 4C4D achieves PSNR of 22.29, outperforming prior methods such as 4DGS and Ex4DGS by a significant margin, with lower DSSIM and LPIPS, indicating both better geometric and photometric reconstruction.
• On ENeRF-Outdoor, which presents challenging uncontrolled lighting and large object motion, 4C4D attains PSNR 24.32 and lowest perceptual dissimilarity.
• On Mobile-Stage, capturing complex multi-person performance, 4C4D not only improves upon baselines but reconstructs fine structures and dynamic motion otherwise lost.

  Figure 2: Visual comparison on Neural3DV, showing 4C4D’s reconstruction quality and temporal coherence over competing baselines.

Qualitative results support these findings: 4C4D’s reconstructions display sharp, temporally stable, and geometry-faithful renderings, while previous methods exhibit noisy, drifting, or incomplete reconstructions especially on unseen or sparsely observed views.

Figure 3: Visual comparisons on ENeRF-Outdoor revealing 4C4D’s superior temporal and geometric consistency over complex dynamics.

Results on the fully portable Dyn4Cam dataset, captured with consumer GoPro cameras and marked by pronounced motion and occlusions, further underline the robustness of the approach under practical, non-laboratory conditions.

Figure 4: Qualitative comparison on Dyn4Cam, illustrating 4C4D’s resilience to fast action and partial occlusion.

Ablation and Analysis

A series of ablation studies dissect the impact of key components:

• Removing Neural Decay yields substantial geometry overfitting and appearance bias, as evidenced by PSNR drop and increased structural dissimilarity.
• Omitting visibility-driven masking (naively decaying all Gaussians) leads to geometric confusion and loss of training stability.
• Substituting the neural decay with analytic decay functions (power or exponential) is consistently inferior across all quantitative measures; the neural, learnable decay strategy outperforms even finely hand-tuned alternatives.

These findings establish the necessity of both the neural decaying and the visibility-aware assignment for enabling balanced and effective 4DGS learning under sparse-view constraints.

Implications, Limitations, and Future Prospects

Practically, 4C4D dramatically lowers the hardware and computational barrier for time-varying novel-view synthesis, enabling dynamic scene capture with minimal consumer equipment. Theoretically, the work underscores the importance of adaptive, learned regularization and visiblity reasoning in neural rendering pipelines, especially under supervision scarcity. The neural decaying paradigm is generic, and could be extended to other radiance field modalities or incorporated into future hybrid volumetric/implicit representations.

Limitations include reliance on accurate multi-view camera extrinsics, and the possible need for further regularization or explicit priors in environments with even sparser coverage or more severe occlusion. 4C4D’s design points towards future directions:

• Integration with single-image or monocular priors to move towards three or fewer cameras.
• Joint optimization with SLAM or scene flow estimation for automated camera pose recovery.
• Extension to text-to-4D generation, editability, or open-world indoor/outdoor domain generalization by combining with recent diffusion-based 4D generative models [e.g., 4DGen, DreamGaussian4D, Stag4D, Diffusion4D].

Conclusion

4C4D establishes a new state of the art for dynamic scene reconstruction from highly sparse camera arrays by introducing a learned, visibility-driven opacity regularization mechanism into 4D Gaussian Splatting. This enables robust recovery of temporally consistent geometry and appearance using only four input videos, supported by strong empirical benchmarks and thorough analysis. The contributions of learned gradient allocation for geometric optimization and principled treatment of visibility are broadly applicable, with significant implications for democratizing 4D content creation and spatio-temporal scene understanding.