Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction (2510.12768v1)

Abstract: Reconstructing dynamic 3D scenes from monocular input is fundamentally under-constrained, with ambiguities arising from occlusion and extreme novel views. While dynamic Gaussian Splatting offers an efficient representation, vanilla models optimize all Gaussian primitives uniformly, ignoring whether they are well or poorly observed. This limitation leads to motion drifts under occlusion and degraded synthesis when extrapolating to unseen views. We argue that uncertainty matters: Gaussians with recurring observations across views and time act as reliable anchors to guide motion, whereas those with limited visibility are treated as less reliable. To this end, we introduce USplat4D, a novel Uncertainty-aware dynamic Gaussian Splatting framework that propagates reliable motion cues to enhance 4D reconstruction. Our key insight is to estimate time-varying per-Gaussian uncertainty and leverages it to construct a spatio-temporal graph for uncertainty-aware optimization. Experiments on diverse real and synthetic datasets show that explicitly modeling uncertainty consistently improves dynamic Gaussian Splatting models, yielding more stable geometry under occlusion and high-quality synthesis at extreme viewpoints.

• The paper introduces a novel uncertainty-aware Gaussian splatting method that robustly reconstructs dynamic 4D scenes from monocular video.
• It formulates an anisotropic, depth-aware uncertainty matrix to reweight Gaussian nodes and mitigate motion drift under occlusion.
• Experiments demonstrate superior geometry preservation and rendering quality on multiple datasets, outperforming state-of-the-art methods.

Uncertainty-Aware Dynamic Gaussian Splatting for Monocular 4D Reconstruction

Introduction and Motivation

The paper addresses the challenge of reconstructing dynamic 3D scenes from monocular video, a fundamentally ill-posed problem due to occlusions and the lack of multi-view constraints. While dynamic Gaussian Splatting (DGS) has become a standard for efficient, photorealistic 4D scene representation, existing methods uniformly optimize all Gaussian primitives, disregarding the reliability of their observations. This uniform treatment leads to motion drift under occlusion and poor synthesis for extreme novel viewpoints. The central thesis is that explicit modeling of per-Gaussian uncertainty is essential for robust monocular 4D reconstruction.

Dynamic Uncertainty Estimation

The proposed framework, G2DSplat, introduces a principled method for estimating time-varying uncertainty for each Gaussian primitive. The uncertainty $u_{i,t}$ is derived from the closed-form variance of the photometric loss, modulated by a convergence indicator to penalize unconverged pixels. This scalar uncertainty is further extended to an anisotropic, depth-aware uncertainty matrix $\mathbf{U}_{i,t}$, which propagates image-space errors into 3D, accounting for the lower reliability of depth in monocular settings. This formulation ensures that uncertainty is directionally sensitive, mitigating overconfidence along the camera axis and preserving geometric fidelity.

Figure 1: Overview of the G2DSplat pipeline, illustrating uncertainty estimation, graph construction, and uncertainty-weighted optimization for dynamic Gaussian Splatting.

Uncertainty-Encoded Spatio-Temporal Graph Construction

To propagate reliable motion cues, G2DSplat organizes Gaussians into a spatio-temporal graph, partitioning nodes into key (low-uncertainty) and non-key (high-uncertainty) sets. Key nodes are selected via 3D gridization and temporal thresholding, ensuring both spatial coverage and temporal stability. Edges are constructed using an Uncertainty-Aware $k$NN (UA-$k$NN) metric, which employs the Mahalanobis distance weighted by uncertainty matrices, favoring connections between spatially close and reliable nodes. Non-key nodes are attached to their nearest key node across the sequence, enabling motion regularization via stable anchors.

Uncertainty-Aware Optimization

The optimization objective is decomposed into key and non-key node losses. Key nodes are regularized to their pre-optimized positions with uncertainty-weighted penalties, while non-key nodes are interpolated from key nodes using Dual Quaternion Blending (DQB) and regularized to both their initialization and interpolated trajectory. The total loss combines photometric, key, and non-key node losses, with uncertainty serving to reweight deviations, guide interpolation, and balance influence in the overall objective. This design mitigates drift under occlusion and maintains geometric consistency for novel view synthesis.

Experimental Results

DyCheck Dataset

G2DSplat demonstrates consistent improvements over state-of-the-art DGS methods (SoM, MoSca) on the DyCheck dataset, both in standard validation views and extreme novel viewpoints. Quantitative metrics (PSNR, SSIM, LPIPS) show that G2DSplat outperforms baselines, with the most pronounced gains under severe viewpoint shifts. Qualitative results highlight superior preservation of geometry and pose, especially in occluded or ambiguous regions.

Figure 2: Qualitative results on validation views of DyCheck, showing improved geometry and pose preservation by G2DSplat compared to SoM and MoSca.

Figure 3: Extreme novel view synthesis on DyCheck, where G2DSplat maintains fine structures and occluded regions under large viewpoint shifts.

DAVIS and Objaverse Datasets

On the DAVIS dataset, G2DSplat yields clearer reconstructions under challenging conditions such as fast motion and self-occlusion, outperforming baselines in qualitative comparisons.

Figure 4: Extreme novel view synthesis on DAVIS, demonstrating G2DSplat's robustness to fast motion and occlusion.

On the synthetic Objaverse benchmark, G2DSplat achieves higher PSNR/SSIM and lower LPIPS than SoM and MoSca, with improvements scaling with the angular range of novel views. The method preserves geometry and texture under extreme viewpoint changes, where baselines often fail.

Figure 5: Objaverse results showing G2DSplat's superior geometry and texture preservation under extreme novel views.

Figure 6: Additional Objaverse results, further illustrating G2DSplat's robustness across diverse objects and motions.

Ablation Studies

Ablations confirm that uncertainty is critical across all components: removing uncertainty from key node selection, edge construction, or loss weighting degrades performance (PSNR, SSIM, LPIPS). The key/non-key ratio and key node selection strategy are robust to parameter choices, and the runtime overhead is minimal relative to the gains in reconstruction quality.

Theoretical and Practical Implications

The explicit integration of uncertainty into dynamic Gaussian Splatting represents a significant methodological advance. By treating uncertainty as a first-class modeling component, G2DSplat enables reliable motion propagation, mitigates drift, and enhances rendering quality under occlusion and extreme viewpoints. The framework is model-agnostic and can be incorporated into existing DGS pipelines with per-Gaussian motion parameterization. The approach is particularly relevant for AR/VR, robotics, and digital content creation, where robust monocular 4D reconstruction is essential.

Limitations and Future Directions

G2DSplat inherits errors from the underlying DGS model; if the initial 4D Gaussians fail to provide stable tracking, the uncertainty-guided graph cannot fully recover geometry. Future work may explore joint uncertainty estimation and motion modeling, integration with multi-modal priors, and scaling to larger, more complex scenes. The framework's principled uncertainty modeling opens avenues for active view selection, adaptive data acquisition, and uncertainty-aware downstream tasks.

Conclusion

G2DSplat establishes that uncertainty modeling is essential for dynamic Gaussian Splatting in monocular 4D reconstruction. By estimating per-Gaussian, time-varying uncertainty and propagating reliable motion cues via a spatio-temporal graph, the method achieves superior geometric consistency and rendering quality, especially under occlusion and extreme novel views. The approach is broadly applicable and sets a new standard for uncertainty-aware dynamic scene reconstruction.