Multi4D: High-Fidelity Dynamic Gaussian Splatting via Multi-Level Competitive Allocation

Abstract: Dynamic 3D Gaussian splatting faces a fundamental tension between motion consistency and visual fidelity. Deformation-based approaches preserve temporal correspondence but suffer from motion over-factorization, oversmoothing high-frequency dynamics. In contrast, 4D-primitive methods capture fine visual details yet incur temporal overparameterization, breaking object identity and leading to severe storage overhead. To resolve this, we introduce Multi4D, a framework for high-fidelity dynamic Gaussian Splatting based on multi-level competitive allocation. Instead of a monolithic representation, we distribute modeling capacity across three structured levels: static structure, persistent dynamic geometry, and transient appearance primitives. Through shared rasterization and residual-driven optimization, these levels dynamically compete to explain photometric error, enabling adaptive specialization without pre-assigned decomposition. This allocation preserves long-term motion consistency while capturing fine dynamic detail, achieving state-of-the-art rendering quality and real-time performance with significantly fewer dynamic primitives. Furthermore, because our representation explicitly tracks compact persistent Gaussians over time, semantic features can be embedded afterward, enabling Multi4D to achieve state-of-the-art 4D segmentation accuracy with an order-of-magnitude speedup. Project page: https://batfacewayne.github.io/Multi4D.io/

• The paper introduces a multi-level allocation pipeline that decomposes scene modeling into static, persistent dynamic, and transient Gaussian subsets for balanced motion and detail.
• Methodology employs competitive photometric error minimization, velocity-aware lifting, and mask-aware pruning to ensure robust, efficient dynamic scene reconstruction.
• Empirical results demonstrate state-of-the-art 4D segmentation accuracy at 161–217 FPS with a 25× reduction in model complexity compared to previous 4DGS approaches.

Multi4D: High-Fidelity Dynamic Gaussian Splatting via Multi-Level Competitive Allocation

Motivation and Background

Dynamic 3D Gaussian Splatting (3DGS) techniques in novel view synthesis have faced a core representational tension: deformation-based approaches enforce temporal correspondence but over-smooth high-frequency motion, while 4D-primitive-based models capture appearance details but produce temporal over-parameterization and break object identity. Existing works—across neural deformation fields, explicit trajectory modeling, and feature-based rendering—either sacrifice spatial detail for motion consistency or vice versa, and become computationally prohibitive when scaling to fine dynamic details. The need to reconcile motion consistency, photometric fidelity, and compact, high-accuracy segmentation motivates a structured decomposition of scene modeling capacity.

Methodology

Multi4D introduces a multi-level competitive allocation pipeline, organizing model capacity across three Gaussian subsets: static ($\mathcal{G}_s$), persistent dynamic ($\mathcal{G}_d$), and transient ($\mathcal{G}_t$) primitives. The static subset anchors time-invariant scene structure, persistent dynamic models track holistic motion via geometric-only deformation (HexPlane grids), and transients capture high-frequency appearance changes and short-lived phenomena.

A bottom-up, self-regularized optimization schedule is employed. Each subset is initialized inversely proportional to its expressiveness—static densely from COLMAP points, persistent dynamic sparsely, transient initially empty. Competitive photometric error minimization, cross-subset self-supervision, and joint rasterization induce emergent specialization. Spatial-temporal normalization ensures stable optimization against gradient imbalance.

Figure 1: Multi4D pipeline overview with bottom-up allocation across the three Gaussian subsets and cross-set self-supervision.

Unified blending via a differentiable renderer generates composite, persistent, and transient outputs, leveraging shared transmittance for coupled gradients and occlusion reasoning. Velocity-aware periodical lifting—sampling activated persistent Gaussians and promoting them with inherited velocity—enables controlled introduction of transients, while mask-aware utility-based pruning eliminates low-contribution or redundant primitives. Densification, pruning, and Adam-based optimization are applied independently to each subset.

Self-supervised dynamic-static decomposition leverages mask logits and spatially-aware opacity regularization to segregate persistent actors from static background without explicit tracking labels, providing a spatial template for pruning and compositional rendering.

Figure 2: Subset specialization visualization—background, persistent motion, and high-frequency transients are modeled distinctly across Neu3D, Technicolor, and NeRF-DS sequences.

Downstream Application: 4D Segmentation

Multi4D's representation supports efficient, temporally consistent 4D segmentation. Semantic features are optimized exclusively on the persistent subset ($\mathcal{G}_p = \mathcal{G}_s \cup \mathcal{G}_d$) after appearance convergence, using soft-mined contrastive objectives and KNN smoothing. Instance tracking is performed via deformation fields, and open-vocabulary segmentation is enabled through Grounding-DINO and SAM-based mask clustering.

Figure 3: 4D semantic segmentation pipeline—persistent structure enables mask-driven feature clustering and temporally consistent object tracking.

Empirical Evaluation

Benchmarks include Technicolor, Neu3D, NeRF-DS, and Neu3D-Mask for multi-view, monocular, and segmentation tasks. Multi4D consistently outperforms baselines in PSNR, DSSIM, and LPIPS, and achieves state-of-the-art segmentation accuracy and runtime performance. Notably, Multi4D produces high-fidelity dynamic reconstructions at 161–217 FPS, using only 165k dynamic Gaussians compared to 4.2M for 4DGS—a $25\times$ reduction and $4.6\times$ faster training convergence.

Figure 4: Qualitative comparisons—Multi4D demonstrates superior modeling of fine dynamic details against deformation- and 4D-primitive-based baselines.

In monocular settings (NeRF-DS), Multi4D maintains coherent geometry and specular highlights, addressing failure modes in baseline 4D-primitive frameworks under sparse supervision.

Ablation and Efficiency Analysis

Removed components (persistent dynamics, transients, periodical lifting, diversity loss, and pruning) demonstrate the necessity of each mechanism for fidelity, compactness, and temporal consistency. Mask-aware pruning and diversity loss suppress temporal over-parameterization, while velocity-aware lifting regularizes transient placement. Structural decoupling yields persistent motion correspondence unaffected by transients.

Figure 5: Visualization—over-parameterized transients disrupt persistent motion.

Implications, Limitations, and Future Directions

Multi4D establishes that competitive allocation and emergent specialization across structured Gaussian subsets resolve the longstanding trade-off between motion consistency and appearance fidelity. This decomposition directly supports efficient, high-accuracy 4D segmentation with an order-of-magnitude speedup and offers robustness across multi-view and monocular regimes.

Potential avenues include further exploration of post-training attribute compression, deformation distillation, and transient quantization. Feed-forward dynamic reconstruction and all-point tracking could benefit from Multi4D's decompositional priors, particularly for egocentric and challenging dynamic datasets. The methodology recommends hybrid decomposition as a structural regularization for high-fidelity dynamic scene modeling in future work.

Conclusion

Multi4D advances dynamic Gaussian Splatting by integrating structured decomposition, bottom-up competitive allocation, and subset-specialized optimization to achieve compact, temporally consistent, and photometrically faithful reconstructions. Downstream segmentation benefits from persistent identity and fast inference, underscoring the theoretical and practical value of multi-level representation for dynamic scene understanding.