On the Design of Mixture-of-Experts for Dynamic Gaussian Splatting

Abstract: Dynamic scene reconstruction remains challenging due to the heterogeneous and spatially varying nature of real-world motion. Although recent 3D Gaussian Splatting methods have introduced diverse deformation formulations for dynamic novel view synthesis, each method typically relies on a single deformation model within its representation, which limits robustness across diverse dynamic scenarios. In this work, we study a fundamental problem-multi-deformation modeling for dynamic 3D Gaussian representations-under two distinct integration constraints that differ in when and how multiple deformation experts interact during training. From a Mixture-of-Experts (MoE) perspective, we view multi-deformation modeling as the problem of combining multiple specialized deformation models within a unified 3D representation. We first introduce Mixture of Deformation Experts (MoDE), which integrates multiple deformation experts directly into the deformable Gaussian Splatting pipeline through joint optimization. In MoDE, experts operate on a shared canonical Gaussian representation, enabling multi-deformation modeling without introducing additional training stages or modifying the original optimization schedule. In contrast, we further present Mixture of Experts for Dynamic Gaussian Splatting (MoE-GS) under a different integration constraint, where deformation experts are optimized independently and combined through a separate routing stage. As a result, expert interaction occurs over non-canonical Gaussian representations after individual optimization. Together, these two approaches provide alternative strategies for multi-deformation modeling, clarifying how integration constraints shape the design and behavior of deformation experts in dynamic 3D Gaussian representations. Our code is available at: https://github.com/cvsp-lab/MoE-GS-studio.

• The paper introduces two novel frameworks, MoDE and MoE-GS, that apply mixture-of-experts strategies to optimize dynamic Gaussian Splatting through multi-deformation modeling.
• MoDE employs spline-interpolated gating weights within a shared canonical representation to balance performance across dynamic and static regions while ensuring training stability.
• MoE-GS independently trains heterogeneous experts and utilizes a volume-aware pixel router for spatially and temporally adaptive expert blending, achieving superior rendering fidelity.

Mixture-of-Experts Approaches for Dynamic Gaussian Splatting

Introduction

Dynamic scene representation via 3D Gaussian Splatting (GS) has become pivotal for high-fidelity, real-time rendering and reconstruction in vision and graphics. While significant advances have been made by extending static GS via different deformation models (e.g., MLP-based, polynomial, keyframe-based), empirical evidence demonstrates that no single deformation strategy is universally effective: performance varies across scenes, spatial regions, and temporal slices, inherently due to the inductive bias imposed by each model's deformation parametrization (Figure 1).

Figure 1: Limitations of existing dynamic Gaussian splatting methods: (a) performance varies across scenes, (b) across spatial regions, and (c) over time. Motion trajectories from four experts are shown.

This work rigorously examines multi-deformation modeling from a Mixture-of-Experts (MoE) viewpoint, introducing two orthogonally structured methodologies for dynamic GS: Mixture of Deformation Experts (MoDE) and Mixture of Experts for Dynamic Gaussian Splatting (MoE-GS). The analysis contextualizes these methods in terms of scene adaptability, training stability, and efficiency, providing a formal framework for understanding and optimizing dynamic GS under heterogeneous scene motion.

Limitations of Single-Deformation Dynamic GS

Through systematic analysis, it is shown that each deformation prior (HexPlane, per-Gaussian, polynomial, keyframe, etc.) induces pronounced spatial, temporal, and scene-level preferences. These result from how deformation fields encode regularity or flexibility, demonstrated by:

• Scene-level variations: No single method dominates across all scenes.
• Spatial-level inconsistencies: Optimal deformation models differ by region.
• Temporal fluctuations: Best-performing method shifts dynamically within sequences.

These findings indicate that limitations are not due to insufficient capacity, but are a direct result of enforcing a single, homogeneous deformation inductive bias (Figure 1, bottom).

Formulation: Multi-Deformation Modeling in GS

1. Mixture of Deformation Experts (MoDE)

MoDE composes multiple canonical deformation experts within a shared 3D Gaussian representation (Figure 2). All experts operate on the same canonical primitives; their predictions are combined via temporally continuous, spline-interpolated gating weights. Gradient propagation to the canonical representation is restricted to a baseline expert for training stability.

Figure 2: MoDE architecture—multiple deformation experts operate on a shared canonical Gaussian set, with time-varying spline-based gating and controlled gradient flow.

Key properties:

• Only canonical deformation methods (e.g., 4DGaussians, Grid4D, E-D3DGS) are eligible as experts.
• Gating weights are spline-based and Top-K sparse, providing temporally smooth and balanced expert allocation.
• Training procedure involves only baseline expert updating canonical parameters, while others are decoupled in the gradient flow.

2. Mixture of Experts for Dynamic Gaussian Splatting (MoE-GS)

MoE-GS adopts a decoupled expert optimization, training each (possibly heterogeneous) expert (keyframe, canonical, polynomial, static) independently, with later image-space composition via a learned router (Figure 3). The Volume-aware Pixel Router operates by aggregating per-Gaussian volumetric features, splatting them, and refining pixel-wise expert weights with a lightweight MLP (Figure 4). This enables spatially and temporally adaptive expert blending across independently trained models.

Figure 3: MoE-GS framework—experts are independently trained, router dynamically blends expert outputs with spatial-temporal gating.

Figure 4: Router designs: pixel-level, volume-level, and volume-aware pixel router. The latter balances geometry and optimization tractability.

Key properties:

• No shared canonical representation is required, supporting any GS method as an expert.
• Pixel/router weights are computed from splatted Gaussian attributes, ensuring geometry awareness in the absence of explicit Gaussian correspondence.
• Gate-aware Gaussian pruning and single-pass multi-expert rendering are employed for scalability.
• Distillation enables teacher-student compression for deployment efficiency.

Experimental Results

MoDE

MoDE provides moderate improvements in PSNR and motion expressiveness, particularly in dynamic regions, owing to flexible expert-level composition (Figure 5). However, performance gains are heavily dependent on the baseline; e.g., with an E-D3DGS base, static-region degradation can offset dynamic-region improvements (Figure 6). Integrated spline-gated composition at the representation level enables enhanced per-Gaussian motion expressiveness.

Figure 5: Qualitative comparison of MoDE and baselines: MoDE captures more expressive dynamics in articulated areas.

Figure 6: Trade-off analysis in E-D3DGS-based MoDE, showing dynamic-region improvement and static-region trade-offs.

MoE-GS

MoE-GS consistently outperforms all single-expert baselines and prior dynamic GS methods over heterogeneous, real-world dynamic datasets (N3V, Technicolor), with the effect robust across varying numbers and types of experts (Tables in paper, Figure 7). Pruning and distillation strategies ensure the method remains practical despite increased inference costs. The router enables fine-grained, spatially and temporally adaptive expert selection; visualization of motion trajectories reveals strong expert specialization (Figure 8).

Figure 7: MoE-GS qualitative results; routing weights strongly follow underlying motion statistics, blending experts adaptively.

Figure 8: Expert-specific motion trajectories, illustrating diverse, model-dependent priors even for same scene regions.

Adding static experts yields pronounced improvements in detail-rich, low-motion regions, while deformation experts dominate in areas of complex articulation (Figure 9). Deformation-level composition via MoDE increases per-Gaussian motion diversity (Figure 10), and multi-view geometry consistency remains high in MoE-GS mixtures due to volume-aware routing (Figure 11).

Figure 9: Effect of adding a static expert—fusion benefits static regions but still cedes high-motion zones to dynamic experts.

Figure 10: Deformation-level composition in MoDE amplifies motion expressiveness (e.g., pouring liquid dynamics).

Figure 11: MoE-GS matches or exceeds baseline methods in multi-view depth consistency, highlighting geometric fidelity.

Discussion

The structured comparison delineates that MoDE and MoE-GS instantiate fundamentally different expert composition constraints:

• MoDE: Canonical, representation-level, jointly optimized; efficient but less flexible.
• MoE-GS: Heterogeneous, image-level, decoupled; maximally flexible and robust, enables model selection post-hoc.

Quantitative results: MoDE gains are modest but efficient in training. MoE-GS achieves the highest PSNR/SSIM and lowest LPIPS, but at higher (albeit scalable via pruning/distillation) compute cost.

These results also clarify that expert specialization emerges due to architectural priors: HexPlane-based methods favor smooth, low-frequency motion; per-Gaussian embeddings excel at coherent, high-velocity motion; keyframe and polynomial methods capture irregular, non-rigid or globally smooth movement, respectively.

Theoretical and Practical Implications

The research provides a foundation for further mixture-based approaches in GS and dynamic neural rendering:

• Theory: Demonstrates how mixture composition (representation- vs. image-space) affects inductive bias, specialization, and optimization trajectories.
• Practice: MoE-GS enables robust deployment over highly heterogeneous scenes; MoDE offers a template for efficient, compositional augmentation of canonical pipelines.
• Future directions: Automatic expert pool selection, continuous expert parametrization, or learning deformation priors themselves, as well as further scaling distillation to accommodate large expert pools without substantial overhead.

Conclusion

This work presents a comprehensive framework and empirical analysis of mixture-of-experts composition for dynamic Gaussian Splatting in vision, giving practitioners a principled design space for handling heterogeneous and complex dynamic scenes. Both deformation-level (MoDE) and model-level (MoE-GS) expert composition are validated, each best suited for specific trade-offs among fidelity, efficiency, and flexibility, crucial for the next generation of real-time, photorealistic dynamic 3D reconstruction systems.

---