MVFusion-GS: Motion-Variance Guided Temporal Attention for High-Quality Dynamic Gaussian Splatting

Abstract: 3D Gaussian Splatting (3DGS) enables real-time novel view synthesis for static scenes. Extending it to dynamic scenes via deformation fields has recently attracted significant attention, particularly for dynamic scene reconstructionband distractor-free. However, existing deformation networks lack explicit motion awareness: they neither capture long-term motion intensity nor exploit short-term temporal coherence, leading to inaccurate foreground deformation and pseudo-static residuals in the background. We present MVFusion-GS, a method that enhances deformation networks with two complementary motion-aware mechanisms. The Motion-Variance Guided Refinement aggregates per-Gaussian deformation statistics across time to estimate motion variance and uses it to guide dynamic-static separation during deformation prediction. The MotionFormer Temporal Attention module applies Transformer self-attention over neighboring timesteps to model local motion dependencies and improve temporal consistency. Extensive experiments on both dynamic scene reconstruction and distractor-free reconstruction benchmarks demonstrate state-of-the-art performance, showing that explicit motion awareness improves both foreground motion modeling and static background reconstruction.

• The paper introduces a novel motion-aware plug-in that integrates motion variance guided refinement and temporal attention into 3D Gaussian Splatting.
• The methodology combines a global motion descriptor with transformer-based temporal aggregation to effectively separate dynamic content from static background.
• Experimental results demonstrate enhanced PSNR, SSIM, and LPIPS metrics, yielding sharper dynamic details and cleaner static reconstructions.

MVFusion-GS: Motion-Variance Guided Temporal Attention for Dynamic 3D Gaussian Splatting

Introduction

Recent advances in 3D Gaussian Splatting (3DGS) have established it as a real-time solution for high-fidelity novel view synthesis in static scenes. Extending 3DGS to dynamic, real-world settings necessitates accurate modeling of object motion and effective separation of transient distractors from the static background. Existing dynamic splatting methods typically rely on deformation networks trained solely with local spatiotemporal cues, which often lack explicit motion awareness. This results in two critical shortcomings: imprecise foreground deformation estimates and the leakage of "pseudo-static" foreground residuals into the static background representation, degrading the fidelity of both dynamic reconstructions and background removal.

MVFusion-GS, introduced in "MVFusion-GS: Motion-Variance Guided Temporal Attention for High-Quality Dynamic Gaussian Splatting" (2607.01578), addresses these limitations by proposing effective motion-aware plug-in modules for 3DGS deformation networks. The method combines global motion statistics and transformer-based short-term temporal reasoning, significantly improving both dynamic scene fidelity and distractor-free composition. This essay details the method's technical innovations, numerical results, and implications for future dynamic scene reconstruction research.

Motion-Aware Deformation: Methodological Overview

MVFusion-GS is built on a decoupled dynamic-static 3DGS architecture, where the dynamic branch models temporally evolving foreground content and the static branch captures time-invariant background. The core hypothesis is that explicit injection of motion-descriptive features can resolve ambiguities that arise when local spatiotemporal features are insufficient for reliable Gaussian decomposition.

The approach introduces two complementary modules into the deformation network:

1. Motion-Variance Guided Refinement (MVG): This module periodically samples the deformation trajectories of each Gaussian across multiple timesteps, estimating global position, scale, and rotation statistics. The resulting "motion descriptor" provides a low-dimensional, global summary of each primitive's motion behavior, which is injected as prior into the deformation feature space.
2. MotionFormer Temporal Attention (MFTA): To model local temporal dependencies, MFTA employs a lightweight transformer to aggregate features across a short time window centered at the current timestep. This enhances deformation prediction by conditioning on both recent history and instantaneous motion context.

The proposed system operates as a feature-space plug-in and is compatible with existing Gaussian Splatting pipelines.

Figure 1: MVFusion-GS architecture: baseline deformation provides coarse updates, refined by motion-variance guided and transformer-based temporal modules, followed by dynamic-static rendering and mask-based blending.

Motion-Variance Guided Feature Construction

Global Trajectory Signature & Local Motion Variance

The MVG module extracts rich, interpretable motion statistics from simulated deformation trajectories:

• Global signature: For each Gaussian, statistics such as temporal mean and variance of position displacement, rotation magnitude, and both isotropic and anisotropic scale changes are computed, forming a compact 13D motion descriptor. A weighted sum (default: 0.7 position, 0.2 scale, 0.1 rotation) is used for an overall motion "intensity" score, empirically shown to best separate dynamic from static content.
• Local variance dictionary: To support rapid lookup of instantaneous motion cues, the system caches localized variance values at sparse time intervals, supporting efficient interpolation during training and inference.

  Figure 2: Motion descriptor extraction: deformation trajectories yield global motion signatures and cached local variance for per-timestep motion characterization.

  

  Figure 3: Each attribute’s motion variance and the fused intensity score highlight clearer dynamic–static separation.

The representation is injected into the deformation network features via a small neural encoder.

Figure 4: Motion descriptor (A) and temporal attention (B) are injected into the deformation features to enable motion-aware refinement.

MotionFormer Temporal Attention

The MFTA module is inspired by transformer architectures for temporal sequence modeling. For a short temporal neighborhood around the current timestep, temporal tokens are constructed by concatenating deformation features, current deformation predictions, and local motion intensity. A cross-attention mechanism allows each timepoint to attend directly to its neighbors, improving the temporal consistency and motion representation capacity of the deformation head.

Empirical sensitivity analysis indicates that moderate temporal window sizes (e.g., $w=5$) achieve the best trade-off, and cross-attention outperforms naive self-attention or temporal averaging.

Experimental Results

Distractor-Free Background Reconstruction

Benchmarking on NeRF On-the-go and RobustNeRF demonstrates that MVFusion-GS surpasses prior dynamic Gaussian methods—including DeGauss, SpotlessSplats, and DeSplat—on background fidelity. MVFusion-GS achieves the lowest LPIPS and highest SSIM across all tested scenes, especially for those with significant transient distraction.

Figure 5: On NeRF On-the-go, MVFusion-GS eliminates ghostly pseudo-static residuals in the static background, improving cleanliness and visual quality over DeGauss.

Figure 6: MVFusion-GS background compositing leads to sharper full-scene renderings with minimal foreground leakage.

Figure 7: On RobustNeRF, backgrounds reconstructed via MVFusion-GS are demonstrably cleaner than prior methods.

Dynamic Scene Reconstruction

On the Neu3D benchmark, MVFusion-GS not only improves dynamic region reconstruction (e.g., flames, fast-moving objects), but also enhances background consistency. Its dynamic region PSNR, SSIM, and LPIPS all exceed prior SOTA.

Figure 8: On Neu3D, MVFusion-GS reconstructs sharper, less artifact-prone motion details than DeGauss and baseline 4DGS.

Ablation and Sensitivity Studies

Ablations confirm the significance of the global motion signature (MVG) and temporal cross-attention (MFTA) modules. Exclusion of either module decreases PSNR/SSIM and increases perceptual error, both on dynamic and static subregions. Position variance dominates motion separation, but incorporating scale and rotation yields more robust foreground-background discrimination.

Figure 9: Variant analysis of different motion variance fusion weightings shows that combined cues best highlight true dynamic regions.

Failure Cases and Limitations

MVFusion-GS’s effectiveness depends on the adequacy of the baseline deformation network. In challenging settings with highly underfitted or ambiguous foreground dynamics, motion statistics may become unreliable, and some pseudo-static leakage persists.

Figure 10: Branch-level decomposition visualization—removal of MVG/MFTA causes persistent pseudo-static artifacts in the static branch.

Figure 11: Example of a failure case: extremely complex or subtle motion may not be fully separable, even given MVFusion-GS’s motion priors.

Implications and Future Directions

MVFusion-GS introduces rigorous motion statistics and sequence modeling into dynamic Gaussian Splatting, addressing long-standing ambiguities in dynamic-static decomposition. The method’s plug-in structure allows straightforward adaptation to various deformation-based architectures and can serve as a foundation for future approaches incorporating uncertainty, adaptive Gaussian pruning, or learned dynamic-static partitioning. Fast, memory-efficient scene representation is key for AR/VR, robotics, and dynamic environment mapping; improvements here directly facilitate real-world deployment of neural scene representations.

Open directions include jointly optimizing deformation fields and motion statistics in more adversarial or unsupervised settings, integrating broader spatial context for subtle motion, and applying learned motion variance priors beyond Gaussian Splatting to other explicit neural field representations.

Conclusion

MVFusion-GS advances the state of the art in dynamic 3D Gaussian Splatting by augmenting deformation networks with explicit global and local motion descriptors and transformer-based temporal aggregation. Experimental evidence demonstrates consistent improvements in both distractor-free background modeling and dynamic region fidelity, achieving cleaner static reconstructions and sharper dynamic details. The modular, efficient design, coupled with comprehensive empirical validation, underscores the utility of explicit motion-awareness cues for dynamic neural rendering frameworks. Further research on robust deformation modeling and adaptive Gaussian assignment, motivated by the demonstrated failure cases, could yield even more resilient and perceptually high-fidelity dynamic scene reconstructions.