Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (2512.03601v1)

Abstract: Recent advancements in foundation models for 2D vision have substantially improved the analysis of dynamic scenes from monocular videos. However, despite their strong generalization capabilities, these models often lack 3D consistency, a fundamental requirement for understanding scene geometry and motion, thereby causing severe spatial misalignment and temporal flickering in complex 3D environments. In this paper, we present Motion4D, a novel framework that addresses these challenges by integrating 2D priors from foundation models into a unified 4D Gaussian Splatting representation. Our method features a two-part iterative optimization framework: 1) Sequential optimization, which updates motion and semantic fields in consecutive stages to maintain local consistency, and 2) Global optimization, which jointly refines all attributes for long-term coherence. To enhance motion accuracy, we introduce a 3D confidence map that dynamically adjusts the motion priors, and an adaptive resampling process that inserts new Gaussians into under-represented regions based on per-pixel RGB and semantic errors. Furthermore, we enhance semantic coherence through an iterative refinement process that resolves semantic inconsistencies by alternately optimizing the semantic fields and updating prompts of SAM2. Extensive evaluations demonstrate that our Motion4D significantly outperforms both 2D foundation models and existing 3D-based approaches across diverse scene understanding tasks, including point-based tracking, video object segmentation, and novel view synthesis. Our code is available at https://hrzhou2.github.io/motion4d-web/.

• The paper introduces a novel framework that enforces 3D spatiotemporal consistency by unifying 2D segmentation priors with dynamic 3D Gaussian Splatting using iterative motion and semantic refinement.
• It leverages a dual-stage optimization strategy to integrate rigid transformation-based motion and embedded semantic fields, significantly reducing artifacts like flickering and spatial misalignment.
• Empirical evaluations demonstrate enhanced performance in video object segmentation, point tracking, and view synthesis, with improved PSNR, SSIM, and tracking precision over state-of-the-art methods.

3D-Consistent Motion and Semantic Modeling with Motion4D

Motivation and Context

High-fidelity understanding of dynamic scenes from monocular video presents a significant challenge, especially in the context of 3D consistency across spatial and temporal dimensions. Although 2D foundation models, such as the Segment Anything Model (SAM) and its extension SAM2, exhibit strong generalization in image and video segmentation, they fail to enforce 3D coherence. This leads to severe artifacts including temporal flickering, spatial misalignments, and boundary inconsistencies that degrade downstream tasks such as scene reconstruction, tracking, and view synthesis.

Motion4D addresses this bottleneck by leveraging rich 2D priors from foundation models and unifying them within a 3D Gaussian Splatting (3DGS) dynamic representation. The framework introduces a two-part iterative optimization scheme that robustly fuses motion and semantics into a 4D scene understanding model, facilitating tracking, segmentation, and novel view synthesis with strong spatiotemporal consistency. The approach is systematically validated with comprehensive experiments on existing and newly proposed benchmarks.

Model Architecture and Optimization Strategy

Motion4D builds on top of 3DGS, augmenting each Gaussian element with motion and semantic fields. Motion is modeled via rigid transformations parameterized by global motion bases and per-Gaussian coefficients, allowing efficient representation and interpolation of dynamic trajectories. Semantics are embedded into each Gaussian analogously to color, enabling direct rendering of semantic feature images.

The optimization pipeline comprises the following key modules:

1. Iterative Motion Refinement:
   • Establishes 3D-consistency by supervising estimated 3D tracks and depths against 2D priors, utilizing pixel-wise confidence weights. A learnable uncertainty field on each Gaussian modulates supervision according to temporal self-consistency of color and semantic features.
   • Introduces adaptive resampling which densifies the 3DGS representation by inserting Gaussians in underfit regions identified through per-pixel error metrics, specifically targeting areas with high semantic or RGB error.
2. Iterative Semantic Refinement:
   • Alternates between updating the semantic field in the 3DGS and refining the 2D segmentation priors using rendered 3D masks as input prompts for SAM2. Through bounding box and prompt point heuristics, the method improves semantic alignment and resolves ambiguity without over-constraining SAM2.
3. Sequential and Global Optimization:
   • The sequential stage divides the video into temporal chunks, optimizing motion followed by semantic fields within each. This design mitigates error propagation and memory bias in 2D networks, ensuring local consistency before global joint optimization harmonizes all attributes across the entire sequence.

     Figure 1: Motion4D's pipeline details sequential and global optimization, alongside iterative motion and semantic refinement modules that jointly fuse 2D priors with dynamic 3D representations.

Empirical Evaluation

Motion4D demonstrates substantial improvements over the state-of-the-art in multiple tasks:

• Video Object Segmentation (VOS): On the newly introduced DyCheck-VOS benchmark, created specifically for dynamic scene evaluation, Motion4D achieves $\mathcal{J}$, outperforming the best 2D (SAM2, $89.4$) and 3D methods (SADG, $79.2$). Similar gains are observed on DAVIS 2017 validation.

Figure 2: Motion4D segmentation and tracking results reveal superior 3D consistency compared with leading 2D models, which suffer from flicker and misalignment.

• 2D Point Tracking: On DAVIS and DyCheck datasets, Motion4D surpasses TAPIR, CoTracker3, and BootsTAPIR in Average Jaccard and occlusion accuracy, highlighting robustness to occlusion and scene dynamics.
• 3D Point Tracking and Novel View Synthesis: Motion4D records lower 3D end-point errors ($0.072$) and higher precision in track localization ($\delta_{3D}^{.05}=46.7$) than prior works (e.g., Shape of Motion, HyperNeRF). Visual reconstruction is enhanced, evidenced by higher PSNR ($17.91$) and SSIM ($0.69$) with lower LPIPS ($0.42$).

Ablation Analysis

Extensive ablation studies reveal the necessity of key components:

• Removing iterative refinement of 2D priors or error-driven adaptive sampling significantly reduces segmentation and tracking accuracy.
• Bypassing sequential or global optimization stages produces local inconsistencies and temporal drift, emphasizing the importance of the multi-stage pipeline in achieving holistic coherence.

Theoretical and Practical Implications

Motion4D establishes a new paradigm for dynamic scene understanding by robustly tying 2D foundation model predictions to explicit 3D representations. The joint optimization strategies enable cross-domain consistency and establish a path for high-performance tracking and segmentation in casual, dynamic videos. In practical terms, this unlocks utilities for robotics, AR/VR, and autonomous systems where coherent scene interpretation is essential.

On the theoretical side, the explicit fusion of 2D semantic priors—refined within the optimization process—into a fully dynamic 3DGS model suggests future research in:

• Unsupervised error-driven densification in large-scale streaming settings,
• Extending prompt-based refinement approaches to broader multimodal understanding (e.g., language-augmented perception),
• Further decoupling or jointly learning appearance and motion fields with geometric constraints in low-data regimes.

  Figure 3: Iterative semantic refinement and sequential optimization jointly stabilize spatial and temporal predictions, propagating prompt-driven updates to anomalous 2D priors.

Conclusion

Motion4D introduces an integrated framework for learning 3D-consistent motion and semantics from monocular videos by iteratively refining and unifying 2D foundation model priors with explicit dynamic 3D scene representations. It achieves superior performance on segmentation, point tracking, and view synthesis benchmarks, attributed to its multi-stage optimization and error-driven resampling. These methodological advances provide a foundation for further explorations in dynamic scene representation, prompt-based refinement, and real-world deployment of 3D scene models.