MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAE

Abstract: We introduce MotionCrafter, a video diffusion-based framework that jointly reconstructs 4D geometry and estimates dense motion from a monocular video. The core of our method is a novel joint representation of dense 3D point maps and 3D scene flows in a shared coordinate system, and a novel 4D VAE to effectively learn this representation. Unlike prior work that forces the 3D value and latents to align strictly with RGB VAE latents-despite their fundamentally different distributions-we show that such alignment is unnecessary and leads to suboptimal performance. Instead, we introduce a new data normalization and VAE training strategy that better transfers diffusion priors and greatly improves reconstruction quality. Extensive experiments across multiple datasets demonstrate that MotionCrafter achieves state-of-the-art performance in both geometry reconstruction and dense scene flow estimation, delivering 38.64% and 25.0% improvements in geometry and motion reconstruction, respectively, all without any post-optimization. Project page: https://ruijiezhu94.github.io/MotionCrafter_Page

• The paper introduces a feed-forward framework that jointly reconstructs 3D geometry and motion using a unified 4D VAE latent space.
• It leverages diffusion-based denoising with pretrained video priors to achieve up to 38.64% improvement in geometry and 25.0% in motion accuracy.
• The method demonstrates robust zero-shot generalization and challenges strict scale normalization, paving the way for flexible 4D scene reconstruction.

MotionCrafter: Feed-Forward Joint Geometry and Motion Reconstruction via 4D Latent VAE (2602.08961)

Problem Formulation and Motivation

MotionCrafter tackles feed-forward 4D geometry and motion reconstruction from monocular video—simultaneously providing dense world-consistent 3D point maps and scene flow. The methodology reflects the real-world interplay between geometry and motion, crucial for downstream domains (robotics, video understanding, world models). Prior paradigms often separated geometry estimation from motion tracking, or relied on laborious per-scene optimization and post-processing. MotionCrafter asserts the necessity of modeling geometry and motion jointly within a shared world-centric coordinate system, explicitly capturing dynamic deformation and consistent spatiotemporal relationships across sequences.

Unified 4D Geometry-Motion Representation

Key to the framework is the unified representation of both geometry and motion: for each pixel in frame $i$, a world-coordinate 3D point $\mathbf{X}_i$ is predicted, along with a 3D scene flow $\mathbf{V}_i$ from $i$ to $i+1$. The deformation $\mathbf{X}_i^d = \mathbf{X}_i + \mathbf{V}_i$ should spatially correspond to points in the next frame, but one-to-one mappings can be ambiguous due to occlusion and viewpoint changes (Figure 1).

Figure 1: Geometry and Motion representation—world-space 3D points and motion vectors, with inherently imperfect pixel-level correspondence across frames.

This world-centric abstraction eliminates camera-induced effects, simplifies temporal consistency, and enables robust tracking of both background and dynamically appearing objects, setting it apart from prior works that restrict flow estimation to pairwise frame relationships.

Model Architecture: 4D VAE and Diffusion Integration

MotionCrafter's architecture fuses a dedicated 4D VAE into a diffusion-based pipeline. Geometry and motion are encoded in separate VAEs, then concatenated to form the joint 4D latent. Crucially, the architecture leverages pretrained SVD (Stable Video Diffusion) priors—video latents are provided as conditional context to guide the diffusion denoising UNet, but noise is added only to the 4D latent during training (Figure 2).

Figure 2: MotionCrafter architecture—joint geometry-motion latent, conditioned by SVD video latents, with noise addition restricted to the 4D latent.

A central claim is that strict latent alignment with pretrained video VAE distributions is unnecessary. Rather than rescaling 3D data to $[-1,1]$, MotionCrafter normalizes point maps using mean-centric isotropic scaling—preserving metric structure and improving VAE and diffusion generalization. This contradicts widely held beliefs from prior geometric diffusion works, which insisted on range alignment to inherit model priors.

Training Paradigm and Data

Being inherently ill-posed, the approach uses large-scale synthetic datasets for both geometry and joint geometry-motion supervision. The authors train the Geometry VAE first, then Motion VAE with the geometry branch frozen, finally integrating the 4D latent for diffusion UNet training. The batch sizes, learning rates, and backbone initialization are designed for maximal inheritance of pretrained video priors.

Figure 3: Representative training samples for geometry and motion branches drawn from diverse synthetic datasets.

Numerical Results and Ablations

MotionCrafter yields pronounced empirical improvements versus leading feed-forward and optimization-based methods. Geometry reconstruction (relative point error, $\delta^{p}$ inlier ratio) and scene flow estimation (EPE, APD metrics) are reported in the world coordinate system for rigorous evaluation. Across five benchmarks (Kubric, Spring, VKITTI2, Dynamic Replica, Point Odyssey), MotionCrafter outperforms state-of-the-art methods by 38.64% in geometry and 25.0% in motion, without any post-refinement.

Qualitative samples demonstrate superior structural fidelity and temporally coherent scene flow estimation (Figure 4, Figure 5). Ablations reveal:

• Mean normalization and full VAE retraining outperform max rescale and decoder-only fine-tuning (Figure 6).
• Unified geometry-motion latent fusion—albeit yielding slightly lower VAE reconstruction—results in better diffusion-based prediction.
• Deterministic training paradigm improves geometry metrics over denoising diffusion.

  Figure 6: Comparison of normalization/training strategies—mean normalization enables robust scene reconstruction, especially for challenging outdoor depth variations.

  

  Figure 4: Qualitative comparison with Zero-MSF—MotionCrafter produces more accurate scene structure and motion direction.

  

  Figure 5: Comparison with ST4RTrack—cleaner, temporally consistent geometry and scene flow trajectories.

Generalization and Zero-Shot Evaluation

The architecture demonstrates strong zero-shot performance on in-the-wild Davis and dynamic datasets, even in cases where motion supervision is unavailable during training, maintaining robustness across diverse scenes and dynamic regimes (Figure 7).

Figure 7: Zero-shot generalization on Davis—consistent geometry and motion estimation across scenes without any post-optimization.

Implications and Future Directions

Practically, MotionCrafter's feed-forward, world-centric 4D reconstruction pipeline opens avenues for enabling real-time, dense understanding of unconstrained dynamic scenes without reliance on specialized sensors or slow optimization. Theoretical implications are substantial: the relaxed distribution alignment strategy challenges established wisdom regarding the necessity of strict scale matching for inheriting diffusion priors, suggesting a more flexible transfer of generative knowledge across modalities.

The authors highlight potential limitations: the current focus is restricted to geometry and motion; multi-modal integration (e.g., depth, normals, cameras, tracks, view synthesis) is likely to further enhance reconstruction accuracy. Future directions should explore expansion into richer geometric representations, adaptive fusion with semantic/appearance cues, and broader application in embodied AI and closed-loop robotics.

Conclusion

MotionCrafter represents a feed-forward, video diffusion-driven approach for joint geometry and motion estimation, leveraging a unified 4D latent space and relaxed distribution normalization. It achieves state-of-the-art performance with robust generalization, without post-optimization, and proposes a contrary stance to the necessity of strict latent alignment in geometric diffusion models. The findings have strong practical and theoretical implications, advocating for modular latent representations and flexible prior transfer in future 4D modeling architectures.