- The paper introduces a dynamic 3D Gaussian Splatting framework conditioned on video diffusion, using appearance, depth, normals, opacity, and 3D scene flow to generate multiview-consistent virtual observations.
- The paper distills generated videos back into the Gaussian representation through joint appearance, depth, track, motion, and regularization losses, improving DyCheck motion accuracy from 0.824 to 0.862 [email protected].
- The paper achieves state-of-the-art DyCheck results, reaching up to 19.96 mPSNR and 0.218 mLPIPS, while generalizing across reconstruction backbones and enabling visual outpainting and unseen-motion inference.
World from Motion (WfM) addresses a central tension in monocular 4D reconstruction: optimization-based methods such as Shape of Motion (SoM), MoSca, and WorldTree recover well-observed geometry and motion with high fidelity but cannot infer unseen content or dynamics, while generative video models can hallucinate missing regions but lack the multiview consistency needed for precise reconstruction. The paper proposes a framework that resolves this tension by conditioning a video diffusion model on a dynamic 3D Gaussian Splatting (3DGS) representation and distilling its outputs back into that representation, yielding a single freely renderable dynamic 3DGS that improves both novel-view synthesis and 3D motion estimates (2607.01202).
Method overview
The pipeline begins with an off-the-shelf monocular 4D reconstruction (e.g., MoSca, SoM, or WorldTree) producing an initial dynamic 3DGS G0​, whose Gaussians have time-varying centers and orientations mapped from a canonical state by a low-rank motion model (motion bases, scaffolds, or temporal trees). WfM then proceeds in two stages: generation of virtual observations from fixed novel cameras, followed by re-optimization of G0​ using both the input video and the generated videos.
Dynamic 3DGS-conditioned video generation
The core technical contribution is the conditioning scheme. For any camera at time t, the initial reconstruction is rasterized into a per-frame bundle of appearance, opacity, depth, surface normals, and 3D scene flow — the latter obtained by rasterizing per-Gaussian displacements ui​(t)=μi​(t+1)−μi​(t) into a per-pixel flow map. Crucially, these 4D buffers are rendered along both the input trajectory and the target trajectory; conditioning only on the target trajectory, as in prior static work, can fail to anchor generated content to the observed input. The buffers are encoded with a video VAE, concatenated into a temporal sequence, and processed by a VACE-style DiT adapter whose residual features are injected into a frozen-base Wan 2.1 14B video DiT, with additional frame-level camera pose embeddings.
A notable design element is reconstruction guidance: during training, 4D-buffer tokens are randomly replaced with a learned null embedding while input-video and camera conditioning are always retained, so the "unconditional" branch corresponds to a ReCamMaster-like model. At inference, a classifier-free-style guidance update steers generation toward the reconstruction-conditioned prediction, with timestep-dependent scale; the supplement shows velocity-space APG gives the best mPSNR/mSSIM trade-off while the x0​ APG variant achieves the lowest mLPIPS. Because conditioning derives from rendered buffers rather than representation-specific parameters, the model generalizes across underlying 4D representations — a model trained only on SoM-derived data improves MoSca and WorldTree reconstructions by +1.66 and +1.52 PSNR respectively, though SSIM slightly degrades in this cross-representation setting.
Distillation into dynamic 3DGS
Target viewpoints are selected by farthest-point sampling along the input trajectory and held fixed over time; the paper reports this simple strategy suffices, in contrast to the more involved multi-view and temporal sampling of CAT4D. Generated videos are then used to re-optimize the representation: joint depth estimation (Depth Anything 3) across input and generated views back-projects new Gaussians to fill in synthesized content, and the full set of Gaussian parameters and motion parameters are jointly optimized against a combined loss of input appearance, generated-view appearance, motion, depth, track, and as-rigid-as-possible regularization. Re-optimizing the motion representation is critical: it allows generated samples to correct inaccurate 3D tracks from the initial monocular estimate, and on DyCheck the distilled representation improves [email protected] from 0.824 (MoSca) to 0.862, exceeding CoTracker (0.803) and BootsTAPIR (0.779).
Training data construction
The model is trained on MultiCamVideo, forming input/target video pairs and reconstructing an initial dynamic 3DGS from the input view with simulated monocular artifacts. Two data-pipeline contributions are emphasized. First, joint depth alignment across views resolves the metric-scale ambiguity of monocular reconstruction, which otherwise causes the generator to ignore the conditioning signal — its ablation shows the largest single degradation in the model ablation (PSNR dropping from 19.50 to 15.58). Second, temporal cropping and reversal debias the shared-starting-viewpoint bias of MultiCamVideo. Training uses 150K 73-frame pairs at 512×384, fine-tuning the adapter, camera encoder, and base-DiT self-attention layers on 32 GB200 GPUs for roughly 48 hours.
Results
On DyCheck under a corrected static-camera protocol, WfM achieves state-of-the-art results. The paper makes a pointed methodological claim here: prior methods (MoSca, WorldTree, ViDAR) perform per-frame test-time pose optimization that violates DyCheck's static-camera assumption, trading pose accuracy for better dynamic-region metrics; the authors recalibrate baselines to fixed poses and show this improves MoSca's static-region metrics but lowers its overall numbers.
| Method (DyCheck, covisible) |
mPSNR |
mSSIM |
mLPIPS |
| MoSca |
18.69 |
0.696 |
0.272 |
| WorldTree |
19.28 |
0.714 |
0.246 |
| ViDAR |
19.44 |
0.711 |
0.224 |
| WT + Ours (distill) |
19.96 |
0.724 |
0.218 |
| MS + Ours (distill) |
19.78 |
0.710 |
0.180 |
The generated samples alone already outperform all baselines in PSNR and LPIPS, and distillation improves them further. Against generative baselines on DyCheck, WfM (19.89 mPSNR, 0.715 mSSIM, 0.197 mLPIPS) substantially outperforms CAT4D (18.24/0.666/0.227) and concurrent Vista4D (14.14/0.310/0.514), which the authors attribute to conditioning on a geometrically aligned 4D representation rather than weak point-cloud conditioning. On the MultiCamVideo validation split the margin is largest: Ours (sample) reaches 27.43 PSNR, 0.918 SSIM, and 0.083 LPIPS versus 22.14/0.754/0.198 for the best baseline ViDAR. Qualitatively, the method performs visual outpainting, repairs degraded dynamics, and infers out-of-frustum dynamics on in-the-wild videos, including Veo-generated footage, while respecting the static region.
Ablations support each component: removing reference-view G-buffers, joint depth alignment, or debiased data each cause significant drops; scaling the backbone from 1.3B to 14B parameters improves results, indicating the approach benefits from larger video generators. On the refinement side, motion re-optimization and depth-based reinitialization matter most, and down-weighting generated views relative to the input slightly improves robustness. Reconstruction quality improves monotonically with virtual camera count, saturating beyond eight cameras (mPSNR 18.69 → 19.78 from 0 to 8).
Limitations and open questions
The paper concedes two failure modes: complete failure of the initial reconstruction cannot be recovered from, and extremely distant target trajectories break alignment between reconstruction and input. Failure cases include volumetric effects such as fluids and smoke, where artifacts in the input reconstruction propagate to the output. The single round of generate-then-distill is also acknowledged as suboptimal; the authors pose as an open question whether multiple alternating rounds of sampling and refinement would yield higher quality. Evaluation remains confined to DyCheck, a small MultiCamVideo validation split, and qualitative in-the-wild demonstrations, leaving systematic in-the-wild benchmarking unresolved.
Conclusion
World from Motion demonstrates that conditioning a video diffusion model on dense, pixel-aligned renderings of a dynamic 3DGS — spanning appearance, geometry, and 3D scene flow along both input and target trajectories — is sufficient to obtain multiview-consistent generations that can be distilled into an improved 4D representation. The result is a state-of-the-art dynamic reconstruction framework on DyCheck that also generalizes across reconstruction backbones and to unconstrained videos, with the caveat that it remains dependent on a non-degenerate initial reconstruction and a single refinement pass.