V-DPM: 4D Video Reconstruction with Dynamic Point Maps

Abstract: Powerful 3D representations such as DUSt3R invariant point maps, which encode 3D shape and camera parameters, have significantly advanced feed forward 3D reconstruction. While point maps assume static scenes, Dynamic Point Maps (DPMs) extend this concept to dynamic 3D content by additionally representing scene motion. However, existing DPMs are limited to image pairs and, like DUSt3R, require post processing via optimization when more than two views are involved. We argue that DPMs are more useful when applied to videos and introduce V-DPM to demonstrate this. First, we show how to formulate DPMs for video input in a way that maximizes representational power, facilitates neural prediction, and enables reuse of pretrained models. Second, we implement these ideas on top of VGGT, a recent and powerful 3D reconstructor. Although VGGT was trained on static scenes, we show that a modest amount of synthetic data is sufficient to adapt it into an effective V-DPM predictor. Our approach achieves state of the art performance in 3D and 4D reconstruction for dynamic scenes. In particular, unlike recent dynamic extensions of VGGT such as P3, DPMs recover not only dynamic depth but also the full 3D motion of every point in the scene.

• The paper introduces V-DPM, a scalable feedforward model that directly reconstructs 4D dynamic scenes from monocular video by decoupling time-variant and time-invariant point maps.
• It leverages a time-conditioned transformer decoder with adaptive LayerNorm to jointly handle viewpoint and temporal invariance, achieving an order-of-magnitude reduction in 3D endpoint error.
• The method efficiently integrates static pretraining with dynamic adaptation, demonstrating robust 4D tracking and scene flow estimation on challenging benchmarks like PointOdyssey and Waymo.

V-DPM: 4D Video Reconstruction with Dynamic Point Maps

Introduction and Motivation

V-DPM (2601.09499) addresses the challenge of reconstructing dynamic 3D scenes—i.e., full 4D spatiotemporal reconstructions—from video using feed-forward neural models. Previous advances such as DUSt3R and its extensions demonstrated that viewpoint-invariant point maps enable efficient static 3D reconstruction and multi-view geometry prediction in a single pass. However, these methods are fundamentally restricted to static content and, if extended to dynamic scenes, require auxiliary post-processing or tracking heuristics. Dynamic Point Maps (DPM) provided a representation capable of encoding time and viewpoint invariance, facilitating dense dynamic 3D correspondence and scene flow estimation, but until now were constrained to pairwise settings and dependent on optimization for aggregation across longer sequences.

The core contribution of V-DPM is a scalable mechanism for generating fully dynamic, temporally and spatially consistent point cloud reconstructions directly from monocular video snippets—enabling robust feed-forward 4D tracking, depth estimation, and camera pose recovery for challenging dynamic scenes.

Dynamic Point Maps and Multi-View Generalization

Dynamic Point Maps, as formalized here, are tensor-valued functions $P_i(t_j, \pi_k)$ mapping each source image pixel to a 3D point in a canonical coordinate frame at arbitrary time and reference viewpoint. In the two-frame setting, as in prior DPM work, a minimal set of four point maps suffices for globally consistent point tracking and camera estimation. The paper rigorously extends this logic to $N$ frames (arbitrary-length video), and demonstrates that while the full $N^3$ point map volume is representationally redundant, careful architectural design can extract the maximal signal with far fewer predictions.

The proposed V-DPM pipeline decomposes the prediction of all necessary point correspondences into two primary classes: (1) time-variant point maps $\mathcal{P}$, each expressing all points in a fixed canonical viewpoint $\pi_0$ at their respective native timestamp $t_i$; and (2) time-invariant 'aligned' maps $\mathcal{Q}$, which project all points to a reference time $t_j$ for global point correspondence and flow estimation. This separation allows for efficient and modular decoupling of viewpoint and temporal invariance, while remaining faithful to the underlying geometry.

Figure 1: V-DPM point maps. The time-variant maps $\mathcal{P}$ encode per-frame geometry, while time-invariant maps $\mathcal{Q}$ enable global correspondence/alignment.

Network Architecture and Conditioning Mechanisms

Building on the high-performing VGGT backbone (designed for static scene 3D reconstruction), V-DPM introduces adapters and decoding heads to generalize from static to dynamic content. The approach leverages the shared architecture and statistics between static and dynamic point maps to reuse backbone weights and inject motion-awareness via targeted heads and conditioning.

Figure 2: Model architecture of V-DPM. The backbone produces time-variant point maps, and decoders yield time-invariant maps via explicit time conditioning.

The pivotal component is the time-conditioned transformer decoder, integrated atop the backbone token representations. This decoder receives as input both the latent features and an auxiliary target time token (encoding the reference $t_j$) and is conditioned via adaptive LayerNorm (adaLN), inspired by FiLM and DiT. This enables explicit reasoning about temporal alignment across frames and permits inference of scene state at any queried time index, without redundant backbone computation.

Figure 3: Transformer block in the time-conditioned decoder, employing adaptive LayerNorm for explicit temporal conditioning.

This mechanism enables the model to jointly reason about spatial (viewpoint) and temporal (motion) invariances, crucial for dense flow estimation and for generating scene-consistent point clouds at arbitrary time/reference frames.

Training and Data

Fine-tuning is performed atop the large-scale static pre-trained VGGT, requiring only modest quantities of synthetic dynamic data for effective motion adaptation. The model is supervised using a mixture of static (ScanNet++, BlendedMVS) and dynamic (Kubric-F, Kubric-G, PointOdyssey, Waymo) datasets, with ground truth point maps normalized and scale recovery handled as in VGGT.

By decoupling training into static and dynamic signals, and supervising both time-variant and time-invariant point maps, V-DPM benefits from the scale and diversity of static datasets while learning robust dynamic correspondences on a smaller curated set.

Empirical Results and Qualitative Analyses

The 4D tracking and reconstruction performance of V-DPM is evaluated on multiple public benchmarks, including PointOdyssey, Kubric-F, Kubric-G, and Waymo. V-DPM demonstrates an order-of-magnitude reduction in 3D end-point error (EPE) relative to prior state-of-the-art methods, most notably when leveraging the full video context as opposed to pairwise reconstruction.

Qualitative results illustrate the superiority of V-DPM in reconstructing complex scene flow and maintaining static/dynamic background separation in scenarios including manipulation and challenging human motion:

Figure 4: Dynamic point maps of a robot performing a manipulation task, illustrating robust motion capture and background disambiguation.

Figure 5: Qualitative comparison of dynamic 3D tracking on DAVIS. V-DPM yields coherent point cloud trajectories and accurate final state reconstructions.

Notably, V-DPM retains high fidelity in both short- and long-range motion tracking, outperforming methods which either cannot handle novel motions or are limited to static depth with auxiliary 2D trackers. When constructed over entire video snippets, the model's accuracy does not degrade with sequence length, a marked contrast to standard pairwise methods.

Video and depth/camera pose estimation tasks further confirm the utility of point map representations for simultaneous trajectory, camera, and scene geometry recovery—validated by competitive performance relative to methods trained on much more extensive data or using heavier optimization-based processing:

Figure 6: Optimized result for video depth and camera pose recovery, showing globally consistent scene and viewpoint estimation.

Theoretical and Practical Implications

The design of V-DPM demonstrates that (1) dynamic, time-aware point map representations are sufficient for dense 4D tracking, camera localization, and geometry estimation, and (2) these can be predicted efficiently in a feed-forward manner using architectures originally specialized for static scenes. The explicit handling of alignment between per-frame and globally consistent point maps is essential for accurate motion disambiguation, dense correspondence, and multi-frame scene flow.

From a theoretical standpoint, this work elucidates the representational sufficiency of DPMs beyond the pairwise regime, and the practical ability to extract this information in a network-friendly format, opening the way to tractable, large-scale 4D scene understanding from video.

On the application side, V-DPM is immediately relevant for computer vision tasks such as SLAM, robotics (manipulation, navigation in dynamic environments), AR/VR content creation, VFX pipelines, and general world modeling—whenever dense dynamic scene geometry is critical. Owing to its architectural modularity, V-DPM can later be ported to or built atop even stronger backbones (e.g., $\pi^3$ or VGGT-X) for further gains.

Future Directions

Key future directions include scaling V-DPM to longer sequences and larger datasets, leveraging stronger pretrained static backbones, and integrating generative modeling (e.g., diffusion or flow matching frameworks) for robust scene uncertainty quantification and temporal coherence. Integration of semantic and instance-level understanding within DPMs remains largely open and will be crucial for real-world scene parsing and embodied AI.

Furthermore, extending V-DPM towards real-time deployment and resource-constrained inference could open new horizons for adaptive world model-based decision-making agents, e.g., in robotics and embodied perception.

Conclusion

V-DPM introduces an efficient, principled scheme for direct 4D reconstruction from monocular videos using Dynamic Point Maps, with a design that exploits static 3D pretraining for rapid, data-efficient adaptation to dynamic settings. The work substantially advances video-based scene flow and motion-consistent depth estimation, surpassing prior feed-forward methods in both tracking accuracy and computational efficiency. By decomposing temporal and spatial invariance and leveraging modular network extensions, V-DPM paves the way for future general-purpose world models in scene understanding, robotics, and content synthesis.