Efficiently Reconstructing Dynamic Scenes One D4RT at a Time (2512.08924v1)

Abstract: Understanding and reconstructing the complex geometry and motion of dynamic scenes from video remains a formidable challenge in computer vision. This paper introduces D4RT, a simple yet powerful feedforward model designed to efficiently solve this task. D4RT utilizes a unified transformer architecture to jointly infer depth, spatio-temporal correspondence, and full camera parameters from a single video. Its core innovation is a novel querying mechanism that sidesteps the heavy computation of dense, per-frame decoding and the complexity of managing multiple, task-specific decoders. Our decoding interface allows the model to independently and flexibly probe the 3D position of any point in space and time. The result is a lightweight and highly scalable method that enables remarkably efficient training and inference. We demonstrate that our approach sets a new state of the art, outperforming previous methods across a wide spectrum of 4D reconstruction tasks. We refer to the project webpage for animated results: https://d4rt-paper.github.io/.

• The paper introduces a unified feedforward model that decouples spatial and temporal inference to reconstruct dynamic 4D scenes efficiently.
• It employs a global self-attention encoder with Fourier encoded queries to decode dense point clouds, camera parameters, and motion trajectories.
• Quantitative results demonstrate 9–100× speed improvements and superior depth, pose, and point cloud accuracy over traditional models.

D4RT: A Unified Feedforward Model for Dynamic 4D Scene Reconstruction and Tracking

Introduction and Motivation

Dynamic 4D scene reconstruction from video—encompassing spatial, temporal, and camera pose inference—remains a critical but computationally challenging task in computer vision. Prevailing paradigms rely on fragmented, task-specific modules (e.g., MegaSaM uses disparate depth and motion segmentation heads, while VGGT employs modality-specialized decoders), resulting in computational overhead and an inability to recover dense spatio-temporal correspondences in dynamic environments. D4RT introduces a feedforward model leveraging unified querying over a global video scene representation, decoding any point’s 3D position at arbitrary space-time coordinates via a lightweight interface. This avoids the bottlenecks of dense, per-frame decoders and hones scalable, highly efficient training/inference for dynamic scene reconstruction.

Figure 1: D4RT provides unified and efficient feedforward dynamic 4D reconstruction and tracking; point clouds, tracks, and camera parameters are decoded via a single flexible interface.

Methodology

Query-Based Encoding and Decoding

D4RT innovates with a global self-attention encoder that computes a latent representation $F$ for the entire input video sequence. The decoder receives flexible queries $$q = (u, v, t_\text{src}, t_\text{tgt}, t_\text{cam})$$ representing 2D point locations together with temporal and camera reference indices. For each query, the decoder cross-attends independently to $F$, outputting a 3D position in the target reference frame. This design allows:

• Decoupled space-time inference: Queries need not share temporal/camera indices, enabling full disentanglement of spatial and temporal dimensions.
• Arbitrary density and parallelism: Each query is decoded independently, supporting both sparse and dense tasks with maximal compute parallelism.
• Unified multi-modal output: The same querying mechanism yields point clouds, tracks, depth maps, and camera parameters across static and dynamic scenes.

  Figure 2: D4RT model overview—global encoding followed by latent querying yields scalable, independent 3D estimation for arbitrary points in space-time.

Camera Parameter Estimation and Dense Tracking

Camera extrinsics between video frames are estimated by querying grids of correspondences and aligning outputs with rigid transformations (Umeyama algorithm), facilitating pose estimation without needing explicit optimization or costly refinement stages. Intrinsics are computed from distributions of decoded 3D points, optionally incorporating distortion via non-linear refinement. D4RT's dense dynamic correspondence algorithm efficiently tracks all pixels in a video using an adaptive spatio-temporal occupancy grid, driving speedups over prior approaches.

Model Architecture and Training

The encoder is a spatio-temporal Vision Transformer (ViT-g), augmented with aspect ratio tokens. The decoder is a small cross-attention transformer. Queries are Fourier encoded and concatenated with local $9\times9$ RGB patches for improved correspondence and detail recovery. Training proceeds via a weighted composite loss over sampled query batches, with primary supervision on normalized 3D positions, auxiliary losses over 2D projections, normals, motion vectors, visibility, and confidence scores, employing advanced data augmentation and curriculum sampling for regions near depth and motion discontinuities.

Experimental Results and Quantitative Evaluation

Efficiency and Speed

D4RT achieves superior pose estimation accuracy and throughput compared to state-of-the-art baselines. On A100 GPUs, D4RT delivers 200+ FPS for pose estimation—9$\times$ faster than VGGT and 100$\times$ faster than MegaSaM—while maintaining or exceeding accuracy.

Figure 3: D4RT achieves high throughput and state-of-the-art pose accuracy—9–100$\times$ speedup over leading baselines, with higher accuracy.

4D Reconstruction and Dense Tracking

Qualitative and quantitative analyses demonstrate that previous feedforward models (e.g., MegaSaM, $\pi^3$) are fundamentally limited in reconstructing dynamic scenes, producing repeated or incomplete entities. Tracking-based models (SpatialTrackerV2, St4RTrack) capture dynamics but only track sparse points, leaving gaps in reconstructions. D4RT reconstructs full 4D scene representations by tracking all video pixels and capturing dynamic motion trajectories in a unified reference frame.

Figure 4: D4RT uniquely reconstructs complete 4D scenes and dense dynamic tracks, outperforming prior methods which show structural failure on dynamic entities.

Figure 5: D4RT demonstrates robust static and dynamic scene reconstructions in-the-wild, including dense 3D trajectories of moving objects.

Depth Estimation and Point Cloud Reconstruction

Across MPI Sintel, ScanNet, KITTI, and Bonn datasets, D4RT attains lowest mean L1 distance in point cloud reconstruction and top-tier depth estimation under both scale-only and scale-shift alignments, especially excelling on highly dynamic scenes.

Camera Pose Estimation

On static indoor and dynamic outdoor scenes, D4RT consistently outperforms all baselines (MegaSaM, VGGT, $\pi^3$, SpatialTrackerV2) in ATE, RPE, and Pose AUC, with markedly improved Sim(3)-aligned pose estimation.

Ablation Studies

• Local appearance patch: Augmenting queries with local RGB patches leads to sharper object boundaries and improved correspondence.
• Encoder scaling: Larger Vision Transformers (ViT-B to ViT-g) significantly boost depth and pose reconstruction accuracy.
• Auxiliary loss components: All loss terms contribute measurably to overall performance, with critical trade-offs between tasks.
• High-resolution decoding: D4RT supports subpixel precision querying; leveraging high-res source patches recovers fine geometric structure without increasing backbone memory overhead.

Implications and Future Directions

D4RT’s feedforward querying paradigm sets a new standard in unified dynamic scene perception, optimizing for scalability, parallelism, and multi-task capability. Practically, it enables efficient large-scale 4D video analysis for robotics, AR/VR, and autonomous systems without the overhead of iterative or per-frame processing. Theoretical implications include the ability to decouple scene representation from decoding, fostering extensibility to arbitrary tasks (e.g., future spatio-temporal attribute inference, long-horizon predictive modeling). Further, D4RT’s architecture is well-suited for scaling to longer video sequences and higher resolutions without architectural modifications.

Potential directions include embedding physics-informed priors for real-world generalization, extending querying to attribute or semantic inference, integrating with multimodal sensors (e.g., LiDAR, event cameras), and investigating transformer-based meta-learning strategies for domain adaptation in real-time systems.

Figure 6: Full D4RT model overview with explicit inputs and outputs; supporting high scalability via disentangled encoding and querying.

Figure 7: D4RT generalizes to long video sequences (KITTI), yielding robust alignment and consistent reconstruction across thousands of frames.

Figure 8: Additional qualitative results—D4RT preserves reconstruction fidelity for both static and dynamic environments, highlighted in multiple video scenarios.

Figure 9: Comparative baseline reconstructions—D4RT uniquely tracks all video pixels and dynamics, as opposed to the sparsity or redundancy in prior models.

Figure 10: D4RT achieves finer dense depth estimation and geometric accuracy, particularly in scenes with significant motion blur.

Conclusion

D4RT establishes a highly scalable, unified approach for dynamic 4D scene reconstruction and tracking, leveraging feedforward querying over global video representations. Its efficient encoder-decoder design circumvents the limitations of per-frame dense decoders and modular pipelines, attaining state-of-the-art performance in both speed and accuracy across multi-modal 4D scene tasks. D4RT paves the way for next-generation 4D perception systems, combining computational efficiency with strong empirical performance.