Flow4R: Unifying 4D Reconstruction and Tracking with Scene Flow

Abstract: Reconstructing and tracking dynamic 3D scenes remains a fundamental challenge in computer vision. Existing approaches often decouple geometry from motion: multi-view reconstruction methods assume static scenes, while dynamic tracking frameworks rely on explicit camera pose estimation or separate motion models. We propose Flow4R, a unified framework that treats camera-space scene flow as the central representation linking 3D structure, object motion, and camera motion. Flow4R predicts a minimal per-pixel property set-3D point position, scene flow, pose weight, and confidence-from two-view inputs using a Vision Transformer. This flow-centric formulation allows local geometry and bidirectional motion to be inferred symmetrically with a shared decoder in a single forward pass, without requiring explicit pose regressors or bundle adjustment. Trained jointly on static and dynamic datasets, Flow4R achieves state-of-the-art performance on 4D reconstruction and tracking tasks, demonstrating the effectiveness of the flow-central representation for spatiotemporal scene understanding.

• The paper introduces a novel unified framework that leverages scene flow to predict pixel-aligned 3D positions, motion vectors, pose weights, and confidence scores.
• It employs a minimal property set and a symmetric transformer architecture to seamlessly integrate reconstruction and tracking in dynamic environments.
• Extensive benchmarks demonstrate superior APD3D scores and reduced reprojection errors, outperforming traditional methods with a compact design.

Flow4R: Unified 4D Scene Reconstruction and Tracking via Scene Flow

Overview and Motivation

Flow4R presents a unified approach to dynamic scene understanding by leveraging scene flow as a central representation for both 4D reconstruction and tracking. Traditional pipelines in Structure-from-Motion (SfM) and dynamic tracking dissociate geometry and motion, relying on explicit pose estimation or complex bundle adjustment. The Flow4R model reframes this by simultaneously predicting pixel-aligned 3D positions, scene flow vectors, pose weights, and confidence scores through a Vision Transformer, treating motion as a first-class geometric primitive and enabling seamless reasoning across static and dynamic scenes.

Figure 1: Flow4R takes two images as input at a time and predicts a pixel-aligned property set, including point position $\mathbf{P}$, scene flow $\mathbf{F}$, pose weight $\mathbf{W}$, and confidence $\mathbf{C}$.

Methodology

Minimal Property Set Formulation

The Flow4R framework adopts a concise property set for each pixel: $\mathbf{P}$ (3D point), $\mathbf{F}$ (scene flow), $\mathbf{W}$ (pose weight), and $\mathbf{C}$ (confidence). Scene flow $\mathbf{F}$ is decomposed into rigid (camera) and non-rigid (object) components using a weighted least-squares solution. Pose weights, learned in a self-supervised manner, identify reliable pixels for pose estimation, filtering out non-static regions and unreliable areas. This minimal formulation enables geometry and bidirectional motion inference within a single network pass, bypassing explicit pose heads.

Transformer Architecture and Sequence Processing

Taking two-view input pairs, Flow4R utilizes a symmetric transformer with shared encoder and decoder parameters, capitalizing on cross-attention for coherent feature matching and reducing architectural overhead. When operating on sequences, the model aligns predictions to an anchor view, using scale normalization for consistent reconstruction and tracking. The sequence processing paradigms leverage anchored pairwise connections, offering robust scale alignment not achievable in head-bound approaches.

Figure 2: Sequence processing paradigms.

Loss Functions and Training

Flow4R's supervision encompasses:

• Confidence-weighted point position loss for $\mathbf{P}$,
• 3D motion loss for $\mathbf{P}_{vt}$ (driven by scene flow and point tracking),
• 2D motion loss for projected $\mathbf{p}_{vt}$ (optical flow and 2D tracking),
• Self-supervised pose weight loss aligning predicted pose estimates to ground truth,
• Rigid motion loss leveraging camera pose transformations.

Directly regressing $\mathbf{P}_{vt}$ instead of scene flow $\mathbf{F}$ empirically improves tracking and reconstruction accuracy, as demonstrated by ablation studies.

Experimental Evaluation

Benchmarks and Metrics

Flow4R is evaluated on a mix of real and synthetic datasets, focusing on world coordinate 3D point tracking (Aria Digital Twin, Dynamic Replica, Point Odyssey, Panoptic Studio) and dynamic 3D reconstruction (Point Odyssey, TUM-Dynamics). Metrics include APD3D and EPE, assessing both tracking fidelity and geometric reconstruction accuracy.

Strong Numerical Results

Flow4R achieves superior APD3D scores across most datasets, outperforming baselines such as St4RTrack, MonST3R, and POMATO despite a smaller parameter footprint. Reconstruction results similarly demonstrate high accuracy in both feedforward and globally aligned settings.

Figure 3: Qualitative Results. 3D tracking trajectories projected onto 2D, with Flow4R showing reduced reprojection error across background and foreground.

Figure 4: 3D Visualization of the Predictions. Scenes reconstructed and tracked from diverse datasets, illustrating robust handling of both static and dynamic elements.

Architecture Analysis and Ablations

Ablation studies validate that regressing $\mathbf{P}_{vt}$, supervised directly by its corresponding ground truth, is optimal for both tracking and reconstruction. This choice circumvents the errors introduced by intermediate transformations, underscoring the value of aligning network outputs with final evaluation metrics.

Visualization Insights

Comprehensive visualizations reveal interpretable pose weight maps, accurate scene flow predictions across varied scenarios, and consistent 3D tracking of both static structures and dynamic actors. Pose weight maps consistently highlight regions contributing most to camera localization and filter out dynamic content effectively.

Figure 5: 2D Visualizations of Flow4R Predictions across static and dynamic scenes, demonstrating interpretability in predicted geometry, flow, pose reliability, and confidence.

Implications and Future Directions

Practically, Flow4R's unified flow-centric modeling streamlines 4D perception pipelines, enabling robust feedforward inference on mixed static/dynamic datasets without auxiliary pose regressors or iterative optimization. Theoretically, it advances a compact parameterization for geometric and motion reasoning, with implications for scalable multi-view attention architectures and future data-driven tracking frameworks.

Expanding to multi-view input via global transformer attention (as in VGGT) may further enhance performance and support online tracking in resource-constrained environments. The central limitation remains the availability of annotated scene flow data, which constrains flow quality relative to depth; domain adaptation and synthetic-to-real transfer learning are likely avenues for progress.

Conclusion

Flow4R demonstrates that scene flow, in concert with pose weighting and confidence mapping, provides a minimal, sufficient abstraction for unifying 4D reconstruction and tracking tasks. The model attains state-of-the-art results on challenging benchmarks with a compact transformer architecture, offering theoretical clarity and practical flexibility. Its formulation suggests promising directions for holistic spatiotemporal modeling in vision, where reconstruction, tracking, and perception coalesce under a single coherent framework.