Flow4R: Unified 4D Reconstruction & Tracking

• Flow4R is a unified framework for 4D reconstruction and tracking that leverages per-pixel scene flow to jointly infer 3D structure, camera motion, and dynamic object motion.
• It employs a two-view Vision Transformer with interleaved self- and cross-attention to fuse geometric and motion cues efficiently.
• The method achieves state-of-the-art 3D tracking and reconstruction benchmarks by eliminating separate pose regressors and bundle adjustment modules.

Flow4R is a unified framework for 4D reconstruction and tracking that formulates camera-space scene flow as a central representation, enabling joint inference of 3D structure, camera motion, and object motion in both static and dynamic scenes using a feed-forward Vision Transformer. It predicts at every image location a set of minimal per-pixel properties—3D point, scene flow, pose reliability, and confidence—thereby connecting geometric and motion cues in a manner that obviates separate pose regression or bundle adjustment frameworks. Flow4R achieves state-of-the-art results in 3D tracking and spatiotemporal reconstruction benchmarks, demonstrating the viability and robustness of a flow-centric geometric formulation (Qian et al., 15 Feb 2026).

1. Motivation and Conceptual Foundations

Traditional multi-view 3D reconstruction pipelines rely on sequential estimation under the assumption of scene rigidity: geometry (scene structure) and motion (camera/objects) are inferred in separate modules using techniques such as structure-from-motion (SfM) and bundle adjustment, while typical dynamic scene trackers dissociate camera motion estimation from per-object or per-pixel motion modeling. This dichotomy yields brittle systems and degrades performance in realistic, dynamic environments where static-scene assumptions fail and isolated motion estimation cannot reconstruct reliable geometry.

Flow4R proposes an alternative design: treat the camera-space scene flow as a unifying representation. Scene flow captures per-pixel 3D displacement between two views, directly encoding both camera-induced and object-induced motion. By predicting a per-pixel property set—comprising 3D position, scene flow vector, pose weight, and confidence—Flow4R enables symmetric, bidirectional, and local inference of both spatiotemporal coherence and structural consistency from view pairs. This eliminates the need for explicit camera pose regressors or separate bundle adjustment modules (Qian et al., 15 Feb 2026).

2. Per-Pixel Property Set and Representation

For a pixel index $i$ in source image $\mathbf I$ (with target $\mathbf I'$), Flow4R predicts:

• $\mathbf P^i \in \mathbb R^3$: 3D point position in the camera frame of $\mathbf I$.
• $\mathbf F^i \in \mathbb R^3$: Scene flow vector (3D displacement from $\mathbf I$ to $\mathbf I'$).
• $W^i \in (0,1)$, $\sum_i W^i = 1$: Pose weight, quantifying the reliability of pixel $\mathbf I$0 for camera pose estimation.
• $\mathbf I$1: Confidence score, dynamically learned to reflect reliability for each pixel.

Notation:

$\mathbf I$2

with $\mathbf I$3, $\mathbf I$4, and $\mathbf I$5.

This decoding enables direct reconstruction of dense 3D geometry (via $\mathbf I$6), camera motion (by fitting a global rigid transform), object motion (from non-rigid scene flow), and pointwise tracking across time, all in a single forward pass.

3. Transformer-based Architecture

Flow4R adopts a two-view Vision Transformer architecture inspired by DUSt3R, with shared encoder and decoder weights for both source and target images (Qian et al., 15 Feb 2026). The workflow comprises:

3.1 Encoder:

Each view is partitioned into non-overlapping patches, embedded as tokens. A stack of $\mathbf I$7 self-attention layers processes each set of tokens to yield per-patch features $\mathbf I$8.

3.2 Cross-Frame Fusion:

Cross-attention is alternated with self-attention in the decoder. Given encoded tokens $\mathbf I$9 and $\mathbf I'$0, cross-attention is computed as:

$\mathbf I'$1

Self- and cross-attention blocks are interleaved to facilitate exchange and comparison of geometric and appearance features between views.

3.3 Output Heads:

The final per-patch features are upsampled to the pixel grid using convolutional projection heads, predicting the four target properties—3D position, scene flow, pose weight, and confidence—enforcing parameteric efficiency and full symmetry between views.

4. Geometric Formulation and Losses

The central geometric principle in Flow4R is to predict the "view-and-time-transferred" point:

$\mathbf I'$2

This point represents the predicted 3D location in the target view/camera after applying the estimated scene flow.

Camera pose is inferred by solving a weighted least-squares problem over all pixels:

$\mathbf I'$3

Scene flow is decomposed as:

• Rigid (camera) motion: $\mathbf I'$4
• Non-rigid (object) motion: $\mathbf I'$5

Loss functions include:

1. Point-Position Loss:

$\mathbf I'$6

2. 3D Scene Flow Loss:

$\mathbf I'$7

3. 2D Optical Flow Loss:

$\mathbf I'$8

4. Pose Weight Loss:

$\mathbf I'$9

5. Rigid Motion Loss:

$\mathbf P^i \in \mathbb R^3$0

The total loss combines these terms:

$\mathbf P^i \in \mathbb R^3$1

with coefficients $\mathbf P^i \in \mathbb R^3$2, $\mathbf P^i \in \mathbb R^3$3, $\mathbf P^i \in \mathbb R^3$4.

5. Training and Evaluation Protocol

Training strategy consists of two stages, leveraging both static and dynamic datasets:

• Stage 1: 100 epochs at $\mathbf P^i \in \mathbb R^3$5 resolution with a linear prediction head; 900K frame pairs per epoch.
• Stage 2: 100 epochs at $\mathbf P^i \in \mathbb R^3$6 random aspect ratio; DPT head, 84K pairs per epoch. Adam optimizer with linear warmup and cosine decay is employed. Batch sizes are 256 (Stage 1) and 64 (Stage 2) across 8 × A100/H100 GPUs.

Diverse datasets are included: Habitat, BlendedMVS, MegaDepth (static), Virtual KITTI 2, Spring, PointOdyssey, Dynamic Replica, Kubric, OmniWorld-Game (dynamic). Pairs of frames are sampled uniformly, with anchoring for scale consistency.

Evaluation benchmarks target both 3D point tracking (WorldTrack) and 4D reconstruction (Point Odyssey, TUM-Dynamics). Metrics include APD3D (percentage of points within a set 3D error threshold) and EPE (mean end-point error). Flow4R achieves top APD3D on most splits—e.g., 78.6 (ADT), 78.5 (DR), 71.1 (PO), 64.3 (PS)—with fewer parameters (0.4B) than alternatives.

On Point Odyssey, Flow4R yields APD3D = 81.0 with EPE = 0.182, outperforming MonST3R (APD3D = 72.3, EPE = 0.263) and other transformer-based methods (Qian et al., 15 Feb 2026).

6. Comparative Results and Ablations

Flow4R is compared with MonST3R, St4RTrack, POMATO, DUSt3R, MASt3R, and SpaTracker. Its unified, flow-centric architecture consistently delivers higher APD3D across datasets and maintains competitive EPE without external post-optimization procedures.

Ablation studies address model variants:

Variant	ADT	DR	PO	PS	PO Reconst APD	TUM APD
Predict $\mathbf P^i \in \mathbb R^3$7, supervise $\mathbf P^i \in \mathbb R^3$8	78.0	73.3	60.2	55.8	69.4	79.8
Predict $\mathbf P^i \in \mathbb R^3$9, supervise $\mathbf I$0	77.7	76.4	61.2	63.7	66.3	80.1
Predict $\mathbf I$1, supervise $\mathbf I$2	78.5	78.5	67.9	67.2	77.2	80.3

Direct supervision on the point-centric target $\mathbf I$3 yields the best tracking and reconstruction outcome, confirming alignment with evaluation protocols.

Qualitative visualizations show that Flow4R learns discriminative pose weights that down-weight unreliable (e.g., dynamic or untextured) regions and confidence maps that attenuate occlusions or out-of-view areas.

7. Impact and Extensions

Flow4R demonstrates that a single camera-space scene-flow representation with jointly predicted per-pixel geometry and motion properties suffices to unify 4D spatiotemporal understanding without reliance on explicit pose heads or post-hoc adjustment (Qian et al., 15 Feb 2026). The model’s transformer-based, symmetric architecture allows extensibility to mixtures of static and dynamic scenes and provides a robust basis for future research in feed-forward 4D reconstruction and tracking.

Possible extensions—though not detailed in current results—include exploration of larger-capacity or multi-scale transformers, inclusion of additional sensor modalities, or adaptation for unsupervised or self-supervised regimes. A plausible implication is that such flow-centric approaches may set new baselines for domains where both fine-grained geometric and temporal coherence are required.