ARROW: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild

Abstract: Dynamic scenes may be captured by a moving camera, multiple video streams, or images taken at different times. These observations reveal complementary aspects of scene geometry and motion, yet bringing them together requires establishing correspondence across viewpoints, capture times, and visibility changes. We introduce ARROW, a feed-forward model that unifies 3D reconstruction and 3D point tracking from arbitrary image sets. At its core is a novel order-invariant querying approach, which allows the association of queries with observations across arbitrary inputs. We show that exposing the model to more diverse sets of inputs during training results in improved task performance. Moreover, the resulting model is capable of generalization to a wider range of tasks including multi-view tracking. Trained with this strategy, ARROW establishes a new state of the art in 3D tracking on WorldTrack and TAPVid-3D and outperforms dedicated multi-view trackers on an adapted RGB-only MVTracker benchmark, while remaining competitive across 3D reconstruction tasks. Code and weights are publicly available.

• The paper introduces ARROW, a unified model for 4D reconstruction and point tracking, leveraging content-derived ID tokens for order-invariant queries, and achieves top results and competitive results compared to specialized 3D trackers.
• ARROW achieves the state-of-the-art tracking on datasets such as Aria Digital Twin, Dynamic Replica, PointOdyssey, and Panoptic Studio outperforming specialized baselines.
• The model demonstrates versatility across various applications including dense reconstruction, monocular 3D tracking, visibility prediction, surface normals, and camera recovery using a single decoding architecture.

Problem formulation and central contribution

ARROW addresses the separation between multi-view reconstruction and point tracking in dynamic scenes. Existing 3D trackers generally assume temporally ordered video, while multi-view reconstruction methods commonly operate on unordered image sets but do not preserve point identity across observations. The paper argues that these assumptions are unnecessarily restrictive: observations of a scene may originate from multiple cameras, non-contiguous times, or static image collections, and a unified model should support all three cases without imposing an artificial temporal order (2610.01314).

The proposed model represents an input as an unordered set of RGB observations. A query specifies a source pixel and three observation roles: source, target, and reference. The source pixel identifies the physical point, the target observation specifies the requested capture time, and the reference observation defines the coordinate system in which the predicted 3D position is expressed. This interface supports sparse point trajectories, dense reconstruction, visibility prediction, surface normals, camera recovery, and source-to-target displacement within one decoder.

The principal architectural contribution is an order-invariant addressing mechanism. D4RT-style models use learned temporal embeddings to associate query roles with positions in an ordered video representation. Such embeddings are unsuitable when observations have no canonical order or when multiple cameras share a timestamp. ARROW therefore derives an identity token for each observation from its visual content. These ID tokens are produced by repurposing the per-image camera tokens of a Depth Anything 3 encoder. Because both the global image-set representation and the ID tokens occupy the same learned feature space, query embeddings can identify source, target, and reference observations without positional encodings.

Figure 1: ARROW jointly supports 4D reconstruction, monocular 3D tracking, and RGB-only multi-view tracking from arbitrary observation sets.

Order-invariant querying

The paper’s key conceptual distinction is between indexed correspondence and input-dependent correspondence. In an ordered video, a query can refer to an observation through a learned timestep embedding. That mechanism implicitly assumes that the $i$-th embedding always denotes the $i$-th temporal position. Once the encoder is made permutation-invariant, this association disappears: the global context contains image information but no fixed index that a query can exploit.

ARROW resolves this mismatch by using content-derived identities. The multi-view encoder processes each image independently before applying global attention across all patches. It outputs a shared context and one ID token per observation. The query encoder combines four components:

• a source-centered $9 \times 9$ RGB crop;
• Fourier features of the normalized source coordinate $(u,v)$;
• the source-observation ID token;
• the target- and reference-observation ID tokens.

The resulting query token cross-attends independently to the global context. Queries do not self-attend, which permits arbitrary numbers of sparse queries and allows the same encoded scene representation to be reused for different decoding demands.

Figure 2: Content-derived ID tokens replace order-dependent temporal embeddings, allowing queries to address observations by visual identity rather than by index.

This design has an important representational consequence. The source, target, and reference roles are relational rather than sequential. The model can therefore distinguish two observations captured simultaneously by different cameras, request a point state at one observation time in the coordinate system of another, and process unordered image collections without introducing a spurious temporal direction.

The decoder predicts a 3D point, confidence, target image coordinates, visibility, surface normal, and displacement. All quantities are conditioned on the same query and are expressed at the target time; 3D quantities are represented in the reference-observation coordinate system. Camera poses are not predicted by a separate pose head. Instead, they are recovered by confidence-weighted rigid alignment of corresponding predicted point sets.

Figure 3: The architecture combines a set-based multi-view encoder, content-derived observation identities, and independent sparse query decoding.

Training strategy and supervision

ARROW is trained with a multi-task objective dominated by a scale-invariant, confidence-weighted 3D loss. Auxiliary terms supervise image coordinates, visibility, surface normals, and displacement. The formulation allows static geometry and dynamic tracks to contribute to a common objective without requiring a dedicated tracking head or a globally shared world coordinate system.

The training mixture contains approximately 78,000 scenes and 22.8 million camera frames, with 78% of the sampling probability assigned to dynamic data and 22% to static data. It combines monocular videos, multi-camera sequences, synthetic dynamic scenes, and static multi-view reconstruction datasets. The model is trained on image sets containing between two and 24 observations, with image selection extending beyond local temporal neighborhoods. For dynamic data, the candidate sampling window can be up to ten times larger than the requested observation set. Multi-camera sampling may also select images from different streams.

This sampling strategy is not merely a data-scaling choice. It directly changes the correspondence distribution seen during training. The model encounters temporally adjacent frames, distant observations, cross-camera views, and unordered static images under the same query interface. The paper’s central claim is therefore stronger than simple multitask sharing: training on non-sequential and multi-view configurations improves conventional temporal tracking as well as generalization to arbitrary observation sets.

The sampling policy also targets difficult geometric regions. Queries are oversampled near depth and normal discontinuities, motion boundaries, dynamic interiors, and visible tracks. In scenes without reliable cross-time annotations, such as nuScenes, cross-time motion supervision is avoided and same-time geometry queries are retained. This restriction is methodologically important: it prevents the model from learning incorrect motion targets while still using the data for geometric supervision.

Ablation evidence

The cumulative ablations separate the contributions of identity conditioning, removal of temporal encodings, non-sequential sampling, and broader training data. Replacing order-dependent query embeddings with content-derived identities improves both monocular and multi-view tracking. The benefit is particularly pronounced when inference exceeds the training context length and temporal embeddings must otherwise be interpolated.

Removing explicit encoder temporal embeddings has mixed effects but does not materially damage monocular tracking. This supports the paper’s claim that temporal order is not necessary for the principal tasks once observation identity is content-derived. The largest multi-view improvement comes from non-sequential sampling: relative to the preceding ablation, it improves multi-view APD by 28.85 percentage points. Adding four static multi-view datasets then improves every reported metric.

The ablations also establish that the gains do not arise solely from freezing or replacing the geometric backbone. Freezing the encoder reduces average WorldTrack APD by 8.71 percentage points and increases EPE by 0.0914 m, showing that end-to-end adaptation of the DA3 representation is important. A patch-feature query initialization converges faster initially but produces lower final tracking accuracy than the independent RGB-crop and Fourier-coordinate design. This result favors explicit query-local appearance and coordinate encoding over direct interpolation of encoder features.

Monocular tracking results

On WorldTrack, ARROW achieves the best overall and dynamic-only APD and EPE on Aria Digital Twin, Dynamic Replica, PointOdyssey, and Panoptic Studio. Its all-point APD scores are 93.55, 88.00, 85.26, and 85.76, respectively. The corresponding EPE values are 0.0818, 0.1426, 0.1532, and 0.1332 m. Dynamic-only APD reaches 90.63 on Aria Digital Twin, 91.47 on Dynamic Replica, 86.72 on PointOdyssey, and 85.76 on Panoptic Studio.

These results are notable because ARROW is not restricted to temporally local video training. It exceeds specialized temporal or 4D baselines while using the same feed-forward query interface for static reconstruction and multi-view correspondence. The implication is that broader observation diversity does not necessarily dilute temporal tracking supervision; in this setting, it strengthens the model’s ability to resolve appearance changes and long-range correspondence.

On TAPVid-3D, ARROW obtains the highest average camera-coordinate APD and AJ among the reported methods, with average values of 0.341 and 0.248. It also leads in world-coordinate APD and normalized L1, reaching 0.385 and 0.041. Its occlusion accuracy is weaker than the strongest baselines, however: the average camera-coordinate OA is 0.863, below D4RT’s 0.899. Thus the model’s stronger geometric localization does not uniformly translate into superior visibility classification.

The benchmark includes sequences of up to 300 frames and arbitrary source frames. ARROW evaluates targets against the full encoded context rather than relying on window handoffs. Full-context inference outperforms windowed inference for ARROW at longer sequence prefixes, although this comparison does not isolate the effects of occlusion recovery from handoff drift.

Figure 4: Full-context inference gives ARROW stronger long-range tracking than 48-frame windowed inference, with the gap increasing on longer prefixes.

RGB-only multi-view tracking

The paper adapts the MVTracker benchmark to remove depth and calibration inputs. Each example contains RGB images from four static cameras over 24 synchronized timesteps, and queries are specified by 2D source locations in visible views. The adapted task therefore requires ARROW to infer geometry, associate observations across cameras, track points through time, and estimate visibility without external geometric inputs.

ARROW leads on 11 of 12 dataset–metric combinations. Its mean scores across Panoptic Studio, DexYCB, and Multi-View Kubric are:

Metric	ARROW average
APD	61.0
EPE	0.1252
Occlusion accuracy	88.5
AJ	49.2

On Panoptic Studio, ARROW achieves 79.2 APD, 0.0929 m EPE, 89.9 OA, and 66.9 AJ. On Multi-View Kubric, it reaches 49.0 APD, 0.1121 m EPE, 86.2 OA, and 36.5 AJ. On DexYCB, it leads APD, OA, and AJ but not EPE: TAPIP3D obtains a lower EPE of 0.0991 m, compared with ARROW’s 0.1705 m.

This discrepancy is informative. DexYCB EPE is sensitive to a small number of large errors, whereas APD and AJ measure the quality of a larger fraction of correspondences. The fact that ARROW leads DexYCB APD and AJ despite losing EPE indicates that its failure distribution may contain fewer moderate errors but more severe outliers. The paper also reports that the DA3 geometric prior can produce a dataset-specific mismatch, reinforcing that joint architecture alone does not eliminate dependence on pretraining-domain compatibility.

The multi-view results support the paper’s strongest systems claim: a model without supplied calibration, depth, explicit camera indexing, or synchronized-video-specific query embeddings can outperform dedicated multi-view trackers on an RGB-only protocol. Direct comparison with the official MVTracker benchmark remains inappropriate because the adapted protocol changes the input assumptions and scoring procedure.

Reconstruction and geometric estimation

Although ARROW is optimized primarily for tracking and unified querying, it remains competitive on conventional geometry tasks. It achieves the lowest Chamfer distance and highest normal consistency on NRGBD, with values of 0.0263 and 0.933. On 7-Scenes, it obtains a Chamfer distance of 0.0487 and normal consistency of 0.762, remaining close to specialized reconstruction models.

For camera pose estimation, ARROW performs particularly well on dynamic Panoptic Studio data, achieving an absolute trajectory error of 0.0714, relative translation error of 0.1301, and relative rotation error of 1.406. It also obtains the lowest Sintel ATE, 0.0613, and ties the best reported relative rotation error on Bonn at 0.211. These results demonstrate that pointwise 3D consistency is sufficient to recover useful camera poses even without a dedicated pose head.

Video depth results show a similar pattern. ARROW obtains the best Bonn AbsRel, 0.041, and $\delta_1$, 0.987; the best KITTI AbsRel, 0.036; and the best 7-Scenes $\delta_1$, 0.924. The implication is that the tracking-oriented training objective preserves substantial geometric accuracy, although the model is not uniformly superior to dedicated depth or reconstruction systems on every benchmark.

Qualitatively, the model maintains static background points during camera motion, tracks articulated objects through occlusion, and predicts target image coordinates alongside visibility. These outputs are relevant because they expose whether the 4D representation preserves point identity rather than merely producing plausible per-frame geometry.

Figure 5: Qualitative UV predictions demonstrate occlusion handling, articulated-motion tracking, and consistency of static points under camera motion.

Limitations and open questions

ARROW’s order-invariant formulation removes explicit temporal interpolation and extrapolation. The model supports target states corresponding to observations supplied as input; it does not natively predict an intermediate or future time absent from the observation set. This limitation distinguishes ARROW from trajectory-field and continuous-time formulations.

Full-context inference is computationally demanding. Sparse queries are efficient because the encoded context can be reused, but dense decoding requires many independent cross-attention operations and can become a memory bottleneck. The absence of self-attention among query tokens may also weaken predictions in distant or low-confidence regions where joint consistency could be useful.

The reported multi-view robustness is conditional on the training distribution. The paper observes degradation when in-distribution views are combined with out-of-distribution egocentric inputs, indicating that content-derived identity tokens do not by themselves remove domain gaps. Fine-grained pose and image-coordinate precision also remains sensitive to local variation, despite strong global accuracy; the authors identify local feature matching or optimization as possible remedies but do not evaluate them.

Finally, the long-context advantage is demonstrated using full-context inference, but the experiments do not fully disentangle context length, occlusion recovery, and error propagation. A specific open question is whether a memory-efficient approximation to full-context attention can preserve ARROW’s gains without reverting to windowed handoffs.

Conclusion

ARROW presents a unified feed-forward formulation for 3D reconstruction and point tracking over arbitrary RGB observations. Its central mechanism—content-derived ID tokens for order-invariant query addressing—removes the dependence on temporal indices while preserving independent source, target, and reference selection. Combined with non-sequential sampling and a heterogeneous training mixture, this design yields leading results on WorldTrack and TAPVid-3D, strong RGB-only multi-view tracking, and competitive reconstruction and pose estimation.

The evidence supports the paper’s principal claim that broadening correspondence supervision beyond temporally local video can improve temporal tracking itself. The remaining constraints are explicit: ARROW does not interpolate or extrapolate time, dense decoding is expensive, query interactions are absent, and performance remains sensitive to domain shift and local geometric precision.