World-Centric 3D Trajectories

• World-Centric 3D trajectories are sequences of 3D points in a fixed global coordinate system that differentiate intrinsic scene motion from camera-induced dynamics.
• Techniques like TrackingWorld and DELTA use sparse 2D tracking, efficient upsampling, and 3D back-projection to achieve accurate and computationally efficient trajectory estimation.
• This approach enhances long-term tracking in dynamic scenes, supporting applications in AR/VR, autonomous navigation, and video analysis.

World-centric 3D trajectories define the motion of visual entities—typically represented at the pixel or feature level—within a single, global Euclidean coordinate system anchored to the physical world, rather than the camera or image frame. This paradigm is foundational for applications requiring temporally consistent tracking of both rigid and non-rigid objects in dynamic scenes, as it separates egocentric (camera-induced) motion from intrinsic scene dynamics. Recent advances in monocular 3D tracking have focused on dense, pixel-wise world-centric trajectory estimation from single-camera video, overcoming previous limitations of sparsity, computational cost, and the inability to track newly emerging or rapidly moving foreground regions (Lu et al., 9 Dec 2025).

1. Definition and Conceptual Foundations

World-centric 3D trajectories are sequences of 3D points described in a fixed, global coordinate frame, typically aligned with the scene or “world.” In contrast to camera-centric 3D tracking—which defines trajectories relative to a moving camera’s local frame—world-centric trajectories unambiguously capture motion in terms of absolute scene location. Given a video sequence, the goal is to reconstruct, for each tracked pixel or region, a coordinate trajectory $\{\mathbf{X}_t\}$ in world coordinates, where $t$ indexes time.

This shift is crucial for unmixing the effects of camera motion and scene dynamics. Applications include structure-from-motion, AR/VR, autonomous vehicle perception, and long-term video understanding in which world-aligned consistency is required across varying camera poses (Lu et al., 9 Dec 2025).

2. Pipeline Overview: TrackingWorld and DELTA Frameworks

Recent systems such as TrackingWorld (Lu et al., 9 Dec 2025) and DELTA (Ngo et al., 31 Oct 2024) implement a multi-stage pipeline for world-centric 3D trajectory estimation from monocular video:

1. Sparse 2D Tracking Backbone: Initial sparse correspondence fields on a downsampled grid are produced via backbones such as CoTrackerV3 or the DELTA global-local transformer, which operate at $4\times$ spatial stride for efficiency.
2. Tracking Upsampler: A dedicated upsampler module lifts sparse 2D tracks to dense, per-pixel correspondences. This module is critical for converting tractable, low-resolution features into high-resolution tracking maps while preserving computational practicality.
3. 3D Back-Projection and World Registration: Using both the dense 2D trajectories and estimated camera poses, tracks are back-projected into 3D space, yielding trajectories in the world coordinate frame. Rigorous optimization separates egomotion from scene dynamics.
4. Redundancy Reduction and Generalization: To extend dense tracking to dynamic, newly appearing regions, the upsampler is applied to every frame; tracks overlapping in space-time are reduced to minimize redundancy (Lu et al., 9 Dec 2025).

These pipelines achieve state-of-the-art dense and accurate world-centric tracking with favorable runtime and memory characteristics.

3. Tracking Upsampler: Architecture and Algorithmic Principles

The tracking upsampler is the principal module enabling efficient, dense world-centric 3D trajectory estimation:

• Input/Output Specification: Given sparse 2D tracks $$\mathbf{P}_{\mathrm{sparse}}\in\mathbb{R}^{(H/s)\times(W/s)\times T\times 2}$$ and sparse deep feature maps $\mathbf{F}_{\mathrm{sparse}}$, the upsampler outputs a full-resolution dense trajectory tensor $$\mathbf{P}_{\mathrm{dense}}\in\mathbb{R}^{H\times W\times T\times 2}$$.
• Mechanism: Instead of independent per-pixel tracking (which is infeasible at scale), the upsampler predicts a weight matrix $\mathbf{W} \in \mathbb{R}^{N_s \times (HW)}$, representing convex combinations of local sparse tracks to reconstruct each dense trajectory:

$$\mathbf{P}_{\mathrm{dense}} = \mathbf{W}^T \mathbf{P}_{\mathrm{sparse}}$$

• Network Structure: In TrackingWorld (adopting DELTA's upsampler), a shallow U-Net or MLP consumes the flattened sparse feature maps; its output, after row-wise softmax normalization, provides the weights $\mathbf{W}$ for dense reconstruction. No additional nonlinearity is applied beyond standard ReLU or GELU activations.
• Training Regime: DELTA’s upsampler is trained end-to-end with the global 3D tracking pipeline—without an explicit upsampling loss. TrackingWorld directly reuses DELTA-trained weights (“frozen form”) without further finetuning, leveraging the robust prior learned by the original system (Lu et al., 9 Dec 2025, Ngo et al., 31 Oct 2024).

A summary of the upsampler’s data flow is given in the following table:

Step	Input	Output
Feature flattening	Sparse feature maps $\mathbf{F}_{\mathrm{sparse}}$	Flattened features $F_\text{flat}$
Weight prediction	$F_\text{flat}$ (optionally $\mathbf{P}_{\mathrm{sparse}}$)	Weight logits $W_\text{logits}$
Softmax normalization	$W_\text{logits}$	Weights $\mathbf{W}$
Weighted combination	$\mathbf{P}_{\mathrm{sparse}$, $\mathbf{W}$	$\mathbf{P}_{\mathrm{dense}}$

The upsampler improves both speed and accuracy: CoTrackerV3+Upsampler reduces EPE from 1.51 to 1.24 and boosts IoU from 75.5% to 80.9%, with %%%%19$\mathbf{P}_{\mathrm{sparse}}$20%%%% runtime reduction on HD video (Lu et al., 9 Dec 2025).

4. Transformer-Based Upsampling in DELTA

DELTA introduces a transformer-based upsampler with two stacked local cross-attention blocks to reconstruct high-resolution tracks from coarse tracks:

• Local Cross-Attention: For each fine-scale pixel, a query is constructed from its upsampled decoder feature and bilinearly sampled coarse track. The attention mechanism aggregates $\kappa^2$ ($\kappa=3$) local coarse features/values using Alibi-style static positional bias.
• Convex Weight Prediction: An MLP post-attention produces convex interpolation weights for each fine pixel over its coarse neighbors; the upsampled trajectory is their convex combination.
• Losses: Supervision is applied at both coarse and fine resolutions, with $\lambda_{2D}=100$ for 2D track error, $\lambda_{depth}=1$, and $\lambda_{vis}=0.1$ for visibility.
• Efficiency: Each upsampling layer operates in $O(HW)$ time with only 9 neighbors per query, retaining computational tractability for video-scale data. DELTA’s full pipeline provides over 8$\times$ speedup versus prior dense trackers (Ngo et al., 31 Oct 2024).

Ablation experiments show that DELTA's attention-based upsampler (with Alibi bias) achieves the best EPE (3.67) compared to bilinear (5.31), nearest-neighbor (5.34), and convolutional upsamplers (4.27) (Ngo et al., 31 Oct 2024).

5. Separation of Camera and Foreground Motion

World-centric 3D trajectory estimation enables the explicit disentanglement of camera motion from dynamic foreground motion. TrackingWorld emphasizes this capability by optimizing for both camera poses and 3D track positions, allowing dense tracking to remain robust even as new objects and complex scene dynamics emerge. This separation is critical for accurate long-term reconstruction, as monocular baselines without world-centric anchoring conflate these two classes of motion (Lu et al., 9 Dec 2025).

The upsampler’s ability to generalize across frames and dynamically extend track coverage to newly emergent image regions further enhances the completeness of world-centric 3D trajectory maps.

6. Limitations, Open Challenges, and Future Directions

Current world-centric dense monocular 3D tracking methods, including those in TrackingWorld and DELTA, inherit limitations from their sparse tracking backbones, such as susceptibility to large occlusion gaps and textureless regions, as well as the increased memory overhead from maintaining large convex weight tensors at high spatial resolutions (Lu et al., 9 Dec 2025).

Potential avenues for improvement include:

• Local (as opposed to global) kernel learning for upsampling to optimize memory/performance tradeoffs.
• End-to-end fine-tuning of the entire pipeline (sparse tracker, upsampler, 3D optimizer) for improved robustness.
• Explicit uncertainty estimation to downweight unreliable dense tracks in ambiguous image regions.

This suggests ongoing research is required to fully address robustness, scalability, and uncertainty quantification in world-centric 3D trajectory estimation.

7. Impact and Benchmarks

Empirical results from both TrackingWorld and DELTA demonstrate significant improvements in 2D/3D tracking accuracy and efficiency, setting new state-of-the-art benchmarks on datasets such as CVO-Clean, CVO-Final, and CVO-Extended (Lu et al., 9 Dec 2025, Ngo et al., 31 Oct 2024). The use of transformer-based upsampling and world-centric optimization has enabled, for the first time, dense long-range 3D tracking at the scale and fidelity required for large-scale video-driven applications.

In practice, these advances have transformed the feasibility of achieving pixel-level 3D tracking across entire video sequences, supporting a wide range of vision applications where accurate world-referenced motion understanding is essential.