Trace Anything: Representing Any Video in 4D via Trajectory Fields (2510.13802v1)

Abstract: Effective spatio-temporal representation is fundamental to modeling, understanding, and predicting dynamics in videos. The atomic unit of a video, the pixel, traces a continuous 3D trajectory over time, serving as the primitive element of dynamics. Based on this principle, we propose representing any video as a Trajectory Field: a dense mapping that assigns a continuous 3D trajectory function of time to each pixel in every frame. With this representation, we introduce Trace Anything, a neural network that predicts the entire trajectory field in a single feed-forward pass. Specifically, for each pixel in each frame, our model predicts a set of control points that parameterizes a trajectory (i.e., a B-spline), yielding its 3D position at arbitrary query time instants. We trained the Trace Anything model on large-scale 4D data, including data from our new platform, and our experiments demonstrate that: (i) Trace Anything achieves state-of-the-art performance on our new benchmark for trajectory field estimation and performs competitively on established point-tracking benchmarks; (ii) it offers significant efficiency gains thanks to its one-pass paradigm, without requiring iterative optimization or auxiliary estimators; and (iii) it exhibits emergent abilities, including goal-conditioned manipulation, motion forecasting, and spatio-temporal fusion. Project page: https://trace-anything.github.io/.

• The paper presents a unified feed-forward model that maps every pixel in a video to a continuous 3D trajectory, offering a novel 4D representation.
• It employs cubic B-spline parameterization and a fusion transformer to capture spatio-temporal dynamics without relying on auxiliary estimators.
• The approach achieves state-of-the-art accuracy and efficiency on a large-scale synthetic benchmark, enabling robust motion forecasting and spatio-temporal fusion.

Trace Anything: A Unified 4D Video Representation via Trajectory Fields

Introduction and Motivation

The paper introduces Trajectory Fields as a principled, atomic-level 4D representation for dynamic video scenes, where each pixel in every frame is mapped to a continuous 3D trajectory over time. This formulation directly models the spatio-temporal evolution of scenes, moving beyond traditional approaches that reconstruct disjoint per-frame point clouds and rely on auxiliary estimators such as depth, optical flow, or 2D tracking. The Trace Anything model is a feed-forward neural network that predicts the entire trajectory field in a single pass, parameterizing each pixel’s trajectory as a spline curve defined by a set of control points.

Figure 1: Any video$^{*}$ can be represented in 4D with a Trajectory Field, a dense mapping assigning each pixel in each frame to a parametric 3D trajectory.

This representation is designed to be versatile, supporting monocular videos, image pairs, and unordered image collections, and is geometrically grounded to enable scalable modeling of scene dynamics.

Methodology

Trajectory Field Formulation

A trajectory field $\mathcal{T}$ is defined as a mapping:

$$\mathcal{T}: [N] \times [H] \times [W] \to C([0,1], \mathbb{R}^3)$$

where $N$ is the number of frames, $H$ and $W$ are image dimensions, and $C([0,1], \mathbb{R}^3)$ is the space of continuous 3D trajectories. Each pixel $(u,v)$ in frame $i$ is assigned a parametric curve $\mathbf{x}_{i,u,v}(t)$, typically a cubic B-spline with $D$ control points:

$$\mathbf{x}_{i,u,v}(t) = \sum_{k=0}^{D-1} \mathbf{P}^{(k)}_{i,u,v} \phi_k(t)$$

where $\phi_k(t)$ are basis functions.

Network Architecture

Trace Anything employs a geometric backbone consisting of an image encoder and a fusion transformer to process input frames. The control point head outputs dense control point maps for each frame, parameterizing the trajectory of every pixel. Temporal index embeddings are used for sequential inputs, but the architecture is agnostic to input ordering, supporting unordered image sets.

Figure 2: Trace Anything pipeline. Input frames are processed by a geometric backbone consisting of an image encoder and a fusion transformer. The control point head outputs dense control point maps $\mathbf{P}_i$.

The model predicts all trajectories jointly in a shared world coordinate system, eliminating the need for iterative optimization or external estimators.

Training and Loss Functions

Supervision is provided by directly minimizing the 3D endpoint error between predicted and ground-truth trajectories, with confidence-weighted losses to downweight uncertain predictions. Regularization terms enforce static degeneracy (static pixels collapse to degenerate trajectories), rigidity (internal distances in rigid regions are preserved), and correspondence (matched pixels across frames share control points).

Data Platform and Benchmark

A Blender-based synthetic data platform was developed to generate large-scale, photo-realistic dynamic scenes with dense annotations, including 2D/3D trajectories, depth, semantics, flow, and camera poses. The Trace Anything dataset comprises over 10,000 videos for training and a benchmark of 200 curated videos for evaluation. The benchmark protocol requires all-to-all trajectory prediction, evaluating the model’s ability to jointly capture dynamics across entire sequences.

Figure 3: Sample renderings from our data platform.

Experimental Results

Qualitative Evaluation

Trace Anything demonstrates robust trajectory field estimation on diverse dynamic scenes, accurately reconstructing both static and dynamic components, handling non-rigid deformations, occlusions, and preserving global structure.

Figure 4: Video-based trajectory field estimation on DAVIS. Trace Anything predicts trajectory fields that yield dynamic point cloud sequences and dense 3D trajectories.

The model also supports goal-conditioned manipulation by inferring plausible 3D trajectories from image pairs, aligning with robotic manipulation tasks.

Figure 5: Image-pair-based trajectory field estimation (goal-conditioned manipulation) on Bridge. Trace Anything predicts a trajectory field interpolating the 3D motion of both the robot arm and manipulated objects.

Trace Anything generalizes to unordered image sets, estimating plausible trajectory fields and camera poses without sequence information.

Figure 6: Trajectory fields and camera poses estimated from an unstructured, unordered image set. No sequence information is provided to the model.

Quantitative Evaluation

On the Trace Anything benchmark, the model achieves the lowest end-point errors (EPE) for both static and dynamic regions, and the best scores for static degeneracy deviation (SDD) and correspondence agreement (CA), indicating strong compliance with geometric consistency conditions. The model runs over an order of magnitude faster than optimization-based baselines, with runtime scaling linearly with the number of frames.

Figure 7: Stage-wise runtime vs. number of input frames.

Emergent Capabilities

The trajectory field representation enables several emergent capabilities:

• Motion Forecasting: Per-pixel trajectories can be extrapolated for dense motion forecasting without additional predictors.
• Instruction-based Forecasting: By leveraging video generation models, Trace Anything can lift future state predictions into trajectory fields conditioned on natural language instructions.

  Figure 8: Instruction-based forecasting. Future states are generated using Seedance 1.0.

• Spatio-temporal Fusion: Dynamic entities observed across multiple frames can be fused into a canonical frame, aggregating partial observations and overcoming occlusions.

  Figure 9: Spatio-temporal fusion. The trajectory field can be leveraged to fuse observations of the dynamic entity across different frames into a canonical frame.

Implementation Considerations

• Computational Requirements: Training requires large-scale synthetic data and significant GPU resources (e.g., 32 NVIDIA A100 80GB GPUs for 7.22 days). Inference is efficient, with single-pass runtime scaling linearly with input size.
• Curve Parameterization: Cubic B-splines with 10 control points yield optimal performance, balancing expressivity and computational cost.
• Backbone Initialization: Pretraining the backbone (e.g., with Fast3R) is critical for convergence and accuracy.
• Generalization: The model generalizes to out-of-distribution data and supports dense per-pixel tracking, outperforming sparse 3D tracking methods in efficiency.

Limitations and Future Directions

• Domain Gap: Reliance on synthetic data introduces a domain gap with real-world scenarios; incorporating real data annotations may improve generalization.
• Expressive Power: The fixed number of control points limits modeling of highly complex or long-range motions; larger datasets and adaptive curve parameterization may address this.
• Precision: Dense per-pixel estimation may be less precise than sparse point-level tracking; hybrid approaches could enhance accuracy.

Conclusion

Trace Anything establishes a unified, feed-forward paradigm for 4D video representation via trajectory fields, enabling efficient, geometrically consistent modeling of scene dynamics. The approach achieves state-of-the-art accuracy and efficiency, supports diverse input modalities, and unlocks new capabilities in motion forecasting and spatio-temporal fusion. The trajectory field representation provides a foundation for future research in dynamic scene understanding, robotic manipulation, and integration with appearance-centric models for novel view synthesis.