Forecasting Motion in the Wild

Abstract: Visual intelligence requires anticipating the future behavior of agents, yet vision systems lack a general representation for motion and behavior. We propose dense point trajectories as visual tokens for behavior, a structured mid-level representation that disentangles motion from appearance and generalizes across diverse non-rigid agents, such as animals in-the-wild. Building on this abstraction, we design a diffusion transformer that models unordered sets of trajectories and explicitly reasons about occlusion, enabling coherent forecasts of complex motion patterns. To evaluate at scale, we curate 300 hours of unconstrained animal video with robust shot detection and camera-motion compensation. Experiments show that forecasting trajectory tokens achieves category-agnostic, data-efficient prediction, outperforms state-of-the-art baselines, and generalizes to rare species and morphologies, providing a foundation for predictive visual intelligence in the wild.

• The paper presents a novel approach using dense 2D trajectory tokens and diffusion transformers to forecast diverse animal behaviors.
• It employs a velocity-based diffusion model combined with a permutation-invariant transformer to overcome traditional pixel-level and category-specific limitations.
• Results on the MammalMotion dataset demonstrate superior performance in metrics like ADE, FDE, and FVMD, with robust out-of-distribution generalization.

Forecasting Motion in the Wild: Trajectory Tokens and Diffusion Transformers for Animal Behavior Prediction

Introduction

This work addresses the challenge of visual forecasting in natural environments by introducing a scalable, structured mid-level representation for non-rigid agent motion: dense point trajectories as visual "tokens" for behavior. The motivation stems from limitations of conventional approaches—pixel-level video prediction models obfuscate physical dynamics with confounding visual factors, while category-specific 3D parameterizations lack generality and data efficiency. Through the discrete, yet dynamic representation of 2D surface trajectories, the paper establishes a tokenization akin to language modeling for motion forecasting. This paradigm is realized via a diffusion transformer capable of reasoning over unordered sets of trajectories, integrating robust visual features and explicit occlusion reasoning, and enabling high-fidelity prediction for highly diverse animal behaviors.

Figure 1: Dense point trajectories act as visual tokens for behavior; the model predicts future motion from a single RGB image, motion history, and high-level motion vectors across diverse species, including rare long-tail categories.

Problem Formulation and Architecture

Motion is represented as a set of $N$ point tracks, where each trajectory encodes a 2D path over $T$ time steps, leaving visibility/occlusion as an explicit variable. The principal modeling challenge is to forecast $(\mathbf{X}_{T_c+1:T}, \mathbf{O}_{T_c+1:T})$ conditioned on an initial image, motion and occlusion history, and an optional displacement prompt. Rather than autoregressive regression of coordinates, the paper employs a denoising diffusion probabilistic model (DDPM) that operates on a reparameterized velocity-based representation, mitigating issues of correlation and occluded values.

The key architectural design is a permutation-invariant transformer (DiT) over trajectory tokens. Each token concatenates:

• DINOv3-derived visual features at the track’s initial location,
• A motion history embedding,
• Occlusion state sequence,
• The noisy (diffused) velocity and visibility to be denoised.

Crucially, location-based positional encoding is used instead of sequential encoding for input order invariance. Conditioning is injected globally using adaptive layer normalization for both diffusion step and motion prompt.

Figure 2: The architecture encodes DINO features, motion/occlusion history, and noisy diffusion targets per token; position encodings are derived from spatial track origin; a DiT transformer predicts clean track futures.

Dataset Construction: MammalMotion

To establish a comprehensive and challenging testbed, the authors curate MammalMotion, a 300-hour dataset distilled from MammalNet, employing robust automated pipelines that combine:

• Spatio-temporal shot segmentation via tracking failures,
• Per-animal detection using Grounding-DINO and VideoSAM segmentation,
• Dense point tracking (BootsTAPIR) with an emphasis on thin structures,
• Homography-based camera stabilization to disentangle camera and animal motion.

This processing enables the extraction of log-normal distributed natural animal trajectories, a property not previously demonstrated over taxonomically diverse, large-scale video.

Figure 3: Camera stabilization separates entangled animal/camera motion: pixel-space tracks (middle) versus stabilized coordinates (right) for training.

Experimental Setup and Metrics

Training and inference operate with synthetic noise schedules (DDPM, with accelerated DDIM sampling at test time) on up to 320 point tracks per trajectory, leveraging DINOv3 for visual features. Evaluation is rigorous and multidimensional:

• Distributional metrics: Fréchet Distance (FD) for velocities/accelerations, Trajectory Variance, and Fréchet Video Motion Distance (FVMD) to measure fidelity against ground truth natural motion distributions.
• Example-level metrics: Average and Final Displacement Error (ADE/FDE), Points Within Threshold (PWT), and example-level VMD, employing a best-of-$K$ protocol to accommodate the stochasticity of generative diffusion models.

Baselines considered include non-learned predictors (no motion, constant/oracle velocity), regression models (ATM, Track2Act), and state-of-the-art point-token diffusers. Additional qualitative comparison is drawn with stable diffusion-based video generation.

Results

Qualitative Results

The model produces realistic, high-frequency articulated motion across a large spectrum of behaviors and morphologies, even for species highly underrepresented in training. Sampling diversity is demonstrated by stochastic generation, modulated by conditional displacement prompting, which yields plausible changes in motion magnitude and direction.

Figure 4: Diverse samples for the same initial condition reflect the model's stochastic generative capability over animal motion.

Figure 5: Conditioning the model on different displacement prompts enables explicit control over forecasted motion magnitude and direction.

Notably, the model exhibits robust out-of-distribution transfer—generating physically plausible motion for non-mammals and even artificial agents.

Figure 6: Zero-shot generalization to humans and non-mammals shows the model's invariance to morphology and category, leveraging learned motion structure.

Comparison with Video Generation

Conventional video diffusion models, such as Stable Diffusion, are outperformed with respect to physically plausible motion, especially on rare species; they often fail catastrophically, conflating morphology and failing to maintain animal identity.

Figure 7: Standard diffusion-based video models struggle with rare or fine-grained animal motion, while trajectory token models consistently preserve plausible kinematics.

Quantitative Results

Across all metrics and motion regimes (low, medium, high motion buckets), the proposed approach yields dominant performance. For high-motion regimes, the model achieves lowest error (e.g., ADE, FDE, FVMD), highest PWT, and best motion/acceleration distribution alignment.

Combining conditional input (ground-truth or estimated displacement) further strengthens results. Training on broader (all-species) data provides positive transfer for specialized categories (e.g., Panthera), outperforming category-limited training.

Notable bold results include:

• Substantial reduction in FVMD (distribution-level): e.g., 49.3 vs. 76.3 for Track2Act in high-motion, and improvements continue with conditioning (40.2).
• Higher point-wise accuracy (PWT): exceeding 26% on high-motion, eclipsing Track2Act (21.8%) and non-learned baselines (<17%).
• Lower ADE/FDE, and lower FD in multiple regimes.

  Figure 8: Log-normal distributions of displacement, x-, and y- motion underscore the scaling structure of natural animal movement as captured by the dataset.

Discussion and Implications

This paper advances the state of behavior forecasting in visual intelligence. The proposed point-trajectory tokenization:

• Establishes a transferable and general representation agnostic to appearance, object class, or morphology, permitting unified modeling of behavior.
• Achieves category-agnostic, data-efficient motion prediction—outperforming both direct video models and regression-based track forecasting.
• Enables explicit control of motion generation via input prompts, facilitating practical use-cases that require goal- or intent-conditioned behavior forecasting.
• Alleviates the need for costly, manually annotated behavior datasets, leveraging large-scale unlabeled video and automated tracking pipelines.
• Scales to long-tail and rare behaviors, critical for general-purpose ecological or biological informatics.

The robust performance and demonstrated OOD transfer indicate immediate applicability for large-scale, label-efficient behavioral analysis. Moreover, the model construction paves the way for integrating motion forecasting into downstream pipelines—either for ecological study, simulated agent behavior in RL, or dynamic scene understanding in robotics and autonomous driving.

Future developments may include extending to full 3D trajectory tokenization as 3D pose estimation matures for unconstrained video, hierarchical modeling of group behavior, or tighter integration with intent/instruction-driven motion control.

Conclusion

The work establishes dense trajectory tokens combined with diffusion transformers as a powerful and scalable paradigm for behavior forecasting in the wild. Its technical contributions in representation, architecture, and dataset construction yield superior empirical performance. By bridging the structural gap between pixels and rigid 3D models, it both provides theoretical insight and delivers practical benefits for predictive visual intelligence, setting a foundation for the next generation of motion modeling systems.