TrackMAE: Video Representation Learning via Track Mask and Predict

Abstract: Masked video modeling (MVM) has emerged as a simple and scalable self-supervised pretraining paradigm, but only encodes motion information implicitly, limiting the encoding of temporal dynamics in the learned representations. As a result, such models struggle on motion-centric tasks that require fine-grained motion awareness. To address this, we propose TrackMAE, a simple masked video modeling paradigm that explicitly uses motion information as a reconstruction signal. In TrackMAE, we use an off-the-shelf point tracker to sparsely track points in the input videos, generating motion trajectories. Furthermore, we exploit the extracted trajectories to improve random tube masking with a motion-aware masking strategy. We enhance video representations learned in both pixel and feature semantic reconstruction spaces by providing a complementary supervision signal in the form of motion targets. We evaluate on six datasets across diverse downstream settings and find that TrackMAE consistently outperforms state-of-the-art video self-supervised learning baselines, learning more discriminative and generalizable representations. Code available at https://github.com/rvandeghen/TrackMAE

• The paper introduces explicit motion supervision through trajectory prediction and motion-aware masking in masked video modeling.
• It employs a dual-decoder architecture that jointly reconstructs spatial features and motion trajectories, yielding robust temporal representations.
• Experiments show state-of-the-art improvements on motion-centric benchmarks with significant gains in low-shot and fine-grained settings.

TrackMAE: Explicit Motion Supervision for Masked Video Modeling

Introduction

TrackMAE introduces a masked video modeling (MVM) paradigm which explicitly incorporates motion supervision to enhance video representation learning. Traditional MVM methods, such as VideoMAE, rely on reconstructing spatial tokens from heavily masked video data, using vision transformer (ViT) architectures and random tube masking. While these methods are effective at encoding appearance-based information, they implicitly supervise motion, thus failing to adequately capture fine-grained temporal dynamics, which are crucial for motion-centric tasks. TrackMAE addresses this limitation by leveraging motion trajectories extracted via off-the-shelf point trackers (e.g., CoTracker3), introducing two architectural modifications: a trajectory decoder for direct motion prediction and a motion-aware masking policy that balances visible token sampling from both static and dynamic regions.

Figure 1: TrackMAE improves masked video modeling by jointly predicting spatial features and motion trajectories in a mask-and-predict fashion.

Methodology

TrackMAE departs from previous work by simultaneously optimizing for spatial and motion reconstructions in a self-supervised manner. The method decomposes an input video $$\mathbf{V} \in \mathbb{R}^{T \times H \times W \times 3}$$ into non-overlapping spatiotemporal tubelets, embedding each into tokens processed by a ViT encoder. Instead of reconstructing only the masked pixels or feature embeddings, TrackMAE also predicts sparse motion trajectories of patch-center points tracked over time, using an auxiliary trajectory decoder. This trajectory prediction provides an additional, explicit learning signal for modeling temporal correspondences.

Motion-aware masking is implemented by partitioning spatiotemporal tokens according to the magnitude of their tracked displacements, ensuring that visible tokens cover both static and dynamic regions in each video clip.

Figure 2: Overview of TrackMAE. The upper branch extracts sparse motion trajectories while the lower branch reconstructs spatial features. Both branches are supported by dedicated decoders per modality.

Motion-Aware Masking

TrackMAE advances beyond uniform masking strategies by introducing motion-aware masking. The tracking output yields an average per-tubelet displacement distribution. Visible tokens are then sampled with a fixed proportion from high-motion and low-motion regions, controlled by a motion ratio hyperparameter. This strategy not only provides broader spatiotemporal coverage but also increases the difficulty of the prediction task, pushing the model towards learning spatiotemporal dependencies.

Figure 3: Masking comparison. Motion-based tube masking ensures visible tokens systematically cover both high-motion and static regions, contrasting with random tube masking.

Dense Motion Target Upsampling

Since computational constraints limit the number of tracked points, TrackMAE introduces spatial upsampling of sparse trajectory grids to simulate denser motion targets. Interpolation leverages the piecewise-smooth nature of local video motion, allowing dense supervision with minimal incremental cost. Ablations confirm that upsampling from $14 \times 14$ to $28 \times 28$ grid points notably improves downstream recognition accuracy without incurring the high cost of dense tracking.

Experimental Evaluation

Linear Probing

TrackMAE sets new state-of-the-art (SOTA) results on several benchmarks under the linear probing protocol. Pretraining a ViT-B model on Kinetics-400 with the full TrackMAE pipeline yields consistent gains versus baselines such as VideoMAE, MGMAE, MGM, and feature-based objectives (CLIP, HOG, DINO). Notably, TrackMAE significantly outperforms all baselines on motion-centric datasets (e.g., Something-Something V2, FineGym), affirming the hypothesis that explicit motion supervision yields more temporally-aware representations.

Full Finetuning

End-to-end finetuning on both Kinetics-400 and SSv2 substantiates these improvements, with TrackMAE+CLIP achieving 72.8 top-1 on SSv2 and 83.9 on K400—outperforming next-best SOTA by 0.8 points on each. Scaling experiments with ViT-L backbones yield further gains (75.7 on SSv2, 86.7 on K400), confirming that TrackMAE benefits from increased model and data scale.

SEVERE Benchmark Generalization

TrackMAE exhibits strong generalization on the SEVERE benchmark, which probes representation robustness across distributional shift, sample efficiency, action granularity, and task shift. TrackMAE consistently delivers superior or competitive results in all facets, demonstrating especially pronounced gains in low-shot and fine-grained settings.

Ablation Studies

Ablations confirm the complementary value of motion trajectory predictions combined with pixel or CLIP-based targets, with up to 4-point gains over singular objectives. Motion-aware masking and trajectory upsampling further drive consistent improvements. The method exhibits robustness to noisy trajectory supervision and generalizes effectively to alternative tracking modules (e.g., TAPNext).

Training Dynamics

Training curve analysis reveals that augmenting CLIP reconstruction with trajectory prediction yields superior accuracy at all pretraining epochs, with the improvement most pronounced at lower epoch counts before converging with extended pretraining.

Figure 4: Training evolution. Top-1 accuracy on K400 and SSv2 improves significantly when trajectory prediction is combined with CLIP reconstruction.

Masking and Sampling Strategies

TrackMAE evaluates a comprehensive suite of masking policies informed by trajectory statistics—spanning motion bins, Bernoulli and sorted sampling, and exclusion-based variants. While the main reported improvements use the balanced motion bins approach (equal high/low-motion sampling), alternative policies are outlined for future exploration.

Figure 5: Motion information can be systematically exploited to define diverse sampling distributions for token masking.

Figure 6: Examples of motion-aware masking maps, including high- and low-motion token selection and "exclude-topk" strategies.

Discussion and Implications

TrackMAE demonstrates that dense, explicit, data-driven motion supervision addresses limitations of pixel/feature reconstruction in MVM frameworks, achieving pronounced gains on action recognition and robustness benchmarks without introducing synthetic motion cues. The architecture remains compatible with future improvements in point tracking, embraces a plug-and-play supervision framework, and can flexibly scale in model and data regime.

Computational overhead from point tracking (~50% increased pretraining time) is mitigated by upsampling, and empirical analysis suggests resilience to tracker noise. The dual-decoder architecture prevents gradient interference between spatial and temporal objectives.

Practically, the approach is valuable for transfer learning in downstream video analysis tasks sensitive to motion (e.g., fine-grained action localizations, low-shot settings, non-standard objectives), as well as any application requiring robust and general spatiotemporal representations.

Conclusion

TrackMAE defines a new standard for masked video representation learning by supplementing spatial reconstruction with explicit, dense motion trajectory supervision and integrating motion-aware masking. Explicit modeling of motion yields more discriminative, temporally-aware representations as evidenced by SOTA performance in both full finetuning and linear probing across appearance-centric and motion-centric datasets, as well as robust generalization on the SEVERE benchmark. Future directions include optimizing masking policies, reducing tracker cost, adopting emerging point tracking frameworks, and extending to denser motion signals such as flow or scene flow.