TrackMAE: Masked Autoencoding for Tracking

• TrackMAE is a self-supervised framework that employs masked autoencoding specially tailored for both visual object tracking and video representation learning.
• The approach uses task-specific masking schemes with a two-stream input structure—fully visible templates and masked search regions—to enhance feature extraction and tracking accuracy.
• Empirical results demonstrate that incorporating explicit motion trajectory prediction and motion-aware masking leads to notable improvements in tracking benchmarks and generalization across dynamic scenes.

TrackMAE encompasses two distinct lines of research in self-supervised learning for both visual object tracking and masked video representation learning. The core principle in both variants is masked autoencoding adapted to tracking-specific settings, but implemented with different architectures, masking schemes, and targeted data modalities.

1. Overview and Definitions

TrackMAE refers to a family of self-supervised pre-training objectives tailored for tracking tasks, leveraging masked autoencoding to promote effective representation learning. In the context of image-based visual object tracking, TrackMAE is designed to utilize the interplay between a fully visible template and a partially masked search region to reconstruct the original input and implicitly force the encoder to extract features relevant for downstream tracking (Cui et al., 2023). In the broader video modeling domain, TrackMAE introduces explicit motion supervision via point trajectory prediction and motion-aware masking in masked video modeling, producing representations highly sensitive to temporal dynamics (Vandeghen et al., 28 Mar 2026).

The implementations share three structural elements:

• Masked autoencoding: High-ratio, structured masking of spatio-temporal tokens is applied prior to encoding, with the learning signal delivered via reconstruction losses.
• Tracking-tailored input structure: The input always includes a decomposition into target and search (image tracker), or dynamic masking emphasizing regions of motion (video counterpart).
• Auxiliary task coupling: In the video regime, motion trajectory prediction provides additional temporal supervision.

2. TrackMAE for Visual Object Tracking

Originating in the MixFormer architecture, TrackMAE for image-based tracking introduces a masked autoencoder pretext task specialized for target localization (Cui et al., 2023). The approach comprises the following steps:

• Each training example is a paired tuple of a target template (fully visible) and a search region (randomly 75% masked).
• A Vision Transformer (ViT)-style encoder processes all template tokens and only the visible search tokens.
• The decoder, lightweight in depth and parameter count, receives encoder outputs plus mask tokens for missing search regions, and reconstructs the original pixel values of the masked search tokens only.
• The loss is computed as the per-patch mean squared error over just the masked search tokens:

$$L_{\mathrm{TrackMAE}} = \frac{1}{\sum_i (1-M_i)}\sum_{i=1}^{N_s}(1-M_i)\, \left\| \hat x^s_i - x^s_i \right\|_2^2$$

• Pre-training is performed solely on tracking datasets (LaSOT, TrackingNet, GOT-10k, COCO), never ImageNet.
• The masking strategy prohibits masking the template while aggressively masking the search, ensuring the model learns to reconstruct search content conditioned on a visible reference (Cui et al., 2023).

Architecturally, TrackMAE modifies the standard MAE pipeline only by this two-stream split and masking regimen. All other architectural components (patch and positional embeddings, encoder-decoder division) remain intact.

Empirical ablations highlight that TrackMAE pre-training on tracking data confers a +0.9 percentage point (pp) improvement in LaSOT AUC over generic MAE (66.0% vs 65.1%), narrowing the gap to ImageNet MAE pre-training. Training from scratch yields 56.2% AUC, illustrating the pronounced benefit of both masked pre-training schemes and, in particular, task-specific masking (Cui et al., 2023).

3. TrackMAE for Masked Video Representation Learning

TrackMAE for masked video modeling explicitly incorporates motion supervision via point tracking and motion-conditioned masking schemes (Vandeghen et al., 28 Mar 2026). Its workflow can be summarized as follows:

• A video clip $V\in \mathbb{R}^{T\times H\times W\times 3}$ is partitioned into non-overlapping 3D tubelets $c_i$ of shape $t\times p\times p\times 3$.
• Masking is applied to 80–95% of tubelet tokens, where visible locations are chosen either randomly or by motion-aware sampling.
• A point tracker (CoTracker3 or TAPNext) generates sparse motion trajectories $m_i$ using a grid of tracked points across frames. These displacements provide ground-truth for an auxiliary prediction task.
• To obtain denser motion targets, sparse tracks are upsampled spatially.
• Two separate decoder heads reconstruct (i) pixel values or external semantic features (e.g., CLIP, DINO descriptors), and (ii) 2D trajectories from masked tokens.
• The masking strategy, parameterized by a motion-ratio $\rho_{\text{motion}}$, ensures that both dynamic (high motion) and static (low motion) regions are adequately represented among visible tokens.

Losses are defined as:

$$\begin{align*} \ell_{\text{pixel}} &= \frac{1}{|\text{Mask}|} \sum_{i\in\text{Mask}} \|c_i - \hat c_i\|_2^2 \ \ell_{\text{feature}} &= \frac{1}{|\text{Mask}|} \sum_{i\in\text{Mask}} \|f_i - \hat f_i\|_2^2 \ \ell_{\text{motion}} &= \frac{1}{|\text{Mask}|} \sum_{i\in\text{Mask}} \|m_i - \hat m_i\|_2^2 \ \ell &= \ell_{\text{spatial}} + \lambda \cdot \ell_{\text{motion}} \end{align*}$$

Default $\lambda$ values are 1.0 (pixel) and 0.25 (CLIP+upsampled feature targets).

Training uses large-scale video data (Kinetics-400/700), ViT-B/L backbones, AdamW, and 800-1600-epoch schedules with mask ratio 90% and $\rho_{\text{motion}}=50\%$ (Vandeghen et al., 28 Mar 2026). Data augmentations and detailed optimizer hyperparameters are specified.

4. Motion-Aware Masking and Trajectory Supervision

The hallmark innovation in masked video TrackMAE is explicit use of motion-coupled supervision:

• Trajectory supervision: Sparse or upsampled tracker displacements are used as direct regression targets for masked tokens, forcing temporal feature extraction.
• Motion-aware masking: The visible tokens for encoding are balanced between regions of high and low motion, parameterized by motion magnitude $\bar M_i = \|m_i\|_2$ and motion ratio $V\in \mathbb{R}^{T\times H\times W\times 3}$0.
• Algorith- mic summary: For each example, rank tokens by $V\in \mathbb{R}^{T\times H\times W\times 3}$1, sample $V\in \mathbb{R}^{T\times H\times W\times 3}$2 visible tokens from both the top $V\in \mathbb{R}^{T\times H\times W\times 3}$3 (high motion) and remaining low-motion tokens, mask the rest. This facilitates better learning of both static and dynamic content under high-masking regimes.

Ablation studies show that adding explicit trajectory prediction (+motion loss) substantially improves representation quality compared to baseline pixel-only masked video models (e.g., +5.0 to +6.1 pp linear accuracy gains in Kinetics-400 and SSv2 linear probing) (Vandeghen et al., 28 Mar 2026).

5. Quantitative Results and Comparisons

A range of controlled studies benchmark TrackMAE against prior masked video modeling (MVM) baselines and feature-target MVMs:

• Linear Probing (ViT-B, K400 pre-train):
  • Pixel targets: VideoMAE Top-1 K400 20.7%, SSv2 17.5%; TrackMAE: 25.7% (+5.0), 23.6% (+6.1)
  • CLIP targets: TrackMAE matches SMILE on K400, outperforms on SSv2 and FineGym
• Full Fine-tuning: VideoMAE K400→K400 80.0%; TrackMAE 80.8% (+0.8). K400→SSv2: VideoMAE 68.5%, TrackMAE 70.1% (+1.6)
• SEVERE Generalization: TrackMAE surpasses VideoMAE and MGMAE in pixel regimes, matches or exceeds SMILE and other clip-feature techniques in downstream settings involving severe domain shifts, low-sample regimes, and action taxonomy granularity tasks.

Ablation findings confirm best motion ratio at 50%, declining performance at more skewed ratios. Separate spatial/motion decoders surpass unified heads by ≈0.7%. Motion-aware masking yields consistently higher linear accuracies than random masking (Vandeghen et al., 28 Mar 2026).

6. Practical Training Protocols and Implementation

TrackMAE implementations for both tracking and video settings share standard routines, with targeted modifications:

• Tracking MAE (Cui et al., 2023):
  • Patchify both template (unmasked) and search (masked 75%)
  • Encoder receives only visible tokens, decoder reconstructs search-region pixels
  • No schedule details beyond referencing vanilla MAE
• Video MAE (Vandeghen et al., 28 Mar 2026):
  • 3D patchification, spatial and motion targets
  • Optimizer: AdamW, lr=1.5×10⁻⁴, batch 512, 800 epochs+, cosine schedule
  • Motion extraction: grid tracking (G=14 for 224×224 with p=16), upsample factor ν=2 for dense supervision

Feature-space pretraining uses CLIP extractors applied to masked tubelets. Both modalities require extensive storage and compute for trajectory extraction and dense tubelet masking.

7. Significance and Implications

The introduction of TrackMAE demonstrates that tracker-tailored masked autoencoding, either via coupled template/search streams (visual tracking) or explicit motion trajectory prediction and motion-aware masking (video), consistently yields more discriminative and generalizable representations. These improvements are evidenced across multiple benchmarks, transfer learning tasks, and action recognition regimes. Explicit motion supervision is key to overcoming the static-bias limitations of prior masked pretext tasks, and the approach scales favorably with compute and model size (Cui et al., 2023, Vandeghen et al., 28 Mar 2026).

A plausible implication is that future self-supervised video and tracking models may increasingly incorporate dynamically derived or external signals (such as optical flow or multi-object tracking cues) in both masking policies and auxiliary objectives, extending the effective learning signal and enforcing richer spatio-temporal feature extraction.