DINO-Tracker: Self-Supervised Dense Tracking

• The paper presents DINO-Tracker, a self-supervised framework that refines pre-trained DINOv2 features for dense long-term point tracking.
• It employs a lightweight Delta-DINO module and CNN refiner to adapt representations for robust matching despite large motions and occlusions.
• Extensive evaluations show state-of-the-art performance among self-supervised trackers, balancing semantic stability with video-specific adaptation.

Searching arXiv for the primary paper and closely related work on DINO-based tracking. DINO-Tracker is a self-supervised framework for dense long-term point tracking in a single video. Given a video $\{I^t\}_{t=1}^T$ and a query pixel $x_q$ in a reference frame $I^k$, it estimates the full trajectory $\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$ together with visibility, without ground-truth trajectories or labels. Its central idea is to combine test-time training on the test video with localized semantic features from a pre-trained DINOv2 ViT, then refine those features into trajectory embeddings through a lightweight residual module called Delta-DINO while preserving DINO’s semantic prior. The resulting system is designed for large motions, appearance change, and long-term occlusions, and it achieves state-of-the-art results among self-supervised methods while remaining competitive with supervised trackers (Tumanyan et al., 2024).

1. Problem setting and conceptual scope

DINO-Tracker addresses dense long-term point tracking rather than box tracking or class-level detection. The target is a trajectory for an arbitrary query pixel, not a category instance. This distinguishes it from tracking-by-detection pipelines and from systems that use DINO primarily as a detector, re-identification encoder, or semantic prior for another task.

The problem is difficult because points undergo large motions and deformations, occlusions can be long and repeated, and appearance changes make pixel-level similarity unreliable. The framework is motivated by a specific tension in prior work: self-supervised trackers that rely only on the test video can adapt to video-specific motion, but they lack a strong semantic prior for re-identification after long gaps; pre-trained visual features provide semantics, but in raw form they are not sufficiently localized or temporally consistent for sub-pixel tracking (Tumanyan et al., 2024).

DINO-Tracker resolves this tension by combining two ingredients. The first is test-time training on a single video, which adapts the representation to the motion statistics of that video. The second is a frozen DINOv2 ViT backbone, whose localized semantic descriptors serve as anchors for long-range correspondence. This suggests a hybrid formulation: semantic correspondences provide long-range identity stability, while video-specific self-supervision provides localization precision and temporal consistency.

A common misconception is to read the name “DINO-Tracker” as referring to a DINO detector or a DETR-style tracking architecture. In this context, the term refers instead to a tracker built around DINO-ViT features, specifically pre-trained DINOv2 features that are refined at test time for dense point trajectories (Tumanyan et al., 2024).

2. Architectural composition

The backbone is DINOv2 ViT-L/14. Its patch embedding stride is reduced from 14 to 7 to increase spatial resolution, and token features from layer 16 are used as localized semantic descriptors. For a frame $I$, the backbone produces a feature map

$$F_{\text{DINO}}(I) \in \mathbb{R}^{H' \times W' \times C}.$$

On top of the frozen ViT, DINO-Tracker introduces Delta-DINO, a lightweight fully convolutional residual network that predicts

$R(I) \in \mathbb{R}^{H' \times W' \times C},$

and forms refined features by

$F_{\text{rfn}}(I) = F_{\text{DINO}}(I) + R(I).$

Delta-DINO has 4 convolutional layers with channels $[3 \to 64 \to 128 \to 256 \to 1024]$, with Conv2d $\to$ BatchNorm2d $x_q$0 ReLU $x_q$1 BlurPool except in the last layer, and it is zero-initialized so that early in optimization $x_q$2. This residual design is central: it adapts the representation without destabilizing the frozen DINO prior (Tumanyan et al., 2024).

Tracking itself is matching-based. For a query pixel $x_q$3 in reference frame $x_q$4, the corresponding refined feature $x_q$5 is sampled from $x_q$6. For a target frame $x_q$7, DINO-Tracker computes a cosine-similarity cost volume

$x_q$8

where $x_q$9.

This cost volume is processed by a small CNN-refiner with architecture Conv2d $I^k$0 ReLU $I^k$1 Conv2d and channels $I^k$2, followed by a spatial softmax to obtain a probability heatmap $I^k$3. The tracked position is then produced by a localized soft-argmax,

$I^k$4

where $I^k$5 is a neighborhood around the maximum of $I^k$6. The complete trainable component therefore consists of Delta-DINO, with about 7.59M parameters, and the CNN-refiner, with about 300 parameters, while the ViT backbone remains frozen (Tumanyan et al., 2024).

3. Test-time training and self-supervised objective

DINO-Tracker is optimized separately for each test video. It does not rely on offline supervised tracker training. Instead, it performs test-time optimization of the Delta-DINO parameters $I^k$7 and CNN-refiner parameters $I^k$8 using only self-supervised signals extracted from the video. The objective is

$I^k$9

The training signals are heterogeneous. RAFT optical flow between consecutive frames is precomputed and chained into reliable short tracklets, producing $\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$0. Frozen DINO best-buddy correspondences are precomputed across more distant frames, producing $\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$1. During optimization, the current refined features are also used to mine refined best-buddy pairs $\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$2, and the tracker’s own trajectories are used to extract cycle-consistent coordinate pairs $\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$3 (Tumanyan et al., 2024).

The flow loss supplies sub-pixel short-term supervision through a Huber penalty on predicted correspondences. The DINO best-buddies loss is a SimCLR-style contrastive term over mutual nearest neighbors obtained from frozen DINO features, thereby distilling long-range semantic correspondences into the refined representation. The refined best-buddies loss repeats that mechanism in the learned feature space, reinforcing newly formed reliable correspondences. The cycle-consistency loss enforces bidirectional agreement of the tracker’s own predictions.

The prior-preservation term prevents the refined representation from drifting too far from the frozen backbone. It penalizes both angular deviation and norm deviation between $\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$4 and $\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$5. This regularization is what makes the phrase “taming DINO” technically precise: the method does not replace DINO’s semantics, but constrains adaptation to remain close to them while becoming trajectory-aware (Tumanyan et al., 2024).

Optimization is performed with Adam. The typical schedule is 10k iterations for videos up to 100 frames and 20k iterations for 250-frame Kinetics videos. Some losses, notably refined best-buddies and cycle-consistency, are activated only after 5k iterations, once the refined features are sufficiently stable. Training uses batches of 8 frames with 512 flow correspondence pairs, up to 1024 DINO best-buddy pairs, and up to 1024 refined best-buddy and cycle-consistent pairs, with foreground-balanced sampling (Tumanyan et al., 2024).

4. Semantic correspondences, occlusion robustness, and limitations

The distinctive contribution of DINO-Tracker is the use of DINOv2’s localized semantic features as a long-range correspondence prior. Best-buddy pairs are defined by mutual nearest neighbors in feature space:

$\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$6

These correspondences are valuable precisely where optical flow is weak: large temporal gaps, long occlusions, and substantial appearance change (Tumanyan et al., 2024).

The method therefore combines two complementary regimes. Optical flow provides accurate local displacement when points remain visible and motion is short-term. DINO-derived correspondences provide semantic continuity when local continuity breaks. The training procedure explicitly exploits this complementarity by using DINO-based supervision where flow is unreliable and avoiding redundancy where valid flow supervision already exists.

The paper characterizes the resulting refined descriptors as trajectory embeddings. A feature sampled along a trajectory becomes tightly clustered across time, whereas raw DINO features are more scattered and may overlap with other trajectories. This implies that the refined representation is not merely more discriminative in the standard metric-learning sense; it is discriminative specifically with respect to temporal identity under the motion distribution of the current video (Tumanyan et al., 2024).

Empirically, this is most visible under long-term occlusions. As occlusion rate increases on TAP-Vid-DAVIS, DINO-Tracker degrades less than Omnimotion, TAPIR, and Co-Tracker, and its Average Jaccard and positional accuracy remain substantially higher in the high-occlusion regime. The paper explicitly attributes this to the semantic prior from DINO, the long-range best-buddy correspondences, and the progressive reinforcement of reliable self-mined correspondences (Tumanyan et al., 2024).

Its limitations are correspondingly specific. The method does not explicitly model trajectories while a point is fully hidden behind an occluder; it is strong at re-identification before and after occlusion, not at reconstructing hidden motion. It can also confuse multiple semantically similar instances, because the prior is dominated by semantics rather than explicit instance separation. This suggests that DINO-Tracker’s strength is semantic persistence, whereas its remaining weakness is instance-level disambiguation under semantic equivalence (Tumanyan et al., 2024).

5. Evaluation, ablations, and computational profile

DINO-Tracker is evaluated on TAP-Vid-DAVIS, TAP-Vid-Kinetics, and BADJA. The reported metrics are position accuracy $\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$7, occlusion accuracy (OA), and Average Jaccard (AJ) for TAP-Vid, and $\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$8 and $\mathcal{T}_q = \{\hat{x}^t\}_{t=1}^T$9 for BADJA (Tumanyan et al., 2024).

Benchmark	Metrics	DINO-Tracker
TAP-Vid-DAVIS-480	$I$0, OA, AJ	80.4, 88.1, 64.6
TAP-Vid-Kinetics-480	$I$1, OA, AJ	74.3, 89.2, 60.9
BADJA	$I$2, $I$3	72.4, 14.3

These results place the method above Omnimotion on both TAP-Vid-DAVIS-480 and TAP-Vid-Kinetics-480, and at state of the art on BADJA among the compared methods. The paper also notes that raw DINOv2 nearest-neighbor tracking is already unexpectedly strong: it beats RAFT and even TAP-Net on DAVIS-256. This is important context, because it shows that a substantial part of the method’s effectiveness derives from the intrinsic localization and semantics of DINOv2 features before any test-time adaptation (Tumanyan et al., 2024).

The ablation studies are particularly informative. Removing DINO and training Delta-DINO purely on appearance reduces DAVIS-480 $I$4 from 80.4 to 71.4. LoRA fine-tuning of DINO is slower and produces less localized, more jittery heatmaps than the residual CNN design. Removing the DINO-bb, refined-bb, cycle-consistency, or prior-preservation terms causes measurable degradation. Removing the flow loss reduces positional accuracy only slightly, by about 2%, which underscores how much of the final performance comes from semantic supervision plus refined features rather than from optical flow alone (Tumanyan et al., 2024).

The computational profile is also central to the method’s positioning. For a roughly 100-frame video, test-time training takes about 1.6 hours on a single A100 GPU, which the paper describes as about 10 times faster than Omnimotion. By contrast, LoRA fine-tuning of DINO for tracking takes about 9 hours per video. DINO-Tracker is therefore not a real-time method, but within the class of per-video optimization methods it is comparatively efficient (Tumanyan et al., 2024).

6. Relation to later DINO-based tracking paradigms

Subsequent work shows that DINO-Tracker inaugurated a broader design pattern in which DINO-family features are not treated as generic embeddings alone, but as structural priors to be adapted, constrained, or converted into explicit correspondence mechanisms.

In medical imaging, "DINOMotion: advanced robust tissue motion tracking with DINOv2 in 2D-Cine MRI-guided radiotherapy" specializes this idea for registration under large, nonrigid motion. It uses DINOv2 with LoRA layers to discover corresponding landmarks and then computes rigid, affine, or thin-plate spline registration analytically, without iterative optimization at test time. On volunteer and patient datasets it reports Dice scores of 92.07% for the kidney, 90.90% for the liver, and 95.23% for the lung, processes each scan in approximately 30 ms, and emphasizes interpretability through explicit visual correspondences (Salari et al., 14 Aug 2025). This is a different problem setting from DINO-Tracker, but it preserves the same conceptual core: DINO features act as a correspondence engine.

In dynamic 3D scene modeling, "DINO_4D: Semantic-Aware 4D Reconstruction" uses frozen DINOv3 features as structural priors to suppress semantic drift during dense 3D point tracking and 4D reconstruction. It maintains linear time complexity $I$5 and improves APD on Point Odyssey from 35.1% to 41.8% at 0.1 m and from 67.4% to 78.1% at 0.3 m relative to St4RTrack, while also reducing Chamfer Distance on TUM-Dynamics to 5.11 cm with its full model (Yang et al., 10 Apr 2026). Here, DINO features are again used to stabilize temporal identity, but the tracked object is a dense set of 3D points rather than a 2D query pixel.

Other descendants redistribute the role of DINO within multi-object systems rather than dense point tracking. "DINO-CoDT: Multi-class Collaborative Detection and Tracking with Vision Foundation Models" uses DINOv2 for appearance-based tracklet re-identification in collaborative 3D detection and tracking, rather than as the primary dense correspondence representation (He et al., 9 Jun 2025). "Zero-Shot Multi-Animal Tracking in the Wild" uses Grounding DINO as a promptable zero-shot detector together with SAM 2 for segmentation-based tracking, yielding HOTA scores of 58.6 on ChimpAct, 74.8 on Bird Flock Tracking, 58.0 on AnimalTrack, and 62.4 on GMOT-40-Animal without retraining or hyperparameter adaptation (Meier et al., 4 Nov 2025). These systems are architecturally different, but they reinforce a broader interpretation: DINO-family models can support tracking as semantic priors, re-identification features, promptable detectors, or dense descriptors.

Taken together, these works suggest that DINO-Tracker (Tumanyan et al., 2024) is best understood not only as a specific dense point tracker, but as an early and technically influential instance of a larger paradigm: tracking by constraining video-specific adaptation with frozen, semantically structured representations from self-supervised vision foundation models.