Track-On2: Transformer Online Point Tracking

• Track-On2 is an online transformer-based model for long-term point tracking that uses per-query memory to maintain temporal coherence with only past frames.
• It employs a coarse patch-level classification followed by local refinement to predict point positions and visibility under challenging conditions.
• The model scales efficiently due to its single expandable memory and multi-scale visual features, enabling real-time processing in streaming applications.

to=arxiv_search 天天中彩票中奖json 天天爱彩票中奖ant{ "query": "Track-On2 Enhancing Online Point Tracking with Memory arXiv 2025", "max_results": 5, "sort_by": "relevance" } Track-On2 is a transformer-based model for online long-term point tracking that follows queried points across video frames under large appearance changes, significant motion, and occlusion while operating strictly causally, i.e. frame by frame without access to future frames. It extends Track-On by replacing a dual-memory design with a single expandable per-query memory, introducing multi-scale visual features through a ViT-Adapter and FPN stack, and retaining a classification-first tracking formulation in which coarse patch-level localization is followed by local refinement. In the reported formulation, given a video $V=\{I_1,\dots,I_T\}$ and a set of point queries $Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$, the tracker predicts point coordinates $\hat{p}_t$, visibility $\hat{v}_t$, and an updated memory state $M_t$ through

$$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$

with $M_t$ serving as the mechanism for temporal coherence in the absence of bidirectional context (Aydemir et al., 23 Sep 2025).

1. Problem formulation and causal setting

Track-On2 addresses the problem of long-term point tracking, in which each query point is specified by a start time and pixel coordinates and must be tracked across all subsequent frames while also predicting whether it is visible or occluded. The target outputs are the 2D point position $\hat{p}_t^i \in \mathbb{R}^2$ and visibility $\hat{v}_t^i \in \{0,1\}$ for each queried point at each time step. The model is designed for the online setting, meaning that at time $t$ it uses only the current frame $Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$0 and past memory $Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$1, not future frames (Aydemir et al., 23 Sep 2025).

This causal formulation distinguishes Track-On2 from recent offline or windowed trackers such as TAPIR, CoTracker, TAPTR, and TAPNext, which use all frames in a clip or a large temporal window and exploit bidirectional context. Track-On2 instead processes frames sequentially and maintains temporal continuity through memory attention, making it suitable for real-time, streaming, and long-video deployment. The method is explicitly motivated by the difficulties of long-term tracking under feature drift, large appearance changes, motion, and occlusion, especially when future frames are unavailable (Aydemir et al., 23 Sep 2025).

A central design choice is that tracking is not treated as direct coordinate regression over the full frame. As in Track-On, the problem is cast first as patch classification over the current frame and only then as local coordinate refinement. This suggests a discrete-to-continuous inference strategy: identify a plausible coarse region through similarity classification, then refine to sub-patch precision. A plausible implication is that this formulation improves robustness against distractors and ambiguous local evidence relative to purely continuous regression (Aydemir et al., 30 Jan 2025).

2. Relation to Track-On

Track-On2 is the successor to "Track-On: Transformer-based Online Point Tracking with Memory" (Aydemir et al., 30 Jan 2025). Track-On introduced a causal transformer architecture in which point queries attend to current-frame features and two explicit memory modules: a spatial memory storing local regions around past predicted locations and a context memory storing past decoded query embeddings. It already used patch classification, top-$Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$2 re-ranking, local offset prediction, and visibility/uncertainty heads, and it demonstrated that memory was responsible for large gains over a vanilla transformer without memory (Aydemir et al., 30 Jan 2025).

Track-On2 simplifies this design while extending its temporal capacity. The most visible architectural change is the replacement of the two-memory system with a single per-query memory that stores a compact embedding history $Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$3 with up to $Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$4 entries per query, managed by FIFO replacement. Track-On2 also replaces the earlier single-scale feature design with a genuine multi-scale encoder built from DINOv3 or DINOv2 together with ViT-Adapter and FPN. The visibility head is simplified to an MLP operating on decoded query features rather than the earlier deformable-attention-based head (Aydemir et al., 23 Sep 2025).

The temporal training regime is also extended. Track-On used a training horizon of 24 frames and memory size 12, whereas Track-On2 trains on 48-frame clips with memory size 24. The later model explicitly studies the effect of training clip length, training memory size, inference-time memory extension, and frame sampling strategy, and concludes that training clip length is the dominant factor for long-term generalization, with memory capacity playing a secondary role (Aydemir et al., 23 Sep 2025).

These changes are accompanied by empirical gains over Track-On on the same backbone family. Using the same DINOv2 backbone, Track-On2 improves over Track-On on TAP-Vid DAVIS, TAP-Vid Kinetics, RoboTAP, Dynamic Replica, and PointOdyssey; on PointOdyssey, for example, the reported $Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$5 score rises from $Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$6 to $Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$7 and Survival from $Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$8 to $Q=\{(t^1,p^1),\dots,(t^N,p^N)\}$9 (Aydemir et al., 23 Sep 2025).

3. Architecture: visual encoding, unified memory, and decoder design

Track-On2 takes as input the current frame $\hat{p}_t$0, the static set of point queries $\hat{p}_t$1, and the previous memory $\hat{p}_t$2, and outputs point coordinates, visibility, uncertainty, and updated memory. Its visual encoder uses DINOv3 ViT-S+ by default, with DINOv2 ViT-S as an alternative, together with ViT-Adapter to produce four feature scales

$\hat{p}_t$3

which are fused by an FPN into

$\hat{p}_t$4

For each query $\hat{p}_t$5, the initial query feature is obtained by sampling $\hat{p}_t$6 at the starting coordinate,

$\hat{p}_t$7

This defines a static query embedding set $\hat{p}_t$8 (Aydemir et al., 23 Sep 2025).

The unified memory has shape

$\hat{p}_t$9

so each query has its own temporal buffer of up to $\hat{v}_t$0 embeddings. Memory entries are augmented with a learnable temporal positional embedding $\hat{v}_t$1, allowing the decoder to distinguish recent from older states. The decoder uses

$\hat{v}_t$2

At inference, Track-On2 supports Inference-time Memory Extension (IME), in which memory is extended from training size $\hat{v}_t$3 to a larger $\hat{v}_t$4 by linearly interpolating $\hat{v}_t$5 to the new length without modifying network weights. This keeps memory footprint bounded by $\hat{v}_t$6 and enables longer temporal context on long videos (Aydemir et al., 23 Sep 2025).

The query decoder is transformer-based and applies three attention stages at each time step. First, feature cross-attention lets queries attend to the current frame features $\hat{v}_t$7. Second, query self-attention allows different tracked points to interact jointly, following the idea that tracking multiple points simultaneously can exploit spatial correlations. Third, memory cross-attention lets each query attend to its own temporal history in $\hat{v}_t$8. The memory stage is the principal causal mechanism for temporal coherence: it lets the tracker retrieve updated appearance information and latent state history without requiring access to future frames (Aydemir et al., 23 Sep 2025).

4. Patch classification, re-ranking, refinement, and memory update

Track-On2 predicts point locations through a coarse-to-fine pipeline. From decoded queries $\hat{v}_t$9 and the multi-scale features $M_t$0, it computes cosine similarity maps across scales, aggregates them into a single similarity map $M_t$1, and applies temperature scaling followed by a spatial softmax, thereby turning localization into a $M_t$2-way patch classification problem. The ground-truth class is the patch containing the true point (Aydemir et al., 23 Sep 2025).

The model then applies a re-ranking module motivated by the observation that the correct patch is often not the top-1 patch but is usually present among the top-$M_t$3 candidates. It selects the top-$M_t$4 patches from $M_t$5, retrieves their features using deformable cross-attention from the multi-scale maps $M_t$6, fuses them back into the query representation, and produces refined queries

$M_t$7

A second similarity map $M_t$8 is then computed from $M_t$9. The highest-scoring patch center becomes the coarse estimate $$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$0 (Aydemir et al., 23 Sep 2025).

Local refinement is performed by a 3-layer deformable transformer decoder $$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$1, which takes the refined queries $$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$2, the fused feature map $$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$3, and the coarse patch centers as reference points: $$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$4 The offsets are constrained to $$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$5 via $$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$6, where $$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$7 is the patch stride. Visibility $$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$8 and global uncertainty $$\hat{p}_t,~ \hat{v}_t,~ M_t = \Phi(I_t, Q, M_{t-1}; \Theta),$$9 are predicted from $M_t$0 by a lightweight MLP. During training, points whose localization error exceeds $M_t$1 pixels or are occluded are labeled uncertain; at inference, visibility is thresholded by $M_t$2 (Aydemir et al., 23 Sep 2025).

The refined queries $M_t$3 are then appended to the per-query memory. When the memory is full, the oldest entry is evicted, so the tracker maintains a bounded temporal state. This update rule makes the memory effectively a history of updated appearance-conditioned query embeddings, not merely a cache of raw coordinates. This suggests that memory primarily functions as a learned representation of evolving point identity under changing visual conditions (Aydemir et al., 23 Sep 2025).

The loss used to train Track-On2 combines coarse and refined patch classification, local offset regression, visibility prediction, uncertainty prediction, and supervision over top-$M_t$4 candidates: $M_t$5 Here $M_t$6 masks visibility-dependent terms, $M_t$7 weights the patch-classification term, and deep supervision is used for the offset head (Aydemir et al., 23 Sep 2025).

5. Training regime, synthetic data, and benchmark results

Track-On2 is trained purely on synthetic data, specifically TAP-Vid Kubric in the CoTracker3 variant, comprising approximately 6k synthetic sequences of 120 frames with ground-truth trajectories and visibilities. The default training configuration uses input resolution $M_t$8, clip length $M_t$9, memory size $\hat{p}_t^i \in \mathbb{R}^2$0, top-$\hat{p}_t^i \in \mathbb{R}^2$1 patches for re-ranking, and up to $\hat{p}_t^i \in \mathbb{R}^2$2 queries per clip. Queries are sampled so that 75% come from the first or middle frame and the remainder from other frames. Optimization uses AdamW with learning rate $\hat{p}_t^i \in \mathbb{R}^2$3, cosine decay, 1% warmup, and 200 epochs on $\hat{p}_t^i \in \mathbb{R}^2$4A100 64GB GPUs with mixed precision. Memory attention also uses key masking with random mask ratio $\hat{p}_t^i \in \mathbb{R}^2$5 (Aydemir et al., 23 Sep 2025).

The paper studies several temporal design choices. Increasing training clip length from 24 to 36 to 48 frames substantially improves long-term performance, whereas increasing training memory size from 18 to 24 yields only small gains or negligible effects, depending on the benchmark. The ablations therefore identify training video length as the primary determinant of long-term generalization. IME further improves performance on long sequences such as RoboTAP and Dynamic Replica when inference memory $\hat{p}_t^i \in \mathbb{R}^2$6 is increased, though overly long memory can slightly hurt short-sequence performance on DAVIS. Frame sampling strategy also matters: uniform sampling with stride ratio $\hat{p}_t^i \in \mathbb{R}^2$7 is best on average, while random sampling with $\hat{p}_t^i \in \mathbb{R}^2$8 performs best on RoboTAP (Aydemir et al., 23 Sep 2025).

Evaluation is reported on TAP-Vid DAVIS, TAP-Vid Kinetics, RoboTAP, Dynamic Replica, and PointOdyssey using the queried-first protocol. Metrics include AJ (Average Jaccard), $\hat{p}_t^i \in \mathbb{R}^2$9 / $\hat{v}_t^i \in \{0,1\}$0, OA (Occlusion Accuracy), MTE (Median Trajectory Error), and Survival. Reported results include:

Benchmark	Track-On2 result
TAP-Vid DAVIS (DINOv3)	AJ 67.0, $\hat{v}_t^i \in \{0,1\}$1 79.9, OA 92.0
TAP-Vid Kinetics (DINOv3)	AJ 55.3, $\hat{v}_t^i \in \{0,1\}$2 69.3, OA 89.6
RoboTAP (DINOv3)	AJ 68.1, $\hat{v}_t^i \in \{0,1\}$3 80.5, OA 93.4
Dynamic Replica (DINOv2)	$\hat{v}_t^i \in \{0,1\}$4 74.6
PointOdyssey (DINOv2)	$\hat{v}_t^i \in \{0,1\}$5 47.4, MTE 20.5, Survival 57.8

These results are presented as state-of-the-art across five synthetic and real-world benchmarks, including cases where Track-On2 surpasses offline methods that use bidirectional context. On long sequences, PointOdyssey is particularly significant because some full-video models reportedly run out of memory, whereas Track-On2 continues to operate with bounded memory and competitive or superior accuracy (Aydemir et al., 23 Sep 2025).

6. Efficiency, limitations, and significance

Track-On2 is explicitly designed for streaming efficiency. Reported runtime and memory figures on A100 in FP32 include tracking 64 points at more than 35 FPS with approximately 0.64 GB GPU memory, 256 points at about 35 FPS with less than 1 GB, and 1024 points at more than 20 FPS with about 1.5 GB when using $\hat{v}_t^i \in \{0,1\}$6. Even for more than 1000 points with extended memory, GPU memory remains below 2.5 GB. The paper attributes these properties to the unified memory design, the coarse patch-level classification strategy, deformable attention restricted to top-$\hat{v}_t^i \in \{0,1\}$7 candidates and local refinement regions, and a frozen DINOv2/v3 backbone with approximately 23.6M learnable parameters, compared with about 194M for BootsTAPNext-B (Aydemir et al., 23 Sep 2025).

The model’s main strengths are therefore twofold. First, it achieves online long-term tracking without access to future frames while retaining strong temporal robustness through memory attention. Second, it scales naturally to very long videos because compute and memory are bounded per frame rather than growing with the number of processed frames. This combination is the core reason Track-On2 is positioned as a practical model for robotics, augmented reality, streaming perception, and other long-context video applications (Aydemir et al., 23 Sep 2025).

The reported limitations are also specific. Track-On2 remains appearance-centric: patch classification is driven by feature similarity, and memory stores embeddings rather than explicit motion states. In textureless or repetitive regions this can lead to jitter or visually unstable predictions. The method is also trained only on synthetic data, and the paper notes that further gains may be achievable through real-world fine-tuning, pseudo-labeling, or distillation from real videos. Future directions proposed include explicit motion modeling, real-world fine-tuning, and extending the causal memory architecture to downstream tasks such as 3D reconstruction, visual odometry, robotics policy learning, and long-context video understanding (Aydemir et al., 23 Sep 2025).

In the broader development of point tracking, Track-On2 represents a specific synthesis: a causal transformer, a classification-first localization scheme, and a single expandable per-query memory whose behavior is shown to depend strongly on training horizon. Relative to Track-On, it simplifies the memory architecture, improves speed and accuracy, and provides a more systematic account of how synthetic training and temporal design shape long-range robustness (Aydemir et al., 30 Jan 2025).