Track-On: Video & 3D Ion Tracking

Updated 3 July 2026

Track-On is a dual-context framework that enables online video point tracking via transformer architectures and high-precision 3D ion track reconstruction in radiobiology.
In video tracking, it employs strictly causal decoding and memory-augmented inference to maintain consistent point correspondences under motion, occlusion, and appearance variations.
For 3D ion trajectory reconstruction, it uses fluorescent nuclear track detectors with intensity-weighted centroiding and regression techniques to achieve sub-micron accuracy.

Track-On refers to a set of methodologies and systems for online point tracking, encompassing both cutting-edge transformer-based models for real-time video tracking and high-precision 3D ion track reconstruction in radiobiology. While these approaches span disparate domains—computer vision and clinical dosimetry—they share core themes of robust temporal correspondence and memory-augmented inference to maintain track integrity under challenging conditions.

1. Problem Definitions

The term "Track-On" is used in two primary contexts:

A. Online Point Tracking in Video:

Track-On and Track-On2 are transformer-based architectures designed for online, frame-by-frame identification and tracking of points across video sequences. The objective is to maintain consistent point correspondences throughout long-term sequences, even in the presence of substantial appearance variation, motion, and occlusion, using only information up to the current frame—no future frames are accessed (Aydemir et al., 30 Jan 2025, Aydemir et al., 23 Sep 2025).

B. 3D Ion Track Reconstruction:

Track-On also denotes a framework for reconstructing ion trajectories in three dimensions using alumina-based fluorescent nuclear track detectors (FNTDs), with the goal of mapping physical ion paths to biological outcomes in radiotherapy research. Here, Track-On provides a direct, quantitative link between radiation dose deposition and biological effect at the sub-cellular level (Niklas et al., 2013).

2. Track-On in Online Video Point Tracking

Track-On and its successor Track-On2 are transformer-based, strictly causal models for consistent long-term point tracking in video.

Architecture Details:

Visual Encoder: A frozen Vision Transformer (ViT; DINOv2 or DINOv3 ViT-S+) with ViT-Adapter extracts dense, multi-scale features for each incoming frame (Aydemir et al., 30 Jan 2025, Aydemir et al., 23 Sep 2025).
Query Initialization: For each query point, initial features are bilinearly sampled at the given coordinates from the current frame feature map.
Causal Decoding: Transformer decoder blocks handle cross-attention with current frame features, self-attention among queries, and cross-attention to memory. No access to future frames is permitted, ensuring true online operation.
Two-stage Coarse-to-Fine Inference:

Coarse Patch Classification: Each query computes cosine similarity to backbone features at multiple scales, followed by a spatial softmax for coarse localization.
Re-ranking and Offset Prediction: Top-k candidate patches are refined using deformable attention; a detector head predicts sub-patch offsets, yielding precise point localization. Visibility and uncertainty are also regressed (Aydemir et al., 30 Jan 2025, Aydemir et al., 23 Sep 2025).

Memory Mechanism (Track-On2):

A single, per-query expandable FIFO memory buffer retains up to L most recent refined query embeddings, augmented with learnable time-positional encodings. Memory attention is incorporated via cross-attention in the decoder, allowing the model to recall past context efficiently (Aydemir et al., 23 Sep 2025).

Loss and Metrics:

The total per-frame loss includes patch classification (cross-entropy), offset regression (L1 loss), and binary cross-entropy for visibility and uncertainty estimates. Evaluation metrics include "x" (localization PCK at {1,2,4,8,16} px), average Jaccard (AJ), occlusion accuracy (OA), mean track error (MTE), and survival rate on specialized datasets (Aydemir et al., 30 Jan 2025, Aydemir et al., 23 Sep 2025).

3. Synthetic Training Paradigms and Model Robustness

Track-On models are exclusively trained on synthetic data (TAP-Vid-Kubric), with no reliance on real-video fine-tuning (Aydemir et al., 30 Jan 2025, Aydemir et al., 23 Sep 2025). Synthetic clip length and memory size at training have significant impact on long-term tracking robustness:

Increasing training sequence length (e.g., T from 24 to 48) confers greater benefits than expanding memory size alone, especially for long-term datasets.
Uniform and random frame sampling strategies affect benchmark performance, with best settings dataset-dependent.
Inference-time memory expansion (IME)—interpolating positional encodings to allow longer test-time memory—boosts performance on long videos but can degrade results in short-clip scenarios if memory becomes redundant (Aydemir et al., 23 Sep 2025).

4. Architectural Evolution: Track-On to Track-On2

Track-On initially used two separate per-query FIFO memory modules:

Spatial Memory: Stores local features around previous predictions, facilitating correction of drift.
Context Memory: Stores past decoded queries for longer-term temporal reasoning (Aydemir et al., 30 Jan 2025).

Track-On2 consolidates memory into a single expandable buffer for per-query embeddings (shape N×L×D), resulting in:

25% faster decoding and lower GPU memory footprint.
More effective multi-scale feature fusion using a ViT-Adapter with FPN replacing single-scale pooling.
Streamlined visibility/offset heads to reduce latency.
Upgraded backbone (optionally DINOv3 ViT-S+), yielding strong real-video results without fine-tuning.
Training on substantially longer clips (up to 200 epochs) to foster deep temporal reasoning (Aydemir et al., 23 Sep 2025).

5. Experimental Results and Benchmark Performance

Track-On and Track-On2, trained solely on synthetic data and operating strictly causally, establish superior results across both synthetic and real-world tracking benchmarks.

Benchmark	Track-On2 (DINOv3)	Track-On (v2 DINOv2)	Offline SOTA
TAP-Vid DAVIS	AJ=67.0, x=79.9, OA=92.0	AJ=66.8, x=79.8, OA=91.7	BootsTAPNext-B (AJ=65.2)
TAP-Vid Kinetics	AJ=55.3, x=69.3, OA=89.6	—	TAPTRv3/matches real-finetune
RoboTAP	AJ=68.1, x=80.5, OA=93.4	—	—
Dynamic Replica	x=74.6	—	CoTracker3 (x=72.3)
PointOdyssey	x=47.4, MTE=20.5, survival=57.8	—	Many offline run out of memory

Track-On2 tracks 256 points at over 30 FPS on a single 40 GB GPU with under 1 GB memory, scaling to 1,024 points and extended memory (Lᵢ=72) with less than 1.5 GB usage (Aydemir et al., 23 Sep 2025).

6. Track-On for 3D Ion Trajectory Reconstruction

Distinct from video-tracking models, Track-On also references an experimental framework for reconstructing single-ion tracks using FNTDs, enabling direct correlation between ion traversals and biological endpoints such as DNA damage (Niklas et al., 2013).

Key Methodology:

Imaging and Readout:

FNTDs of α‐Al₂O₃:C,Mg serve as both passive track recorders and substrates for live-cell experiments. Confocal microscopy with 633 nm excitation and precise optical sectioning (~3 μm steps) enables 3D trajectory acquisition.

Track-Spot Detection:

Segmentation and centroiding isolate the track core, discarding δ-electron signal. Intensity-weighted centroiding with dynamic thresholds yields lowest residuals.

3D Regression and Angle Estimation:

Least-squares fits yield straight-line models for each trajectory; angles (θ, φ) are derived from fitted slopes. Error metrics include 95% confidence and prediction intervals for axial (θ) and lateral (x, y) coordinates.

Axial Correction:

Refractive-index mismatches across oil, cell, and Al₂O₃ interfaces introduce focal distortions, modeled and corrected with analytical formulas.

Quantitative Performance:

Perpendicular irradiation (θ≈0°): θ error < 0.10°, PIₓ, PI_y ≈ 0.12 μm.
Angular irradiation (θ≈60°): θ error ≈ 0.89°, PIₓ=0.28 μm, PI_y=1.72 μm.
Spot sampling: ≥21 track spots (Δz=3 μm) recommended for Δθ<1° at θ=60°. A plausible implication is that system performance degrades at high θ and coarse sampling, largely due to elongated asymmetric spot morphology (Niklas et al., 2013).

Applications:

Track-On supports radiobiological mapping in live-cell/FNTD hybrids (e.g., A549 epithelial monolayers) and precise clinical heavy-ion beam monitoring, achieving sub-micron and sub-degree accuracy (Niklas et al., 2013).

7. Implementation Recommendations and Use Cases

Online Video Tracking:

Employ strictly causal transformer decoding with memory modules for long-sequence robustness.
Train on synthetic data with increased clip length for resilience to drift and occlusion.
Exploit inference-time memory extension in long videos, carefully balancing memory size to avoid redundancies (Aydemir et al., 23 Sep 2025).

3D Ion Track Reconstruction:

Use intensity-weighted centroiding or dynamic thresholding for spot localization.
Acquire at least 21 track spots per trajectory with fine z-sectioning (Δz=3 μm).
Minimize focal depth bias by mounting in refractive-matched media or applying analytical corrections.
Apply perpendicular irradiation (θ≤30°) in live-cell radiobiology for maximal angular and lateral accuracy.
Prioritize high SNR acquisition parameters to balance photobleaching against spatial resolution (Niklas et al., 2013).

Track-On, in both video tracking and radiobiological ion-track reconstruction, exemplifies memory-augmented, causal correspondence under resource, temporal, and signal constraints, setting benchmarks for real-time operation and quantitative precision across domains (Aydemir et al., 30 Jan 2025, Aydemir et al., 23 Sep 2025, Niklas et al., 2013).