TrackFormer: Transformer Tracking Models

• TrackFormer is a suite of Transformer-based models that use self-attention for end-to-end track assignment in both multi-object video tracking and large-scale particle tracking.
• The architectures replace conventional pipelines with encoder-only or encoder-decoder designs, achieving high accuracy (up to 94% fit accuracy) and rapid inference (sub-10 ms per event) in complex settings.
• Enhancements like block-sparse attention and joint contrastive losses boost performance in real-time scenarios, making TrackFormer effective for HL-LHC events and crowded video scenes.

TrackFormer refers to a family of Transformer-based architectures for solving tracking problems in both high-energy physics (HEP) and computer vision, with primary instantiations in two lines of research: (1) multi-object tracking (MOT) in video (TrackFormer, Meinhardt et al.), and (2) large-scale particle tracking for collider experiments (TrackFormers, Brüning, Sliwa, et al.). Across domains, TrackFormer approaches replace conventional tracking-by-matching pipelines with end-to-end trainable architectures, leveraging self-attention to perform both assignment and identity association in a single forward pass.

1. Problem Formulation and Motivation

In high-energy physics, the challenge is the partitioning of unordered 3D sensor hits from a collision event, $H = [h_1, ..., h_N] \in \mathbb{R}^{N \times 3}$, into candidate trajectories (“tracks”). Each hit $h_i$ is represented by its spatial coordinates $(x_i, y_i, z_i)$. The goal is to assign each hit a track label $\ell_i \in \{1, ..., M\}$, or equivalently, group hits into sets $T_1, ..., T_M$ corresponding to true particles. The computational bottleneck is the global, non-local association of hits, made more severe with increasing pile-up and detector complexity as foreseen for the High-Luminosity LHC upgrade (Caron et al., 9 Jul 2024).

In video MOT, the task is to maintain consistent object identities across frames by predicting trajectories, requiring joint spatiotemporal reasoning. Here, bounding boxes or segmentation masks are assigned object IDs over time, demanding robust data-association even under occlusions and appearance changes (Meinhardt et al., 2021).

Across both settings, the central challenge is learning global (often permutation-invariant) associations efficiently and accurately from high-dimensional, set-structured inputs, motivating an attention-based description.

2. Core Architectures and Design Choices

High-Energy Physics: TrackFormers

TrackFormers introduce several Transformer-based models, unified by input featurization, self-attention, and task-specific output heads. The principal variants are:

• Encoder–Classifier (EncCla, "one-shot"): An encoder-only Transformer processes all hits in parallel, outputting per-hit logits over quantile-binned track classes. No positional encoding is applied to the unordered hit set. Prediction is performed for all hits simultaneously. The classification loss is multi-class cross-entropy,

$$L_{CE} = -\sum_{i=1}^N \sum_{c=1}^C y_{i,c} \log p_{i,c}$$

where $p_{i,c}$ is the softmax output for the $i$th hit and $c$th class (Caron et al., 9 Jul 2024).

• Encoder–Regressor (EncReg): Also encoder-only, but regresses continuous track parameters per hit, followed by density-based clustering (HDBSCAN) to group hits. Regression targets are e.g., $\theta, \phi, p, q$, and the loss is mean squared error.
• Encoder–Decoder (EncDec, autoregressive): A traditional encoder–decoder Transformer recursively predicts the next hit conditioned on an input segment (analogous to next-token prediction in LLMs).
• Sparse U-Net: Detectors are voxelized, and a sparse convolutional (U-Net) network performs per-voxel classification.

Enhancements in TrackFormers Part 2 (Caron et al., 30 Sep 2025) include:

• Block-sparse attention with FlexAttention to restrict attention to geometrically nearby hits, reducing computational demand.
• InfoNCE contrastive loss to jointly embed hits from the same particle close in latent space:

$$L_i = -\frac{1}{|P(i)|} \sum_{j \in P(i)} \log \frac{\exp(\mathrm{sim}(z_i, z_j)/\tau)}{ \sum_{k \neq i} \exp(\mathrm{sim}(z_i, z_k)/\tau)}$$

where $P(i)$ are positive hits (same true track), and $\mathrm{sim}(\cdot, \cdot)$ is cosine similarity.

• "Joint" models (JM) inject regressed physics features into the classifier, further boosting accuracy.

Computer Vision: TrackFormer

TrackFormer in computer vision (Meinhardt et al.) follows an encoder–decoder Transformer structure (Meinhardt et al., 2021):

• Feature Encoding: Framewise image features are extracted by a ResNet-50 backbone, followed by a Transformer encoder over flattened patches with 2D positional encodings.
• Query-based Decoding: The decoder input consists of:
  • Static object queries: learned embeddings for initialization of new tracks at every frame.
  • Autoregressive track queries: output embeddings from previous frames, carried forward to represent and update existing tracks.
• Decoder layers alternate self-attention (over queries) with encoder–decoder attention (queries to frame tokens), enabling global data association in both time and space.
• Assignment of predictions to ground truth is framed as a set-prediction (Hungarian matching) problem.

3. Data, Preprocessing, and Experimental Setup

HEP TrackFormers operate on two primary data sources:

• REDVID Framework: Toy simulator producing events in $(r, \theta, z)$ space, with variable track multiplicity and no noise, to isolate baseline performance.
• TrackML Reductions: Realistic, large-scale events with tens of thousands of tracks per event. Preprocessing involves coordinate transformations and quantile-binning for classification targets (Caron et al., 9 Jul 2024). New datasets from ACTS-Pythia-Fatras chains enable training on genuine physics processes for variable pile-up (μ) in (Caron et al., 30 Sep 2025).

Computer vision TrackFormer is trained on MOT17, MOT20 (bounding box tracking), and MOTS20 (mask tracking), with pretraining on CrowdHuman.

4. Quantitative Performance and Metrics

HEP TrackFormers

Performance is summarized using FitAccuracy (fraction of hits assigned to ≥50% pure tracks with ≥4 hits) and mean inference time per event.

Model/Data	REDVID (10–50)	REDVID (helical)	TrackML (10–50)	TrackML (200–500)
EncCla	93%	93%	94%	78%
EncReg	97%	92%	93%	70%
U-Net	68%	62%	—	—

• Encoder–Classifier achieves <10 ms inference per HL-LHC scale event, supporting O(10⁴) hits/event on commodity GPUs—two orders of magnitude faster than Kalman Filter or graph neural network (GNN) pipelines (Caron et al., 9 Jul 2024).
• TrackFormers Part 2 attains TrackML score of 91.4% and sub-100 ms inference with block-sparse attention and joint InfoNCE-augmented networks (Caron et al., 30 Sep 2025).

Computer Vision TrackFormer

• MOT17 (private detections): 74.1 MOTA, 68.0 IDF1 at 7.4 FPS.
• MOT20 (private): 68.6 MOTA, 65.7 IDF1.
• MOTS20: sMOTSA 54.9, IDF1 63.6, MOTSA 69.9.
• Competes with, or outperforms, previous tracking-by-detection and attention-based baselines (Meinhardt et al., 2021).

5. Algorithmic Trade-offs and Computational Considerations

• Encoder–Classifier: Maximizes parallelism, yielding best run-time/accuracy trade-off for HL-LHC scale events (3–6 ms per event). No post-processing required. Accuracy drops gracefully with event complexity.
• Encoder–Regressor: Comparable on simple events, but clustering step introduces significant latency (up to 70 ms CPU per event).
• Encoder–Decoder: High per-track accuracy on small data, but scales linearly with number of hits (several seconds per event at realistic occupancy).
• Block-sparse Attention (FlexAttention) (Caron et al., 30 Sep 2025): Reduces attention cost by a factor ≈400×. Enables deep (up to 15-layer) encoder-only architectures under GPU memory constraints.
• Contrastive Losses/Projection: Enforcing latent proximity for true-track hits yields efficiency ≈90–91% at O(10⁴) hits/event.
• Joint Models: Augmenting EncCla with regressed physics latents (“JM” models) gives 2–2.4% boost in both assignment accuracy and TrackML score.

A plausible implication is that one-shot, encoder-only architectures, particularly when equipped with geometry-motivated sparsity and hybrid loss functions, are the dominant paradigm for HL-LHC track assignment.

6. Extensions, Limitations, and Future Work

TrackFormer approaches in both physics and vision are modular and extensible:

• HEP future directions: Incorporation of full per-hit Softmax posteriors, cascaded encoder–regressor/classifier hybrids, further acceleration of attention layers, and scale-up to hierarchical or end-to-end detection plus tracking (Caron et al., 9 Jul 2024, Caron et al., 30 Sep 2025).
• Computer Vision: Proposed extensions include integrating persistent memory banks for long-term association, multi-frame attention, and leveraging pre-trained appearance embeddings for stronger identity cues (Meinhardt et al., 2021).

Limitations include:

• Wall-clock scaling for autoregressive decoding in large events.
• Performance drop at highest pile-up densities and in more realistic detector geometries.
• For video, TrackFormer is currently limited to frame-pair associations and does not exploit full sequence dynamics.

7. Comparative Impact and Significance

TrackFormer techniques represent a departure from graph-based, hand-crafted, or pipeline-based tracking towards fully data-driven, self-attention-based assignment. In HEP, the encoder-classifier TrackFormer is the first O(1) runtime model capable of near state-of-the-art tracking performance at full HL-LHC scale, with demonstrated efficiency and speed several orders of magnitude ahead of Kalman Filter and GNN approaches (Caron et al., 9 Jul 2024, Caron et al., 30 Sep 2025). In multi-object video tracking, TrackFormer establishes an end-to-end tracking-by-attention paradigm, unifying detection, association, and identity tracking without external matching (Meinhardt et al., 2021). This convergence of attention-based techniques is central to future advances in both domains.