TrackFormers: Transformer-Based Tracking

• TrackFormers are transformer-based models for charged-particle and multi-object tracking that replace combinatorial methods with one-shot, encoder-only approaches.
• They leverage attention mechanisms and sparsification strategies like Flash and FlexAttention to efficiently group detector hits and reduce computational costs.
• Extensions of TrackFormers span high-energy physics and computer vision, demonstrating practical trade-offs in latency, accuracy, and hardware deployment.

Searching arXiv for recent and foundational papers on TrackFormers and closely related transformer-based tracking work. Found and cross-checked the core TrackFormers papers and representative related work, including TrackFormer (Meinhardt et al., 2021), Global Tracking Transformers (Zhou et al., 2022), Unified Transformer Tracker (Ma et al., 2022), 3DMOTFormer (Ding et al., 2023), ODTFormer (Ding et al., 2024), "TrackFormers: In Search of Transformer-Based Particle Tracking for the High-Luminosity LHC Era" (Caron et al., 2024), "Transformers for Charged Particle Track Reconstruction in High Energy Physics" (Stroud et al., 2024), and "TrackFormers Part 2" (Caron et al., 30 Sep 2025). TrackFormers denotes a line of Transformer-based tracking and association models whose most explicit named instantiation is the high-energy-physics program introduced in “TrackFormers: In Search of Transformer-Based Particle Tracking for the High-Luminosity LHC Era” and extended in “TrackFormers Part 2” (Caron et al., 2024, Caron et al., 30 Sep 2025). In that usage, TrackFormers are primarily encoder-centric, one-shot models for charged-particle track reconstruction: they process detector hits and infer hit-to-track structure in a single forward pass, rather than through combinatorial seeding and iterative extension. The surrounding literature also suggests a broader methodological family in which attention replaces hand-engineered association across frames, detections, hits, voxels, or trajectory hypotheses, as in TrackFormer for multi-object tracking and ODTFormer for stereo obstacle detection and voxel tracking (Meinhardt et al., 2021, Ding et al., 2024).

1. Historical and conceptual framing

The modern TrackFormers lineage emerged from two converging developments. In computer vision, transformer-based tracking reformulated association as attention over persistent queries or over detections from multiple frames. TrackFormer cast multi-object tracking as a frame-to-frame set prediction problem, using static object queries for new objects and identity-preserving track queries for continuation; association was performed through attention rather than explicit graph optimization, motion models, or appearance models (Meinhardt et al., 2021). Global Tracking Transformers then moved association to a short temporal window, using object features from all frames and trajectory queries to group detections into trajectories without intermediate pairwise grouping or combinatorial association (Zhou et al., 2022). Unified Transformer Tracker extended the idea further by using a single paradigm for both single-object tracking and multi-object tracking through a shared track transformer (Ma et al., 2022).

In high-energy physics, the same general shift appeared as a response to the HL-LHC regime, where hit assignment becomes the dominant combinatorial bottleneck. The TrackFormers project explicitly drew inspiration from LLMs and studied both autoregressive “next hit” prediction and one-shot event-level assignment (Caron et al., 2024). A later, distinct HEP line adapted Transformer hit filtering and a MaskFormer reconstruction model to jointly optimise hit assignments and charged-particle properties on TrackML, thereby importing object-detection and instance-segmentation ideas into collider reconstruction (Stroud et al., 2024).

A recurrent theme across these works is that attention is used not merely as a feature extractor but as the mechanism that performs association. In vision MOT, the associated entities are detections, object queries, or trajectory hypotheses; in particle tracking, they are detector hits. This suggests a unifying description of TrackFormers as transformer systems in which the correspondence structure is learned directly in the same latent space used for prediction.

2. Core problem formulations

In the original HEP TrackFormers study, four model designs were evaluated on five datasets spanning simple REDVID synthetic tracks to reduced TrackML events: EncDec, EncCla, EncReg, and a Sparse U-Net baseline (Caron et al., 2024). EncDec is a Transformer encoder-decoder that predicts the next hit autoregressively. EncCla is an encoder-classifier that predicts track class labels for all hits in one shot. EncReg is an encoder-regressor that predicts track parameters per hit and then applies HDBSCAN for clustering. The Sparse U-Net baseline treats the event as a sparse image or voxel grid and performs one-shot classification.

The one-shot formulations became the dominant TrackFormers design because they avoid the fragility of sequential decoding. The paper reports that autoregressive EncDec is conceptually close to next-token language modeling but is slow and prone to error accumulation, whereas the one-shot encoder-classifier is the most practical and effective solution among the tested approaches for realistic TrackML settings (Caron et al., 2024). In the largest TrackML reduction, EncCla predicts one of 5,400 classes derived from discretized $(\phi,\theta,p,q)$ bins; EncReg instead predicts track parameters and relies on post-inference clustering.

The later “Part 2” work preserved the encoder-only, one-shot philosophy but split it into two assignment paradigms. One is a contrastive embedding model that maps each hit into a 32-dimensional embedding and groups hits by cosine similarity. The other is a joint regression/classification model in which a regressor predicts $(\theta,\sin\phi,\cos\phi,q)$ plus four learned latent variables, and a classifier consumes the concatenation of raw hit coordinates and those regressed quantities (Caron et al., 30 Sep 2025). This means that in TrackFormers usage, “one-shot” does not imply a single universal decoding style; it refers to event-level inference without iterative track building.

A separate HEP formulation, “Transformers for Charged Particle Track Reconstruction in High Energy Physics,” treats tracks as MaskFormer-style instances. It first filters hits with a Transformer encoder, then reconstructs tracks with object queries, cross-attention, mask prediction over hits, and regression of $(p_x,p_y,p_z,v_z)$ (Stroud et al., 2024). Although this work is not the same project as TrackFormers, it is closely aligned in treating tracking as set prediction over unordered hits.

3. Principal architectures and learning objectives

The initial TrackFormers study used standard Transformer components—multi-head self-attention, normalization, encoder layers, and, for EncDec, a decoder with positional encoding for ordered hit sequences. A key design decision was to remove positional encoding for the encoder in hit-set models because detector hits were treated as unordered sets rather than sequences (Caron et al., 2024). The standard attention form,

$$Attention(Q,K,V)=softmax\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$

remained the computational core, but the semantics of tokens were event hits rather than words or image patches.

Model	Prediction target	Salient property
EncDec	Next hit coordinates $(x,y,z)$	Autoregressive, seed-based
EncCla	Per-hit track class	One-shot encoder-classifier
EncReg	Per-hit track parameters	HDBSCAN post-processing
Sparse U-Net	Per-hit class/bin prediction	Sparse convolution baseline

The original comparison showed that EncCla used cross-entropy loss and was the strongest model overall on realistic TrackML data, while EncReg remained competitive but incurred a clustering penalty (Caron et al., 2024). Part 2 then introduced a multi-positive InfoNCE contrastive loss for the embedding model, where all hits from the same track are positives and all other hits are negatives, and a weighted joint loss for the regression/classification model,

$$\mathcal{L} = \alpha\,\mathcal{L}_{\text{reg}} + \beta\,\mathcal{L}_{\text{cla}}, \qquad \alpha=1,\;\beta=0.3,$$

with $\mathcal{L}_{\text{reg}}$ a per-hit mean squared error on $(\theta,\sin\phi,\cos\phi,q)$ and $\mathcal{L}_{\text{cla}}$ a cross-entropy over quantile-binned $(\phi,\theta,p,q)$ classes (Caron et al., 30 Sep 2025).

The MaskFormer-style HEP model used a different but related architecture. Its hit filtering network employed a Transformer encoder with sliding-window self-attention over hits sorted by $(\theta,\sin\phi,\cos\phi,q)$0, reducing complexity from quadratic to approximately $(\theta,\sin\phi,\cos\phi,q)$1. The downstream reconstruction model used 8 decoder layers, learned object queries, hit masks, a binary track-vs-null classifier, and a regression head. Its total loss was a weighted combination of classification, mask, and regression terms, schematically

$(\theta,\sin\phi,\cos\phi,q)$2

with optimal bipartite matching for permutation-invariant supervision (Stroud et al., 2024).

The main architectural distinction, therefore, is between encoder-only direct assignment models and encoder-decoder or MaskFormer-style instance models. The literature does not present one as universally dominant; instead, the choice reflects the target trade-off between direct hit classification, latent embedding geometry, explicit instance masks, and deployment cost.

4. Scaling, data realism, and systems implementation

Scaling to HL-LHC multiplicities is the central systems problem of TrackFormers. The original TrackFormers paper already emphasized that attention has quadratic $(\theta,\sin\phi,\cos\phi,q)$3 cost in sequence length and explored Flash Attention for EncReg-FA on the largest TrackML reduction (Caron et al., 2024). “Part 2” made scalability the main architectural problem, replacing earlier attention machinery with FlexAttention and geometry-aware sparse masking so that events with more than $(\theta,\sin\phi,\cos\phi,q)$4 hits could be processed (Caron et al., 30 Sep 2025).

The Part 2 pipeline introduced ACTS-based data generation with Monte Carlo simulation, detector transport, digitization, and TrackML-style formatting, and constructed signal $(\theta,\sin\phi,\cos\phi,q)$5 and background inclusive $(\theta,\sin\phi,\cos\phi,q)$6 datasets at pileup levels $(\theta,\sin\phi,\cos\phi,q)$7, with 40k events per pileup setting (Caron et al., 30 Sep 2025). To reduce attention cost, hits were projected onto simplified detector surfaces, clustered by lightweight windowing or DBSCAN, and converted into FlexAttention BlockMask patterns. The paper states that this can cut attention cost by up to roughly $(\theta,\sin\phi,\cos\phi,q)$8.

Deployment studies reinforced that TrackFormers are as much a systems project as a modeling project. “TrackCore-F” used two TrackFormers-derived models, EncCla and EncReg, as the basis for FPGA inference on the AMD Zynq UltraScale+ MPSoC ZCU102 (Blankestijn et al., 30 Sep 2025). Its flow proceeded from pretrained PyTorch to ONNX, then to Vitis HLS, Vivado, and PYNQ, with either monolithic synthesis or partitioned synthesis. The prototype focused on a single encoder layer containing Multi-Head Attention, Add + Normalization, a feed-forward layer, and final Add + Normalization. The reported resource bottleneck was Block RAM rather than DSPs, and quantization was found to be delicate: the unquantized baseline accuracy was 0.97, while INT16/INT16 and INT16/INT8 both gave 0.90, but INT8 activations reduced accuracy to 0.71 or 0.70 depending on weight precision (Blankestijn et al., 30 Sep 2025).

These results indicate that TrackFormers-scale models can be partitioned into deployable kernels, but only with explicit memory-aware design. A plausible implication is that hardware feasibility depends less on the abstract presence of attention than on how aggressively the attention pattern, memory layout, and model partition are constrained.

5. Empirical performance and practical trade-offs

The first TrackFormers study reported both FitAccuracy and computational performance across five datasets. On REDVID 10–50 linear tracks, EncReg reached 97% while EncCla and EncDec each reached 93%; on REDVID 10–50 helical tracks, EncCla reached 93%; on TrackML 10–50 tracks, EncCla reached 94% and EncReg 93%; and on TrackML 200–500 tracks, EncCla reached 78%, EncReg 70%, and EncReg-FA 67% (Caron et al., 2024). EncCla was also the fastest model overall, with CPU time about 0.1 ms and GPU time around 4.0 ms on small and medium datasets and 7.0 ms on TrackML 200–500. The paper estimated total inference time for roughly 10,000 hits per event at about 3–6 ms.

“Transformers for Charged Particle Track Reconstruction in High Energy Physics” reported stronger TrackML-style reconstruction performance in a different architecture. Its best model, MF-750 MeV, achieved approximately 97.1% double-majority efficiency, 0.6% fake rate, and 94.5% perfect-match efficiency, with combined filtering and tracking inference time of 100 ms on an A100 GPU (Stroud et al., 2024). It also reported linear scaling with hit multiplicity, and the hit filter alone reduced raw events from about 57k hits to 12k, 8k, or 6k depending on the $(\theta,\sin\phi,\cos\phi,q)$9 model. This demonstrates that Transformer-based set reconstruction can compete with graph-based HEP methods in both efficiency and latency.

Part 2 emphasized realism and density scaling rather than only reduced TrackML subsets. It reported end-to-end runtime on the order of tens of milliseconds per event—6 ms for clustering in the best case, 2 ms/event for block-mask creation, 20 ms/event for the transformer encoder, and 47 ms/event for track–hit assignment—and about 90% track double-majority efficiency in the barrel and 91% in the endcaps after $(p_x,p_y,p_z,v_z)$0-vertex refinement (Caron et al., 30 Sep 2025). Its joint regression/classification models improved over plain EncCla variants: for example, EncCla 15 reached accuracy 78.5% and TrackML 89.8%, whereas JM 9:15 reached accuracy 80.5% and TrackML 91.4%.

The FPGA study presented a different trade-off surface. EncCla had about 1.5 million parameters with estimated memory 5.69 MB for parameters and 0.07 MB for activations, while EncReg had about 76,484 parameters with estimated memory 0.29 MB for parameters and 0.07 MB for activations (Blankestijn et al., 30 Sep 2025). The deployed single-encoder-layer kernel used 76.1% of BRAM, 2.66% of DSP, 13.5% of FF, and 24.01% of LUT on the ZCU102. These numbers did not establish final end-to-end latency, but they did show that at least partial TrackFormers-style inference can be synthesized on FPGA hardware.

A common misconception is that transformer tracking is inevitably impractical at high multiplicity because of dense quadratic attention. The TrackFormers literature does not support that generalization. Instead, it presents a sequence of increasingly explicit sparsification strategies—Flash Attention, FlexAttention BlockMask, sliding-window self-attention, geometry-aware clustering, and hardware partitioning—used to recover practical latency without abandoning set-based inference.

6. Broader TrackFormers-style ecosystem

Outside HEP, several papers instantiate TrackFormers-style ideas in other domains. ODTFormer performs obstacle detection and tracking from rectified stereo image pairs by constructing a 3D voxel-space cost volume with deformable cross-attention and then matching occupied voxels across consecutive frames under a bounded search window. The paper explicitly states that ODTFormer “fits naturally into the broader family of ‘TrackFormers’,” because detection and tracking share the same learned voxel representation and correspondence is solved in that voxel space (Ding et al., 2024). On DrivingStereo it reported state-of-the-art obstacle detection, and on KITTI tracking it achieved EPE $(p_x,p_y,p_z,v_z)$1 m and foreground EPE $(p_x,p_y,p_z,v_z)$2 m with only 64.47G MACs, compared with substantially larger RAFT-3D variants (Ding et al., 2024).

TrajectoryFormer brought a related idea to LiDAR 3D MOT by generating hybrid trajectory hypotheses from both current-frame detections and temporally predicted boxes. It then used long-term motion features, short-term appearance features, and a Global-Local Interaction Module to score hypotheses and recover detector-missed objects (Chen et al., 2023). 3DMOTFormer likewise formulated online 3D MOT as edge classification on a track-detection bipartite graph, using an Edge-Augmented Graph Transformer and an autoregressive online training strategy designed to reduce training–inference mismatch; with CenterPoint detections it achieved 71.2% AMOTA on nuScenes validation and 68.2% on test (Ding et al., 2023).

In single-object tracking, MixFormerV2 removed convolutional localization heads and complex score prediction modules in favor of four learnable prediction tokens, mixed attention, and MLP heads. The paper described this as shifting transformer tracking toward a more unified and hardware-friendly architecture, and more broadly suggested a TrackFormers-style principle in which specialized prediction tokens and simple heads replace legacy dense heads (Cui et al., 2023). The survey “Transformers in Single Object Tracking” organized the field into CNN-Transformer, two-stream/two-stage fully-Transformer, and one-stream/one-stage fully-Transformer trackers, and concluded that one-stream one-stage fully-Transformer methods were the most accurate and generalizable, though less efficient (Thangavel et al., 2023).

This broader ecosystem clarifies that TrackFormers is not a single fixed architecture. Rather, the literature supports a family resemblance: transformer-based models that internalize association in attention, operate on structured sets rather than handcrafted pairwise pipelines, and increasingly couple prediction with deployment-aware sparsity. In HEP, that family has centered on one-shot encoder-only hit association and its hardware realization; in vision and robotics, it has appeared as tracking-by-attention, graph-transformer association, voxel correspondence, and trajectory-hypothesis reasoning.