Autoregressive Long-term Tracking (ALT)

• ALT is a tracking design pattern that conditions current predictions on previously observed states while retaining long-term contextual memory.
• It employs diverse mechanisms such as GRUs, LSTMs, GNNs, and transformer query spaces to autoregressively propagate trajectories and maintain identity.
• Empirical studies show that ALT improves occlusion handling, re-detection, and overall tracking continuity compared to heuristic short-term models.

Autoregressive Long-term Tracking (ALT) denotes a family of tracking formulations in which the next state, association decision, or identity assignment is generated conditionally on previously tracked states, while long-horizon continuity is preserved through explicit memory, recurrent state, autoregressive trajectory propagation, or repeated subclip fusion. In the literature considered here, ALT appears in online multi-object tracking through generative trajectory models, in single-object tracking through autoregressive query states, in LLM-based identity token generation, and in graph-based tracklet fusion across arbitrary temporal horizons (Fang et al., 2017, Saleh et al., 2020, Xie et al., 2024, Jia et al., 5 Jan 2026, Missaoui et al., 2 Sep 2025). The term is not used uniformly: one 2021 single-object tracker uses ALT to mean “Active Learning-based Tracker,” not autoregressive long-term tracking (Yuan et al., 2021).

1. Definition, scope, and terminology

In the strictest sense used by recent tracking work, ALT combines two properties. The first is autoregression: the model conditions current predictions on previously generated or observed trajectory states. The second is long-term state retention: the model preserves informative history beyond a single frame or a short heuristic template update. This can be realized through recurrent hidden states, explicit external memories, trajectory inpainting, temporal memory tokens, or autoregressive propagation of track sets across subclips.

NOOUGAT makes this interpretation explicit by naming its cross-subclip fusion module the Autoregressive Long-term Tracking (ALT) layer, defined recursively by

$$\mathcal{T}^{1}_{\text{alt}} := \mathcal{T}^{1}_{\text{hicl}}, \qquad \mathcal{T}^{i}_{\text{alt}} := \text{ALT}\big(\mathcal{T}^{i-1}_{\text{alt}}, \mathcal{T}^{i}_{\text{hicl}}\big),$$

so that past trajectories become the structured memory consumed at the next step (Missaoui et al., 2 Sep 2025). Earlier and parallel systems do not always name the mechanism “ALT,” but they instantiate the same pattern. RAN couples a GRU-based internal memory with an explicit sliding external memory (Fang et al., 2017); ArTIST uses an autoregressive motion likelihood and free-running inpainting through long gaps (Saleh et al., 2020); AR-MOT turns association into next-token ID prediction with temporal memory fusion (Jia et al., 5 Jan 2026).

This suggests that ALT is best understood as a design pattern rather than a single architecture. A recurring misconception is that ALT must denote either a particular recurrent model or an explicit long-term re-detection framework. The cited literature shows a wider space: ALT may operate in feature space, velocity-token space, query space, ID-token space, or tracklet graphs. Another common source of confusion is acronym collision: “ALT” in “Active Learning for Deep Visual Tracking” refers to an annotation-efficient DiMP-style tracker and is unrelated to autoregressive long-term tracking (Yuan et al., 2021).

2. Canonical formulations

The technical core of ALT is the factorization of tracking as conditional sequence modeling. The formulations differ primarily in what is autoregressed: appearance and motion features, bounding-box velocities, latent target queries, identity tokens, or track sets.

System	Autoregressive object	Long-term mechanism
RAN (Fang et al., 2017)	Appearance and motion features	GRU internal memory + $K$-step external memory
ArTIST (Saleh et al., 2020)	Quantized velocities / tracklet likelihood	Inpainting and rollout up to 60 unobserved frames
STTrack (Xie et al., 2024)	Target queries	Sliding query window + STM fusion
AR-MOT (Jia et al., 5 Jan 2026)	ID tokens	TMF, RAA, and TCM
NOOUGAT (Missaoui et al., 2 Sep 2025)	Past global tracks fused with incoming tracklets	Repeated ALT layer across arbitrary horizons
RosettaSim (Xiao et al., 30 Jun 2026)	Motion tokens and spawn tokens	Structured rollout with dynamic agent cardinality

RAN is the clearest early feature-space ALT formulation. For a feature vector sequence $\mathbf{x}^{1:t-1}$, it models

$$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$

with a time-varying autoregressive model whose coefficients and variances are generated from the GRU hidden state,

$$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$

The result is a probabilistic appearance-motion model in which the weighting of the last $K$ observations depends on longer history (Fang et al., 2017).

ArTIST instead operates directly on tracklet motion. It factorizes trajectory probability as

$$p(\mathcal{T}_j) = p(b_1)\prod_{t=2}^{T} p(b_t \mid b_{t-1},\dots,b_1),$$

but in implementation models normalized velocity increments $\Delta_t$ with an LSTM and a discrete distribution over quantized motion classes. This gives a likelihood for data association and a generative mechanism for sampling multiple plausible futures when detections are absent (Saleh et al., 2020).

Single-object tracking admits a related latent-state formulation. STTrack maintains a query state $Q_t$ updated from previous queries and current-frame information,

$$Q_t = f_\theta(Q_{t-m+1},\dots,Q_{t-1},Q_{cur}, f_{zx}^N),$$

where temporal attention uses the current query as query and the full query set as key/value. Here autoregression occurs in learned query space rather than on coordinates, and long-sequence robustness is mediated by a sliding temporal window and a spatio-temporal fusion module (STM) (Xie et al., 2024).

AR-MOT reformulates multi-object tracking as next-token generation in an LLM. For object $K$0 in frame $K$1, it predicts the next ID token by maximizing

$K$2

where the context contains previous frames, current image tokens, earlier object-ID pairs in the current frame, and the current object token. Association is therefore internal to the autoregressive decoder rather than externalized to a Hungarian cost matrix (Jia et al., 5 Jan 2026).

NOOUGAT moves the same principle to graph tracking. Local subclip tracklets are first built by a hierarchical GNN, then fused across time by the ALT layer. RosettaSim generalizes the pattern beyond visual tracking to dynamic multi-agent scene evolution, factorizing each transition into parallel motion updates for existing agents and autoregressive generation of new agents (Xiao et al., 30 Jun 2026). A plausible implication is that ALT is not tied to object identity alone; it is a more general mechanism for long-horizon state continuity under iterative prediction.

3. Memory, occlusion handling, and identity preservation

The defining practical challenge of ALT is not one-step prediction but continuity through occlusion, detector failure, appearance corruption, and long temporal gaps. Different systems solve this with different memory abstractions.

RAN uses a dual-memory design. Its external memory explicitly stores the last $K$3 appearance or motion inputs, typically $K$4, while its internal memory is a GRU hidden state updated across the entire trajectory. During occlusion, the learned autoregressive weights can shift away from the newest corrupted input and toward older, cleaner observations. The paper’s qualitative analysis shows that when a pedestrian undergoes occlusion, the weights usually decay with time but shift toward older frames when occlusion starts; after reappearance, the model initially adopts a more averaging-like strategy before returning to recent-frame emphasis (Fang et al., 2017).

ArTIST addresses the same problem through trajectory inpainting. When detections are missing, it switches from teacher-forced scoring to free-running generation: the sampled motion at time $K$5 is fed back as input for time $K$6, and this process can continue for up to 60 unobserved frames. It samples $K$7 inpainted candidates and then applies a tracklet rejection scheme based on IoU consistency and short lookahead, allowing the tracker to bridge long gaps rather than terminating immediately (Saleh et al., 2020).

AR-MOT combines explicit object tokenization with two long-term modules. The Region-Aware Alignment (RAA) module fuses local object tokens with averaged regional image tokens, while Temporal Memory Fusion (TMF) compresses object history into a per-track memory embedding. A Temporal Context Manager (TCM) further retains temporarily missing objects for a controlled number of frames. This means that long-term association is no longer delegated to heuristic retention alone: it is supported by an explicit memory token that persists and is reintroduced into the LLM context at later frames (Jia et al., 5 Jan 2026).

NOOUGAT’s memory is the set of already-fused global trajectories $K$8. This set is not a fixed sliding window: because ALT is applied autoregressively across non-overlapping subclips, it can connect arbitrarily distant subclips without rebuilding a full-video graph. In online mode, it uses a bipartite graph from past trajectories to incoming detections; in offline mode, it uses full graph connectivity within a large horizon such as $K$9. The key long-term mechanism is therefore autoregressive trajectory-set propagation, not recurrent hidden state (Missaoui et al., 2 Sep 2025).

STTrack occupies a narrower point in the design space. It uses a sliding query window, with best results at $\mathbf{x}^{1:t-1}$0, and does not maintain an external memory bank or explicit re-detection state. It is therefore an ALT-like model for long sequences with appearance change and distractors, but not a full long-term tracker for frequent disappearance and re-entry (Xie et al., 2024).

4. Learning objectives and inference procedures

ALT systems differ sharply in their supervision signals. Some are explicitly generative, some are discriminative sequence models, and some combine detection and identity losses.

RAN is trained by maximum likelihood over ground-truth trajectories. For each trajectory, detections with IoU $\mathbf{x}^{1:t-1}$1 to the ground truth are sampled, appearance features and motion features are extracted, and the model minimizes the negative log-likelihood

$\mathbf{x}^{1:t-1}$2

At inference, RAN computes a Gaussian likelihood for appearance and motion, multiplies them into an association score, and then applies bipartite matching with the Hungarian algorithm, as in prior work by Xiang et al., 2015 (Fang et al., 2017).

ArTIST is likewise generative but in discrete velocity space. Each component of the normalized velocity vector is clustered with k-means into $\mathbf{x}^{1:t-1}$3 motion classes, and training uses cross-entropy, i.e. negative log-likelihood over quantized velocities. To stabilize long-horizon generation, it uses gradient clipping per Pascanu et al. and stochastic teacher forcing with probability $\mathbf{x}^{1:t-1}$4 after 70% of the sequence length. At inference it constructs a cost matrix from negative log-likelihoods and runs Hungarian assignment twice, prioritizing tracklets whose last step used a real detection over those currently maintained by inpainting (Saleh et al., 2020).

AR-MOT couples detection losses and token prediction. Its object tokenizer is based on D-DETR with a ResNet-50 backbone, global image tokens come from SigLIP ViT, and the decoder-only LLM is Qwen2-0.5B within LLaVA-OneVision. Training minimizes a sum of D-DETR classification, L1, and GIoU losses together with cross-entropy over the next ID token,

$\mathbf{x}^{1:t-1}$5

without an extra association-specific regularizer (Jia et al., 5 Jan 2026).

NOOUGAT uses a unified GNN training regime. The hierarchical GNN and ALT share weights, training proceeds progressively, and both hierarchy and ALT edge classification use focal loss with $\mathbf{x}^{1:t-1}$6. Training samples random incoming subclips, random past intervals, a temporal skip of up to 8 frames, and random dropping of past tracks or detections so that ALT learns association across gaps rather than only in continuous observations (Missaoui et al., 2 Sep 2025).

STTrack remains closer to standard discriminative SOT training. It uses video-level sampling, focal loss on the classification heatmap, GIoU loss, and L1 regression loss with $\mathbf{x}^{1:t-1}$7 and $\mathbf{x}^{1:t-1}$8. The temporal behavior is learned implicitly because the same decoder and query mechanism are applied over sampled temporal windows (Xie et al., 2024).

RosettaSim adopts causal cross-entropy on a structured token stream. Agent, motion, map, layout, and spawn tokens are concatenated into a single decoder sequence, but loss is applied only to motion-token and spawn-token positions. Most of the Qwen2.5-0.5B-Instruct backbone remains frozen, and only small alignment components or late layers are fine-tuned (Xiao et al., 30 Jun 2026).

5. Empirical evidence

The empirical record across these systems supports the claim that autoregressive long-horizon mechanisms improve either identity preservation, continuity through missing observations, or flexible temporal deployment.

RAN’s window study on 2DMOT2015 shows that increasing the external memory span improves performance up to about ten frames: $\mathbf{x}^{1:t-1}$9 gives MOTA 69.7 and IDS 120, $$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$0 gives MOTA 70.6 and IDS 83, and $$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$1 or $$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$2 gives MOTA 70.7 and IDS 77. In component analysis, $$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$3-RAN reaches MOTA 70.7 and IDS 77, outperforming $$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$4-GRU at MOTA 57.7 and IDS 85, $$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$5-AVE at MOTA 68.6 and IDS 142, and $$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$6-TIV at MOTA 69.3 and IDS 109. On benchmark test sets, RAN reports MOTA 56.5 and IDS 428 on 2DMOT2015 and MOTA 63.0 and IDS 482 on MOT16 (Fang et al., 2017).

ArTIST’s evidence centers on inpainting. On MOT17 train with Faster R-CNN detections, removing inpainting yields MOTA 48.0, IDF1 54.1, and FN 56,838, while adding inpainting yields MOTA 50.3, IDF1 56.7, and FN 52,010. Replacing multinomial sampling plus tracklet rejection with top-1 prediction lowers performance from MOTA 50.3 and IDF1 56.7 to MOTA 50.0 and IDF1 54.1. In the Tracktor-based variant, ArTIST* improves over Tracktor++ on MOT17 train from MOTA 61.9 to 62.4, from IDF1 64.7 to 68.8, and from IDs 326 to 255 (Saleh et al., 2020).

STTrack’s results show that autoregressive query modeling can improve long-sequence single-object tracking even without explicit re-detection. On LaSOT, STTrack-384 reports AUC 72.7%, $$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$7 82.9%, and P 80.2%; on LaSOT$$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$8, AUC 52.7% and $$\Pr(\mathbf{x}^{t} \mid \mathbf{x}^{1:t-1}) = \mathcal{N}(\mathbf{x}^{t} \mid \boldsymbol\mu^t, \boldsymbol\Sigma^t), \qquad \boldsymbol\mu^t = \sum_{k=1}^K \alpha_k^t \mathbf{x}^{t-k},$$9 64.2%; and on GOT-10k, AO 76.0%, SR$$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$0 85.2%, and SR$$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$1 74.9%. Its ablations show AUC 70.5 with $$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$2, 70.8 with $$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$3, 71.4 with $$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$4, 70.6 with $$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$5, and 70.2 with $$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$6, indicating that a small autoregressive window is most effective under the reported training constraints (Xie et al., 2024).

AR-MOT’s most direct long-term evidence comes from retention and memory ablations on DanceTrack validation. Increasing $$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$7 from 1 to 10 raises HOTA from 33.53 to 42.18 and AssA from 17.36 to 25.51; at $$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$8, HOTA reaches 42.39 and AssA 25.88. In the component study, “MLLM only” gives HOTA 33.53 and IDF1 28.03, “+ TMF only” gives HOTA 33.74 and IDF1 29.17, and “+ RAA + TMF” gives HOTA 37.09 and IDF1 31.67. On DanceTrack test, AR-MOT reports HOTA 48.1, DetA 77.2, AssA 30.2, and IDF1 43.9 (Jia et al., 5 Jan 2026).

NOOUGAT provides the most explicit evidence for an ALT layer as a replacement for heuristic association and stitching. It reports online AssA improvements of +2.3 on DanceTrack, +9.2 on SportsMOT, and +5.0 on MOT20, with even greater gains in offline mode. On DanceTrack validation, a Hungarian baseline with the same cues yields HOTA 62.1 and IDF1 65.8, while ALT yields HOTA 64.6 and IDF1 69.0. Replacing SUSHI-style heuristic stitching with ALT raises DanceTrack validation results from HOTA 67.9 and IDF1 72.0 to HOTA 69.8 and IDF1 75.2 (Missaoui et al., 2 Sep 2025).

RosettaSim shows that the same principles extend to dynamic multi-agent simulation. On WOSAC long-term evaluation, it reaches RMM-F1 0.7624, Flow Overall 0.7846, Behavior Overall 0.7414, and Kinematic 0.9176, outperforming InfGen, CAT-K, and SMART on the overall realism score. Its Retrieval-based Traffic Evaluation (RTE) also shows stronger correlation with standard metrics, $$(\hat{\boldsymbol\alpha}^t, \hat{\boldsymbol\sigma}^t) = f(\mathbf{h}^{t-1}).$$9, than prior log-based evaluation, $K$0 (Xiao et al., 30 Jun 2026).

6. Limitations, misconceptions, and broader extensions

Several limitations recur across ALT systems. RAN inherits detector errors, assumes diagonal Gaussian feature distributions, and uses a fixed external window even though the internal GRU is unbounded (Fang et al., 2017). ArTIST’s discrete velocity modeling assumes independence among $K$1, $K$2, $K$3, and $K$4, and its purely geometric formulation can fail when motion alone is insufficiently discriminative (Saleh et al., 2020). AR-MOT remains detector-dependent, trains on relatively short windows of 2–5 frames, and compresses each track’s history into a single memory vector, which may create an information bottleneck (Jia et al., 5 Jan 2026). NOOUGAT’s online mode is bottlenecked by graph construction, and its learned GNN does not explicitly impose global one-to-one flow constraints (Missaoui et al., 2 Sep 2025). STTrack has no explicit lost-state handling or global re-detection and therefore should not be conflated with dedicated long-term re-detection trackers (Xie et al., 2024). RosettaSim handles disappearance with heuristics rather than learned delete tokens and still relies on snapping and overlap control to stabilize long rollouts (Xiao et al., 30 Jun 2026).

A common misconception is that ALT is necessarily appearance-driven. ArTIST is explicitly purely geometry-based, and its central claim is that a learned probabilistic motion prior can score tracklets and bridge long gaps without appearance or re-ID (Saleh et al., 2020). A second misconception is that ALT always requires a recurrent neural network. NOOUGAT’s ALT layer is a GNN over tracklets, AR-MOT’s association is next-token prediction in an LLM, and STTrack performs autoregression in transformer query space (Missaoui et al., 2 Sep 2025, Jia et al., 5 Jan 2026, Xie et al., 2024). A third misconception is terminological: the 2021 “ALT” tracker is an Active Learning-based Tracker using farthest point sampling, nearest-neighbor discrimination, and Tversky loss for annotation-efficient DiMP-style training; it is not an autoregressive long-term tracking method (Yuan et al., 2021).

Broader sequence-modeling work suggests that ALT also has a training problem, not only an architectural one. “Improving Long-term Autoregressive Spatiotemporal Predictions: A Proof of Concept with Fluid Dynamics” proposes the Stochastic PushForward framework, which retains one-step-ahead training while exposing the model to its own multi-step rollout distribution through a supplementary dataset. In the shallow-water experiments, SPF2 and SPF3 use 689 MB GPU memory, compared with 959 MB for ATF2 and 1159 MB for ATF3, while improving long-term stability (Zhou et al., 25 Aug 2025). This suggests that future ALT systems may need both long-horizon architectures and distribution-shift-aware training schemes to control exposure bias under rollout.

Taken together, the literature defines ALT less by any single backbone than by a stable set of principles: conditional next-state generation from tracked history, explicit retention of longer temporal context, and mechanisms that preserve identity or dynamical coherence when short-term evidence is missing, corrupted, or ambiguous.