Long-Horizon State Tracking

• Long-horizon state tracking is the process of maintaining, reconstructing, and forecasting the state of dynamic systems over extended periods while accounting for accumulated uncertainty.
• It integrates trajectory forecasting, memory-augmented inference, and probabilistic modeling to bridge occlusions and manage non-Markovian observation challenges.
• Practical implementations span BEV-based forecasting, state space models, control-theoretic frameworks, and hierarchical abstractions to mitigate state drift and computational complexity.

Long-horizon state tracking is the problem of maintaining, reconstructing, and forecasting the state of one or more dynamical systems with explicit attention to uncertainties and ambiguities that accumulate over extended time intervals and multiple time steps or frames. This task forms the core of robust multi-object tracking under occlusion, persistent control of dynamical systems to a prescribed output trajectory, and planning or reasoning over long action horizons in partially and fully observable environments. The central technical challenge is that short-term feedback and appearance cues degrade quickly over time, rendering myopic or short-window approaches inadequate. As a result, modern long-horizon tracking methods systematically incorporate trajectory forecasting, sophisticated probabilistic modeling, memory-augmented neural inference, or formal duality principles to maintain state estimates and plan actions despite uncertainty growth, non-Markovian observation regimes, and combinatorial association ambiguities.

1. Trajectory Forecasting and Combinatorial Search Pruning

Appearance-based multi-object trackers reliably bridge short occlusions (<1 s), but reconnect fewer than 10% of tracks across longer occlusions (>3 s), due primarily to expanded state association ambiguity and rapid appearance drift. "Quo Vadis" (Dendorfer et al., 2022) identifies trajectory forecasting as a critical mechanism for reducing long-horizon identity fragmentation. By leveraging a learned, uncertainty-aware generative forecaster in bird’s-eye view (BEV) coordinate space, the tracker generates a compact, diverse set of plausible future trajectories for each lost track. This constrains the region in which the re-association search is performed, substantially shrinking the candidate set to be matched and limiting the proliferation of false matches as the occlusion duration increases.

The core architecture consists of a BEV LSTM encoder, a GAN or multi-generator GAN (MG-GAN) to capture multi-modality in future prediction, and explicit uncertainty modeling via either adversarial loss variance or mixture-density networks. The MG-GAN uses a best-of-many (“winner-takes-all”) loss, ensuring prediction diversity by assigning trajectory modes to specialized generator heads. The output set of N predicted trajectories, together with their estimated covariances, is used to gate association in the tracker pipeline: each inactive (lost) track i, after entering the inactive set at time t, spawns N predicted locations at each frame t' up to t+H. These predictions are projected back into image space for candidate matching using a cost function combining IoU, Euclidean distance, and appearance similarity, thus robustifying long-horizon continuity and raising correct re-association rates for long occluders by up to a factor of two relative to prior SOTA (Dendorfer et al., 2022).

2. State Space Models and Memory for Long-Term Context

Robust long-horizon tracking requires maintaining not just local motion history but an efficiently encoded summary of all relevant context over the sequence to date. The "MambaLCT" tracker (Li et al., 2024) exemplifies this paradigm by deploying a unidirectional state space model (SSM)—the Context Mamba—that recursively scans visual features over arbitrarily long time horizons. This mechanism compresses the cumulative target variation cues from the first frame to the current frame into a fixed-size hidden state vector, which is then injected as a context token into a multi-stream vision transformer for joint spatial and temporal reasoning.

The SSM is formulated as a discrete-time update: $$H_{i}^{t} = \overline{A} H_{i}^{t-1} + \overline{B} x_{i}^{t}$$ with frame-scanning and aggregation steps that ensure information retention over hundreds of frames. The unique carry-forward of a "frame-summary" state enables a single "memory" token to encapsulate all past dynamics, allowing the model to robustly localize targets after extensive appearance drift, occlusion, or distractor interference. Empirically, this yields 0.4–2.9% absolute improvements over leading context-limited trackers (e.g., AQATrack, ARTrack) on multiple video tracking benchmarks and superior resilience to full occlusion, motion blur, and deformation (Li et al., 2024).

3. Probabilistic and Stochastic-Process Frameworks

For continuous-valued state estimation under measurement noise and process uncertainty, long-horizon tracking transcends standard Markovian Bayesian filters by exploiting both long-range temporal correlation and decomposable structure. Data-fitting approaches replace stochastic state transition models with continuous-time trajectory fitting (FoT), yielding an estimated trajectory as a function of time that is robust to irregular sampling and unknown motion models (Li et al., 2017). More expressive frameworks model the state as a sample path from a stochastic process (SP), decomposed as a deterministic trend plus a residual SP. The residual can be modeled as a GP or a Student’s-t process to explicitly encode colored temporal covariance and robustify against outliers (Li et al., 3 Mar 2025).

The update structure is as follows:

• Fit the deterministic trend over a sliding window of d observations by least squares.
• Model the residuals using GP or StP regression for the pseudo-measurements, learning kernel hyperparameters recursively and computing posterior mean/variance for any query time.
• This yields closed-form, uncertainty-aware inference of the target state at arbitrary times—well beyond the standard Markovian or one-step filtering frameworks.

Empirical results show that T-FoT+StP outperforms conventional GP motion trackers and GP-only baselines by up to 40% RMSE reductions, especially for complex maneuvering and heavy-tailed noise (Li et al., 3 Mar 2025).

4. Long-Horizon State Tracking in Control Systems

In control-theoretic settings, "tracking controllability" formalizes the requirement not only to reach a target state at a terminal time but to match a prescribed output trajectory over the entire time horizon (Zamorano et al., 2024). A system

$x'(t) = A x(t) + B u(t), \qquad x(0) = x_0$

with output $E x(t)$, is "E–tracking controllable" if for every compatible $f \in H^1$ (i.e., $f(0)=E x_0$), there exists a control u such that $E x(t) = f(t)$ for all t. This requirement is stricter than standard controllability, and its feasibility is characterized by a non-standard observability inequality for the adjoint system.

Optimal controls minimizing the $L^2$ norm subject to output tracking are synthesized via Hilbert Uniqueness Method (HUM) duality, yielding well-posedness and explicit formulas in the scalar control case. The norm of tracking control scales inversely with the minimal singular value of the "tracking Gramian" as a function of the time horizon T, establishing that longer horizons mitigate the ill-conditioning encountered at short times (Zamorano et al., 2024).

5. Hierarchical Abstractions and Skill-Centric Tracking

Long-horizon sequential decision-making often requires hierarchical abstraction to mitigate the curse of dimensionality. Value Function Spaces (VFS) (Shah et al., 2021) construct state embeddings by stacking the value functions of a set of parameterized lower-level skills ("options"). Formally, for a bank of K skills, the embedding

$$\phi(s) = \begin{bmatrix} V^{\pi_1}(s) \ \vdots \ V^{\pi_K}(s) \end{bmatrix} \in \mathbb{R}^K$$

serves as a compact, task-relevant abstraction of state, capturing affordance structure while discarding irrelevant distractors. Hierarchical controllers operating over VFS can robustly chain low-level policies over hundreds of steps. In MiniGrid multi-room and robotic planning tasks with horizons ≳ 200, VFS achieves up to 54% higher success rate than state-of-the-art learned embeddings and demonstrates strong zero-shot generalization (Shah et al., 2021).

6. Limitations, Bottlenecks, and Diagnostic Benchmarks

Systematic limitations in long-horizon state tracking persist, especially for open-ended symbolic domains and interactive environments with partial observability. CubeBench (Gao et al., 29 Dec 2025) exposes foundational bottlenecks in LLM agents’ ability to maintain and update a persistent mental model over long action sequences. For multi-step predictive tasks (e.g., solving a Rubik’s Cube from an arbitrary configuration with horizon H ≥ 8), all leading LLM and policy architectures exhibit 0% success rates, with failures stemming from error accumulation in iterative state updates and inability to coordinate spatial memory or exploration across hundreds of steps.

This diagnostic reveals that current agents are fundamentally limited by the lack of mechanisms for persistent, high-fidelity state tracking and fail to interleave exploration and planning strategies necessary for robust long-horizon reasoning. Additionally, even dense reward signals are insufficient to bootstrap successful long-term performance in the absence of persistent, markovian-compatible state representations (Gao et al., 29 Dec 2025).

7. Practical Implementations and Remaining Challenges

Long-horizon tracking solutions require domain-adapted architecture choices:

• In monocular tracking, BEV mapping and uncertainty-aware, diverse trajectory forecasting are essential (Dendorfer et al., 2022).
• For spatiotemporal video analysis, memory-efficient context aggregation (e.g., SSMs in MambaLCT) outperforms short-window approaches (Li et al., 2024).
• In control, both theoretical guarantees of tracking controllability and practical numerical methods (e.g., penalized HUM) are required to synthesize minimal-norm trajectory-matching controls (Zamorano et al., 2024).
• In sequential planning, value-function-based state embeddings and hierarchical policy composition are necessary for scalability (Shah et al., 2021).
• For nonparametric and model-free target tracking, hybrid deterministic–stochastic decompositions with explicit long-memory kernels are favored over Markovian GP/SSM baselines, especially for heavy-tailed and temporally correlated noise (Li et al., 3 Mar 2025).

However, challenges remain: (1) accumulation of modeling errors and state drift over long horizons; (2) the computational complexity of maintaining diverse hypotheses or tracking high-dimensional belief states; (3) the need for persistent, compositional representations that support generalization and tool integration. Future research directions include explicit uncertainty propagation in non-Euclidean state-spaces, schema-based memory models for structured data, and diagnostic platforms that stress-test tracking under adversarial or combinatorial partial observability.

---

Table: Representative Frameworks for Long-Horizon State Tracking

Method/Model	Domain	Key Mechanism
Quo Vadis (Dendorfer et al., 2022)	2D/3D MOT	Forecasting in BEV, MG-GAN diversity
MambaLCT (Li et al., 2024)	Video tracking	SSM long-term context memory
DSD-GP/StP (Li et al., 3 Mar 2025)	Continuous tracking	Trend + SP residual, colored noise
HUM Tracking (Zamorano et al., 2024)	Linear systems	Observability, Gramian, duality
VFS (Shah et al., 2021)	RL/HRL	Value-function skill abstraction
CubeBench (Gao et al., 29 Dec 2025)	LLM/AI	Spatial reasoning benchmark

These approaches collectively demonstrate that advances in long-horizon state tracking require integrative frameworks that combine probabilistic forecasting, memory augmentation, trajectory generation diversity, hierarchical abstraction, and formal guarantees to overcome the explosion of uncertainty and association ambiguity inherent in extended temporal reasoning.