On the Importance of Multistability for Horizon Generalization in Reinforcement Learning

Abstract: In reinforcement learning (RL), agents acting in partially observable Markov decision processes (POMDPs) must rely on memory, typically encoded in a recurrent neural network (RNN), to integrate information from past observations. Long-horizon POMDPs, in which the relevant observation and the optimal action are separated by many time steps (called the horizon), are particularly challenging: training suffers from poor generalization, severe sample inefficiency, and prohibitive exploration costs. Ideally, an agent trained on short horizons would retain optimal behavior at arbitrarily longer ones, but no formal framework currently characterizes when this is achievable. To fill this gap, we formalized temporal horizon generalization, the property that a policy remains optimal for all horizons, derived a necessary and sufficient condition for it, and experimentally evaluated the ability of nonlinear and parallelizable RNN variants to achieve it. This paper presents the resulting theoretical framework, the empirical evaluation, and the dynamical interpretation linking RNN behavior to temporal horizon generalization. Our analyses reveal that multistability is necessary for temporal horizon generalization and, in simple tasks, sufficient; more complex tasks further require transient dynamics. In contrast, modern parallelizable architectures, namely state space models and gated linear RNNs, are monostable by construction and consequently fail to generalize across temporal horizons. We conclude that multistability and transient dynamics are two essential and complementary dynamical regimes for horizon generalization, and that no current parallelizable RNN exhibits both. Designing parallelizable architectures that combine these regimes thus emerges as a key direction for scalable long-horizon RL.

• The paper demonstrates that multistability in RNN memory systems is necessary for robust temporal horizon generalization in reinforcement learning tasks.
• It formalizes conditions, including compatibility and separation in read-outs, that enable RNNs to preserve cue information across vast temporal gaps.
• Empirical results in T-maze and LookupTreeMaze tasks reveal that hybrid architectures outperform monostable models in complex, long-horizon environments.

Multistability as a Fundamental Mechanism for Temporal Horizon Generalization in Reinforcement Learning

Introduction

Temporal horizon generalization (THG) in reinforcement learning (RL) is the property whereby an agent trained on tasks with a given temporal horizon—that is, with information about required actions delivered several time steps before the action is evaluated—retains optimal behavior as the temporal separation between informative cue and required action grows arbitrarily. While the inability to generalize across longer horizons is a major limitation in deploying RL agents for real-world, sparse-reward POMDPs, a principled framework for understanding and characterizing horizon generalization has been lacking. The paper "On the Importance of Multistability for Horizon Generalization in Reinforcement Learning" (2605.12206) provides a formal analysis of this problem and elucidates the essential role of multistability in recurrent memory systems for supporting THG, both theoretically and empirically.

Formalization of Temporal Horizon Generalization

The authors define THG as the property that a policy, optimized for horizon $T$ (the temporal separation between cue and required action), is also optimal for every $T' \geq 0$. This formalism exposes the central memory bottleneck in RL with long sequences of uninformative steps: unless the internal state of the agent can retain decision-relevant information across unbounded horizons, optimality degrades rapidly.

A precise dynamical condition is derived. An RNN-based policy $\pi = \bar{\pi} \circ \rho$ will achieve THG if and only if $\rho$ is a read-out that is:

• Compatible: invariant under the hidden-state update map $U$, i.e., $\rho \circ U = \rho$, for reachable states; and
• Separating: distinct across encodings of different cue observations, i.e., for cues $\sigma \neq \sigma'$, $\rho(h_0(\sigma)) \neq \rho(h_0(\sigma'))$.

This condition has strong implications for the design space of RL policies, particularly with respect to the architecture of the memory-encoding backbone.

Figure 1: (A) Schematic of temporal horizon generalization; (B) Phase diagram distinguishing monostable and multistable memory systems; (C) Trajectory illustrations showing the preservation or loss of initial cue information.

Dynamical Regimes: Monostability versus Multistability

A central claim is that monostable memory systems—those with a single globally attracting fixed point for the hidden state update under uninformative input—cannot achieve THG. Any information about the past is inevitably forgotten as the system approaches this unique attractor, rendering invariant read-outs trivial and indistinguishable between distinct cues.

In contrast, multistable systems—those possessing multiple stable attractors with disjoint basins of attraction—allow the initial cue to dictate the basin in which the hidden state evolves. If the system remains within this basin throughout the idle phase, an invariant read-out suffices to maintain separability, thus supporting THG. This mechanic is robust even in the presence of perturbations, provided the trajectory does not cross basin boundaries.

RNN Architectures, Parallelizability, and the Limits of Modern Sequence Models

An incisive architectural analysis demonstrates that:

• Classical nonlinear RNNs (e.g., GRU, LSTM, BRC, nBRC) can exhibit multistability, depending on parameter regimes and training.
• Parallelizable state-space models (SSMs) and input-gated linear RNNs (e.g., minGRU, xLSTM, GILR, HGRN), which underlie recent scalable architectures, are necessarily monostable. Their state dynamics always converge to a unique equilibrium under constant input, precluding true persistent memory.

Additionally, the work discusses bistable memory recurrent units (BMRU): a class of parallelizable RNNs capable of multistability but lacking transient dynamics. This enables one-step switching between attractors but sacrifices graded temporal dynamics.

Empirical Validation in Synthetic Long-Horizon Tasks

Experiments in carefully constructed environments assess the mechanistic claims:

T-maze

In the T-maze POMDP, the agent receives a binary cue at the beginning, followed by an extended idle phase before a critical decision (turn up/down) at the end.

Figure 2: (A) Structure of the T-maze environment highlighting the temporal gap between cue and decision; (B) Structure of the LookupTreeMaze task introducing combinations of persistent and frequently updated cues.

Models trained on short (length 1–3) versus long (length 100) T-mazes were evaluated on their ability to retain the initial cue over an extended horizon. Outcomes reveal:

• Models exhibiting multistability, identified via the VAA (variability among attractors) metric, successfully generalize across large horizons, regardless of training regime.
• Monostable models always eventually perform no better than random as the horizon increases.
• Training on short horizons is both more efficient and more effective at inducing multistable regimes that generalize.

  Figure 3: (A) Distributions of successful, random, and failed models and their stability properties; (B) Average reward as a function of T-maze length for nonlinear RNNs separated by stability type; (C) Analogous results for parallelizable RNNs (minGRU, BMRU).

LookupTreeMaze

This compound task extends T-maze by adding multiple sequential subdecision points, combining persistent and frequently updated memory requirements. Key findings:

• High VAA (indicative of robust multistability) correlates with sustained performance across an increasing number of submazes, especially for nonlinear RNNs.
• Neither pure monostability nor pure multistability (BMRU, lacking transients) suffices in this more intricate temporal regime; both transient dynamics and long-term attractor memory are necessary.
• A hybrid RNN combining BMRU and minGRU units outperforms either alone, retaining high reward as the number of temporal dependencies grows into the several-thousand range.

  Figure 4: (A) LookupTreeMaze generalization performance for nonlinear RNNs as a function of VAA and maze number; (B) Same for parallelizable architectures and their hybrids.

Internal Dynamics and Learning Behavior

The temporal evolution of internal states during training and evaluation sheds further light on the mechanisms at play:

• Bistable models encode cues by rapid convergence to distinct attractors; those trained on longer horizons develop slower transients, sometimes failing to reach equilibrium within the task duration.
• Monostable models display fading memory effects, with trajectories converging to identical final states regardless of initial cue.

  Figure 5: Internal state evolution for RNNs with varying architectural stability properties, highlighting monostable fading (top), bistable attractor switching (bottom left), and parallelizable one-step convergence (bottom right).

  

  Figure 6: (A) Training reward curves for short- and long-horizon tasks; (B) Internal state dynamics in bistable models trained on small versus long T-mazes.

Implications and Theoretical/Practical Impact

The necessity and sufficiency of multistability for THG establish a sharp limit: scalable parallelizable memory architectures currently popular in large-scale sequence modeling, including SSM variants such as Mamba and recent input-gated RNNs, are structurally unable to support robust long-range credit assignment across arbitrary horizons. This limitation stands independent of model size and cannot be circumvented by increasing representational capacity or depth, as multistability is a property of the cell dynamics, not the overall depth.

These results strongly advocate for fundamental changes in the architecture of RL agents required to operate at scale and in environments characterized by sparse, delayed rewards:

• Hybrid Memory Architectures: Integrating both multistable and transient regimes can recover THG in complex POMDPs, marrying trainability/parallelization with robust persistent memory.
• Training Regimes: Curriculum-learning over horizon, or explicit objectives enforcing multistability, could substantially improve RL agent generalization and data efficiency on challenging temporal tasks.
• Model Scaling Trajectory: As the RL field migrates to massive sequence models for flexible control and world modeling, the inability of monostable architectures to handle unbounded-horizon dependencies will become increasingly problematic, motivating both new algorithmic designs and theoretical inquiry.

Conclusion

This work provides a rigorous and comprehensive theoretical and empirical analysis linking temporal horizon generalization in RL to the dynamical property of multistability in recurrent memory architectures (2605.12206). The established necessary and sufficient conditions highlight that the current generation of parallelizable RNNs, despite their computational advantages, lack the structural prerequisites for THG. The demonstrated effectiveness of hybrid models integrating both transient and multistable regimes presents a promising avenue for future research, especially in settings where horizon scalability and robust memory are critical. The results underscore the importance of dynamical systems principles in the architectural and algorithmic development of next-generation RL agents.