Recurrent State-Space Models (RSSMs)

• RSSMs are a class of sequence models that integrate state-space dynamics with recurrent neural network principles for effective time-series modeling.
• They employ strategies like variational inference and hybrid deterministic-stochastic transitions to accurately estimate latent states and uncertainties.
• RSSMs are applied in reinforcement learning, forecasting, and control, demonstrating robust performance in environments with partial observability and dynamic changes.

Recurrent State-Space Models (RSSMs) are a principled and highly expressive class of sequence models that combine the temporal structure of state-space models (SSMs) with the flexibility and scalability of modern recurrent neural network parameterizations. RSSMs underpin state-of-the-art approaches in model-based reinforcement learning, time series forecasting, system identification, and control, especially under high-dimensional, partially observed, or dynamically shifting observational regimes. The core paradigm is to encode system trajectories as a sequence of latent states governed by a recurrent transition function, with emissions mapping latent states to high-dimensional observations, all within a probabilistic or hybrid deterministic-probabilistic framework.

1. Mathematical Foundations and Model Structure

The basic form of an RSSM in discrete time is the following latent-variable model for time-indexed observations $o_{1:T}$ and (optionally) controls or actions $a_{1:T}$:

$$\begin{align*} s_0 &\sim p(s_0) \ s_t &\sim p(s_t \mid s_{t-1}, a_{t-1}) \ o_t &\sim p(o_t \mid s_t) \end{align*}$$

Here, $s_t \in \mathbb{R}^d$ is the latent state at time $t$, governed by a transition function (often parameterized by a neural network such as a GRU or LSTM), and $o_t \in \mathbb{R}^D$ is the observation, with a conditional emission model mapping latent states to the observation space. The structure naturally models partially observable, non-Markovian, or nonlinear environments and can be extended to include exogenous covariates, goals, or hidden parameters (Srivastava et al., 2021, Shaj et al., 2022, Kadi et al., 23 Aug 2025, Inzirillo, 21 Jul 2024).

The transition and emission models can be either deterministic or probabilistic, with typical choices including:

• Gaussian transitions: $$p(s_t | s_{t-1}, a_{t-1}) = \mathcal{N}(f_\theta(s_{t-1}, a_{t-1}), Q)$$.
• Deterministic transitions: $s_t = f_\theta(s_{t-1}, a_{t-1})$, typically for deep forecasting or hybrid RSSMs (Inzirillo, 21 Jul 2024).
• Hybrid deterministic-stochastic: augmenting a deterministic state with a stochastic component for improved expressivity and uncertainty modeling (Srivastava et al., 2021, Kadi et al., 23 Aug 2025).

2. Inference and Learning Methodologies

Exact inference in nonlinear RSSMs is tractable only in special cases. Most deep RSSMs employ amortized variational inference to approximate the posterior $q_\phi(s_t \mid s_{t-1}, a_{t-1}, o_t)$, learning a recognition/encoder network to align the posterior and prior distributions using the evidence lower bound (ELBO):

$$\mathrm{ELBO} = \sum_{t=1}^T \mathbb{E}_{q_\phi(s_t)}[\log p_\theta(o_t|s_t)] - \mathrm{KL}[q_\phi(s_t|s_{t-1}, a_{t-1}, o_t) \| p_\theta(s_t|s_{t-1}, a_{t-1})]$$

Regularization is provided by this KL term and by auxiliary heads (e.g., reward, inverse dynamics, regime classification) (Srivastava et al., 2021, Shaj et al., 2022). Training employs backpropagation through time (BPTT), with optimization via Adam and often gradient clipping for stability (Srivastava et al., 2021, Shaj et al., 2022). Extensions to closed-form Gaussian Kalman-style inference arise in linear or locally linear RSSMs (Shaj et al., 2022, Becker et al., 2022), and hybrid recognition models can be incorporated to accelerate initial state inference (Doerr et al., 2018).

Some research demonstrates that the default “filter-only” variational family in RSSMs can lead to overestimated aleatoric uncertainty due to its inability to revise past beliefs in light of future observations. This effect can act as implicit regularization, aiding robustness but potentially impairing uncertainty calibration under partial observability or sensor fusion (Becker et al., 2022).

3. Architectural Variants and Expressivity

RSSMs admit a diversity of architectural instantiations:

• GRU/LSTM RSSMs: Structure the latent transition with gated recurrent units, mapping observations to latent states and supporting flexible emission models (Srivastava et al., 2021, Inzirillo, 21 Jul 2024).
• Gaussian-process RSSMs: Employ a nonparametric transition function via sparse GP with exact temporal correlations, yielding the Probabilistic Recurrent State-Space Model (PR-SSM) (Doerr et al., 2018).
• Hidden-Parameter RSSMs (HiP-RSSMs): Introduce low-dimensional latent variables $c$ that parameterize temporal dynamics across a family of related tasks, learned by amortized context encoders and Kalman-style inference (Shaj et al., 2022).
• Contrastive RSSMs: Replace pixel-wise reconstruction with mutual information maximization in a learned feature space, enforced via symmetric contrastive losses to encourage task-relevant, distraction-robust embeddings (Srivastava et al., 2021).
• Goal-conditioned RSSMs (GC-RSSMs): Incorporate explicit goal-conditioning via concatenated or embedded goal observations, supporting planning via actor-critic or CEM in a latent imagination loop (Kadi et al., 23 Aug 2025).
• Switching regime RSSMs: Multiplex several RSSM instances, mixing outputs with learned regime probabilities and Hamilton-style filtering (Inzirillo, 21 Jul 2024).
• Structured SSM kernels: Use diagonal, low-rank, or HiPPO-based recurrence, optimizing scan- or convolutional-style representations for long-sequence modeling (Tiezzi et al., 13 Jun 2024, Singh et al., 4 Sep 2025).

Latent state, emission, and context dimensions, as well as basis combinatorics (e.g., number of locally linear bases, context size), are crucial hyperparameters influencing both expressivity and computational tractability (Shaj et al., 2022).

4. Applications and Empirical Outcomes

RSSMs deliver state-of-the-art results across several domains, particularly:

• Pixel-based and robotic RL: Contrastive RSSMs (CoRe) achieve superior final validation reward compared to alternatives such as bisimulation, CURL, and SAC+RAD on the Distracting Control Suite, maintaining robustness under severe background, color, and camera perturbations (mean reward 480 vs PSE 386 and CURL 236) (Srivastava et al., 2021).
• System identification and changing dynamics: HiP-RSSMs improve over classical RSSMs and recurrent neural nets in low-data adaptation, system identification, and transfer across regimes, empirically validated in robotic arm and mobile robot settings (Shaj et al., 2022).
• Robust control and task generalization: Goal-conditioned RSSMs enable sample-efficient visual model predictive control for garment handling, achieving >94% normalized coverage in real and simulated manipulation with a single policy (Kadi et al., 23 Aug 2025).
• Financial time series forecasting: Deep RSSMs leveraging GRU, LSTM, or TKAN transition modules outperform traditional models in classification, Sharpe ratios, and drawdown metrics on highly nonstationary market data (Inzirillo, 21 Jul 2024).
• Sequence modeling and long-range dependency: Structured and diagonal RSSMs match or exceed transformers in efficiency, scaling linearly in sequence length while maintaining expressivity via controlled recurrence (Tiezzi et al., 13 Jun 2024, Singh et al., 4 Sep 2025).
• Uncertainty quantification and fusion: RSSMs supporting Kalman-style or fully Bayesian updates accurately propagate uncertainty in partially observed, missing-data, or multimodal fusion tasks, outperforming standard RSSMs under high aleatoric noise (Becker et al., 2022).

5. Comparative Analysis within Sequential Modeling

RSSMs sit at the intersection of classical linear state-space modeling, nonlinear stochastic dynamical systems, general RNNs, and modern attention or structured-kernel models. Compared to LSTM/GRU RNNs, RSSMs impose explicit state transitions with learned, often interpretable, structure that alleviates vanishing/exploding gradients and permits efficient parallelization via techniques such as diagonalization, low-rank correction, FFT, or associative scan (Tiezzi et al., 13 Jun 2024, Singh et al., 4 Sep 2025). While transformers provide unrestricted attention and softmax-mediated memory, RSSMs encode the full sequence into a fixed-dimensional latent, trading maximal expressivity for extreme computational and memory efficiency.

Architectural and algorithmic variants yield a spectrum from fully deterministic to fully probabilistic, from plain minimum risk to amortized variational and hybrid EM/Kalman optimization, with tradeoffs in trainability, interpretability, and robustness across tasks and noise conditions (Tiezzi et al., 13 Jun 2024, Shaj et al., 2022, Becker et al., 2022).

6. Open Challenges, Limitations, and Research Directions

Several challenges remain in the theory and practice of RSSMs:

• Expressivity versus efficiency: Purely linear recurrences struggle with tasks requiring arbitrary span association or copy; input gating, time-varying dynamics, and hybrid local attention-SSM models are active areas of research (Tiezzi et al., 13 Jun 2024).
• Uncertainty decomposition: Standard RSSMs overestimate aleatoric uncertainty under their default variational filter, compromising performance in true sensor fusion or partial observation. Principled smoothing approaches, as in VRKN, restore correct uncertainty decomposition and enable efficient sensor fusion (Becker et al., 2022).
• Online and infinite-stream learning: Existing algorithms rely on BPTT or truncated scan; robust online methods (RTRL, UORO, local/forward, or recurrent backprop fixes) remain relatively unexplored in deep RSSMs (Tiezzi et al., 13 Jun 2024).
• Scalability and parallelization: While scaling can be linear with careful structure (diagonal, low-rank, FFT), mapping RSSMs to modern hardware architectures at the bandwidth/FLOP tradeoff frontier is a continuing challenge (Tiezzi et al., 13 Jun 2024, Singh et al., 4 Sep 2025).
• Hybrid architectures and lifelong adaptation: Combining RSSMs with transformer-style local attention, gating, and latent adaptation yields promising results in simulation, but large-scale, real-world deployments and benchmarks are still lacking (Tiezzi et al., 13 Jun 2024, Kadi et al., 23 Aug 2025).

A central future direction is the deep integration of structured state-space recurrence with global attention and flexible uncertainty-aware inference, along with new methodologies enabling online adaptation and scalable lifelong learning.

7. Summary Table: Key RSSM Variants and Properties

Variant	Transition/Inference	Key Application
Standard RSSM (Srivastava et al., 2021)	GRU/LSTM + amortized variational	RL, pixel-based world modeling
Contrastive RSSM (Srivastava et al., 2021)	GRU + symmetric contrastive loss	Robust RL under distractions
HiP-RSSM (Shaj et al., 2022)	Locally linear + task encoder	Identifying changing dynamics
Probabilistic RSSM (Doerr et al., 2018)	GP transitions + variational	Nonlinear system identification
Goal-conditioned RSSM (Kadi et al., 23 Aug 2025)	GRU + goal embedding	Visual MBRL, manipulation
VRKN (Becker et al., 2022)	Kalman + MC Dropout	Uncertainty, sensor fusion
Deep RSSM (Inzirillo, 21 Jul 2024)	LSTM/GRU/TKAN, deterministic	Financial time series
Structured SSM (Tiezzi et al., 13 Jun 2024Singh et al., 4 Sep 2025)	Diagonal, S4, Koopman, subspace	Long-sequence modeling

Each variant is characterized by its approach to latent transition modeling, uncertainty representation, and targeted domain/applications.

---

RSSMs unify probabilistic latent-state modeling, deep learning-based transition/emission parameterizations, and scalable inference to provide a versatile foundation for time-series, control, and classification tasks, excelling especially where long horizons, uncertainty, and partial observability are dominant constraints. Advances in structure, uncertainty handling, and efficient optimization continue to enhance their theoretical rigor and practical impact.