---
title: Deep State Space Models Overview
url: https://www.emergentmind.com/topics/deep-state-space-models-ssms-c6de70ff-3367-4cea-bbe0-e06a7662f886
type: topic
---

# Deep State Space Models Overview

Deep State Space Models (SSMs) are a class of sequential models that unify the control-theoretic formalism of state-space with deep neural network architectures, aiming to efficiently capture long-range dependencies in sequential data. Recent advances have demonstrated their scalability for language, time-series, and dynamical system modeling, providing not only expressive modeling capacity but also competitive or superior performance to classic Transformer-based architectures in various long-context benchmarks [2403.16899]. Deep SSMs encompass linear, nonlinear, discrete, and continuous-time formulations, and are often differentiated by their parametrization, stacking strategies, architectural variants, and their integration with deep learning pipelines.

## 1. Mathematical Foundations and Core Architectures

A linear state-space model in continuous time is defined by:
\[
\dot x(t) = A x(t) + B u(t), \qquad y(t) = C x(t) + D u(t),
\]
where \( x(t)\in\mathbb{C}^p \) is the state, \( u(t)\in\mathbb{R}^q \) the input, \( y(t)\in\mathbb{R}^q \) the output, and \( A,B,C,D \) learnable matrices. After discretization (e.g., step size \(\Delta\)) this leads to:
\[
x_{k+1} = \bar A x_k + \bar B u_k, \qquad y_k = \bar C x_k + \bar D u_k,
\]
with trainable \(\bar A,\bar B,\bar C,\bar D\) [2403.16899].

When sequence supervision is available the model admits a convolutional form:
\[
y_k = \sum_{\tau=0}^k \bar C \bar A^{k-\tau} \bar B u_\tau,
\]
enabling parallel training.

Deep SSMs are constructed by stacking such layers, with each layer comprising:
- A pre-processing fork projecting \(u\) into two streams.
- A core SSM recurrence, which in time-varying (LTV) models like Mamba (S6) uses input-dependent \(\bar A_k, \bar B_k\):
  \[
  x_{k+1} = \bar A_k x_k + \bar B_k u_k, \quad o_k = \bar C_k x_k + \bar D_k u_k,
  \]
  with selectivity (gating).
- A post-processing gating mechanism:
  \[
  y_k = o_k \odot \sigma(W_{\rm gate} u_k^{\rm (skip)}),
  \]
where \(\odot\) denotes element-wise multiplication and \(\sigma\) is a nonlinear activation (softmax/sigmoid) [2403.16899].

Main architectural variants include S4 (LTI, low-rank update), S4D (diagonal \(A\)), S5 (MIMO, large block), LRU (direct discrete parametrization), and selective time-varying models like S6/Mamba with input-dependent parameters [2403.16899]. These structures share the property of enabling either linear or nearly linear computational cost with respect to sequence length via custom parallel scan or FFT-based algorithms [2303.09489].

## 2. Theory of Expressivity, Depth, and Width

The expressivity of deep linear SSMs is governed by both depth (number of layers) and width (state dimension per layer). It has been rigorously established that:
- Without parameter-norm constraints, depth and width are equivalent: any deep SSM can be "flattened" into a single wide SSM, and vice versa, with only \(O(lm)\) parameters for an \(l\)-layer, width-\(m\) model [2506.19296].
- Under parameter-norm (e.g., spectral, operator norm) constraints, deeper models can implement high-norm behaviors with much smaller per-layer parameter norms than shallow models: the required norm per layer scales as \(c_1^{2/(l+1)}\) for depth \(l\), compared to \(c_1\) in the shallow case, where \(c_1\) is the base norm bound [2506.19296].
- Empirically, deep-narrow SSMs (many layers, small width) achieve lower error on long-range tasks than shallow-wide ones for fixed total state size, at the cost of increased per-step compute.

A notable result is that, for a fixed expressivity target and total parameter count, deeper SSMs are more robust and train more efficiently as the required norm per parameter can be reduced, alleviating optimization and stability challenges [2506.19296].

## 3. Selective and Gated SSMs, Theoretical Guarantees

Recent architectures such as Mamba, S6, GateLoop, and Gated Linear Attention implement *selective* SSMs by introducing input-controlled, multiplicative updates in the recurrence:
\[
h_\ell = A(x_\ell) h_{\ell-1} + B(x_\ell) x_\ell,
\]
where the state-to-state and input-to-state transitions depend explicitly on \(x_\ell\) [2402.19047]. This input-dependent gating mechanism equips deep SSMs with content-aware "selectivity," crucial for tasks requiring variable-length memory and dynamic feature importance.

A fundamental theoretical insight is that selective SSMs can be understood through rough path theory: the hidden state of such models forms a low-dimensional projection of the truncated signature of the input path. Since signatures universally capture all time-ordered interactions, deep and/or chained selective SSMs provably approximate rich classes of nonlinear functionals over sequences, subsuming key capabilities of attention [2402.19047].

Implementation-wise, balance between efficiency and expressivity motivates diagonal gating (per-channel selectivity, \(O(NL)\) time) or small-block structures, with dense blocks stacked for higher-order interaction recovery [2402.19047].

## 4. Training Dynamics, Initialization, and Pruning

The training dynamics of deep linear SSMs have been analytically linked to those of deep linear feedforward networks. In the frequency domain, each frequency or singular-value "mode" is learned independently, with the time constant proportional to data covariance in the mode and, crucially, inversely proportional to latent dimension: over-parameterization (larger hidden state size) accelerates convergence [2407.07279]. For multi-layer SSMs, each layer's frequency response multiplies, producing backpropagated gradient ODEs akin to classic deep networks.

Initialization is critical for stability and memory length. For LTI models (S4), HiPPO-based eigenvalue placements or polar coordinate parameterizations ensure eigenvalues are close to the unit circle but within the unit disk, optimizing both memory and numerical stability [2403.16899].

Model compression is addressed by structure-aware pruning. The Layer-Adaptive State Pruning (LAST) algorithm ranks and removes state variables layer-wise without retraining, using normalized \(\mathcal{H}_\infty\) system norms as a global energy criterion. Empirically, pruning up to 33% of states in deep SSMs typically incurs less than 1% performance loss, exposing substantial redundancy [2411.02824].

## 5. Empirical Performance and Applications

Deep SSMs achieve strong performance in long-sequence modeling benchmarks, often surpassing Transformers and RNNs in both accuracy and efficiency. For example, on the Long-Range Arena, LTI SSMs such as S5 obtain average accuracy of 87.5%, outperforming Transformers (53.7%) and LTV SSMs (Mamba/S6, 66.6%) [2403.16899].

Recent models like SpaceTime employ companion matrix parameterizations for discrete-time SSMs, yielding exact ARIMA/ETS expressivity, effective long-horizon closed-loop forecasting, and greatly accelerated FFT-based inference, outperforming alternatives in forecasting and classification across diverse benchmarks [2303.09489].

SSMs also function as efficient architectural modules for sequence modeling beyond language, notably excelling in time-series settings with mixed frequency or irregular sampling, where classical attention models falter [2412.11211]. The Selective State Space Layer Aggregation (S6LA) module extends this paradigm to deep vision models, aggregating CNN or ViT layer outputs through a learned SSM scaffold, yielding consistent gains in ImageNet and COCO tasks [2502.10463].

## 6. Limitations, Bottlenecks, and Mitigations

Despite their strengths, deep SSMs exhibit intrinsic bottlenecks:
- **Recency bias:** The influence of inputs decays exponentially with distance, controlled by the largest eigenvalue of the transition matrix \(A\) [2501.00658].
- **Over-smoothing:** As depth increases, token representations collapse (become indistinguishable), as the effective smoothing factor compounds with each stacked SSM layer [2501.00658].

Experiments confirm that while increasing depth extends receptive field, it also amplifies over-smoothing, ultimately impairing discriminativeness. The *polarization technique*, which fixes two channels of \(A\) at 1 (pure recurrence) and 0 (pure skip/copy), simultaneously eliminates recency bias and over-smoothing, unlocking further depth scalability—a crucial insight for scaling SSM-based models [2501.00658].

## 7. Interpretability, Theoretical Duality with Attention, and Future Directions

Deep SSMs enable insight into latent representations when equipped with structured priors and linear decoders; e.g., interpretable latent variables can act as random effects with explicit loadings in a linear mixed model framework [2203.02057]. Shrinkage priors further enhance parsimony and robustness.

At an architectural level, the generalized theory of structured semiseparable matrices (SSS) reveals that Transformers and SSMs share a common foundation: attention maps can be represented as state-space recurrences with structured kernel masking, and vice versa. For instance, the Mamba-2 architecture leverages this duality for efficient inference and training, surpassing FlashAttention-2 and classic softmax attention at scale [2405.21060].

Key future challenges include:
- Developing minimal parametrizations and exploiting transfer-function-domain approaches to further compress SSMs [2312.06211].
- Integrating data-driven initialization from control theory for improved sample efficiency.
- Systematic benchmarking on heterogeneous sequence modalities.
- Extending formal analysis and hardware-efficient implementation for general block-structured or hybrid SSM-attention models [2405.21060, 2312.06211].

Source: https://www.emergentmind.com/topics/deep-state-space-models-ssms-c6de70ff-3367-4cea-bbe0-e06a7662f886