Deep State Space Models Overview

• Deep State Space Models (SSMs) are sequential models that integrate control-theoretic state-space formulations with deep neural networks to capture long-range dependencies.
• They employ stacked layers with input-dependent gating and specialized recurrences, achieving both computational efficiency and expressive power for tasks like language and time-series modeling.
• Empirical studies show that deep SSMs can outperform Transformer models on long-context benchmarks while addressing challenges like over-smoothing and recency bias.

Deep State Space Models (SSMs) are a class of sequential models that unify the control-theoretic formalism of state-space with deep neural network architectures, aiming to efficiently capture long-range dependencies in sequential data. Recent advances have demonstrated their scalability for language, time-series, and dynamical system modeling, providing not only expressive modeling capacity but also competitive or superior performance to classic Transformer-based architectures in various long-context benchmarks (Alonso et al., 25 Mar 2024). Deep SSMs encompass linear, nonlinear, discrete, and continuous-time formulations, and are often differentiated by their parametrization, stacking strategies, architectural variants, and their integration with deep learning pipelines.

1. Mathematical Foundations and Core Architectures

A linear state-space model in continuous time is defined by: $$\dot x(t) = A x(t) + B u(t), \qquad y(t) = C x(t) + D u(t),$$ where $x(t)\in\mathbb{C}^p$ is the state, $u(t)\in\mathbb{R}^q$ the input, $y(t)\in\mathbb{R}^q$ the output, and $A,B,C,D$ learnable matrices. After discretization (e.g., step size $\Delta$) this leads to: $$x_{k+1} = \bar A x_k + \bar B u_k, \qquad y_k = \bar C x_k + \bar D u_k,$$ with trainable $\bar A,\bar B,\bar C,\bar D$ (Alonso et al., 25 Mar 2024).

When sequence supervision is available the model admits a convolutional form: $$y_k = \sum_{\tau=0}^k \bar C \bar A^{k-\tau} \bar B u_\tau,$$ enabling parallel training.

Deep SSMs are constructed by stacking such layers, with each layer comprising:

• A pre-processing fork projecting $u$ into two streams.
• A core SSM recurrence, which in time-varying (LTV) models like Mamba (S6) uses input-dependent $\bar A_k, \bar B_k$:

$$x_{k+1} = \bar A_k x_k + \bar B_k u_k, \quad o_k = \bar C_k x_k + \bar D_k u_k,$$

with selectivity (gating).

• A post-processing gating mechanism:

$$y_k = o_k \odot \sigma(W_{\rm gate} u_k^{\rm (skip)}),$$

where $\odot$ denotes element-wise multiplication and $\sigma$ is a nonlinear activation (softmax/sigmoid) (Alonso et al., 25 Mar 2024).

Main architectural variants include S4 (LTI, low-rank update), S4D (diagonal $A$), S5 (MIMO, large block), LRU (direct discrete parametrization), and selective time-varying models like S6/Mamba with input-dependent parameters (Alonso et al., 25 Mar 2024). These structures share the property of enabling either linear or nearly linear computational cost with respect to sequence length via custom parallel scan or FFT-based algorithms (Zhang et al., 2023).

2. Theory of Expressivity, Depth, and Width

The expressivity of deep linear SSMs is governed by both depth (number of layers) and width (state dimension per layer). It has been rigorously established that:

• Without parameter-norm constraints, depth and width are equivalent: any deep SSM can be "flattened" into a single wide SSM, and vice versa, with only $O(lm)$ parameters for an $l$-layer, width-$m$ model (Bao et al., 24 Jun 2025).
• Under parameter-norm (e.g., spectral, operator norm) constraints, deeper models can implement high-norm behaviors with much smaller per-layer parameter norms than shallow models: the required norm per layer scales as $c_1^{2/(l+1)}$ for depth $l$, compared to $c_1$ in the shallow case, where $c_1$ is the base norm bound (Bao et al., 24 Jun 2025).
• Empirically, deep-narrow SSMs (many layers, small width) achieve lower error on long-range tasks than shallow-wide ones for fixed total state size, at the cost of increased per-step compute.

A notable result is that, for a fixed expressivity target and total parameter count, deeper SSMs are more robust and train more efficiently as the required norm per parameter can be reduced, alleviating optimization and stability challenges (Bao et al., 24 Jun 2025).

3. Selective and Gated SSMs, Theoretical Guarantees

Recent architectures such as Mamba, S6, GateLoop, and Gated Linear Attention implement selective SSMs by introducing input-controlled, multiplicative updates in the recurrence: $h_\ell = A(x_\ell) h_{\ell-1} + B(x_\ell) x_\ell,$ where the state-to-state and input-to-state transitions depend explicitly on $x_\ell$ (Cirone et al., 29 Feb 2024). This input-dependent gating mechanism equips deep SSMs with content-aware "selectivity," crucial for tasks requiring variable-length memory and dynamic feature importance.

A fundamental theoretical insight is that selective SSMs can be understood through rough path theory: the hidden state of such models forms a low-dimensional projection of the truncated signature of the input path. Since signatures universally capture all time-ordered interactions, deep and/or chained selective SSMs provably approximate rich classes of nonlinear functionals over sequences, subsuming key capabilities of attention (Cirone et al., 29 Feb 2024).

Implementation-wise, balance between efficiency and expressivity motivates diagonal gating (per-channel selectivity, $O(NL)$ time) or small-block structures, with dense blocks stacked for higher-order interaction recovery (Cirone et al., 29 Feb 2024).

4. Training Dynamics, Initialization, and Pruning

The training dynamics of deep linear SSMs have been analytically linked to those of deep linear feedforward networks. In the frequency domain, each frequency or singular-value "mode" is learned independently, with the time constant proportional to data covariance in the mode and, crucially, inversely proportional to latent dimension: over-parameterization (larger hidden state size) accelerates convergence (Smékal et al., 10 Jul 2024). For multi-layer SSMs, each layer's frequency response multiplies, producing backpropagated gradient ODEs akin to classic deep networks.

Initialization is critical for stability and memory length. For LTI models (S4), HiPPO-based eigenvalue placements or polar coordinate parameterizations ensure eigenvalues are close to the unit circle but within the unit disk, optimizing both memory and numerical stability (Alonso et al., 25 Mar 2024).

Model compression is addressed by structure-aware pruning. The Layer-Adaptive State Pruning (LAST) algorithm ranks and removes state variables layer-wise without retraining, using normalized $\mathcal{H}_\infty$ system norms as a global energy criterion. Empirically, pruning up to 33% of states in deep SSMs typically incurs less than 1% performance loss, exposing substantial redundancy (Gwak et al., 5 Nov 2024).

5. Empirical Performance and Applications

Deep SSMs achieve strong performance in long-sequence modeling benchmarks, often surpassing Transformers and RNNs in both accuracy and efficiency. For example, on the Long-Range Arena, LTI SSMs such as S5 obtain average accuracy of 87.5%, outperforming Transformers (53.7%) and LTV SSMs (Mamba/S6, 66.6%) (Alonso et al., 25 Mar 2024).

Recent models like SpaceTime employ companion matrix parameterizations for discrete-time SSMs, yielding exact ARIMA/ETS expressivity, effective long-horizon closed-loop forecasting, and greatly accelerated FFT-based inference, outperforming alternatives in forecasting and classification across diverse benchmarks (Zhang et al., 2023).

SSMs also function as efficient architectural modules for sequence modeling beyond language, notably excelling in time-series settings with mixed frequency or irregular sampling, where classical attention models falter (Lin et al., 15 Dec 2024). The Selective State Space Layer Aggregation (S6LA) module extends this paradigm to deep vision models, aggregating CNN or ViT layer outputs through a learned SSM scaffold, yielding consistent gains in ImageNet and COCO tasks (Liu et al., 12 Feb 2025).

6. Limitations, Bottlenecks, and Mitigations

Despite their strengths, deep SSMs exhibit intrinsic bottlenecks:

• Recency bias: The influence of inputs decays exponentially with distance, controlled by the largest eigenvalue of the transition matrix $A$ (Wang et al., 31 Dec 2024).
• Over-smoothing: As depth increases, token representations collapse (become indistinguishable), as the effective smoothing factor compounds with each stacked SSM layer (Wang et al., 31 Dec 2024).

Experiments confirm that while increasing depth extends receptive field, it also amplifies over-smoothing, ultimately impairing discriminativeness. The polarization technique, which fixes two channels of $A$ at 1 (pure recurrence) and 0 (pure skip/copy), simultaneously eliminates recency bias and over-smoothing, unlocking further depth scalability—a crucial insight for scaling SSM-based models (Wang et al., 31 Dec 2024).

7. Interpretability, Theoretical Duality with Attention, and Future Directions

Deep SSMs enable insight into latent representations when equipped with structured priors and linear decoders; e.g., interpretable latent variables can act as random effects with explicit loadings in a linear mixed model framework (Wu et al., 2022). Shrinkage priors further enhance parsimony and robustness.

At an architectural level, the generalized theory of structured semiseparable matrices (SSS) reveals that Transformers and SSMs share a common foundation: attention maps can be represented as state-space recurrences with structured kernel masking, and vice versa. For instance, the Mamba-2 architecture leverages this duality for efficient inference and training, surpassing FlashAttention-2 and classic softmax attention at scale (Dao et al., 31 May 2024).

Key future challenges include:

• Developing minimal parametrizations and exploiting transfer-function-domain approaches to further compress SSMs (Bonassi et al., 2023).
• Integrating data-driven initialization from control theory for improved sample efficiency.
• Systematic benchmarking on heterogeneous sequence modalities.
• Extending formal analysis and hardware-efficient implementation for general block-structured or hybrid SSM-attention models (Dao et al., 31 May 2024, Bonassi et al., 2023).