Structured State-Space Models (SSMs)

• Structured State-Space Models (SSMs) are neural architectures that use parameterized recurrences and structured matrices to efficiently model sequential data with long-term dependencies.
• They excel in applications such as language, speech, vision, and algorithmic tasks by outperforming traditional RNNs and Transformers in capturing long-range patterns.
• SSMs leverage mathematical foundations from dynamical systems and innovative discretizations to deliver precise inductive biases, efficient computation, and enhanced model expressivity.

Structured State-Space Models (SSMs) are a class of neural architectures that represent sequences using parameterized recurrences derived from dynamical system theory and linear systems. SSMs combine the expressiveness of state-driven sequence models with highly structured matrices, allowing extremely efficient computation while capturing long-range dependencies with precise inductive biases. Evolving rapidly since the introduction of the S4 model, SSMs now underpin state-of-the-art models for language, speech, vision, system identification, and many algorithmic tasks (Somvanshi et al., 22 Mar 2025, Mehari et al., 2022, Cirone et al., 2024).

1. Mathematical Foundations and Model Structures

SSMs generalize classical linear time-invariant (LTI) systems to the deep learning setting. The continuous-time SSM is defined by

$$\frac{d}{dt} x(t) = A x(t) + B u(t) \,, \qquad y(t) = C x(t) + D u(t)$$

with latent state $x \in \mathbb{R}^N$, input $u \in \mathbb{R}^d$, output $y \in \mathbb{R}^m$, and parameter matrices $A, B, C, D$. Discrete SSMs arise via zero-order hold discretization: $$\bar{A} = \exp(A\Delta)\,,\quad \bar{B} = \int_0^{\Delta} \exp(A\tau) B d\tau,\quad x_k = \bar{A} x_{k-1} + \bar{B} u_k,\quad y_k = C x_k + D u_k.$$ Deep SSMs stack such layers with nonlinearities (often GeLU or Swish), skip connections, and additional MLPs, mapping input sequences into states through successive transformations (Bonassi et al., 2023). Most modern SSMs restrict $A$ to highly structured forms for computational tractability.

Key structural choices include:

• Diagonal $A$: Per-moding recurrent update, optimal for FFT or parallel scan; critical for S4, S5, LRU models.
• Diagonal plus low-rank (DPLR) $A$: Allows controlled expressivity with efficient computation, used in S4 and S5.
• Sparse/Permutation-Diagonal (PD-SSM): Product of column-one-hot (permutation) and (complex-)diagonal matrices, enabling efficient FSA emulation and optimal state tracking (Terzić et al., 26 Sep 2025).
• Block structures: SlotSSMs partition the global state into slots, independently updated and sparsely mixed (Jiang et al., 2024).

2. Parameterizations, Discretizations, and Inductive Bias

Parameterizations of $A, B, C$ are critical to stability, expressivity, and efficiency. SSMs such as S4 employ the HiPPO framework for parameterization: $A$ is initialized to encode projections onto exponentially-warped Legendre (or other orthonormal) bases, granting inductive bias for long-term memory (Gu et al., 2022). This leads to state matrices of the form

$$A_{nk} = -\sqrt{2n+1}\sqrt{2k+1}/(n+k+1),\quad B_n = \sqrt{2n+1}$$

for HiPPO-LegS. Extensions allow general orthogonal bases, including Fourier (HiPPO-FouT), permitting adaptation to locality or periodicity.

Discrete-time instantiation uses either strict ZOH, bilinear, or hybrid discretizations to preserve the memory properties and stability. Recent work introduces modular lag-operator interpretations, where the recurrence update

$$A_{nm} = \langle \phi_n, \phi_m \circ \ell \rangle_{L^2(Z)}$$

follows the induced "lag" between basis projections, giving a geometric and compositional framework to derive new SSMs (Tomonaga et al., 22 Dec 2025).

Complex-valued parameterizations of SSMs have been shown to admit richer oscillatory behavior and strictly more compact representations: any real diagonal SSM can be realized by a complex diagonal SSM of equal (or smaller) dimension, but the reverse is not true without exponential blow-up in state size or parameter norms (Ran-Milo et al., 2024).

3. Algorithmic Realizations and Computational Complexity

Efficient SSM implementations exploit structural recurrence, allowing operations to scale linearly or near-linearly in both sequence length $T$ and state dimension $N$:

• Parallel scan: For diagonal or PD-SSMs, state updates can be executed as parallel prefix-sums (Terzić et al., 26 Sep 2025).
• FFT convolution: For fixed-$A$ SSMs, the input is convolved with the impulse response kernel using FFT, yielding $O(N \log T)$ cost (Mehari et al., 2022).
• Semiseparable and low-rank representations: General SSMs are equivalent to applying N-semiseparable matrices, enabling efficient matrix-vector multiplication for both forward and gradient computation (Dao et al., 2024, Hu et al., 6 Oct 2025).
• Selective SSMs: Mamba and similar architectures allow multiplicative gating: $A_t$ (and sometimes $B_t, C_t$) depend nonlinearly on the input, dramatically increasing expressive power for modest additional cost (Cirone et al., 2024).

The computational footprint of SSMs matches or outperforms RNNs and Transformers for long sequences. For instance, the core recurrence is typically $O(NT)$, with memory consumption sublinear in $T$ due to caching or scan fusion (Mehari et al., 2022, Shakhadri et al., 6 Jan 2025). PD-SSMs admit $O(NT)$ time and space, yet can represent arbitrary finite-state automata (Terzić et al., 26 Sep 2025).

4. Connections to Attention, Duality, and Hybrids

A theoretical breakthrough is the characterization of structured state-space duality: certain SSMs (notably those with scalar-identity or diagonal $A_t$) are algebraically equivalent to masked attention with semiseparable masks. A scalar-multiplied-identity $A_t$ implies a 1-semiseparable causal mask $L$, such that

$y = (L \circ (C B^\top))X$

is equivalent to both a linear-time recurrence and a quadratic-time masked attention (Hu et al., 6 Oct 2025, Dao et al., 2024). Diagonal SSMs generalize this to sums of $N$ one-dimensional semiseparable attention blocks. This duality does not extend to softmax attention due to rank explosion—full softmax attention is not representable by a finite-state SSM.

Hybrid models such as Zamba or SlotSSM interleave SSMs and attention: SlotSSMs maintain K independent SSM slots and bottleneck cross-slot information flow via sparse self-attention (Jiang et al., 2024). GFSSM incorporates grouped FIR filtering and explicit attention-sink mechanisms to stabilize training and maintain information locality (Meng et al., 2024). In NLP and multimodal architectures, Transformer-SSM hybrids exploit the strengths of both frameworks (Terzić et al., 26 Sep 2025).

5. Empirical Performance, Applications, and Practical Insights

SSM variants (S4, S5, Mamba, Jamba, PD-SSM, SlotSSM) excel in long-range sequence modeling, outperforming CNNs and RNNs in domains with long-term dependencies and/or monotonic or periodic patterns. Key domains:

• NLP: Mamba and Mamba-2 match or exceed Transformer results at small to medium scale, with 2–8$\times$ faster evaluation at sequence lengths $\sim$2k tokens and much lower memory (Dao et al., 2024, Shakhadri et al., 6 Jan 2025).
• Time-Series/Signal Processing: SSMs achieve SOTA in ECG analysis (macro-AUC 0.9417 on PTB-XL) and speech recognition (Samba-ASR achieves 1.17% WER on LibriSpeech) (Mehari et al., 2022, Shakhadri et al., 6 Jan 2025).
• Algorithmic Tasks/FSA Tracking: PD-SSM achieves $>$99% accuracy on cycle navigation, parity, and group word problems, unreachable by diagonal SSMs at equal complexity (Terzić et al., 26 Sep 2025).
• Vision/Video: SlotSSMs excel on object-centric unsupervised learning and 3D reasoning benchmarks, with superior scalability to Transformer-based baselines (Jiang et al., 2024).
• Spiking Neural Networks: SSM-inspired parameterizations improve both expressivity and efficiency in event-based speech tasks (Fabre et al., 4 Jun 2025).

Compression techniques exploit SSM redundancy: Mamba-Shedder can prune 20–25% of blocks or SSM modules with $<$5% accuracy loss and $1.4\times$ inference speedup, with further recovery after brief fine-tuning (Muñoz et al., 28 Jan 2025).

6. Robustness, Stability, and Limitations

Ensuring input-output stability and robustness remains a challenge, especially for deep SSM stacks. L2RU provides a free parametrization for $\mathcal{L}_2$-bounded SSMs, guaranteeing stability via internal Lyapunov certificates (Massai et al., 31 Mar 2025). This enables unconstrained optimization without complex projections, crucial for safe deployment in control or safety-critical settings.

Implicit bias in SSMs can break down under clean-label poisoning: the inclusion of carefully chosen, perfectly-labeled outlier sequences can catastrophically distort the inductive bias of SSMs, destroying generalization even in high-dimensional settings (Slutzky et al., 2024). Defenses require data sanitization, regularization, and certified guarantees on impulse response robustness.

Complex-parameter SSMs are strictly more expressive (with exponentially lower dimension/parameter magnitude for oscillatory or frequency-rich tasks) compared to real-parameter SSMs—this separation persists even in overparameterized regimes, except possibly when strong selectivity mechanisms are present (Ran-Milo et al., 2024).

7. Theoretical Advances and Future Directions

Recent theoretical insights include the path signature characterization of selective SSMs: input-dependent recurrences (as in Mamba) allow the model to compute random projections of the input path's signature, a universal set of path functionals, with $O(NT)$ computational cost (Cirone et al., 2024). The lag-operator framework unifies discrete and continuous SSM design by framing each update as a basis lagging geometry, modularizing the choices of time-warp and basis (Tomonaga et al., 22 Dec 2025).

Key open research topics:

• Interpretable and certified SSM blocks: Learning robust, compositional modules with explicit stability certificates (Massai et al., 31 Mar 2025).
• Hybrid attention-SSM architectures: Exploring the design space guided by structured matrix duality (Dao et al., 2024).
• Algorithmic and symbolic reasoning: Leveraging SSMs' capacity for FSA emulation for more complex program parsing or code understanding.
• Data corruption and poisoning robustness: Developing practical defenses against adversarial or accidental data distortions (Slutzky et al., 2024).
• Multi-resolution and adaptive memory: Using modular warping and basis selection to match complex multi-timescale sequences (Tomonaga et al., 22 Dec 2025).