Structured State-Space Models

• Structured state-space models are linear dynamical systems tailored for deep learning that capture complex temporal dependencies with high computational efficiency.
• Modern variants use structured parameterizations such as diagonal, low-rank, and sparse formulations to achieve near-linear training and inference complexity.
• SSMs demonstrate competitive performance across domains like NLP, speech recognition, and control, offering scalable, hardware-friendly architectures.

Structured state-space models (SSMs) are a class of sequence modeling architectures grounded in linear dynamical system theory and tailored for deep learning applications requiring efficient, expressive modeling of long-range dependencies. Defined by the recurrence

$$x_{t+1} = A x_t + B u_t,\qquad y_t = C x_t + D u_t,$$

where $x_t$ is an $N$-dimensional hidden state, $u_t$ the input, $y_t$ the output, and $A,B,C,D$ learnable parameters, SSMs achieve high computational parallelism and scalable memory while capturing complex temporal relationships. Modern SSM variants such as S4, Mamba, S5, and hybrids exploit low-rank, diagonal, or highly structured formulations to reduce both training and inference cost, making them competitive with—and often superior to—transformers and RNNs for long-sequence modeling tasks. Their prominent features include linear or near-linear scaling with sequence length, hardware-friendliness via convolutional or parallel scan formulations, and robust performance across domains such as NLP, speech recognition, time-series forecasting, vision, symbolic music, and control (Mehari et al., 2022, Shakhadri et al., 6 Jan 2025, Yuan et al., 27 Jul 2025, Korkmaz et al., 2024, Zakwan et al., 8 Apr 2026).

1. Mathematical Foundations and Parameterizations

At the core, SSMs are discrete or continuous-time linear time-invariant (LTI) systems, mapped to deep sequence models via appropriate discretization and stacking with nonlinearities: $$\dot{x}(t) = A x(t) + B u(t),\quad y(t) = C x(t) + D u(t)$$ with corresponding discrete recurrence after time-discretization (typically via zero-order hold, exponential map, or bilinear transform): $$x_{t+1} = \bar{A} x_t + \bar{B} u_t,\quad y_t = \bar{C} x_t + \bar{D} u_t.$$ Structured parameterizations such as diagonal, block-diagonal, or diagonal-plus-low-rank (DPLR) for $A$ enable efficient state updates, exploiting fast matrix exponentials and parallel scan (prefix-sum) algorithms for $O(N)$ or $x_t$0 complexity per step (Mehari et al., 2022, Shakhadri et al., 6 Jan 2025, Bonassi et al., 2023). For example, S4 parameterizes $x_t$1 as $x_t$2 (with $x_t$3 diagonal, $x_t$4 low-rank), allowing efficient FFT-based convolution. Mamba and its successors introduce input-dependent gating ("selectivity") in $x_t$5 and $x_t$6, further adapting the state-space evolution to the input content (Muñoz et al., 28 Jan 2025, Dao et al., 2024).

Recent designs leverage even more expressive sparsity structures, such as the PD-SSM’s composition of a column-one-hot permutation matrix $x_t$7 with a diagonal complex-valued matrix $x_t$8, preserving linear cost while emulating arbitrary finite-state automata with tight minimality guarantees (Terzić et al., 26 Sep 2025). The L2RU reparameterizes the SSM to guarantee prescribed $x_t$9-gain bounds via an explicit bounded-real LMI decomposition, introducing robust, unconstrained end-to-end optimization for control-critical applications (Massai et al., 31 Mar 2025).

2. Algorithmic Efficiency and Scalability

SSMs achieve linear or near-linear training and inference complexity through parallel convolutional algorithms or associative scan operations. For dense unstructured $N$0, operations cost $N$1 per step (impractical for deep learning). Diagonal or DPLR $N$2 reduces this to $N$3, and scan-based recurrences enable batched GPU-friendly implementations.

The SSM kernel can be computed either by direct recurrence or by convolution: $N$4 which, for appropriate parameterizations, admits $N$5 complexity via FFT for long-range unrolled kernels. Mamba-2 and related SSMs further optimize for hardware efficiency by grouping computations and reducing activation size, matching or outpacing highly optimized attention kernels such as FlashAttention on long sequences (Dao et al., 2024, Korkmaz et al., 2024).

PD-SSM’s parameterization of $N$6 as $N$7, with $N$8 a one-hot-permutation and $N$9 diagonal, matches the diagonal SSMs in computational cost while providing full expressive capacity for automaton emulation. This structure ensures $u_t$0 per-step cost in parallel scan (Terzić et al., 26 Sep 2025). L2RU exploits an explicit "free" parametrization covering the full set of gain-constrained LTI systems, allowing standard gradient-based optimization with bounded computational overhead ($u_t$1 per Cholesky step) (Massai et al., 31 Mar 2025).

3. Expressiveness: Complex, Real, and Sparse SSMs

The expressiveness of SSMs depends critically on the field ($u_t$2 vs $u_t$3) and the structure of $u_t$4:

• Complex diagonal SSMs can represent oscillatory and highly nontrivial convolution kernels in low state dimension (e.g., $u_t$5 for certain delays, oscillations), whereas real SSMs require $u_t$6 for impulse responses of length $u_t$7. Real parameterizations often suffer exponential blowup in $u_t$8 to mimic certain tasks, making them impractical to optimize in practice due to gradient scaling and finite-precision issues (Ran-Milo et al., 2024).
• Selectivity, i.e., making $u_t$9 and/or $y_t$0 input-dependent via shallow networks (as in Mamba), increases the representational power of real SSMs, allowing them to match or approach complex SSM performance in some high-frequency or specified-delay settings, though complex models retain an advantage for smooth/oscillatory kernels (Ran-Milo et al., 2024, Muñoz et al., 28 Jan 2025).
• Structured sparsity, e.g., the PD-SSM’s $y_t$1 factorization, raises the expressive ceiling: a single-layer PD-SSM with state size $y_t$2 exactly emulates any $y_t$3-state deterministic finite-state automaton (FSA), which is provably minimal for unique state codings. Such models maintain linear scan while achieving strict universality on formal state-tracking tasks (Terzić et al., 26 Sep 2025).

4. Theoretical Frameworks: Duality and Hybridization

Structured state-space duality (SSD) establishes a formal correspondence between recurrent (SSM) and attention (Transformer) sequence models under certain structural conditions. For scalar or diagonal $y_t$4, the SSM recurrence is exactly equivalent to a masked linear attention with a semiseparable (rank-1 or rank-$y_t$5) mask. Explicitly, for scalar $y_t$6 and corresponding recurrences,

$y_t$7

the same mapping is obtained by a $y_t$8-semiseparable masked attention mechanism. Diagonal SSMs extend this as a sum over $y_t$9 independent rank-1 masks. However, standard softmax attention destroys this correspondence due to rank explosion in the output, which cannot in general be captured by low-rank or semiseparable structures (Hu et al., 6 Oct 2025, Dao et al., 2024).

Hybrid models such as Zamba (alternating Mamba and Transformer) and Hymba (parallel Mamba/Attention heads) exploit both long-range memory (via SSM) and in-context/fine-grained reasoning (via attention), facilitating architectural efficiency as well as accuracy (Muñoz et al., 28 Jan 2025). Structured hybridization supports diverse applications where both explicit state-tracking and dynamic, token-wise interactions are needed.

5. Empirical Performance Across Domains

SSMs achieve state-of-the-art or highly competitive performance across multiple domains:

• ECG classification: S4-based models consistently surpass convolutional and RNN baselines in macro-AUC and label-wise AUC, demonstrating robust long-range integration and sampling-rate agnosticity (Mehari et al., 2022).
• Speech recognition: Samba-ASR, using Mamba blocks for both encoder and decoder, outperforms transformer-based models in word error rate (WER), parameter efficiency, and computational speed (Shakhadri et al., 6 Jan 2025).
• MRI reconstruction/vision: MambaRecon integrates structured visual SSM blocks into image-reconstruction cascades, achieving highest PSNR/SSIM among transformer, UNet, and SSM baselines, with substantial parameter and time cost reductions (Korkmaz et al., 2024).
• Music generation: SMDIM leverages a stacked Mamba-FeedForward-Attention (MFA) block to unify near-linear cost with global music coherence, outperforming transformer baselines while maintaining fidelity over long symbol sequences (Yuan et al., 27 Jul 2025).
• In-context reinforcement learning: S4/S5-based agents excel on memory and meta-learning tasks, running 2–6$A,B,C,D$0 faster than LSTMs or transformers while sustaining reward on out-of-distribution evaluation (Lu et al., 2023).
• Automaton/algorithmic tasks: PD-SSM achieves near-perfect state-tracking, effective group emulation, and robust generalization across formal languages and permutations, surpassing both real and complex diagonal SSMs and competing with hand-tuned NCDEs (Terzić et al., 26 Sep 2025).

These empirical results reflect SSMs’ ability to trade off range, fidelity, and resource footprint in scenarios where transformers’ quadratic scaling is prohibitive or insufficient.

6. Extensions: Nonlinearities, Stability, and Control

Stacking SSM layers with nonlinear activations and skip connections forms deep Wiener cascades, extending the system identification framework to deep learning. Each structured state-space layer acts as a (complex or real) Wiener block, and stacking $A,B,C,D$1 such blocks composes a deep, stable architecture amenable to convolutional training (Bonassi et al., 2023).

Recent advances guarantee stability and robustness properties:

• L2RU incorporates bounded-real LMI constraints directly into a free parameterization, ensuring prescribed $A,B,C,D$2-gain for all input sequences and avoiding complex projection or barrier methods (Massai et al., 31 Mar 2025).
• Controller design for SSMs employs contraction theory and dual LMI-based synthesis for observer and controller gains, establishing a nonlinear separation principle and tractable synthesis of output-feedback controllers with convergence guarantees (Zakwan et al., 8 Apr 2026).

For system identification, techniques such as tensor polynomial decoupling retrieve highly structured state-space representations from overparameterized black-box models, improving interpretability and reducing dimension with marginal performance loss (Decuyper et al., 2020). Structured identification of LTI MIMO SSMs leverages rank-constrained and difference-of-convex (DCP) programming, attaining global optimality under mild assumptions (Yu et al., 2015).

7. Open Challenges and Future Directions

Despite significant progress, open challenges remain:

• Optimization and training: Certain parameterizations (e.g., real SSMs mimicking complex oscillatory kernels) are vulnerable to exploding parameter norms and precision issues. Careful initialization (HiPPO-based, eigenvalue shaping), custom learning rate schedules, and hybrid structures (e.g., selectivity) are active areas of development (Ran-Milo et al., 2024, Gu et al., 2022, Massai et al., 31 Mar 2025).
• Interpretability: The rich internal dynamics of SSMs, while theoretically tractable, often lack clear attribution for long-range memory or state-tracking in deep architectures. The emergence of primacy effects suggests nontrivial parallels to biological and psychological models of memory (Morita, 19 Feb 2025).
• Expressivity-minimality tradeoff: Ensuring maximal expressivity (e.g., universal FSA emulation) while retaining computational efficiency drives research into parameter- and compute-optimal SSM structures (Terzić et al., 26 Sep 2025).
• Hybridization: The interface between SSMs and other sequence modules (attention, convolution, NCDEs) presents design and interpretability questions for scaling, modularity, and integration in large-scale foundation models (Muñoz et al., 28 Jan 2025, Dao et al., 2024).
• Control-theoretic certification: For deployment in control and safety-critical applications, embedding formal guarantees (stability, gain bounds, separation) into trainable SSMs is an important direction, as exemplified by L2RU and contraction-based observer/controller results (Massai et al., 31 Mar 2025, Zakwan et al., 8 Apr 2026).

Future research is expected to expand on adaptive structure learning, automatic sparsification and pruning, deep system identification, and rigorous theory for learning and generalization of deep SSMs. Hybrid models integrating SSM blocks with foundation model backbones (e.g., transformers) for multimodal, long-context sequence modeling are a prominent area for both applied and theoretical advances.