Selective State-Space Models

• Selective State-Space Models (SSMs) are sequence models that enhance classical linear dynamics with adaptive gating to filter noise and allocate memory effectively.
• They leverage input-conditioned transition mechanisms to optimize performance in tasks like language modeling, time series forecasting, and vision applications.
• Recent developments combine mathematical rigor with hardware-efficient techniques to ensure stability, predictive sufficiency, and improved computational efficiency.

Selective State-Space Models (SSMs) are a class of sequence modeling architectures that generalize classical linear dynamical systems by introducing input-dependent, adaptive gating mechanisms into the evolution of latent states. This selectivity enables the models to dynamically allocate memory, filter relevant information, and discard spurious or non-causal noise, significantly enhancing both expressivity and computational efficiency relative to traditional models and even Transformers. Over the last years, selective SSMs—exemplified by the Mamba, S6, and their numerous extensions—have established themselves as state-of-the-art sequence backbones across language modeling, time series, vision, and resource-constrained domains.

1. Mathematical and Algorithmic Foundations

The standard state-space model is governed by discrete-time recursions: $$h_t = \bar A h_{t-1} + \bar B x_t,\quad y_t = C h_t + D x_t,$$ where $h_t\in\mathbb{R}^N$ is the hidden state, $x_t$ the input, and $y_t$ the output. In selective SSMs, all or a subset of these transition/read-in/read-out matrices ($\bar A, \bar B, C$) become functions of the current input $x_t$, introducing adaptivity through selection mechanisms. The selection/gating function typically arises from small neural networks generating parameters (e.g., $\Delta_t = \mathrm{softplus}(W_\Delta x_t)$, $B_t = W_B x_t$, $C_t = W_C x_t$), leading to time- and input-dependent updates: $$h_t = \exp(\Delta_t A) h_{t-1} + (\exp(\Delta_t A) - I)A^{-1}B_t x_t, \quad y_t = C_t h_t + D x_t.$$ These models unify continuous-time SSM ideas from control theory with data-driven, learnable selectivity gates (Dao et al., 2024, Wang et al., 5 Aug 2025). When serializing computation for inference, selective SSMs support highly optimized hardware-friendly recurrences, running in true $O(L)$ or $O(\log L)$ wall-clock time via associative scan primitives (Behrouz et al., 2024), and generalize efficiently to multi-dimensional and multi-scale decompositions (Zubić et al., 2024).

2. Principle of Predictive Sufficiency and Information-Theoretic Characterization

Recent theoretical advances have formalized the core principle underlying selective SSMs: minimal predictive sufficiency. The hidden state $h_t$ should be a minimal sufficient statistic of the past input $U_{1:t}$ for predicting the future $Y_{t+1:t+\tau}$, requiring

$$I(U_{1:t}; Y_{t+1:t+\tau}) = I(h_t; Y_{t+1:t+\tau}), \qquad I(U_{1:t}; h_t) \le I(U_{1:t}; h_t') \;\forall h_t'.$$

This criterion expresses that $h_t$ must maximize predictive power while compressing historical information, thus discarding non-causal and spurious structure (Wang et al., 5 Aug 2025). The Minimal Predictive Sufficiency SSM (MPS-SSM) operationalizes this via a composite loss: $$\mathcal{L}_{\rm Total} = \mathcal{L}_{\rm Pred} + \lambda \,\mathcal{L}_{\rm Min},$$ where $\mathcal{L}_{\rm Pred}$ enforces predictive sufficiency (through forecasting error) and $\mathcal{L}_{\rm Min}$ penalizes mutual information between $h_t$ and $U_{1:t}$ (approximated variationally). This approach is theoretically guaranteed (under suited hyperparameter settings) to produce robust, minimal sufficient summary states, with empirical and formal invariance to non-causal perturbations. Notably, ablation studies show a U-shaped performance curve as a function of the compression Lagrange multiplier $\lambda$, featuring an optimal “sweet spot” depending on data complexity and noise levels. The generalization of this regularization to other model architectures yields consistent performance gains (Wang et al., 5 Aug 2025).

3. Mechanisms of Selectivity, Memory Compression, and Theoretical Expressivity

Selective SSMs attain their efficient memory utilization by composing neural selectors and state-space dynamics. The gating mechanisms are typically parametrized by directly input-conditioned projections and nonlinearities such as softplus or sigmoid. Mathematically, the selective update at each step is either a Hadamard gate on the transition, stochastic gate composition, or an attention-style convex combination over a dictionary of transitions (Bhat, 2024, Terzić et al., 2024). Notably, the SD-SSM (Selective Dense State-Space Model) achieves universality for regular languages by maintaining a dictionary of dense transition matrices and choosing the next transition via a softmax router; this architecture can perfectly emulate any finite-state automaton and guarantees length generalization, in contrast to diagonal/selective SSMs which are intrinsically commutative and limited in expressiveness (Terzić et al., 2024).

From an information-theoretic and dynamical systems lens, selective SSMs are shown to capture non-linear sequence dependencies by approximating “path signatures” of input (iterated integrals over time) (Cirone et al., 2024), thus exceeding the linear representational capacity of fixed-parameter state space models (e.g., S4). Generalization error analyses further relate length and stability properties to the spectral abscissa of the (possibly input-dependent) state transition maps; stability ($\mathrm{max}\;\mathrm{Re}\;\lambda(A) < 0$) is essential to avoid unbounded error with increasing sequence length (Honarpisheh et al., 3 Feb 2025).

4. Hardware Efficiency, Quantization, and Compression

Due to their linear sequence-level complexity, selective SSMs offer substantial efficiency advantages over self-attention Transformers, especially for long sequences or on resource-constrained hardware. Detailed profiling reveals the per-step SSM recurrence dominates inference latency and memory usage (Asif et al., 28 Nov 2025). Hardware-aware optimizations, including quantization (Quamba (Chiang et al., 2024), Quamba2 (Chiang et al., 28 Mar 2025)), exploit the structure of SSMs by channel-order preserving and input clustering approaches, yielding robust performance under aggressive 8-bit or mixed-precision post-training quantization. Empirical results include up to 3× generation speedup and 4× memory reduction with ≤2% drop in accuracy, and allow practical deployment on edge devices and cloud-scale inference (Chiang et al., 28 Mar 2025, Chiang et al., 2024, Mandal, 10 Feb 2026).

Model pruning and component-level compression (e.g., Mamba-Shedder (Muñoz et al., 28 Jan 2025), PerfMamba (Asif et al., 28 Nov 2025)) enable removal of low-activity state channels or entire blocks, delivering 10–40% memory and speed improvements with negligible performance loss in the safe-prune regime. Recovery fine-tuning post-pruning can restore most of the lost accuracy (Muñoz et al., 28 Jan 2025).

5. Applications and Empirical Performance

Selective SSMs have achieved state-of-the-art or near-SOTA results on long-term time series forecasting, large language modeling, vision classification/detection/segmentation, and recommendation systems (Liu et al., 2024, Behrouz et al., 2024, Wang et al., 5 Aug 2025, Zubić et al., 2024). Notable findings:

• MPS-SSM outperforms prior SSMs and Transformers in long-horizon forecasting and demonstrates 3× robustness gain against injected noise for large $\lambda$ (Wang et al., 5 Aug 2025).
• Vision models like ViM2 and time series models like TSM2 leverage dual token and channel selection for improved accuracy and compute efficiency versus prior SSMs and Transformers (Behrouz et al., 2024).
• GG-SSM generalizes the scan operation to dynamically constructed graphs (MSTs), substantially improving representational power and sample efficiency in computer vision and non-local interaction domains (Zubić et al., 2024).
• On resource-constrained tasks (e.g., TinyML human activity recognition), lightweight Mamba-inspired SSMs match or exceed competitive baselines with an order-of-magnitude lower energy and parameter count (Mandal, 10 Feb 2026).

Below is a summary table reporting representative empirical gains:

Task / Dataset	Model	Metric	SSM / SSM Variant	Best Baseline	Gain
Long-term Forecasting	MPS-SSM	MSE (ETTm2-720)	0.358	0.385	Lower error
ImageNet-1K	ViM2-S	Top-1 (%)	83.7	81.8 (DeiT-B)	Higher accuracy
Eye Tracking	GG-SSM	p₁₀ (%)	99.50	99.30	Higher, fewer params
TinyML HAR (Opportunity)	BabyMamba	Macro F1 (%)	88.3	86.16	Higher, 11× less MACs

6. Stability, Memory Control, and Design Regularities

Stability and well-posedness in selective SSMs, especially under discontinuous gating, invoke advanced control-theoretic tools such as quadratic storage functions, parametric LMIs, and ISS arguments (Zubić et al., 16 May 2025). Exponential memory forgetting is certified under uniform local dissipativity, and design constraints (e.g., keeping all gating transitions within analytically controlled regions) are essential for reliable learning and deployment. Irreversible forgetting is formalized via monotone increase of the kernel of the quadratic storage, structurally removing unobservable modes as a consequence of passivity (Zubić et al., 16 May 2025).

From the information-theoretic perspective, selective SSMs facilitate explicit rate-distortion and information bottleneck tradeoffs, enabling theoretical prediction of minimal hidden state dimension and compression-induced error bounds (Bhat, 2024, Wang et al., 5 Aug 2025).

7. Extensions, Limitations, and Future Directions

Extensible selection mechanisms have been proposed, such as:

• Residual SSMs (multiple LTI filters with gating inspired by control-fault detection) to overcome the limitations of static selectors and increase dynamical selectivity for higher-order temporal triggers (Casti et al., 23 May 2025).
• Graph-generating SSMs for data-adaptive, sparse, non-local propagation (Zubić et al., 2024).
• Dual token/channel selection and dynamic, input-driven selective pruning (Behrouz et al., 2024, Asif et al., 28 Nov 2025).
• Regularization frameworks that generalize MPS-SSM’s predictive sufficiency principle to other architectures, including Transformers and linear models (Wang et al., 5 Aug 2025).

Current limitations include reduced expressive power of diagonal or commutative SSMs in non-commutative state tasks (Terzić et al., 2024), the need for careful stability control at depth and under gating discontinuities (Zubić et al., 16 May 2025), and the challenge of maintaining efficient and robust selection in highly non-stationary or adversarial settings.

Ongoing directions include hybrid SSM-attention stacks, further hardware-specialized kernels, theoretical extensions to nested state hierarchies, hard selection via annealed Gumbel routers, and robust adaptive gating for highly structured and non-Euclidean data (Dao et al., 2024, Zubić et al., 2024, Wang et al., 5 Aug 2025).