---
title: Selective State-Space Models
url: https://www.emergentmind.com/topics/selective-state-space-models
type: topic
---

# Selective State-Space Models

Selective state-space models (selective SSMs) are recurrent state-space architectures whose effective state-update, input-injection, and readout operators depend on the current input. In contrast to linear time-invariant (LTI) SSMs, which apply a fixed dynamical system at every position and are equivalent to convolution with an input-independent kernel, selective SSMs implement input-controlled linear dynamical systems. Their recurrence remains linear in the hidden state, preserving recurrent scanability and fixed-size inference memory, but input-dependent parameters create multiplicative interactions between current tokens and historical state. Mamba, GateLoop, gated linear attention (GLA), and related architectures are prominent instances. Theoretical analyses connect their hidden states to signature features, attention-like factorizations, adaptive temporal discretization, and content-dependent memory compression [2402.19047] [2412.11211].

## 1. State-space formulation and selectivity

A continuous-time linear SSM is commonly written as

$$
\frac{d h(t)}{dt}=A h(t)+B x(t),\qquad y(t)=C h(t),
$$

where $x(t)$ is the input, $h(t)$ is the latent state, $y(t)$ is the output, and $A$, $B$, and $C$ govern state transition, input injection, and readout. A direct input-output term $D x(t)$ may also be included. Under zero-order-hold discretization,

$$
\overline{A}=\exp(\Delta A),
$$

and

$$
\overline{B}=(\Delta A)^{-1}\left(\exp(\Delta A)-I\right)\Delta B.
$$

The resulting recurrence is

$$
h_t=\overline{A}h_{t-1}+\overline{B}x_t,\qquad y_t=C h_t.
$$

When $\overline{A}$, $\overline{B}$, and $C$ are fixed across positions, the system is LTI and can be expressed as convolution:

$$
y=x*\overline{K},
$$

with kernel

$$
\overline{K}=
\left(
C\overline{B},
C\overline{A}\overline{B},
\dots,
C\overline{A}^{L-1}\overline{B}
\right).
$$

This recurrence–convolution equivalence underlies LSSL, S4, and related structured SSMs. HiPPO initialization supplies structured long-range memory, while S4 uses diagonal-plus-low-rank state matrices, generating functions, resolvent evaluation, and FFT-based convolution. S5 uses a multi-input, multi-output state space and parallel prefix scans [2412.11211].

Selective SSMs remove the strict LTI assumption by making selected parameters functions of the current input:

$$
\overline{A}_t=\overline{A}(x_t),\qquad
\overline{B}_t=\overline{B}(x_t),\qquad
C_t=C(x_t).
$$

In the Mamba parameterization, the continuous transition matrix $A$ is generally structured and input-independent, while the discretization step, input projection, and output projection are input-dependent:

$$
\Delta_t=f_\Delta(x_t),\qquad
B_t=f_B(x_t),\qquad
C_t=f_C(x_t).
$$

The effective transition is

$$
\overline{A}_t=\exp(\Delta_t A),
$$

and a selective recurrence takes the form

$$
h_t=\overline{A}_t h_{t-1}+\overline{B}_t x_t,\qquad
y_t=C_t h_t.
$$

A common step-size parameterization is

$$
\Delta_t=\operatorname{softplus}(c+\operatorname{Linear}(x_t)),
$$

where

$$
\operatorname{softplus}(u)=\log(1+\exp(u)).
$$

The positive step size controls the effective temporal scale: small values preserve more of the preceding state, whereas large values advance the state more aggressively. $B_t$ controls how the current input enters memory, and $C_t$ controls which state components are exposed. Selectivity therefore implements content-dependent writing, retention, forgetting, and reading without making the recurrence nonlinear in the previous state.

The distinction is important. Selective SSMs are neither classical linear RNNs nor standard nonlinear RNNs: they are linear in $h_{t-1}$ but time-varying because their transition operators depend on $x_t$. The unrolled recurrence contains products of input-dependent matrices,

$$
h_L=
A(x_L)\cdots A(x_1)h_0+
\sum_{\ell=1}^{L}
A(x_L)\cdots A(x_{\ell+1})B(x_\ell)x_\ell.
$$

Consequently, the effect of an earlier token depends jointly on later tokens. Products of transition factors generate ordered higher-order interactions such as

$$
x_{r_1}^{i_1}x_{r_2}^{i_2}\cdots x_{r_k}^{i_k},
\qquad
r_1<r_2<\cdots<r_k,
$$

even though the state recurrence itself is linear.

## 2. Expressivity, signatures, and transition algebra

Theoretical work characterizes selective SSMs as linear controlled differential equations (CDEs). A general gated linear CDE is

$$
dZ_t=\sum_{i=1}^{d_\omega}A_i Z_t\,d\omega_t^i+B\,d\xi_t,
\qquad
Z_0=CX_0,
$$

where $\omega$ and $\xi$ are input-dependent control paths. Classical S4 is recovered with a fixed clock $\omega_t=t$ and an integrated input path $\xi_t=\int_0^t X_s\,ds$. In selective systems, the input controls the clock itself; for example,

$$
d\omega_t=\sigma(\widetilde{\alpha}X_t+\widetilde{\beta})\,dt,
$$

and

$$
d\xi_t=\sigma(\widetilde{\alpha}X_t+\widetilde{\beta})X_t\,dt.
$$

The hidden state can be expanded in terms of the path signature. For a path $\gamma$, a signature coordinate indexed by a word $I=i_1\cdots i_k$ is the ordered iterated integral

$$
\operatorname{Sig}(\gamma)_{s,t}^{(I)}
=
\int_{s<u_1<\cdots<u_k<t}
d\gamma_{u_1}^{i_1}\cdots d\gamma_{u_k}^{i_k}.
$$

The full signature is an infinite tensor series containing iterated integrals at every order. Its truncated level-$M$ representation has

$$
1+d+d^2+\cdots+d^M
$$

coordinates, which grows exponentially with $M$. Signature terms exhibit factorial decay:

$$
\left\|
\int_{s<u_1<\cdots<u_k<t}
d\gamma_{u_1}\otimes\cdots\otimes d\gamma_{u_k}
\right\|
\leq
\frac{\|\gamma\|_{1\text{-var},[s,t]}^k}{k!}.
$$

The hidden state of a linear controlled system is a finite-dimensional projection of these signature features. Its ordered exponential has the form

$$
W_{s,t}
=
\sum_I A_I\operatorname{Sig}(\omega)_{s,t}^{(I)},
$$

where $A_I$ is the ordered product of transition matrices associated with the word $I$. Variation of constants then expresses $Z_t$ as a linear map of signature coordinates and integrals involving $\xi$.

Under compactness, continuity, and suitable time augmentation, dense input-controlled recurrences are universal in a closure sense for functionals of the form

$$
\Psi(\omega_{[0,t]})\cdot X_0+
\int_0^t\Phi(\omega_{[s,t]})\cdot d\xi_s.
$$

This is an approximation theorem: a finite state does not exactly store the entire infinite signature. Exact representation of all signature words through level $M$ requires a state dimension on the order of

$$
\sum_{k=0}^{M}d^k.
$$

A probabilistic result further states that randomly initialized dense recurrences can, with probability tending to one as the state dimension grows, provide sufficiently rich random projections of signature features so that only a final linear readout needs to be trained. This connects selective SSMs with reservoir computing [2402.19047].

Transition structure determines which signature information is accessible. Dense matrices need not commute, so ordered path interactions and noncommutative products can be represented. Diagonal transitions commute:

$$
A_{i_k}\cdots A_{i_1}
=
A_{i_{\sigma(k)}}\cdots A_{i_{\sigma(1)}}.
$$

They therefore identify signature coordinates that differ only by permutations and retain symmetric increment statistics rather than the full ordered signature. This limitation is less consequential for a one-dimensional fixed clock, where the signature reduces to powers of the interval,

$$
\operatorname{Sig}(\omega)_{s,t}^{(k)}
=
\frac{(t-s)^k}{k!},
$$

but becomes significant for multidimensional input-controlled paths.

Depth and pointwise mixing can partially compensate for diagonalization. A chain of approximately $|I|-1$ diagonal layers can approximate a signature term of word length $|I|$. Thus, dense transitions obtain higher-order noncommutative interactions within one layer, whereas diagonal systems can construct increasingly complex interactions progressively [2402.19047].

A related analysis of regular languages distinguishes representability from learnability. A finite-state automaton can be embedded exactly in a sufficiently general selective SSM by assigning one dense transition matrix to each input symbol:

$$
x_{t+1}=A(u_t)x_t.
$$

However, diagonal transition families are simultaneously diagonalizable only under restrictive conditions, and their multiplicative transitions commute. Under the natural automaton-to-SSM mapping, a single-layer diagonal selective SSM can therefore emulate only commutative automata. The Selective Dense State-Space Model (SD-SSM) addresses this restriction with a dictionary of dense matrices:

$$
A(u_t)
=
\operatorname{OpNorm}
\left(
\sum_{i=1}^{k}
\operatorname{softmax}(Su_t)[i]A_i
\right).
$$

On the reported regular-language tasks, SD-SSM achieved at least $99.9\%$ accuracy over evaluated lengths up to $500$ while training on lengths up to $40$, including noncommutative automata such as $D_4$ and $A_5$. This is an empirical result on the tested tasks, not a universal guarantee of arbitrary-length regular-language generalization [2412.19350].

## 3. Stability, memory compression, and generalization

Selective SSMs compress a sequence prefix into a fixed-dimensional state. This compression creates a trade-off between information retention and computational efficiency. A proposed selective-memory recurrence is

$$
h_t=
G(x_t,h_{t-1})\odot(Ah_{t-1}+Bx_t)
+
\bigl(1-G(x_t,h_{t-1})\bigr)\odot h_{t-1}
+w_t,
$$

where

$$
G:\mathbb{R}^{m}\times\mathbb{R}^{d}\rightarrow[0,1]^d.
$$

A small gate preserves the old state, while a large gate allows the candidate update to overwrite or modify it. The effective dimensionality

$$
\dim_{\mathrm{eff}}(h_t)=\sum_{i=1}^{d}G_i(x_t)
$$

is a weighted measure of how many coordinates are actively updated; for soft gates it is not literally a vector-space dimension.

Information-theoretic formulations describe compression through mutual information, rate-distortion, and information bottleneck objectives. Examples include

$$
\min_{\dim(h_t)}\dim(h_t)
\quad
\text{subject to}
\quad
I(h_t;x_{1:T})\geq\tau,
$$

and

$$
\min I(z;h_t)
\quad
\text{subject to}
\quad
I(z;x_{1:T})\geq\tau.
$$

These formulations express the principle that irrelevant or redundant information should be removed while task-relevant information is retained. The associated theoretical analysis is conditional: gate computation itself has a cost, soft gates do not necessarily yield sparse computation, and the stated convergence arguments require stronger boundedness and coupling assumptions than the basic exposition provides [2410.03158].

Stability is governed by the transition spectrum, discretization, gate behavior, and input magnitude. For a continuous-time state matrix $A_c$, the spectral abscissa is

$$
s_{A_c}=\max_i\{\Re(\lambda_i(A_c))\}.
$$

The regimes are:

- $s_{A_c}<0$: exponential decay of the homogeneous state;
- $s_{A_c}=0$: marginal behavior and polynomial accumulation;
- $s_{A_c}>0$: exponential growth of states and perturbations.

A generalization analysis for selective SSMs derives a covering-number bound whose sequence-length dependence is controlled by this spectral abscissa. In the stable regime, the stated bound is independent of sequence length $T$. At the marginal boundary, it grows polynomially, while in the unstable regime it grows exponentially. The analysis also gives a lower bound showing that length dependence cannot generally be removed when $s_{A_c}\geq 0$ [2502.01473].

The same analysis emphasizes that length-independent generalization does not imply length-independent accuracy. Longer sequences may contain more task-relevant information, as in IMDb sentiment classification, even when the train–test gap remains stable. Conversely, an unstable initialization can cause losses and gradients to become so large on long sequences that optimization fails before the model reaches a stable regime.

Token-dynamics analysis provides a complementary view. In an idealized one-dimensional continuous-depth S6 model, the input-output coupling

$$
\mu=S_C^\top S_B
$$

determines asymptotic behavior. If $\mu<0$, nonzero tokens converge toward zero, with

$$
x_l(t)=O(t^{-1/2}),
\qquad
P_{lj}(t)=O(t^{-1}),
$$

where $P_{lj}$ are hidden-attention coefficients. If $\mu>0$ and the softplus arguments are initially negative, tokens diverge at infinite time; under stronger ordering conditions,

$$
x_l(t)=O((\log t)^l).
$$

If $\mu>0$ and some initial step-size argument is positive, finite-time blow-up occurs in the idealized continuous system. These theorems apply to a simplified one-dimensional model that omits major Mamba components such as normalization, convolution, projections, and gating. They do not establish identical behavior for fully implemented Mamba networks [2410.03292].

## 4. Computational mechanisms and architectural variants

The central computational advantage of selective SSMs is linear scaling in sequence length. Attention explicitly forms token-to-token interactions and conventionally requires quadratic time and attention-memory growth. A selective recurrence instead maintains a fixed-size state and applies input-dependent transitions through a scan. The affine recurrence

$$
h_t=A_t h_{t-1}+b_t
$$

can be represented as an affine transformation. Composition is associative:

$$
(A_2,b_2)\circ(A_1,b_1)
=
(A_2A_1,\;A_2b_1+b_2).
$$

This permits parallel prefix scans during training while preserving stepwise recurrent execution during inference. Hardware-aware implementations fuse parameter generation, discretization, scanning, and memory movement, using kernel fusion, SRAM-resident intermediates, recomputation, and selective materialization.

The computational characteristics of common variants are as follows.

| Variant | Principal transition structure | Main computational mechanism |
|---|---|---|
| S4 | Diagonal-plus-low-rank, fixed | Generating functions, resolvents, FFT convolution |
| S5 | Diagonal, fixed, MIMO | Parallel prefix scan |
| Mamba/S6 | Structured, input-dependent | Selective scan |
| SD-SSM | Dense dictionary with input-dependent selection | Dense recurrence and parallel scan |

Mamba’s canonical block typically includes input and output projections, a short causal convolution, SiLU nonlinearities, an SSM branch, an input-derived multiplicative gate, and an output projection. Mamba’s exact block differs across implementations and papers; some application papers specify only the selective recurrence and do not document the complete internal parameterization.

Several architectures extend selectivity across additional dimensions. MambaMixer introduces a Selective Token Mixer and a Selective Channel Mixer. The token mixer scans along the sequence dimension, while the channel mixer transposes the representation and scans across channels, generally bidirectionally. It also uses learned weighted averages to give later layers direct access to early token and channel features. Its stated complexity is linear in both sequence length and channel count, summarized as

$$
\mathcal{O}\left(EB(N+L)\right).
$$

Vision MambaMixer uses multidirectional scans over image patches, hierarchical downsampling, depth-wise convolution, and selective channel mixing. Time Series MambaMixer treats time points as tokens and variates as channels, using a unidirectional time mixer and bidirectional variate mixer. Experiments on ImageNet, ADE20K, COCO, and multivariate forecasting report improvements over corresponding SSM and MLP alternatives, although the reported results are architecture- and task-specific [2403.19888].

SeRpEnt extends selectivity from state updates to sequence resampling. It interprets the learned interval $\Delta_l$ as both an effective sampling interval and a local proxy for input influence. The proposition

$$
\|h_L-h_L^i\|\sim c\Delta_i,
\qquad
\Delta_i\to0,
$$

is a local asymptotic result for diagonal state matrices and single-element removal. SeRpEnt constructs nonuniform times

$$
t_l=\sum_{i=0}^{l}\Delta_i,
$$

then resamples the input onto a shorter uniform grid by learned interpolation over nearby points. Multiple branches with different compression rates are reverse-resampled and combined through a residual path. Experiments improve several LRA text tasks and WikiText-103 results, but not the image-oriented LRA tasks, and no complete end-to-end runtime analysis is provided [2501.11729].

Mamba-Shedder demonstrates that selective SSMs themselves contain structured redundancy. Training-free pruning can remove entire Mamba blocks, S6 or SSD modules, Transformer subblocks in hybrids, and groups of MLP channels. Sensitivity is architecture-dependent: Mamba-1 is relatively tolerant of whole-block removal but sensitive to S6 removal, whereas Mamba-2 is more tolerant of SSD-module removal. In Zamba2, combining fine-grained pruning with recovery tuning produced decoding speedups up to approximately $1.39\times$ in the reported settings. These results concern module-level compression, not pruning of individual selective parameters or state dimensions [2501.17088].

## 5. Applications across sequential and multimodal systems

Selective SSMs have been applied to recommendation, speech, gesture synthesis, audio processing, graphs, vision-language models, and generative audio.

In sequential recommendation, Mamba4Rec models a chronologically ordered interaction history and generates input-dependent $\boldsymbol{B}$, $\boldsymbol{C}$, and $\Delta$ parameters. Its Mamba block combines input projection, local one-dimensional convolution, SiLU, selective recurrence, a SiLU gating branch, and output projection. On MovieLens-1M, Mamba4Rec reported lower GPU memory, shorter training time per epoch, and lower inference time than SASRec and BERT4Rec under the reported implementation. Across MovieLens-1M, Amazon-Beauty, and Amazon-Video-Games, it obtained the best listed metric except Beauty HR@10, where SASRec was higher. These results are empirical and depend on the datasets, hardware, sequence lengths, and implementation settings [2403.03900].

MambaTalk applies selective SSMs to holistic speech-driven gesture synthesis. Direct selective-scan prediction of continuous gestures produced severe jitter, so the system first trains a VQ-VAE over body and facial motion and then predicts discrete motion latents. Four independently parameterized Mamba branches model facial, upper-body, lower-body, and hand motion. The system uses audio and FastText text features, separate audio encoders, VQ decoders, and a pretrained global-motion predictor. On the BEATX-standard dataset, MambaTalk achieved the best reported Beat Constancy, facial MSE, and Learned Vertex Distance, while EMAGE achieved the best FGD and diversity. The discrete motion prior is essential to the complete system; the reported improvements cannot be attributed to Mamba alone [2403.09471].

Dual-path Mamba replaces recurrent LSTMs or Transformer blocks in time-domain speech separation. Encoded waveform features are divided into overlapping chunks, and bidirectional Mamba units process both intra-chunk and inter-chunk axes. This separates short-term from long-term dependencies while avoiding full attention over the complete encoded sequence. On WSJ0-2mix, DPMamba-M achieved $22.6$ dB SI-SNR improvement and $22.7$ dB SDR improvement with $15.9$ million parameters, exceeding the reported SepFormer values with fewer parameters. The model is primarily suited to offline separation because of its bidirectional processing [2403.18257].

Selective SSMs have also been used for causal black-box modeling of optical compressors. The reported architecture combines a 64-sample input buffer, linear projection, S6 block, FiLM and Temporal FiLM conditioning, GLU-like gating, a second S6 block, and a sample-level gain output

$$
y_n=g_nx_n.
$$

The model uses approximately $1{,}000$ trainable parameters and an explicit latency of approximately $1.33$ ms at $48$ kHz. On the TubeTech CL 1B and Teletronix LA-2A, S6 outperformed the tested LSTM, S4D, ED, and TCN alternatives under the reported objective and subjective evaluations. The study does not isolate S6 from the surrounding FiLM, GLU, convolutional, spectral, and two-block architecture [2408.12549].

MambaFoley places a bidirectional Mamba bottleneck inside a raw-waveform diffusion U-Net. The model retains diffusion, convolutional multiscale processing, categorical FiLM, and temporal BFiLM conditioning. On the DCASE 2023 Task 7 Foley Sound Synthesis dataset, MambaFoley achieved the best FAD under three embedding extractors and the highest overall-quality MOS among the compared models. Temporal-fidelity scores were similar across models, supporting the interpretation that BFiLM primarily controls temporal-envelope alignment while the bottleneck sequence model affects timbral and waveform quality [2409.09162].

DG-Mamba uses selective SSMs as graph-conditioned temporal memory. Kernelized message passing first learns intra-snapshot and inter-snapshot structures. The learned inter-snapshot adjacency then modifies the input-dependent discretization step before a selective scan propagates information across graph snapshots. Its dominant complexity is reported as approximately

$$
\mathcal{O}\left(T(|\mathcal{V}|+|\mathcal{E}|)\right),
$$

rather than quadratic temporal or spatial attention. The model combines this mechanism with Principle of Relevant Information regularization, which suppresses redundant and noisy graph structures. Experiments on COLLAB, Yelp, and ACT report robustness against adversarial attacks and near-linear scaling in the tested settings [2412.08160].

State Space Memory Integration (SSMI) inserts lightweight Mamba-based modules into a pretrained large vision-language model between multi-head self-attention and the feed-forward network. The paper reports updating approximately $0.5\%$ of total LVLM parameters and improvements over full fine-tuning, adapters, LoRA, and VPT on COCO captioning, VQA, and Flickr30k retrieval. However, the SSMI paper does not specify the actual selective $\Delta$, $B$, and $C$ parameterization or selective-scan implementation. Its formal equations describe a generic SSM, so its connection to selective SSMs is architectural rather than fully documented mathematically [2412.09875].

## 6. Interpretation, explainability, limitations, and research directions

Selective SSMs can be interpreted as adaptive recurrent filters. The input controls the effective clock, state transition, state writing, and state readout. This yields a computational compromise between fixed SSMs and attention: unlike S4-like systems, selective SSMs can alter their dynamics according to content; unlike full attention, they compress history into a fixed-size state and avoid explicit all-pairs interactions.

The relationship to attention is especially clear after unrolling the recurrence. The current output projection can act as a query, an input projection as a key, and the historical input as a value, while state-transition products and step sizes modulate contributions from earlier positions. Under special parameter settings, the selective recurrence reduces to causal linear attention. This is an attention-like factorization, not an assertion that general selective SSMs are equivalent to Transformers [2502.01473].

The principal trade-offs are:

- **Efficiency versus explicit retrieval**: selective SSMs have linear sequence scaling and compact recurrent inference memory, but cannot generally preserve every pairwise relation exactly.
- **Selectivity versus convolutional simplicity**: input-dependent parameters remove the single fixed convolution kernel, requiring scan-based implementations.
- **Diagonal efficiency versus ordered expressivity**: diagonal transitions are inexpensive but commute; dense transitions preserve noncommutative interactions at higher per-step cost.
- **Memory retention versus stability**: small effective steps preserve state, but unstable spectra or excessive accumulation can cause exploding states, gradients, or outputs.
- **Compression versus task information**: reducing state updates or resampling the sequence can remove redundancy, but may also discard information needed for prediction.
- **Depth versus transition structure**: stacking diagonal blocks can recover some higher-order interactions, but depth does not automatically overcome an unsuitable transition algebra.

Interpretability requires model-specific treatment. Standard Gradient $\times$ Input and naive LRP can violate relevance conservation in Mamba because SiLU, input-dependent SSM parameters, and multiplicative gates introduce additional attribution pathways. MambaLRP treats realized selective coefficients as fixed during backward propagation, detaches the coefficient-generating paths, and distributes multiplicative-gate relevance equally between branches. It reports improved conservation and perturbation faithfulness over tested baselines, while also revealing recency bias, spurious image cues, gender associations, and misleading evidence in retrieval tasks [2406.07592].

The principal theoretical and methodological limitations remain substantial. Expressivity results are often approximation theorems or apply to idealized systems. Stability analyses depend on spectral, boundedness, regularity, or compactness assumptions. The strongest token-dynamics results are one-dimensional and omit much of the implemented Mamba block. Diagonal models lose order-sensitive information unless depth, mixing, input injection, or other mechanisms restore it. Dense selective models incur higher state-update cost. Claims of perfect length generalization apply only to evaluated automata and sequence ranges. Reported application results frequently combine selective SSMs with domain-specific components, making causal attribution to selectivity difficult.

Several implementation questions remain open: the exact interaction between realistic Mamba gates and signature recovery; sharp finite-width bounds for random projections; principled integration of physical elapsed time with learned step sizes; higher-dimensional stability criteria for nonsymmetric and complex transition families; efficient dense selective transitions; state compression with task-relevant rather than input-history information objectives; and direct comparisons under matched hardware, parameter, memory, and latency budgets.

The general progression is

$$
\text{fixed continuous SSM}
\longrightarrow
\text{structured LTI recurrence}
\longrightarrow
\text{input-controlled dynamics}
\longrightarrow
\text{signature-like memory}
\longrightarrow
\text{adaptive compression and scan}.
$$

Selective state-space models preserve the recurrent and structured advantages of SSMs while introducing content-dependent temporal dynamics. Their effectiveness depends not only on selectivity itself, but also on transition algebra, spectral stability, state dimension, depth, normalization, scan implementation, and the requirements of the target task.

Source: https://www.emergentmind.com/topics/selective-state-space-models