Selective State-Space Dynamics

• Selective state-space dynamics is a sequence modeling method that employs learned, input-dependent gating to dynamically update state transitions.
• It distinguishes between convergent, slow-diverging, and finite-time blow-up regimes, providing a framework for analyzing model stability and performance.
• Refinements like constraining parameter matrices and employing token reordering with SoftSort yield improved language modeling perplexity and image classification accuracy.

Selective state-space dynamics describes a class of sequence modeling mechanisms where the parameters governing system evolution—typically state transition, input, and output matrices—are rendered input-dependent via learned, often nonlinear projections. This enables the dynamic selection of memory update pathways at each time step, conferring both enhanced expressiveness and computational efficiency compared to fixed-parameter state-space models (SSMs). Empirical and theoretical studies of deep selective SSMs, notably in the context of models such as Mamba, have elucidated key dynamical regimes, stability criteria, and practical modifications to optimize model fidelity across diverse tasks, including language modeling and vision.

1. Mathematical Foundations and Continuous-Time Limit

Selective SSMs are defined by discretized recurrences of the form

$$h_l = e^{Δ(x_l)A} h_{l−1} + Δ(x_l) S_B x_l, \qquad y_l = (S_C x_l)^⊤ h_l$$

where the step size $Δ(x_l)$ and readout/input matrices $S_B, S_C$ are functions of the current input, providing a selection (gating) mechanism over the state update. The continuous-time limit interprets sequence index $l$ as time $t$ and yields a system of ODEs,

$$\frac{d}{dt} x_l(t) = \sum_{j=1}^l P_{lj}(t) x_j(t), \qquad \text{with parameter-dependent dynamics:} \quad Δ(x) = \ln(1+e^{S_Δ x}), \quad \mu = S_C^⊤ S_B$$

The hidden state elimination gives rise to a lower-triangular "hidden-attention" matrix $P$, revealing intricate nonlocal dependencies among tokens in the sequence (Vo et al., 2024).

2. Asymptotic Regimes: Trichotomy and Dynamical Implications

The one-dimensional case with $D=1$ exposes a trichotomy in asymptotic dynamics determined by $\mu$ and the sign of $S_Δ x_{l0}$:

• Convergent scenario ($\mu<0$): All tokens monotonically contract to zero, with $x_l(t) = O(t^{-1/2})$. The associated kernel $P_{lj} \to 0$, leading to vanishing diversity in latent activations and gradients.
• Slow-diverging regime ($\mu>0, S_Δ x_{l0}<0~\forall l$): All tokens diverge logarithmically, as $x_l(t) = O((\ln t)^l)$. This introduces an update hierarchy—later tokens dominate, motivating reordering for training efficiency.
• Finite-time blow-up ($\mu > 0, \exists l: S_Δ x_{l0} > 0$): Selected tokens exhibit finite-time divergence, with associated entries in $P$ diverging rapidly. Parameter-based criteria delineate these regimes unambiguously: $$\begin{cases} \mu<0 & :~ \text{all tokens } \to 0 \ \mu>0,\;S_Δ\,x_{l0}<0~\forall l & :~ \text{all tokens } \to \infty\text{ (logarithmically)} \ \mu>0,~\exists\,l: S_Δ\,x_{l0}>0 & :~ \text{finite-time blow-up} \end{cases}$$ The scenario where $\mu<0$ is empirically deleterious, increasing perplexity and diminishing gradient signal (Vo et al., 2024).

3. Model Refinements and Performance Enhancement

Two refinements are proposed to mitigate suboptimal dynamical scenarios:

• Exclusion of the Convergent Regime: Constrain $S_C^⊤S_B$ to be positive definite, e.g., via LDL$^⊤$ factorization and softplus parametrization, ensuring $\mu\succ 0$.
• Token Reordering by Importance: Introduce learnable scores $s_l = \langle K, S_Δ x_l\rangle$ (with $K$ learnable), sort tokens such that high-score (fast-diverging) tokens appear later, and implement differentiable sorting (SoftSort) for gradient flow.

Empirical results confirm these interventions: On WikiText103, constraining $\mu>0$ lowers perplexity (PPL), and on ImageNet-1K, token reordering produces measurable accuracy improvements and faster convergence (Vo et al., 2024).

4. Algorithmic Workflow and Differentiable Sorting

The overall process underlying token reordering is as follows:

1. Compute importance scores $s_l = K \cdot (S_Δ x_l)$ for all tokens.
2. Use the SoftSort algorithm to generate a permutation matrix $P$, yielding reordered token sequence $\tilde{X} = P X$.
3. Feed reordered tokens into the S6 (deep selective SSM) block. This algorithm preserves differentiability, allowing end-to-end optimization (Vo et al., 2024).

5. Empirical and Theoretical Validation

Quantitative evaluation demonstrates that removing negative eigenvalues from $S_C^⊤S_B$ (ensuring $\mu>0$) reduces perplexity in language modeling tasks: | Model Setting | Perplexity (WikiText103) | |------------------|-------------------------| | Negative-only μ | 17.26 | | Mixed μ | 16.84 | | Positive-only μ | 16.71 |

For vision (ImageNet-1K with MambaVision-T), token reordering increments Top-1 and Top-5 accuracy and accelerates training. These gains corroborate the theoretical predictions and confirm the criticality of proper dynamical regime selection and token ordering for optimal performance (Vo et al., 2024).

6. Implications for Model Design and Theory

Interpreting deep selective SSMs within a continuous-time dynamical systems framework exposes a dichotomous behavior, rigidly governed by $\mu$. Excluding convergence ($\mu<0$) and aligning training with token divergence rates (reordering) are effective, architecture-agnostic design principles. These insights extend to related model classes and inform guidelines for parameter initialization, hyperparameter selection, and block composition in deep sequence models. The proposed refinements generalize across domains, yielding consistent improvements in both language and vision benchmarks (Vo et al., 2024).

7. Broader Research Context

The selective state-space framework surveyed here fits into a larger trend of replacing quadratic-complexity attention mechanisms with content-adaptive linear recursions. By dynamically selecting latent update pathways, deep selective SSMs such as Mamba secure robust asymptotic stability, expressive memory, and computational scalability, offering a viable route for sequence modeling at scale. The explicit identification of dynamical regimes and algorithmic remedies in (Vo et al., 2024) represent a foundational advance, significantly demystifying the operational principles governing these architectures.