Pre-trained SSM Models: Efficient Sequence Modeling

• Pre-trained SSM Models are neural sequence models leveraging state-space theory to capture long-range dependencies with high computational efficiency.
• They employ mathematically principled dynamic operators, such as HiPPO-based projections and low-rank parameterizations, to ensure stable long-term memory.
• These models are pre-trained on large unsupervised corpora and fine-tuned for diverse tasks including NLP, time-series forecasting, speech, and code understanding.

Pre-trained Structured State Space Models (SSMs) are a class of neural sequence models that utilize state-space theory to capture long-range dependencies in sequential data with high computational efficiency. Pre-trained SSMs—such as S4, Mamba, and derivatives like ES-SSM, Time-SSM, and CodeSSM—are typically initialized and optimized on large-scale unsupervised objectives, analogous to Transformers, and subsequently fine-tuned to a wide spectrum of downstream tasks. These models have established themselves as both rivals and complements to attention-based architectures in NLP, time-series forecasting, code understanding, vision, and speech domains. Their defining properties are mathematically principled dynamic operators, convolutional or recurrent parameterizations for stable long-term memory, and, depending on the variant, dynamic adaptivity via input-driven gating.

1. Mathematical and Algorithmic Foundations of SSMs

Pre-trained SSM models are founded on continuous- and discrete-time dynamical systems represented by:

$$\dot h(t) = A h(t) + B x(t), \quad y(t) = C h(t) + D x(t)$$

where $h(t)$ is the latent state, $x(t)$ is the input, and $y(t)$ is the output. Discretization yields:

$$h_k = \overline{A} h_{k-1} + \overline{B} x_k,\quad y_k = C h_k$$

Structured SSMs such as S4 introduce HiPPO-based projections (e.g., Legendre polynomial basis), and efficient diagonal plus low-rank (DPLR) parameterizations:

• $A = \Lambda - P Q^\ast$, with $\Lambda$ diagonal (or block diagonal) and $P, Q$ low-rank.
• The convolutional kernel view: $y = K * x$, with $K_n = C A^n B$.
• Fast evaluation via FFT and Woodbury identities reduces complexity to nearly linear in sequence length.

Selective SSMs (e.g., Mamba) extend this with token-wise input-dependent parameters $(A_k, B_k, C_k)$, thus supporting non-linearity and adaptivity. Mamba2 further abstracts the scan-based computation into semiseparable matrices $M$, enabling Transformer-like batched computation.

For foundation SSMs designed for efficient scale and deployment, as in ES-SSM, spectral decompositions using the Hankel matrix and associated eigenmodes $(\sigma_k, \phi_k)$ concentrate the modeling power in the leading modes, enabling post hoc truncation for runtime adaptation (Song et al., 30 Jan 2026).

2. Architectural Innovations for Pre-training

The transition from early SSMs to practical, pre-trainable architectures centers on two axes: parallelization and memory retention.

• S4 architecture: HiPPO-legS block provides orthogonal polynomial-based recurrence for long-term stationarity and supports parallel, kernelized sequence processing (Lv et al., 14 Mar 2025).
• Mamba/S6: Input-selective parameters ($A_k,B_k,C_k$ from a learned sub-network per token) implement dynamic recurrence, acting analogously to sparse, learnable attention. Hardware-aware parallel scan and semiseparable kernels recover much of the transformer’s throughput (Lv et al., 14 Mar 2025).
• ES-SSM: Introduces a lightweight input-adaptive gate and budget dropout, allowing a single full-capacity checkpoint to be elastically truncated at inference time with graceful performance degradation and without retraining. Predictive capacity is explicitly concentrated into the lowest-index (leading spectral) modes (Song et al., 30 Jan 2026).
• Time-SSM: Factorizes SSMs in real and complex planes, leveraging shared or parameterizeable matrices, minimalist block architectures, and multi-scale spectral operators to achieve high performance with $\sim$1/7-th the parameters of benchmark transformer-like SSMs (Hu et al., 2024).

3. Pre-training Protocols and Methodologies

The canonical self-supervised SSM pre-training workflow is closely patterned after transformer routines:

• Objective: Next-token prediction (causal LM) or masked-token modeling.
• Corpora: Web-scale text (Wikipedia, BookCorpus, C4), code (StarCoder), and, for Time-SSM, patchified multi-domain time series.
• Optimization: AdamW; typical LR $10^{-4}$, with warm-up and cosine decay; weight decay and dropout for regularization.
• Batch and scale: Per-GPU batch 8–256; aggregate global batch up to 2K; 200K–1M steps; 8–64 GPUs.
• Parameter regimes: Pre-trained SSMs at 20M–220M parameters for foundation models, scaling up to 10–50M for ES-SSM with full-budget truncation; S4 and Mamba at 10M–100M (practically up to 1B+).
• Specializations: ES-SSM budget dropout—randomized $K$ modes per step; Time-SSM multi-horizon supervision; CodeSSM BERT-style masked code modeling (Wu et al., 6 Feb 2026, Hu et al., 2024, Song et al., 30 Jan 2026).

4. Empirical Performance and Application Domains

Pre-trained SSMs are evaluated across a broad range of benchmarks:

• Language modeling: S4 and Mamba attain perplexities rivaling LSTMs and small transformers; Mamba-based models achieve end-to-end speedups exceeding 20x in long-sequence regime (Lv et al., 14 Mar 2025).
• Long Range Arena (LRA): DSS and S4 outperform linear/sparse transformer baselines on sequence recall tasks (Lv et al., 14 Mar 2025).
• Speech/audio: SSMs (esp. structured SSMs) outperform attention models in SI-SDR and streaming efficiency.
• Time-series forecasting: Time-SSM achieves state-of-the-art mean squared error (MSE) and mean absolute error (MAE) on 31/64 ETS/crypto/weather horizons using only ~1M parameters (versus ~7M for Mamba4TS) (Hu et al., 2024).
• Code understanding: Pre-trained CodeSSM exceeds transformers on direct AST and data-flow probes and maintains a better hierarchy of long- and short-range syntactic knowledge. Fine-tuning on localized tasks sometimes induces a spectral shift, weakening global dependency retention; architectural corrections (parallel local CNN path, multi-kernel SSM) mitigate this (Wu et al., 6 Feb 2026).
• Budgeted inference: ES-SSM delivers monotonic accuracy-vs-budget curves, e.g., on PG19 and Speech Commands V2 datasets, with >98% retention at 10–20% spectral channel utilization and no retraining (Song et al., 30 Jan 2026).

5. Interpretability, Spectral Analysis, and Model Comparison

Advanced interpretability for pre-trained SSMs is pioneered in frequency-domain diagnostics:

• SSM-Interpret: Discrete Fourier analysis of SSM convolution kernels reveals layerwise specialization—pre-trained SSMs typically form complementary pairs of low-pass (global) and high-pass (local) filters, while fine-tuning can induce undesirable bias toward high-frequency structures (Wu et al., 6 Feb 2026).
• Ablations: Kernel variant selection, time-varying $B$ parameterization, and unitary transforms in SSMs critically impact long-range context retention and noise robustness (Hu et al., 2024).
• Comparison with Transformers: Across domains, SSMs match or surpass attention models with two- to ten-fold higher throughput, better scaling for long sequences, and less memory requirement, particularly when global, stationary dependencies dominate (Lv et al., 14 Mar 2025, Hu et al., 2024, Wu et al., 6 Feb 2026).

Model	Core Mechanism	Notable Pre-training Properties	Downstream Strengths
S4	HiPPO, DPLR kernel	Linear, HiPPO initialization, FFT-based	Long stationary sequences, low-compute
Mamba	Input-selective SSM	Tokenwise dynamic parameters, SRAM scan	Nonlinear adaptivity, close to attention
ES-SSM	Spectral truncation	Budget dropout, input-adaptive gate	Elastic deployment, budgeted inference
Time-SSM	Dynamic spectral op.	Minimalist, patchwise, shared/complex SSM	TSF, parameter efficiency
CodeSSM	S4D + frequency path	Code/data pretraining, freq. diagnostics	Hierarchical code syntax/semantics

6. Practical Deployment, Extensions, and Future Directions

Pre-trained SSMs are conducive to heterogeneous deployment scenarios:

• ES-SSM supports cloud-to-edge adaptation, with a single checkpoint yielding budget-accuracy tradeoffs by spectral truncation; degradation is monotonic and predictable (Song et al., 30 Jan 2026).
• Time-SSM proposes extensions to multi-patch, multivariate, and nonlinear regimes for broader time-series foundation modeling (Hu et al., 2024).
• Hybrid architectures: Mamba and its successors are increasingly embedded in mixed attention-SSM networks for maximal context and expressivity benefits (Lv et al., 14 Mar 2025).
• Research trajectories: Future priorities include scaling SSMs to multi-billion parameter regimes, closed-loop adaptive budgeting, multivariate time/frequency modeling, improved theoretical guarantees for dynamic and structured kernels, and integrating learned attention with spectral operators for further gains (Hu et al., 2024, Song et al., 30 Jan 2026).

7. Comparative Strengths, Limitations, and Selection Criteria

Strengths of pre-trained SSMs include:

• Linear or near-linear scaling with respect to sequence length (S4, ES-SSM)
• Long-range memory retention that mitigates vanishing gradients (HiPPO/S4, S4D)
• Parameter, inference, and hardware efficiency (Mamba, Time-SSM)
• Flexibility and adaptivity to data and deployment constraints (ES-SSM, Mamba2)

Key limitations and considerations are:

• Purely linear SSMs (S4, Time-SSM) may underperform on highly nonstationary or strongly localized tasks without hybrid extensions.
• Input-adaptive SSMs (Mamba, CodeSSM) entail higher hardware and engineering complexity due to dynamic recurrence or specialized scan requirements.
• Interpretability remains a challenge; frequency-domain analysis only partially explains model behavior and does not bridge to all downstream errors (Wu et al., 6 Feb 2026).

Selection should be governed by the stationarity and context distribution of the target domain, desired deployment flexibility, and compute constraints:

• S4/time-SSM for resource-constrained, stationary, long-context modeling;
• Mamba/Mamba2 for high-expressivity tasks with dynamic input structure;
• ES-SSM for runtime-heterogeneous or on-device applications where elastic inference is essential.

Pre-trained SSMs thus encapsulate a unified spectrum of foundation sequence models, formally grounded in control and spectral theory, and pragmatically adapted to the computational realities of modern AI deployment (Lv et al., 14 Mar 2025, Song et al., 30 Jan 2026, Hu et al., 2024, Wu et al., 6 Feb 2026).