S5 Model: Simplified State Space Layer

• S5 is a simplified state space sequence layer derived from continuous-time linear SSMs, using one multi-input, multi-output system instead of multiple SISO systems.
• It replaces S4's convolution-kernel machinery with a diagonalized recurrent system evaluated via efficient parallel scan operations.
• Empirical results demonstrate S5's strong performance on benchmarks like Long Range Arena and Path-X, while also simplifying model initialization and quantization.

S5, introduced in "Simplified State Space Layers for Sequence Modeling" (Smith et al., 2022), is a sequence modeling layer built from a continuous-time linear state space model (SSM). It was proposed as a simplification of S4: whereas an S4 layer uses many independent single-input, single-output SSMs, the S5 layer uses one multi-input, multi-output SSM. The resulting layer keeps the favorable linear-time scaling associated with structured state space models, but replaces S4’s more specialized convolution-kernel machinery with a diagonalized recurrent system evaluated by parallel scan. In the empirical results reported for the original model, S5 averages 87.46 on Long Range Arena and reaches 98.58 on Path-X (Smith et al., 2022).

1. Historical placement and relation to S4

S5 emerges directly from the S4 line of structured state space sequence layers. In the formulation given for S4, an input sequence $\mathbf{u}_{1:L} \in \mathbb{R}^{L \times H}$ is processed by $H$ independent SISO SSMs of state size $N$, each handling one feature channel; the outputs are then mixed by a nonlinear layer. S5 replaces that bank of independent SISO systems with one MIMO SSM whose input and output dimensions are both typically $H$, while the latent dimension is $P$ (Smith et al., 2022).

This architectural change is the core reason S5 is described as “simplified.” The model keeps the continuous-time SSM formulation and HiPPO-inspired initialization associated with S4, but removes the need for S4’s frequency-domain convolution-kernel evaluation and its extra position-wise linear mixing layer. The stated design goals are both architectural simplicity and computational simplicity: one dense MIMO system instead of many independent SISO systems, and a purely recurrent, time-domain implementation based on widely available associative scan primitives rather than FFT-based kernel machinery (Smith et al., 2022).

The original paper also formalizes a connection between S4 and S5. Under simplifying assumptions—tied state matrices across S4 channels, tied discretization timescales across channels, and an S5 input matrix formed by concatenating the S4 input vectors—the S5 latent state corresponds to the sum of the latent states of the independent S4 subsystems:

$\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$

The corresponding S5 output can be written using an equivalent output projection

$$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$

This relationship is used to justify transferring HiPPO-based initialization strategies from S4 to S5 (Smith et al., 2022).

2. State-space formulation and learned parameters

S5 starts from the standard continuous-time linear SSM

$$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$

In the paper’s notation, $\mathbf{u}(t) \in \mathbb{R}^{U}$ is the input, $\mathbf{x}(t) \in \mathbb{R}^{P}$ is the latent state, and $H$0 is the output, with parameter matrices $H$1, $H$2, $H$3, and $H$4. For the sequence layer used in practice, $H$5 and $H$6, so the S5 layer is an $H$7 map with latent size $H$8 (Smith et al., 2022).

To make the recurrent computation scan-friendly, S5 diagonalizes the continuous-time state matrix:

$H$9

where $N$0 is diagonal and $N$1 contains eigenvectors. In the eigenbasis,

$N$2

so the dynamics become

$N$3

The learned parameters are $N$4, $N$5, $N$6, $N$7, and $N$8, a learnable vector of timescales (Smith et al., 2022).

Because diagonalization of a real matrix produces complex conjugate eigenpairs, S5 enforces conjugate symmetry. Only half the eigenvalues and states are represented explicitly, and the paired structure guarantees real outputs while reducing runtime and memory by about a factor of $N$9 (Smith et al., 2022).

3. Discretization, parallel scan, and initialization

For sequence modeling, the continuous-time SSM is discretized into the recurrence

$H$0

S5 uses zero-order hold (ZOH) discretization in the diagonalized basis:

$H$1

Operationally, with per-state timescales,

$H$2

The discrete recurrence then takes the form

$H$3

Because $H$4 is diagonal, the recurrence remains inexpensive per step (Smith et al., 2022).

The scan formulation is based on tuples $H$5 with

$H$6

combined by the associative operator

$H$7

For diagonal $H$8, transition composition becomes elementwise or otherwise low-cost vector arithmetic. The paper states that if latent size $H$9, then S5 and S4 have the same order of magnitude complexity in runtime and memory. Concretely, S4 offline is

$P$0

whereas S5 offline is

$P$1

which becomes

$P$2

when $P$3 (Smith et al., 2022).

Initialization is a second defining component. S4 used HiPPO-LegS, but that matrix is not stably diagonalizable, so S5 instead uses the normal component of HiPPO-LegS, called HiPPO-N. The practical recipe is to initialize $P$4 as $P$5, eigendecompose it to obtain $P$6 and $P$7, and initialize $P$8 and $P$9 by

$\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$0

A further practical variation is block-diagonal initialization, in which $\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$1 is initialized with several smaller HiPPO-N blocks on the diagonal. Timescales are learned in log-space, with each component of $\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$2 sampled uniformly from $\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$3 using defaults $\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$4 and $\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$5; for Path-X, the paper uses $\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$6 and $\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$7 (Smith et al., 2022).

4. Layer architecture, training configuration, and empirical results

An S5 layer takes an input sequence $\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$8, discretizes the diagonalized continuous-time SSM, computes latent states by parallel scan, forms outputs with $\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.$9 and $$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$0, and then applies a nonlinearity. In the minimal code path, the output is

$$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$1

In the full architecture used experimentally, the layer output is

$$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$2

The broader network uses a linear input encoder, a stack of S5 layers, mean pooling over sequence length for classification tasks, and a linear output decoder. The experimental models also use dropout, either layer normalization or batch normalization, and either pre-norm or post-norm; for many tasks, the reported choice is batch norm and pre-norm. Bidirectional models are used for LRA and speech tasks, while unidirectional models are used where causal comparison is required (Smith et al., 2022).

Benchmark or task	Reported S5 result
Long Range Arena average	87.46
Path-X	98.58
Speech Commands, 16kHz	96.52%
Speech Commands, zero-shot 8kHz	94.53%
Irregularly sampled pendulum MSE	$$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$3
sMNIST	99.65
psMNIST	98.67
sCIFAR	90.10

These results place S5 among the strongest linear-complexity sequence models reported in the original study. On LRA, the detailed task scores are 62.15 on ListOps, 89.31 on Text, 91.40 on Retrieval, 88.00 on Image, 95.33 on Pathfinder, and 98.58 on Path-X. The paper also reports 86× relative application speed on the irregularly sampled pendulum task, with S5 MSE 3.41 $$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$4 compared to CRU (our run) at 3.94 $$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$5 (Smith et al., 2022).

The ablation results identify two implementation choices as decisive. First, continuous-time parameterization with HiPPO-N initialization is the only configuration that consistently works on all tasks, including Path-X. On Path-X, the ablation values reported are 55 for discrete Gaussian, 55 for discrete antisymmetric, 55 for discrete HiPPO-N, 55 for continuous Gaussian, 55 for continuous antisymmetric, and 98.58 for continuous HiPPO-N. Second, replacing a scalar $$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$6 with a vector $$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$7 improves performance consistently, and larger latent size together with block-diagonal initialization improves performance further (Smith et al., 2022).

5. Quantization studies and later architectural adaptations

Later work has treated S5 not only as a long-sequence model but also as a precision-sensitive recurrent system. "Q-S5: Towards Quantized State Space Models" (Abreu et al., 2024) studies quantization-aware training (QAT) and post-training quantization (PTQ) for S5 and emphasizes that the recurrent/state-space dynamics—especially the diagonal transition $$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$8—are substantially more fragile than the rest of the network. In that study, the discretized recurrence is written

$$\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].$$9

The reported empirical pattern is that recurrent/state-space weights, especially $$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$0, are the most sensitive to quantization; PTQ works well only on the language-based LRA tasks Text and Retrieval; and mixed precision is markedly better than uniform low-bit quantization. On sMNIST, the full-precision S5 baseline is 99.65%, while QAT gives 99.54% for W8A8, 99.63% for W4A8SSM8, 99.26% for W4A8$$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$18, 99.56% for W2A8SSM8, but only 12.68% for W4A8 and 54.75% for W2A8. The paper also states that fully quantized S5 models can have test accuracy drops of less than 1% on sMNIST and most of the LRA (Abreu et al., 2024).

"Slot State Space Models" (Jiang et al., 2024) places S5 alongside S4 and Mamba as part of the modern SSM family, but its actual implementation uses Mamba-style selective SSM blocks rather than an S5 layer. Its relevance to S5 is architectural rather than parametric: it preserves the efficient, diagonal or block-diagonal, scan-friendly recurrence associated with modern SSMs, but replaces a single monolithic hidden state with multiple slot states updated independently and coupled only through a self-attention bottleneck. This suggests a line of development in which the core S5 intuition—structured long-range recurrence with efficient parallel execution—is retained while the hidden state is reorganized around explicit modularity (Jiang et al., 2024).

6. Terminological ambiguity and other uses of “S5 model”

The term “S5 model” is not unique to state space sequence modeling. In modal logic, the phrase denotes multi-agent Kripke models whose accessibility relations are equivalence relations. In "Formalization of PAL$$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$2S5 in Proof Assistant" (Li, 2020), the Lean structure $\mathbf{u}(t) \in \mathbb{R}^{U}$2 formalizes an S5 model with valuation $$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$3, agent-indexed accessibility relation view, and the condition that each agent’s accessibility relation is reflexive, symmetric, and transitive. That paper develops the syntax and semantics of PAL·S5, proves reduction axioms for public announcements, and formalizes soundness and completeness for static S5 (Li, 2020).

A second proof-theoretic line, "Base-extension Semantics for S5 Modal Logic" (Eckhardt et al., 2024), gives a base-extension semantics for multi-agent S5 in which atomic validity is grounded in derivability from a base of atomic rules rather than in truth assignments. There, $$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$4 is valid at a base $$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$5 iff $$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$6 is valid at every related base $$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$7 such that $$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$8, with the relations $$x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).$$9 constrained to satisfy the semantic conditions appropriate for S5 (Eckhardt et al., 2024).

A computational third line, "Modal Logic S5 Satisfiability in Answer Set Programming" (Alviano et al., 2021), studies S5 satisfiability with worlds represented as a list $\mathbf{u}(t) \in \mathbb{R}^{U}$0 of propositional interpretations under total accessibility, so that

$\mathbf{u}(t) \in \mathbb{R}^{U}$1

That work exploits the total accessibility of S5 to construct parsimonious ASP encodings based on reachability relations rather than naïvely copying every atom into every world (Alviano et al., 2021).

The acronym is also used outside both sequence modeling and modal logic. "S5: Scalable Semi-Supervised Semantic Segmentation in Remote Sensing" (Lv et al., 17 Aug 2025) introduces an unrelated framework for remote-sensing foundation models, so references to “S5” require domain-specific disambiguation (Lv et al., 17 Aug 2025).