---
title: 'S5 Model: Simplified State Space Layer'
url: https://www.emergentmind.com/topics/s5-model
type: topic
---

# S5 Model: Simplified State Space Layer

S5, introduced in "Simplified State Space Layers for Sequence Modeling" [2208.04933], is a sequence modeling layer built from a continuous-time linear state space model (SSM). It was proposed as a simplification of S4: whereas an S4 layer uses many independent single-input, single-output SSMs, the S5 layer uses one multi-input, multi-output SSM. The resulting layer keeps the favorable linear-time scaling associated with structured state space models, but replaces S4’s more specialized convolution-kernel machinery with a diagonalized recurrent system evaluated by parallel scan. In the empirical results reported for the original model, S5 averages **87.46** on Long Range Arena and reaches **98.58** on Path-X [2208.04933].

## 1. Historical placement and relation to S4

S5 emerges directly from the S4 line of structured state space sequence layers. In the formulation given for S4, an input sequence $\mathbf{u}_{1:L} \in \mathbb{R}^{L \times H}$ is processed by $H$ independent SISO SSMs of state size $N$, each handling one feature channel; the outputs are then mixed by a nonlinear layer. S5 replaces that bank of independent SISO systems with one MIMO SSM whose input and output dimensions are both typically $H$, while the latent dimension is $P$ [2208.04933].

This architectural change is the core reason S5 is described as “simplified.” The model keeps the continuous-time SSM formulation and HiPPO-inspired initialization associated with S4, but removes the need for S4’s frequency-domain convolution-kernel evaluation and its extra position-wise linear mixing layer. The stated design goals are both architectural simplicity and computational simplicity: one dense MIMO system instead of many independent SISO systems, and a purely recurrent, time-domain implementation based on widely available associative scan primitives rather than FFT-based kernel machinery [2208.04933].

The original paper also formalizes a connection between S4 and S5. Under simplifying assumptions—tied state matrices across S4 channels, tied discretization timescales across channels, and an S5 input matrix formed by concatenating the S4 input vectors—the S5 latent state corresponds to the sum of the latent states of the independent S4 subsystems:
$$
\mathbf{x}_k = \sum_{h=1}^{H} \mathbf{x}_k^{(h)}.
$$
The corresponding S5 output can be written using an equivalent output projection
$$
\mathbf{C}^{\mathrm{equiv}} = [\, \mathbf{C} \mid \cdots \mid \mathbf{C} \,].
$$
This relationship is used to justify transferring HiPPO-based initialization strategies from S4 to S5 [2208.04933].

## 2. State-space formulation and learned parameters

S5 starts from the standard continuous-time linear SSM
$$
x'(t) = A x(t) + B u(t), \quad y(t) = C x(t) + D u(t).
$$
In the paper’s notation, $\mathbf{u}(t) \in \mathbb{R}^{U}$ is the input, $\mathbf{x}(t) \in \mathbb{R}^{P}$ is the latent state, and $\mathbf{y}(t) \in \mathbb{R}^{M}$ is the output, with parameter matrices
$\mathbf{A} \in \mathbb{R}^{P \times P}$,
$\mathbf{B} \in \mathbb{R}^{P \times U}$,
$\mathbf{C} \in \mathbb{R}^{M \times P}$,
and $\mathbf{D} \in \mathbb{R}^{M \times U}$.
For the sequence layer used in practice, $U = H$ and $M = H$, so the S5 layer is an $H \to H$ map with latent size $P$ [2208.04933].

To make the recurrent computation scan-friendly, S5 diagonalizes the continuous-time state matrix:
$$
\mathbf{A} = \mathbf{V} \mathbf{\Lambda} \mathbf{V}^{-1},
$$
where $\mathbf{\Lambda}$ is diagonal and $\mathbf{V}$ contains eigenvectors. In the eigenbasis,
$$
\tilde{\mathbf{x}}(t)= \mathbf{V}^{-1}\mathbf{x}(t), \qquad
\tilde{\mathbf{B}}= \mathbf{V}^{-1}\mathbf{B}, \qquad
\tilde{\mathbf{C}}=\mathbf{C}\mathbf{V},
$$
so the dynamics become
$$
\dfrac{d \tilde{\mathbf{x}}(t)}{d t} = \mathbf{\Lambda} \tilde{\mathbf{x}}(t) + \tilde{\mathbf{B}}\mathbf{u}(t), \quad
\mathbf{y}(t) = \tilde{\mathbf{C}}\tilde{\mathbf{x}}(t) + \mathbf{D}\mathbf{u}(t).
$$
The learned parameters are $\tilde{\mathbf{B}} \in \mathbb{C}^{P \times H}$, $\tilde{\mathbf{C}} \in \mathbb{C}^{H \times P}$, $\mathrm{diag}(\mathbf{D}) \in \mathbb{R}^{H}$, $\mathrm{diag}(\mathbf{\Lambda}) \in \mathbb{C}^{P}$, and $\mathbf{\Delta} \in \mathbb{R}^{P}$, a learnable vector of timescales [2208.04933].

Because diagonalization of a real matrix produces complex conjugate eigenpairs, S5 enforces conjugate symmetry. Only half the eigenvalues and states are represented explicitly, and the paired structure guarantees real outputs while reducing runtime and memory by about a factor of $2$ [2208.04933].

## 3. Discretization, parallel scan, and initialization

For sequence modeling, the continuous-time SSM is discretized into the recurrence
$$
\mathbf{x}_k = \bar{\mathbf{A}} \mathbf{x}_{k-1} + \bar{\mathbf{B}} \mathbf{u}_k, \quad
\mathbf{y}_k = \bar{\mathbf{C}} \mathbf{x}_k + \bar{\mathbf{D}} \mathbf{u}_k.
$$
S5 uses zero-order hold (ZOH) discretization in the diagonalized basis:
$$
\overline{\mathbf{\Lambda}} = e^{\mathbf{\Lambda}\Delta}, \quad
\overline{\mathbf{B}} = \mathbf{\Lambda}^{-1}(\overline{\mathbf{\Lambda}}-\mathbf{I})\tilde{\mathbf{B}}, \quad
\overline{\mathbf{C}} = \tilde{\mathbf{C}}, \quad
\overline{\mathbf{D}} = \mathbf{D}.
$$
Operationally, with per-state timescales,
$$
\Lambda_{\text{bar}} = \exp(\Lambda \odot \Delta), \qquad
B_{\text{bar}} = \left(\frac{\Lambda_{\text{bar}} - 1}{\Lambda}\right)\odot \tilde{B}.
$$
The discrete recurrence then takes the form
$$
\mathbf{x}_k = \overline{\mathbf{\Lambda}} \mathbf{x}_{k-1} + \overline{\mathbf{B}} \mathbf{u}_k, \quad
\mathbf{y}_k = \tilde{\mathbf{C}} \mathbf{x}_k + \mathbf{D} \mathbf{u}_k.
$$
Because $\overline{\mathbf{\Lambda}}$ is diagonal, the recurrence remains inexpensive per step [2208.04933].

The scan formulation is based on tuples $(A_k, b_k)$ with
$$
A_k = \overline{\mathbf{\Lambda}}, \qquad b_k = \overline{\mathbf{B}}\mathbf{u}_k,
$$
combined by the associative operator
$$
(A_i, b_i) \bullet (A_j, b_j) = (A_j A_i,\; A_j b_i + b_j).
$$
For diagonal $A$, transition composition becomes elementwise or otherwise low-cost vector arithmetic. The paper states that if latent size $P = O(H)$, then S5 and S4 have the same order of magnitude complexity in runtime and memory. Concretely, S4 offline is
$$
O(H^2L + HL\log L),
$$
whereas S5 offline is
$$
O(PHL + PL),
$$
which becomes
$$
O(H^2L + HL)
$$
when $P = O(H)$ [2208.04933].

Initialization is a second defining component. S4 used HiPPO-LegS, but that matrix is not stably diagonalizable, so S5 instead uses the normal component of HiPPO-LegS, called HiPPO-N. The practical recipe is to initialize $\mathbf{A}$ as $\mathbf{A}_{\mathrm{LegS}^{\mathrm{Normal}}}$, eigendecompose it to obtain $\mathbf{\Lambda}$ and $\mathbf{V}$, and initialize $\tilde{\mathbf{B}}$ and $\tilde{\mathbf{C}}$ by
$$
\tilde{\mathbf{B}} = \mathbf{V}^{-1}\mathbf{B}, \qquad
\tilde{\mathbf{C}} = \mathbf{C}\mathbf{V}.
$$
A further practical variation is block-diagonal initialization, in which $\mathbf{A}$ is initialized with several smaller HiPPO-N blocks on the diagonal. Timescales are learned in log-space, with each component of $\log \Delta$ sampled uniformly from $[\log \delta_{\min}, \log \delta_{\max})$ using defaults $\delta_{\min}=0.001$ and $\delta_{\max}=0.1$; for Path-X, the paper uses $\delta_{\min}=0.0001$ and $\delta_{\max}=0.1$ [2208.04933].

## 4. Layer architecture, training configuration, and empirical results

An S5 layer takes an input sequence $\mathbf{u}_{1:L} \in \mathbb{R}^{L \times H}$, discretizes the diagonalized continuous-time SSM, computes latent states by parallel scan, forms outputs with $\tilde{C}$ and $D$, and then applies a nonlinearity. In the minimal code path, the output is
$$
u'_k = \mathrm{GELU}(y_k).
$$
In the full architecture used experimentally, the layer output is
$$
\mathbf{u}'_k = \mathrm{GELU}(\mathbf{y}_k) \odot \sigma\!\left(\mathbf{W} \, \mathrm{GELU}(\mathbf{y}_k)\right).
$$
The broader network uses a linear input encoder, a stack of S5 layers, mean pooling over sequence length for classification tasks, and a linear output decoder. The experimental models also use dropout, either layer normalization or batch normalization, and either pre-norm or post-norm; for many tasks, the reported choice is batch norm and pre-norm. Bidirectional models are used for LRA and speech tasks, while unidirectional models are used where causal comparison is required [2208.04933].

| Benchmark or task | Reported S5 result |
|---|---:|
| Long Range Arena average | 87.46 |
| Path-X | 98.58 |
| Speech Commands, 16kHz | 96.52% |
| Speech Commands, zero-shot 8kHz | 94.53% |
| Irregularly sampled pendulum MSE | $3.41 \times 10^{-3}$ |
| sMNIST | 99.65 |
| psMNIST | 98.67 |
| sCIFAR | 90.10 |

These results place S5 among the strongest linear-complexity sequence models reported in the original study. On LRA, the detailed task scores are **62.15** on ListOps, **89.31** on Text, **91.40** on Retrieval, **88.00** on Image, **95.33** on Pathfinder, and **98.58** on Path-X. The paper also reports **86×** relative application speed on the irregularly sampled pendulum task, with S5 MSE **3.41 $\times 10^{-3}$** compared to **CRU (our run)** at **3.94 $\times 10^{-3}$** [2208.04933].

The ablation results identify two implementation choices as decisive. First, continuous-time parameterization with HiPPO-N initialization is the only configuration that consistently works on all tasks, including Path-X. On Path-X, the ablation values reported are **55** for discrete Gaussian, **55** for discrete antisymmetric, **55** for discrete HiPPO-N, **55** for continuous Gaussian, **55** for continuous antisymmetric, and **98.58** for continuous HiPPO-N. Second, replacing a scalar $\Delta$ with a vector $\boldsymbol{\Delta} \in \mathbb{R}^{N}$ improves performance consistently, and larger latent size together with block-diagonal initialization improves performance further [2208.04933].

## 5. Quantization studies and later architectural adaptations

Later work has treated S5 not only as a long-sequence model but also as a precision-sensitive recurrent system. "Q-S5: Towards Quantized State Space Models" [2406.09477] studies quantization-aware training (QAT) and post-training quantization (PTQ) for S5 and emphasizes that the recurrent/state-space dynamics—especially the diagonal transition $\bar A$—are substantially more fragile than the rest of the network. In that study, the discretized recurrence is written
$$
x_k = \bar{A}x_{k-1} + \bar{B}u_k, \qquad
y_k = \bar{C}x_{k-1} + \bar{D}u_k.
$$
The reported empirical pattern is that recurrent/state-space weights, especially $\bar A$, are the most sensitive to quantization; PTQ works well only on the language-based LRA tasks Text and Retrieval; and mixed precision is markedly better than uniform low-bit quantization. On sMNIST, the full-precision S5 baseline is **99.65%**, while QAT gives **99.54%** for W8A8, **99.63%** for W4A8SSM8, **99.26%** for W4A8$\bar A$8, **99.56%** for W2A8SSM8, but only **12.68%** for W4A8 and **54.75%** for W2A8. The paper also states that fully quantized S5 models can have test accuracy drops of less than **1%** on sMNIST and most of the LRA [2406.09477].

"Slot State Space Models" [2406.12272] places S5 alongside S4 and Mamba as part of the modern SSM family, but its actual implementation uses Mamba-style selective SSM blocks rather than an S5 layer. Its relevance to S5 is architectural rather than parametric: it preserves the efficient, diagonal or block-diagonal, scan-friendly recurrence associated with modern SSMs, but replaces a single monolithic hidden state with multiple slot states updated independently and coupled only through a self-attention bottleneck. This suggests a line of development in which the core S5 intuition—structured long-range recurrence with efficient parallel execution—is retained while the hidden state is reorganized around explicit modularity [2406.12272].

## 6. Terminological ambiguity and other uses of “S5 model”

The term “S5 model” is not unique to state space sequence modeling. In modal logic, the phrase denotes multi-agent Kripke models whose accessibility relations are equivalence relations. In "Formalization of PAL$\cdot$S5 in Proof Assistant" [2012.09388], the Lean structure
```lean
structure worlds (α agent W : Type) : Type :=
(f : W → α → Prop)
(view : agent → W → W → Prop)
(equiv : ∀(a : agent), equivalence (view a))
```
formalizes an S5 model with valuation $f$, agent-indexed accessibility relation `view`, and the condition that each agent’s accessibility relation is reflexive, symmetric, and transitive. That paper develops the syntax and semantics of PAL·S5, proves reduction axioms for public announcements, and formalizes soundness and completeness for static S5 [2012.09388].

A second proof-theoretic line, "Base-extension Semantics for S5 Modal Logic" [2403.19431], gives a base-extension semantics for multi-agent S5 in which atomic validity is grounded in derivability from a base of atomic rules rather than in truth assignments. There, $K_a \phi$ is valid at a base $\mathscr B$ iff $\phi$ is valid at every related base $\mathscr C$ such that $\mathfrak R_a \mathscr B \mathscr C$, with the relations $\mathfrak R_a$ constrained to satisfy the semantic conditions appropriate for S5 [2403.19431].

A computational third line, "Modal Logic S5 Satisfiability in Answer Set Programming" [2108.04194], studies S5 satisfiability with worlds represented as a list $\mathbf{I} = [I_0,\ldots,I_n]$ of propositional interpretations under total accessibility, so that
$$
(\mathbf{I},i) \models \Box\phi \iff (\mathbf{I},j) \models \phi \text{ for all } j \in [0..n].
$$
That work exploits the total accessibility of S5 to construct parsimonious ASP encodings based on reachability relations rather than naïvely copying every atom into every world [2108.04194].

The acronym is also used outside both sequence modeling and modal logic. "S5: Scalable Semi-Supervised Semantic Segmentation in Remote Sensing" [2508.12409] introduces an unrelated framework for remote-sensing foundation models, so references to “S5” require domain-specific disambiguation [2508.12409].

Source: https://www.emergentmind.com/topics/s5-model