---
title: 'Liquid-S4: Adaptive Sequence Modeling'
url: https://www.emergentmind.com/topics/liquid-s4
type: topic
---

# Liquid-S4: Adaptive Sequence Modeling

Liquid-S4 is a structural state-space model (SSM) designed for high-performance representation learning on long-range sequential data. It is constructed by integrating a linear liquid time-constant (LTC) ODE with a diagonal plus low-rank decomposition (DPLR) of state transition matrices, leveraging methodology from S4 (Structured State Spaces). Liquid-S4 is characterized by its input-dependent state transitions and kernel structure that encodes similarities and correlations within the sequence, enabling state-of-the-art results across image, text, audio, and medical signal domains [2209.12951].

## 1. Mathematical Structure of Liquid-S4

Liquid-S4 is founded upon a continuous-time linearized LTC ODE, specified for an $N$-dimensional hidden state $x(t) \in \mathbb{R}^N$, scalar input $u(t)$, and scalar output $y(t)$:
\[
\frac{d\,x(t)}{dt} = \left[ A + B\,u(t) \right] x(t) + B\,u(t), \qquad y(t) = C\,x(t)
\]
where $A \in \mathbb{R}^{N\times N}$ is the transition matrix, $B \in \mathbb{R}^{N\times 1}$ is both the bias and input modulation, and $C \in \mathbb{R}^{1 \times N}$ computes the readout.

Discretization via the bilinear (trapezoidal) rule—using step size $\Delta t$—yields:
\[
\bar{A} = (I - \tfrac{\Delta t}{2}A)^{-1} (I + \tfrac{\Delta t}{2}A), \quad
\bar{B} = (I - \tfrac{\Delta t}{2}A)^{-1} \Delta t\,B, \quad
\bar{C} = C
\]
The resulting discrete-time recurrence for each timestep $k$:
\[
x_k = [\bar{A} + \bar{B}\,u_k] x_{k-1} + \bar{B}\,u_k, \qquad y_k = \bar{C}\,x_k
\]

## 2. Diagonal-Plus-Low-Rank Parameterization

The state transition matrix $A$ is initialized using the HiPPO-LegS matrix (scaled Legendre measure) and rewritten in the Normal + Low-Rank (NPLR) decomposition:
\[
A = V\,\Lambda\,V^* - P\,Q^\top, \qquad
A = \text{diag}(\Lambda) + P_{\text{eff}} Q_{\text{eff}}^T
\]
with $V$ unitary, $\Lambda \in \mathbb{C}^N$ diagonal (initially left half-plane for stability), $P,Q \in \mathbb{R}^{N\times r}$ low-rank ($r=1$ in practice), and $P_{\text{eff}} = V^*P$, $Q_{\text{eff}} = V^*Q$.

This DPLR form ensures computational efficiency, numerical stability, and the flexibility to learn long temporal dependencies. Input dependence is injected strictly via the $B\,u$ modulation.

## 3. Kernel Construction and Correlation Structure

Liquid-S4 introduces a kernel framework comprising both linear and higher-order correlation terms. For $u$-independent recurrence, the convolutional kernel is:
\[
y_k = \sum_{i=0}^k \bar{C} \bar{A}^{k-i} \bar{B} u_i = (\bar{K} * u)_k, \quad \bar{K}_n = \bar{C} \bar{A}^n \bar{B}
\]
Efficient computation via FFT leverages the Cauchy kernel pipeline, offering nearly linear-time complexity in sequence length.

The LTC recurrence introduces additional “liquid” kernels representing auto-correlations:
\[
y_k \supset \sum_{0 \leq i < j \leq k} \bar{C} \bar{A}^{k-1-j} \bar{B}^2 (u_i u_j) + \cdots + \sum_{0 \leq i < j < \cdots < p \leq k} \bar{C} \bar{A}^{k-p} \bar{B}^p (u_i u_j \cdots u_p)
\]
For each chosen order $p$:
\[
K_{\text{liquid}}^{(p)}[n] = (\bar{K}[N-1-n]) \odot (\bar{B}^{p-1}[N-1-n]), \quad n=0,\dots,N-1
\]
Or in PB mode:
\[
K_{\text{liquid}}^{(p)}[n] = C B^p \quad \text{(constant over $n$)}
\]
Liquid-S4 concatenates the $p=1$ S4 kernel with higher-order liquid kernels ($p=2,\dots,\mathcal{P}$), computed in $\tilde O(N+L+\mathcal{P}\tilde L)$ time.

## 4. Model Architecture and Training Protocols

Liquid-S4 constructs deep sequence models by stacking multiple state space blocks, each comprising the Liquid-S4 kernel convolution, residual connections, feed-forward layers, and pointwise nonlinearities (GeLU, ReLU). The architecture is causal—each output $y_k$ depends exclusively on past and current inputs $u_{0:k}$.

Training employs cross-entropy loss for classification or $\ell_2$ loss for regression. Regularization strategies include weight decay on learnable parameters $(\Lambda, P, Q, B, C)$, dropout in feed-forward submodules, and gradient penalties on eigenvalues to enhance stability. Optimization is via Adam/AdamW with learning rates tuned per task, typically smaller for Liquid-S4 compared to S4.

The kernel’s input dependence—modulating the recurrence at each timestep—enables Liquid-S4 to re-weight historical inputs adaptively, yielding improved generalization in non-stationary and highly correlated data regimes.

## 5. Empirical Evaluation and Performance Benchmarks

Liquid-S4 establishes new state-of-the-art results on multiple long-range sequence modeling benchmarks:

### Long-Range Arena (1K–16K sequences)

| Method               | ListOps | IMDB   | AAN    | CIFAR  | PathFinder | Path-X | Average |
|----------------------|---------|--------|--------|--------|------------|--------|---------|
| S4-LegS (reprod.)    | 59.60   | 86.82  | 90.90  | 88.65  | 94.20      | 96.35  | 86.09   |
| Liquid-S4 (PB, $p\le6$) | 62.75   | 89.02  | 91.20  | 89.50  | 94.80      | 96.66  | 87.32   |

### Raw Speech Commands (35 classes, 16 kHz)

| Model            | Params  | Accuracy |
|------------------|---------|----------|
| S4-LegS          | 307 K   | 96.08%   |
| S4D-Lin          | 306 K   | 96.25%   |
| Liquid-S4        | 224 K   | 96.78%   |

### BIDMC Vital Signs (RMSE)

| Model            | HR     | RR     | SpO₂   |
|------------------|--------|--------|--------|
| S4-LegS          | 0.332  | 0.247  | 0.090  |
| S4D-Inv          | 0.373  | 0.254  | 0.110  |
| Liquid-S4 (best $p$) | 0.303  | 0.158  | 0.066  |

These results indicate consistent 1–3% gains over S4, with Liquid-S4 frequently achieving higher accuracy and reduced parameterization (e.g., 30% fewer parameters on Speech Commands).

## 6. Implementation Details and Hyperparameter Choices

Standard settings are:
- Number of state space blocks (“depth”): 4–9, task-dependent
- Hidden units per block ($H$): 128–512
- State dimension ($N$): 7 (ListOps/IMDB) to 512 (CIFAR)
- Low-rank factor ($r$): $1$
- Liquid kernel order ($\mathcal{P}$): typically 2–4, default at 3
- FFT-based convolution via PyKeops for memory efficiency
- Forward pass complexity: $\tilde O(N + L + \mathcal{P}\tilde L)$; memory similar to S4 with minor overhead for correlation kernels

## 7. Interpretations and Relevance

Liquid-S4’s lightweight kernel modulation introduces input-correlation structure into SSMs, directly encoding similarities of input samples during both training and inference. This property facilitates data-dependent adaptive filtering and generalization under long-sequence, highly-correlated, or non-stationary conditions. The empirical improvement at negligible computational or parameter cost suggests further applicability in domains typified by long-range dependencies and dynamic signal correlation.

Liquid-S4 extends S4’s diagonal-plus-low-rank parametrization by input-dependent kernel construction, resulting in a robust and scalable sequence modeling framework with consistent empirical benefits across modalities [2209.12951].

Source: https://www.emergentmind.com/topics/liquid-s4