HiPPO Framework: Optimal Memory for SSMs

Updated 26 January 2026

HiPPO framework is a family of mathematical algorithms that optimally compress sequential data into finite-dimensional, orthogonal polynomial representations for online state-space models.
It underpins SSMs like S4 and S5, achieving high performance (up to 96% accuracy on long-range benchmarks) through adaptive timescale and perturb-then-diagonalize methodologies.
The framework addresses numerical stability and noise robustness by integrating data-driven initialization, PTD techniques, and uncertainty-aware extensions in sequential modeling.

The HiPPO framework is a family of mathematical constructions and algorithms enabling online, finite-dimensional summaries of sequential data via optimal orthogonal polynomial projections. Its primary domain of impact is the initialization and analysis of state-space models (SSMs) for long-sequence modeling in machine learning, but its methods have also been adopted in Bayesian optimization, online Gaussian processes, and sequence-based representation learning. HiPPO formulations yield fast, timescale-adaptive, and theoretically robust memory representations, which have driven advances in structured SSMs such as S4 and inspired algorithmic refinements to overcome ill-posedness and achieve numerical stability.

1. Mathematical Foundations: Optimal Polynomial Projections

The core mathematical problem addressed by HiPPO is the incremental compression of a continuous-time signal $x(t)$ or discrete sequence $x_n$ into a compact vector $c(t)$ containing the coefficients of the best-degree- $(N-1)$ polynomial approximation of the history, under a specified time-dependent weighting $\mu^{(t)}$ . Formally, HiPPO derives $c_k(t) = \langle x_{\leq t}, \phi_k^{(t)} \rangle_{\mu^{(t)}}$ , where $\{\phi_k^{(t)}\}$ are orthonormal polynomials on the history interval and $\langle\cdot,\cdot\rangle_{\mu^{(t)}}$ is the corresponding $L^2$ inner product. The time evolution of $c(t)$ is governed by a linear ODE: $\dot c(t) = A(t)\,c(t) + B(t)\,x(t)$ The form of $A(t)$ and $B(t)$ depends on the chosen orthogonal polynomial family and the normalization induced by $\mu^{(t)}$ . Prominent specializations include:

HiPPO-LegS, using exponentially-warped Legendre polynomials under uniform history weighting, yielding timescale-invariant memory (Gu et al., 2020, Gu et al., 2022).
Sliding window variants (translated-Legendre, Laguerre) tie memory to windows of fixed past duration, recovering models such as the Legendre Memory Unit (LMU).

Discrete-time implementations are obtained via stable discretizations of the ODE, such as forward Euler or bilinear transforms. The state update becomes: $c_{k+1} = W_c\,c_k + W_x\,x_k$ with precomputed $W_c, W_x$ derived from $(A, B, \Delta t)$ (Gu et al., 2020).

2. Application to State-Space Sequence Models

The structured state-space sequence (S4) layer and its extensions employ HiPPO-initialized SSMs to capture long-range dependencies in sequences. For single-input, single-output (SISO) LegS systems, the continuous-time SSM is

$x'(t) = A_H x(t) + B_H u(t),\quad y(t) = C x(t)$

with the HiPPO matrix $A_H$ given by

$(A_H)_{jk} = -\begin{cases} \sqrt{(2j-1)(2k-1)} & j > k \ j & j = k \end{cases}$

and $B_H$ , $C$ set according to the basis (Yu et al., 2023, Gu et al., 2020, Gu et al., 2022).

The key property is that the impulse response kernels of this system form an orthonormal basis, enabling online function approximation and memory retention across arbitrarily long sequences. HiPPO's formulation is responsible for S4 and S5's empirical ability to solve long-context sequence tasks, achieving performance such as 86–96% on the Long Range Arena benchmark (Gu et al., 2022, Yu et al., 2023).

3. Diagonalization, Ill-Posedness, and the PTD Methodology

Direct diagonalization of the HiPPO matrix $A_H$ is numerically unstable due to its extreme nonnormality—the eigenvectors are exponentially ill-conditioned, quantified via the pseudospectrum $\sigma_\varepsilon(A)$ . Small perturbations can cause large spectral shifts, rendering naive exact diagonalization unreliable (Yu et al., 2023). This leads to the backward-stable "perturb-then-diagonalize" (PTD) methodology:

Perturb $A_H$ by a small matrix $E$ (with $\|E\| \leq \epsilon$ ) to obtain $\widetilde{A} = A_H + E$ , making it diagonally well-conditioned.
Diagonalize $\widetilde{A}$ as $V \Lambda V^{-1}$ .
Construct the PTD-based S4-PTD or S5-PTD layer as

$H_{\text{PTD}}(z) = \overline{C} (zI - \Lambda)^{-1}\overline{B} + \overline{D}$

This process yields uniform (strong) convergence of the transfer function to the ideal HiPPO response, whereas purely diagonal (S4D/S5) schemes provide only weak, pointwise convergence and can experience catastrophic Fourier-mode amplification (Yu et al., 2023).

Empirical validation on synthetic frequency tests and the Long Range Arena benchmark (S5-PTD averaging 87.6% accuracy) confirms the practical superiority and robustness of PTD initialization (Yu et al., 2023).

4. Extensions: Timescale, Conditioning, and Initialization Schemes

Recent work analyzes the effect of sequence statistics, especially autocorrelation, on SSM initialization. Timescale (step size $\Delta$ ) should be set adaptive to both sequence length $L$ and maximum autocorrelation eigenvalue $\lambda_{\mathrm{max}}$ : $\Delta \approx 1/\sqrt{L\lambda_{\mathrm{max}}}$ Rather than the heuristic $\Delta \sim 1/L$ , this data-driven rule improves memory stability for uncorrelated or highly autocorrelated data (Liu et al., 2024).

Additionally, choosing zero real parts for $W$ (eigenvalues) in the diagonal SSM setting prevents exponential decay in memory kernels—alleviating the "curse of memory," as required for tasks needing spikes at large lags. Imaginary parts $v_j = \mathrm{Im}(w_j)$ can be chosen to either:

Maximize approximation accuracy by matching dominant frequencies of the target sequence (at the potential cost of poor Gram matrix conditioning if frequencies are close),
Or maintain Gram matrix conditioning by separating $v_j$ , manifesting a tradeoff between approximation power and optimization tractability (Liu et al., 2024).

5. Robustness to Noise and Uncertainty-aware Variants

The original HiPPO SSMs assume noise-free inputs, rendering them brittle to measurement or observation noise. The "UnHiPPO" extension interprets HiPPO as a linear stochastic control problem and reformulates the initialization as posterior inference in a linear Gaussian (Kalman filter) setting: $dc_t = (1/t) A c_t dt + dw_t,\quad y_k = C c_{t_k} + \epsilon_k, \quad \epsilon_k \sim \mathcal{N}(0, \sigma^2)$ The resulting UnHiPPO recurrence precomputes uncertainty-aware dynamics matrices $(A_{\mathrm{uncert},k}, B_{\mathrm{uncert},k})$ via the Kalman filter. This initialization improves resistance to noise, as demonstrated on speech classification benchmarks with up to 10 percentage point accuracy gains under test noise (Lienen et al., 5 Jun 2025).

HiPPO's memory-projection framework enables novel approaches in online Gaussian processes. In "Online HiPPO Sparse Variational Gaussian Process" (OHSGPR), HiPPO's time-varying orthogonal polynomials serve as interdomain inducing variables, allowing efficient online updating of the GP inducing states and kernel blocks via ODE recurrences, surpassing conventional fixed-point methods in long-term memory preservation and speed (Chen et al., 12 Feb 2025).

Similarly, HiPPO is used as a regularization mechanism in high-dimensional Bayesian optimization frameworks such as HiBBO. Here, the VAE encoder/decoder is augmented with a HiPPO-based regularizer to enforce the space consistency of functional representations between the latent and original spaces, controlling kernel mismatch and accelerating convergence across high-dim tasks (Xuan et al., 10 Oct 2025).

7. Algorithmic Variants in Reinforcement Learning and Hierarchical Models

HiPPO has also been instantiated as an acronym for Hierarchical Proximal Policy Optimization in hierarchical RL. However, in this context, it refers to a two-level hierarchy of policies trained with a clipping-based PPO objective, not to polynomial projection memory models (Li et al., 2019). This use is orthogonal to the polynomial projection-based HiPPO framework central to SSMs and sequence modeling.

In summary, the HiPPO framework provides precise prescriptions for the construction of SSMs whose states optimally summarize the past via polynomial projections, supports robust initialization for deep sequence models (notably S4 and its PTD variants), offers algorithmic guidelines adaptive to data autocorrelation and noise, and has been extended to related domains in online GP, Bayesian optimization, and beyond. Empirical and theoretical advances continually refine its initialization, conditioning, and stability properties, ensuring broad impact across sequential modeling disciplines (Gu et al., 2020, Gu et al., 2022, Yu et al., 2023, Liu et al., 2024, Lienen et al., 5 Jun 2025, Chen et al., 12 Feb 2025, Xuan et al., 10 Oct 2025).