---
title: 'HiPPO-LegS Matrix: Online Projection Operator'
url: https://www.emergentmind.com/topics/hippo-legs-matrix
type: topic
---

# HiPPO-LegS Matrix: Online Projection Operator

A HiPPO-LegS matrix arises within the HiPPO (High-order Polynomial Projection Operators) framework as a closed-form, lower-triangular state-space operator for maintaining the online $L^2$-projection of past continuous input signals onto a degree-$N$ Legendre polynomial basis supported on $[0, t]$ under a uniform measure. The HiPPO-LegS memory update mechanism provides a theoretically robust, efficient, and timescale-agnostic solution for compressing cumulative signal history in sequential or trajectory data. Its formulation delivers essential advancements in recurrent state space models (SSMs), especially for modeling long-range temporal dependencies in deep learning.

## 1. The HiPPO Framework and the LegS Specialization

The HiPPO framework seeks the optimal online projection of the truncated input history $f_{\leq t}$ onto an $N$-dimensional polynomial space with respect to a time-varying measure $\mu^{(t)}$. At each instant $t$, given an orthonormal polynomial basis $\{p_n^{(t)}\}_{n=0}^{N-1}$ on $[0, t]$,

\[
g^{(t)}(x) = \sum_{n=0}^{N-1} c_n(t) p_n^{(t)}(x), \qquad c_n(t) = \langle f_{\leq t}, p_n^{(t)} \rangle_{\mu^{(t)}}
\]

For HiPPO-LegS, the measure is uniform on $[0, t]$: $\mu^{(t)}(x) = \frac{1}{t} \mathbf{1}_{[0, t]}(x)$, yielding rescaled Legendre polynomials. The projection coefficients $\mathbf{m}(t) = [c_0(t),...,c_{N-1}(t)]^\top$ evolve according to a linear, time-varying ODE, which underpins the HiPPO-LegS matrix structure [2008.07669].

## 2. Formulation of the HiPPO-LegS Matrix

The continuous-time HiPPO-LegS state equation is

\[
\frac{d}{dt} \, \mathbf{m}(t) = -\frac{1}{t} A_{\mathrm{LegS}} \, \mathbf{m}(t) + \frac{1}{t} B_{\mathrm{LegS}} f(t)
\]

where $f(t)$ is the scalar input, $A_{\mathrm{LegS}} \in \mathbb{R}^{N \times N}$, and $B_{\mathrm{LegS}}\in\mathbb{R}^N$ are time-invariant matrices given by:

\[
(A_{\mathrm{LegS}})_{n,k} =
\begin{cases}
  \sqrt{(2n+1)(2k+1)}, & n > k\\
  n+1, & n = k\\
  0, & n < k
\end{cases}
\qquad
(B_{\mathrm{LegS}})_n = \sqrt{2n+1}
\]

This closed-form, lower-triangular structure emerges from differentiating the orthogonal projection and exploiting Legendre polynomial recurrence relations [2008.07669, 2206.12037, 2412.08595]. The update dynamics optimally compress the entire signal history onto the polynomial basis at all timescales, avoiding explicit timescale priors.

## 3. Dynamics, Singularities, and Well-Posedness

Letting $P_n(x)$ denote standard Legendre polynomials on $[-1, 1]$, the normalized basis on $[0, t]$ is

\[
\phi_n(s; t) = \sqrt{\frac{2n+1}{t}} \, P_n\left(2\frac{s}{t} - 1\right)
\]

The projected state $x_n(t)$ satisfies

\[
\dot{x}(t) = A(t)x(t) + b(t)u(t), \qquad A(t) = \frac{1}{t}\widetilde{A}, \quad b(t) = \frac{1}{\sqrt{t}}\widetilde{b}
\]

where $\widetilde{A}$ and $\widetilde{b}$ are derived from Legendre recurrence integrals. At $t \to 0$, $A(t)$ develops a singularity, precluding arbitrary initial conditions. The only permissible initial state is $x_n(0) = 0$ for all $n$, corresponding to zero prior history. Under this requirement, the ODE is well-posed on $(0, T]$ for Riemann-integrable inputs [2412.08595].

## 4. Discretization and Numerical Analysis

Time discretization converts the continuous ODE to a discrete recurrence suitable for algorithmic implementation. For unit steps ($\Delta t = 1$), forward Euler yields:

\[
m_{k+1} = (I - \frac{1}{k}A_{\mathrm{LegS}}) m_k + \frac{1}{k}B_{\mathrm{LegS}} x_{k+1}
\]

Alternative schemes such as backward Euler and midpoint (second-order Runge-Kutta) have also been shown to converge, with respective global accuracy $O(h)$ and $O(h^2)$ for step size $h$ and suitable resolution near $t=0$. Stability is enhanced for implicit schemes due to the stiffness introduced by the $1/t$ scaling [2412.08595]. Each matrix-vector update can be performed in $O(N)$ time by leveraging the sparsity and structure of $A_{\mathrm{LegS}}$ [2008.07669].

## 5. Theoretical Properties and Memory Capacity

HiPPO-LegS exhibits several key properties critical for sequential modeling:

- **Timescale robustness:** The evolution is equivariant to time dilations: if $f(t)\to f(\alpha t)$, then $m_h(t) = m_f(\alpha t)$, ensuring invariance to sampling rate changes and obviating timescale hyperparameter tuning [2008.07669].
- **Stability:** The gradient decay in the mapping from inputs to memory coefficients is $\Theta(1/t)$, avoiding vanishing or exploding gradients typical in recurrent networks.
- **Approximation error:** For Lipschitz or smoother $f$, the error in the $L^2$ projection decays as $O(t/\sqrt{N})$ or $O(t^k/N^{k-1/2})$ for $k$-smooth $f$, reflecting classical Legendre approximation bounds [2008.07669].
- **Spectral structure:** The underlying $\widetilde{A}$ is nearly skew-Hermitian under the Legendre norm, leading to eigenvalues on the imaginary axis and precluding exponential growth of the homogeneous flow $e^{\int A \, dt}$. The condition number of the eigenvector matrix scales polynomially with $N$, ensuring practical stability for moderate $N$ [2412.08595].

## 6. Comparison with Related Operators

The Legendre Memory Unit (LMU, also referenced as LegT) projects onto a fixed-length window $[t-\theta, t]$ using a translated Legendre polynomial basis, yielding an ODE

\[
A_{\mathrm{LegT}}^{(n, k)} = \frac{1}{\theta}
\begin{cases}
  (-1)^{n-k}(2n+1), & n \geq k\\
  2n+1, & n < k
\end{cases}
,\quad
B_{\mathrm{LegT}}^{(n)} = \frac{1}{\theta}(2n+1)(-1)^n
\]

This approach introduces a timescale hyperparameter $\theta$, requires nontrivial boundary treatment, and is less robust under varying sampling rates. HiPPO-LegS eliminates the need for such priors by adapting the window to $[0, t]$ at all timescales and empirically outperforms LMU/LegT in benchmarks targeting long-range dependency [2008.07669]. HiPPO-LegS matrices also play a crucial foundational role in state space models (notably S4), where they serve as initialization for the state matrix, enabling effective modeling of long sequences [2206.12037].

## 7. Implementation and Empirical Performance

HiPPO-LegS matrices can be precomputed for fixed $N$. The memory update at each timestep is performed as:

```python
m = (I - A_legS / k) @ m + (B_legS / k) * x_next
```

or with optimized prefix-sum kernels for the lower-triangular matvec. These updates yield RNN memory cells that optimally track the Legendre coefficients of the ongoing input history [2008.07669].

Empirically, HiPPO-LegS achieves state-of-the-art results on tasks such as permuted MNIST (98.3% accuracy) and trajectory classification, with substantial margins over RNN and neural ODE baselines, especially in regimes requiring timescale robustness or with missing data [2008.07669].

---

**Table 1: HiPPO-LegS and LMU Matrix Structures**

| Operator           | Measure/Window         | $A_{n,k}$ Pattern (closed form)             |
|--------------------|-----------------------|---------------------------------------------|
| HiPPO-LegS         | Uniform on $[0, t]$   | Lower triangular, $(2n+1)(2k+1)$, $n\geq k$ |
| LMU (LegT)         | Uniform on $[t-\theta,t]$ | Alternating sign, explicit $\theta$ dependence |

For further technical details and proofs, see [2008.07669], [2206.12037], and [2412.08595].

Source: https://www.emergentmind.com/topics/hippo-legs-matrix