A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

Abstract: Linear-attention and state-space LLMs compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--value associations compete, earlier facts are overwritten and needle recall degrades. Inspired by Complementary Learning Systems, we give linear attention a hippocampal complement. HOLA (Hippocampal Linear Attention) keeps the usual delta-rule state as a compressive memory and adds a bounded exact KV cache, forming a semiparametric test-time memory: the state models linearly compressible structure, while the cache stores associations that should not be forced through that state. The cache writes without a learned eviction module, keeping tokens with large beta * ||e||, the prediction residual actually committed to the state; a decoupled RMSNorm-gamma cache read then turns these exact KV pairs into sharp retrieval rather than soft averaging. At 340M parameters trained on 15B SlimPajama tokens, HOLA lowers Wikitext perplexity from 27.32 to 22.92 (-16.1%), below a full-attention Transformer++ (26.88), and improves LAMBADA perplexity from 30.95 to 30.26. It also achieves the best linear in-context retrieval and remains much more robust than GDN or a matched HOLA+recency cache on RULER needle-in-a-haystack recall out to 32k tokens (16x its training length).

• The paper introduces HOLA, a mechanism combining a parametric recurrent state with a bounded, surprise-based KV cache for exact memory retrieval.
• It employs a delta rule derived surprise metric to selectively store high-utility tokens, enabling precise, context-sensitive recall without sacrificing efficiency.
• Experimental results demonstrate significant improvements in perplexity and long-context retrieval over existing linear attention models with minimal additional overhead.

A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

Motivation and Problem Statement

Linear attention and state-space models (SSMs) offer sub-quadratic complexity and fixed $O(1)$ memory, but achieve this by summarizing their prefix history in a fixed-size recurrent state, which induces strong capacity bottlenecks for associative memory. This bottleneck, evident in architectures like DeltaNet and Gated DeltaNet (GDN), causes catastrophic interference: as the recurrent state saturates, new associations overwrite old ones, leading to severe degradation in tasks demanding exact recall, such as needle-in-a-haystack retrieval, copy tasks, and associative recall. Existing mitigations, including fixed-size recency caches, remain fundamentally limited due to their inability to retain information about distant, non-redundant tokens.

Complementary Learning Systems (CLS) theory, originating from cognitive neuroscience, posits that episodic (hippocampal) memory supports one-shot, exact event storage, complementing the generalizing function of the neocortical system. The paper leverages this analogy to propose a hippocampal mechanism for recurrent LMs: a bounded, cache-based memory that persistently stores high-utility, surprising token-specific associations, specifically targeting what the recurrent state cannot compress.

Method: HOLA—Hippocampal Linear Attention

The Hippocampal Linear Attention (HOLA) architecture augments a linear-attention backbone (GDN) with a semiparametric memory framework comprising two key memory mechanisms:

1. Parametric recurrent state ($S_t$): Continues compressing history using the delta rule, capturing the linearly predictable structure in the sequence.
2. Bounded, exact KV cache ($\mathcal{A}_t$): Stores a fixed-size set of surprising token-key/value (KV) pairs for high-precision, context-sensitive recall.

The critical distinction is that the cache is not a sliding window (recency-based), but rather selects tokens according to an intrinsic surprise signal—namely, the magnitude ${m_t} = \beta_t \|e_t\|$, where $\beta_t$ is the delta rule write strength and $e_t$ is the prediction residual. This metric quantifies how much each token changed the recurrent state during processing. High-magnitude updates are prioritized for cache storage, reflecting their novelty or lack of compressibility.

Semiparametric Test-Time Memory Regression

The model's output at time $t$ is

$o_t = q_t S_t + \lambda_t g_t(q_t)$

where $q_t S_t$ is the parametric estimate and $\lambda_t g_t(q_t)$ is a non-parametric correction computed by performing (sharpened) softmax retrieval over the bounded cache. Full softmax attention appears as the special case where $S_t$0 is unbounded and $S_t$1 is disabled.

Cache Reading Mechanism

A critical practical observation is that naively reusing backbone-normalized queries and keys for cache lookup leads to retrieval indistinguishable from soft averaging—failing to recover sharp, near-argmax recall necessary for exact memory. HOLA addresses this by introducing a decoupled RMSNorm-$S_t$2 normalization (with learnable $S_t$3) for the cache path, boosting the effective logit scale from $S_t$4 to $S_t$5 and enabling sharp attention over cache entries. This design is orthogonal to the recurrent state update and does not interfere with delta-rule stability.

Instantiation and Overhead

HOLA is instantiated with Gated DeltaNet as the backbone, attaching a KV cache of size $S_t$6 per layer. The introduced trainable parameters are minimal—per-layer cache-path normalization and gating—totaling $S_t$7 of the parameter count. Inference-time memory use increases slightly but remains flat in context length, and does not challenge the $S_t$8 promise of the recurrent memory path.

Experimental Results

HOLA is evaluated at 340M parameters trained on 15B SlimPajama tokens, compared to baseline GDN, KDA, DeltaNet, GLA, GSA, and a full-attention Transformer++. Several findings are emphasized:

• Perplexity: HOLA achieves 22.92 on Wikitext-103, an absolute reduction of $S_t$9 (from 27.32) over GDN, outperforming all linear and full-attention baselines (Transformer++ at 26.88).
• LAMBADA: Perplexity drops from 30.95 (GDN) to 30.26, yielding the best sub-quadratic result.
• Retrieval: On in-context extraction, HOLA obtains the highest scores among all linear models (FDA 20.1 vs. GDN 11.7, SWDE 35.9 vs. GDN 29.0).
• Long-context Recall: On RULER S-NIAH-1 (needle retrieval), GDN collapses ($\mathcal{A}_t$0 at 32k), while HOLA maintains robust recall ($\mathcal{A}_t$1 at 32k, well beyond its 2k training context).
• Commonsense QA: Minor differences across models; HOLA remains competitive but gains are concentrated in perplexity and retrieval regimes.
• Scaling: Gains in perplexity and retrieval persist across backbone sizes (46M, 170M, 340M).
• Ablations: Recency-based caches yield marginal improvements for long-context recall, highlighting the necessity of surprise-based eviction. Both the cache write (intrinsic surprise) and sharp read (RMSNorm-$\mathcal{A}_t$2) are essential for observed gains.

Theoretical and Practical Implications

HOLA offers a formalization of linear attention as a semiparametric sequence estimator, where a recurrent state processes compressible structure and a non-parametric bounded correction retrieves non-compressible, high-information tokens. This resolves the recall-throughput tradeoff for efficient sequence models by explicitly separating generalization and verbatim memory.

From a systems perspective, HOLA demonstrates that with minimal additional inference overhead and frozen backbone parameters, one can recover a significant subset of the recall benefits of softmax attention—without losing the linear efficiency desirable for long-context and high-throughput inference.

The finding that the delta rule's intrinsic residual already supplies an effective surprise metric obviates the need for learned cache-eviction modules, simplifying implementation and training.

Future Directions

Future research may explore:

• Adaptive or learned cache dimensioning and mixing coefficients
• Integration of hierarchical or cross-layer hippocampal caches
• Generalization to multimodal or hierarchical state-space models
• Robustness analysis in extremely dense associative recall regimes and distributed memory deployment
• Examination of learned eviction mechanisms and joint training with cache structure (cf. LTE)

Conclusion

HOLA introduces an architectural mechanism for conjunction of efficient linear attention with bounded, exact non-parametric memory, resolving key limitations in the recall capacity of existing state-space and linear-attention LMs. Empirically, HOLA achieves state-of-the-art perplexity and retrieval performance in the fixed-size memory regime, bridging the gap with full attention and preserving robustness during long-context extrapolation. The approach leverages the internal dynamics of the backbone itself ("what the state forgets") to make memory selection both principled and highly efficient, representing, in theoretical and practical terms, a substantial advance in the design of efficient memory-augmented sequence models (2607.02303).