Hippocampal Linear Attention (HOLA)

• HOLA is a semiparametric memory method that combines a fixed-size recurrent state with an exact key–value cache to overcome lossy compression in linear attention.
• It employs a delta-rule backbone and surprise-based cache writing to select and store high-surprise tokens, ensuring robust in-context recall.
• Empirical results demonstrate improved language modeling and needle-in-a-haystack retrieval, validating HOLA as an effective memory augmentation technique.

Searching arXiv for papers directly relevant to HOLA and related linear-attention memory mechanisms. Hippocampal Linear Attention (HOLA) is a semiparametric memory augmentation for linear-attention and delta-rule LLMs in which a fixed-size recurrent state is retained as a compressive memory and complemented by a bounded exact key–value cache for associations that the recurrent state does not preserve exactly. Introduced as “a hippocampus for linear attention,” HOLA is explicitly inspired by Complementary Learning Systems: the recurrent state models linearly compressible structure, while the cache stores exact support points selected by the prediction residual actually committed to the state. The design targets the failure mode whereby fixed-capacity recurrent memories overwrite earlier facts, interfere along shared key directions, and degrade on needle-in-a-haystack recall as context length increases (Cui, 2 Jul 2026).

1. Problem setting and conceptual basis

HOLA starts from the observation that linear attention, DeltaNet, Gated DeltaNet (GDN), and related state-space models summarize the entire prefix in a fixed-size recurrent state $S_t$. This yields attractive efficiency because memory does not grow with sequence length in the same way as full attention, but it also forces the model to compress all prior key–value associations into a finite matrix. The paper characterizes this compression as inherently lossy: if many distinct associations compete for representation in the same finite state, earlier facts get overwritten, similar key directions interfere, exact one-shot facts are not preserved, and “needle-in-a-haystack” recall degrades.

The paper’s diagnostic is cast in state-space terms. The recurrent state is a fixed-capacity online estimator of the mapping from keys to values. It works when the mapping is linearly compressible, but not when many specific associations must be remembered individually. The underlying limitation is capacity: the state has rank at most $d_k$, so once the number of distinct associations exceeds that capacity, new writes overwrite old ones along shared key directions. A distant $(k,v)$ pair may therefore no longer be exactly recoverable from $S_t$.

HOLA’s response is explicitly CLS-inspired. The recurrent state is retained as a compressive, generalizing, “neocortex-like” memory, while a separate bounded exact KV cache acts as a “hippocampal” complement for surprising items that should not be forced through compression. This division is presented as a way to avoid catastrophic interference between “generalize over many examples” and “remember this exact event.”

2. Two-memory architecture

HOLA is a two-memory design inside each recurrent layer. One component is the standard delta-rule or GDN recurrent state $S_t$, which stores a compressed summary of the prefix. The second is a persistent cache $\mathcal{A}_t$ of exact key–value pairs with fixed capacity $w$, described as a bounded exact KV cache and a test-time semiparametric memory. In the main 340M setup, the default persistent cache size is $w=64$ per layer.

Component	Definition	Responsibility
Recurrent state $S_t$	Fixed-size delta-rule / GDN state	Model linearly compressible structure
Exact cache $\mathcal{A}_t$	Persistent top-$d_k$0 exact KV pairs	Preserve exact support points and high-surprise items
Visible set $d_k$1	Persistent cache, current causal block items, and one null sink	Bounded read set for causal retrieval

The model combines the two memories through

$d_k$2

where $d_k$3 is the recurrent-state output and $d_k$4 is a non-parametric read over the cache. In this decomposition, the state functions as the parametric estimator and the cache supplies a non-parametric correction.

Although the cache is framed as a test-time memory state, HOLA modifies both training and inference. The cache is inference state rather than model parameters, but training and inference use the same cache semantics because the top-$d_k$5 set by write magnitude can be maintained online or blockwise without order dependence. The paper emphasizes that HOLA adds only small learned parameters for the cache read path—cache-path RMSNorm scales, per-head sink, and per-head cache gate—so the architecture remains essentially iso-parametric with its GDN backbone.

3. Delta-rule backbone and semiparametric formulation

The baseline recurrence used by HOLA is the delta rule:

$d_k$6

Here $d_k$7 is the recurrent state matrix, $d_k$8 are the query, key, and value, $d_k$9 is the write strength, $(k,v)$0 is the residual or innovation, and $(k,v)$1 is the recurrent readout. The operational interpretation is “predict, then write the residual”: the model predicts the value using the current state, computes what is missing, and writes that missing part into memory scaled by $(k,v)$2.

For GDN, the prediction is $(k,v)$3 with a decay gate $(k,v)$4, so the residual becomes

$(k,v)$5

This preserves the same residual-writing logic while making the prediction itself decay-gated.

The paper formalizes HOLA as a semiparametric memory estimator. Given prefix observations $(k,v)$6, pure state memory reads as

$(k,v)$7

For semiparametric memory, the state becomes $(k,v)$8, and the read is

$(k,v)$9

Within this framework, full attention is treated as the unbounded non-parametric reference case:

$S_t$0

The comparison is structurally important: full attention stores every token exactly in an unbounded KV cache, whereas HOLA is a bounded semiparametric approximation with a compressive recurrent state plus a small exact cache.

4. Surprise-based cache writing

The cache write mechanism is defined directly from the delta-rule update. Writing is expressed as

$S_t$1

and the write magnitude is measured by the Frobenius norm

$S_t$2

Because $S_t$3 has unit norm, the cache score reduces to $S_t$4. The paper describes this as the token’s intrinsic “surprise” or committed write magnitude.

The rationale is that the delta-rule model already computes the signal needed for cache selection. The residual $S_t$5 indicates what the state failed to predict, while $S_t$6 indicates how strongly the model committed that correction to memory. A large $S_t$7 therefore means that the token both exposed a weakness in the recurrent state and materially changed that state. HOLA uses this as a criterion for exact storage.

Cache maintenance is parameter-free. The cache writes without a learned eviction module and instead maintains the top-$S_t$8 exact $S_t$9 pairs seen so far according to $S_t$0. Operationally, a new token is scored, inserted with its exact $S_t$1, and if the cache exceeds capacity, the item with the smallest score is evicted. The persistent exact memory is therefore a top-$S_t$2 set by committed residual magnitude rather than by recency.

This choice is empirically contrasted with recency-based selection. The matched control HOLA+recency uses the same cache machinery but evicts by position instead of $S_t$3. The reported pattern is that recency helps somewhat locally, but for long-range needle recall it is much weaker than surprise-based selection: at 32k on S-NIAH-1, HOLA reaches $S_t$4, HOLA+recency $S_t$5, and GDN $S_t$6. This supports the claim that cache usefulness depends more on what is stored than on simply retaining the latest tokens.

5. Decoupled cache retrieval and sharp exact-memory readout

HOLA reads the cache through a sharpened softmax over the visible set $S_t$7:

$S_t$8

The paper fixes cache temperature $S_t$9 and relies on RMSNorm-$\mathcal{A}_t$0 to set the effective logit scale. Retrieval is therefore still weighted softmax attention rather than hard nearest-neighbor lookup, but the intention is to make it behave as near-argmax retrieval rather than ordinary soft averaging.

The motivation for decoupling the cache read from the recurrent path is numerical. If one naively reuses the backbone’s unit-L2-normalized $\mathcal{A}_t$1, the effective cache logit is roughly $\mathcal{A}_t$2, and over $\mathcal{A}_t$3 entries even a perfect match gets only about $\mathcal{A}_t$4 of the softmax mass. Under those conditions the cache behaves almost like uniform averaging over stored values, which defeats the purpose of exact storage. RMSNorm-$\mathcal{A}_t$5 keeps the vector norm around $\mathcal{A}_t$6, so with $\mathcal{A}_t$7 the effective logits become around $\mathcal{A}_t$8, making the softmax much more concentrated on the best match.

The decoupling is also required for stability. The recurrent state-update path continues to use unit-L2 normalized $\mathcal{A}_t$9, because the delta rule relies on $w$0 so that the update operator $w$1 has eigenvalues in $w$2. If the norm were $w$3 on the state path, those eigenvalues would become $w$4, which could diverge. HOLA therefore uses unit-L2 normalized queries and keys for state recurrence and RMSNorm-$w$5 transformed queries and keys only for cache read.

The read-side sharpening ablation is one of the paper’s central analyses. At 46M parameters, adding the cache with a weak unit-L2 read changes Wiki perplexity only slightly, from $w$6 to $w$7, whereas the same score rule plus RMSNorm-$w$8 reduces it to $w$9. On the same ablation, far $w=64$0 recall increases from $w=64$1 for GDN to $w=64$2 with $w=64$3+unit-L2 read and to $w=64$4 with $w=64$5+RMSNorm-$w=64$6. The paper therefore identifies sharpening as the single largest lever in the design.

6. Empirical profile and computational properties

The main model reported for HOLA has 340M parameters, uses a GDN architecture with $w=64$7, 24 layers, 4 heads, head dimension 256, and is trained on 15B SlimPajama tokens with context length 2048. HOLA-specific settings are per-layer cache, $w=64$8, chunk $w=64$9, eviction score $S_t$0, cache normalization RMSNorm-$S_t$1, frozen $S_t$2, and gate init $S_t$3. The benchmarks include Wikitext-103 perplexity, LAMBADA perplexity, commonsense tasks, in-context retrieval on FDA, SWDE, and SQuAD, and long-context retrieval on RULER from 2k to 32k.

Benchmark	GDN	HOLA
Wiki perplexity	27.32	22.92
LAMBADA perplexity	30.95	30.26
FDA	11.7	20.1
SWDE	29.0	35.9
SQuAD	32.5	33.8

On language modeling, HOLA lowers Wikitext perplexity from $S_t$4 to $S_t$5, a $S_t$6 change, and beats a full-attention Transformer++ result of $S_t$7. The same trend appears across scale: Wikitext perplexity improves from $S_t$8 to $S_t$9 at 46M, from $\mathcal{A}_t$0 to $\mathcal{A}_t$1 at 170M, and from $\mathcal{A}_t$2 to $\mathcal{A}_t$3 at 340M. The reported interpretation is that the exact cache is not merely a niche retrieval gadget; it also improves language modeling.

On in-context retrieval, the gains are concentrated in extraction-heavy tasks. FDA improves from $\mathcal{A}_t$4 to $\mathcal{A}_t$5, described as a $\mathcal{A}_t$6 relative gain; SWDE improves from $\mathcal{A}_t$7 to $\mathcal{A}_t$8, described as a $\mathcal{A}_t$9 relative gain; SQuAD improves from $d_k$00 to $d_k$01. By contrast, the six-task commonsense average changes only modestly, from $d_k$02 for the GDN anchor to $d_k$03 for HOLA, while HOLA+recency reaches $d_k$04. The empirical pattern is therefore consistent with the paper’s claim that HOLA primarily helps exact memory rather than all reasoning uniformly.

The long-context evidence is given on RULER needle-in-a-haystack tasks. On S-NIAH-1 at 32k tokens, HOLA records $d_k$05, HOLA+recency $d_k$06, and GDN $d_k$07, despite training context only 2k. At 2k, S-NIAH-2 rises from $d_k$08 for GDN to $d_k$09 for HOLA and S-NIAH-3 from $d_k$10 to $d_k$11. At 4k, S-NIAH-2 rises from $d_k$12 to $d_k$13, and at 8k S-NIAH-1 rises from $d_k$14 to $d_k$15. The paper summarizes these results by stating that HOLA achieves the best linear in-context retrieval and remains much more robust than GDN or a matched HOLA+recency cache on RULER needle-in-a-haystack recall out to 32k tokens, which is $d_k$16 its training length.

The added memory and parameter cost remains bounded. During decoding, HOLA stores roughly $d_k$17 KV pairs per layer, with $d_k$18, $d_k$19, and one sink. For the 340M model in bf16, the cache overhead is reported as about 31 MB plus negligible scores. Peak GPU allocation is $d_k$20 GB for HOLA and $d_k$21 GB for GDN at both 32k and 128k, described as nearly flat with context and only about 5% overhead. Cache-specific learned parameters total

$d_k$22

trainable scalars, less than $d_k$23 of model size, plus 96 frozen temperature scalars.

7. Relation to adjacent research and limitations

Within the broader literature on efficient long-context modeling, HOLA occupies a distinct position. CHELA decomposes long-range modeling into a structured, stable, multiscale global context pathway and a hardware-efficient linear-attention pathway, using short-long convolutions plus tiled linear attention; this suggests a division of labor between cheap global memory/compression and selective data-dependent retrieval, but CHELA remains a dense hybrid mixer rather than a bounded exact-memory system (Liu et al., 2024). LAVO replaces the full key/value history with a fixed-size orthogonal memory complemented by a local sliding-window attention path; its orthogonal basis bank explicitly enforces pattern separation in a compressed global memory, but the resulting memory is a running average projection rather than an exact episodic cache (Zhang et al., 2023). CCQ addresses a different failure mode by modifying only the read step: it contracts the query along high-density directions of memory using the running key covariance, thereby treating retrieval weakness as a read-selectivity problem rather than only a write-capacity problem (Le et al., 31 May 2026). A separate hippocampal line in reinforcement learning uses sparse event encoding and fixed linear sequence propagation $d_k$24 as a temporal memory buffer for egocentric visual navigation; this suggests a complementary state-space view of hippocampal computation, but it is autonomous and temporally indexed rather than query-driven and content-addressable (Lin et al., 11 Oct 2025).

Relative to these neighboring designs, HOLA’s defining novelty is the combination of semiparametric framing, parameter-free surprise-based cache selection using $d_k$25, decoupled RMSNorm-$d_k$26 cache read for sharp retrieval, and attachment inside every recurrent layer. The architecture does not abandon recurrent compression; instead, it makes exact memory a bounded complement to that compression.

The paper is also explicit about limitations. The cache is deliberately small, with persistent size $d_k$27 and visible read span $d_k$28, so not every relevant fact can be retained in very long or needle-dense settings. This is reflected in the fact that 32k single-needle recall reaches $d_k$29 rather than perfect recall. The gap to full attention remains on pure extraction tasks such as FDA, where every token may matter. The design depends on hyperparameters including cache size $d_k$30, chunk size $d_k$31, read normalization choice, normalization scales $d_k$32, cache gate, and score rule $d_k$33. The authors also note that they did not run a matched-memory comparison against learned eviction systems such as LTE’s CNN-based evictor. A plausible implication is that HOLA is best understood not as a general replacement for all attention behavior, but as a targeted correction for the exact-recall deficiencies of fixed-state recurrent memory.