- The paper introduces SleepGate, a sleep-inspired KV-cache architecture that tags superseded entries, learns retention scores, and compresses stale memories to reduce proactive interference.
- On a 793K-parameter transformer with synthetic updates, SleepGate achieved 99.5% retrieval accuracy at interference depth 5 and 97.0% at depth 10, while all five baselines stayed below 18%.
- The method’s gains collapse beyond depth 15, reaching 16.5% at depth 30, highlighting unresolved challenges in semantic disambiguation, residual attention mass, and transfer to pretrained production LLMs.
The problem: proactive interference as an architectural bottleneck
The paper addresses proactive interference (PI) in LLMs, building on the PI-LLM paradigm of Wang and Sun, in which a model receives a stream of key-value updates to the same semantic key and is queried for the most recent value. Under this regime, retrieval accuracy degrades log-linearly toward chance as superseded associations accumulate, regardless of context length, and prompt-engineering interventions provide only marginal relief. The paper's central diagnosis is that this is a working-memory bottleneck rather than a capacity limitation: because standard softmax attention admits every cached key into competition with no mechanism for selective inhibition, stale entries suppress current ones through sheer numerical mass.
The proposed remedy draws on sleep neuroscience—specifically synaptic homeostasis (global downscaling), selective replay during slow-wave sleep, and active forgetting—and maps these onto three computational modules operating over the key-value (KV) cache. The resulting framework, SleepGate, is positioned explicitly as an architecture-level solution that prompting cannot replicate, since instructing a model to "ignore earlier values" requires a suppression lever that vanilla attention does not possess.
Framework design
SleepGate augments each cache entry with metadata—position timestamp, semantic signature, binary superseded flag, and cumulative attention received—and adds three coordinated mechanisms:
- Conflict-aware temporal tagger: computes semantic signatures via a learned projection of keys plus locally pooled context, marking entries as superseded when a later entry exceeds a cosine-similarity threshold δ. Amortized O(1) per-token cost is achieved via locality-sensitive hashing.
- Forgetting gate: a 2-layer MLP (dh=128, under 0.01% of base-model parameters) mapping entry features to a retention score, discretized into keep/compress/evict actions via thresholds. Two inference variants are supported: hard eviction with Gumbel-softmax training, and soft attention biasing, which converts retention scores into additive pre-softmax biases bi=βlog(max(ri,ε)); with β=5, an entry with ri=0.01 receives a bias near −23, effectively zeroing its attention weight while remaining recoverable if the gate errs.
- Consolidation module: clusters compress-marked entries by semantic signature and merges each cluster into a single key-value pair using recency-biased cross-attention weights, preserving the most recent value within the cluster.
Sleep micro-cycles are triggered adaptively by rising attention entropy (a proxy for interference buildup), high conflict density, or a fallback periodic interval. Training uses a dual-phase objective combining wake-phase language modeling loss, post-consolidation retrieval ("sleep") loss, a compression penalty on mean retention, and a gate-alignment BCE term against tagger labels, delivered through a four-stage curriculum (warm-start, gate pre-training, joint soft-biasing training, optional threshold calibration).
Theoretical analysis
The paper formalizes why PI produces roughly $1/n$ retrieval probability under uniform attention across n same-key entries—the log-linear decay observed empirically—and proves that SleepGate can eliminate it. Under assumptions that the gate identifies superseded entries with probability pc≥1−ϵ and cycles occur every O(1)0 tokens, the expected number of surviving stale entries is bounded by O(1)1, i.e., constant in O(1)2 when O(1)3 is close to 1. A corollary gives retrieval probability at least O(1)4, independent of update count. A further proposition bounds steady-state cache size by approximately O(1)5 for typical eviction and compression parameters. These results depend on the gate's correctness assumption; the empirical failure mode below shows exactly where that assumption breaks down.
Experimental results
Validation uses a deliberately small proof-of-concept: a 4-layer transformer with 793K base parameters trained from scratch on synthetic PI sequences (100 entities, 500 values), evaluated at seven PI depths with 200 episodes each, against five baselines given identical training budgets.
| Method |
O(1)6 |
O(1)7 |
O(1)8 |
O(1)9 |
dh=1280 |
| SleepGate |
99.0 |
99.5 |
97.0 |
73.5 |
16.5 |
| StreamingLLM |
18.0 |
10.0 |
6.0 |
6.0 |
5.5 |
| Sliding window |
12.5 |
8.0 |
8.5 |
6.5 |
2.0 |
| Decay only |
12.5 |
7.0 |
7.5 |
8.5 |
4.5 |
| Full KV cache |
9.0 |
3.5 |
2.0 |
3.0 |
2.5 |
| H₂O |
7.5 |
1.0 |
4.0 |
3.0 |
0.5 |
SleepGate reaches 99.5% accuracy at depth 5—a tenfold improvement over the best baseline—and 97.0% at depth 10, while all baselines remain below 18% everywhere. Notably, H₂O performs worst: cumulative attention scores are anti-correlated with freshness under PI, so retaining heavy hitters actively preserves interference. The decay-only ablation matches other baselines, establishing that learned gating, not exponential decay alone, drives the gains. An instructive anomaly appears at dh=1281: accuracy (82.5%) falls below dh=1282 (99.0%), confirming that the soft bias improves retrieval precisely when there is conflict to resolve.
Failure modes
Performance collapses sharply beyond depth ~15, dropping to 16.5% at dh=1283 with stale retrieval rates of 62%. The authors attribute this to two factors: residual attention mass from many suppressed-but-present stale entries overwhelming the correct one despite strong individual bias, and insufficient capacity of the 64-dimensional semantic signatures to disambiguate 15–30 near-identical entries. This directly violates the theoretical assumption of near-perfect gate correctness, so the logarithmic-horizon guarantee does not hold empirically at extreme depths. The paper also concedes that the evaluation is confined to a small from-scratch model on synthetic data; transfer to pre-trained production models and naturalistic interference patterns (e.g., document corrections) remains unvalidated, as does composability with GQA, PagedAttention, and quantized caches.
Conclusion
SleepGate demonstrates that learned, content-dependent cache curation—analogous to sleep-dependent consolidation—can resolve proactive interference at the architectural level where prompting and existing KV-cache efficiency methods fail. The soft attention biasing mechanism is the practical centerpiece, enabling fully differentiable joint training without threshold calibration. The principal open questions left by the paper are concrete: whether the mechanism survives scaling to pre-trained LLMs and naturalistic PI, and how to extend effective resistance past the depth-15 cliff via larger signature dimensions, hybrid soft-bias/hard-eviction schemes, or multi-scale sleep scheduling.