Cache Enrichment Oracle

• Cache Enrichment Oracle is a conceptual framework that defines optimal cache utilization and augmentation strategies in transformer KV-caches and hardware hierarchies.
• It demonstrates that semantic enrichment can recover most few-shot performance gains using fixed cache size, achieving improvements such as a +3.92 percentage point increase.
• Oracle-guided methods, including learned gating and heavy-hitter retention, inspire practical policies that significantly boost cache hit rates and overall system efficiency.

A Cache Enrichment Oracle is a conceptual and experimental framework for establishing best-possible strategies for cache utilization and augmentation in computational settings, notably in transformer-based LLMs and hardware cache hierarchies. Across these contexts, enrichment oracles provide either absolute upper bounds or practical policies on how cache content can be optimally enhanced to maximize downstream utility (such as accuracy or cache hit rate) under specific constraints, e.g., fixed cache size or associativity. This article surveys cache enrichment oracles in (1) semantic augmentation of transformer KV-caches for model communication and inference, (2) learned cache replacement in hardware systems, and (3) dynamic cache eviction strategies founded on theoretical and empirical heavy-hitter identification.

1. Semantic Enrichment Oracle in Transformer KV-Caches

The cache enrichment oracle in transformer-based LLMs is designed to empirically isolate and measure the maximal attainable improvement in response quality achievable by enriching the semantic content of the key-value (KV) cache—without increasing its length—during language understanding and generation (Fu et al., 3 Oct 2025).

Given a prompt token sequence $X$ of length $|X|$, standard practice fills the cache during prefill as $$C(X) = [c_0, ..., c_{|X|-1}] \in \mathbb{R}^{|X| \times d}$$. The model decodes using the conditional distribution: $P(y_{i+1} \mid C(X) \parallel C(Y[0:i]))$ where $\parallel$ is sequence concatenation.

The enrichment oracle operates as follows:

• Augmentation is performed by prefilling on $[E; X]$, with $E$ a set of exemplars.
• Only the cache slice corresponding to $X$ is retained: $C^*(X) = C([E; X])[|E| : |E| + |X|]$ The decoding phase then uses $C^*(X)$—identical in length to $C(X)$ but "enriched" by the exemplars' semantic influence—to isolate the effect of enrichment from mere cache extension.

Empirically, this enrichment recovers the majority of the few-shot performance gain while keeping cache size fixed. On MMLU-Redux with the Qwen3-0.6B model:

• Baseline ("Direct") cache: $58.42\%$ accuracy.
• Few-Shot (longer cache): $63.39\%$.
• Oracle-enriched (fixed size): $62.34\%$.

This demonstrates a +3.92 percentage point gain attributable strictly to semantic enrichment at constant cache length (Fu et al., 3 Oct 2025).

2. Layer-Wise Sensitivity and Gating

The cache enrichment oracle further reveals heterogeneous, layer-dependent sensitivity to enrichment. Targeting enrichment to only the top-performing transformer layers increases accuracy beyond blanket enrichment of all layers, while enriching the least-responsive layers can decrease overall performance.

This empirical finding underpins the design of learnable gating mechanisms in the full Cache-to-Cache (C2C) framework. For each layer $l$, the gating function is: $$g^{(l)} = \sigma\left(W_g^{(l)} \cdot [K_s^{(l)}; V_s^{(l)}] + b_g^{(l)}\right)$$ where $[K_s^{(l)}, V_s^{(l)}]$ are the sharer's layer-$l$ key/value tensors, and $g^{(l)}\in[0,1]$ modulates the information injected into the receiver.

Incorporating gating into C2C enables dynamic, selective cross-model cache fusion through elementwise blending: $$C_F^{(l)} = (1 - g^{(l)}) \odot C_r^{(l)} + g^{(l)} \odot \mathcal{F}_{\text{proj}}([C_r^{(l)}; C_s^{(l)}])$$ with $\mathcal{F}_{\text{proj}}$ a projection network and $\odot$ denoting elementwise product.

3. Cache Enrichment Oracles for Hardware Replacement Policies

In hardware contexts, a cache enrichment oracle is instantiated by Belady's optimal replacement policy (Liu et al., 2020). For a cache set of associativity $W$ at time $t$, containing lines $l_1, ..., l_W$, the oracle computes for each $l_w$ its reuse distance $d_t(l_w)$—the number of future accesses until $l_w$ is next referenced. Belady's policy elects to evict the line with maximal $d_t(l_w)$, maximizing cache hits over the execution trace.

Since true future access is unavailable at runtime, "Parrot" leverages imitation learning:

• It casts cache replacement as an MDP, with state composed of current cache contents, incoming access, and truncated history.
• A neural network policy is trained by DAgger to mimic Belady's eviction actions, using ranking losses on predicted reuse distances and benchmarking against the raw oracle policy.

Experimentally, Parrot achieves substantial gains:

• +16 percentage points in raw hit rate versus LRU on SPEC CPU2006 tasks.
• Achieves $\sim$20\% higher normalized hit rates than the best prior IL baseline (Glider) (Liu et al., 2020).

This oracle-guided imitation learning strategy narrows the achievable gap between learned and optimal cache replacement.

4. Dynamic Submodular Cache Enrichment via Heavy-Hitter Oracles

For autoregressive LLM inference, the KV-cache eviction problem admits a formalization in submodular maximization, enabling oracle-like strategies for minimizing utility loss when cache is bounded (Zhang et al., 2023). At each generation step $i$, the model must select a token subset $S_i\subseteq [i]$, $|S_i|\leq k$, to retain in cache.

The oracle objective is to maximize: $F(S_i) = \sum_{j \in S_i} w_j(i)$ where $w_j(i)$ represents the accumulated (unnormalized) attention score assigned to token $j$ when generating token $i$. This utility is provably submodular, and optimal retention of high-$w$ tokens yields minimal degradation in attentional fidelity.

Empirically, a tiny minority of tokens—“heavy hitters” (H$_2$)—account for the majority of total attention. Evicting these causes severe quality loss, while retaining them (even at strict cache budgets) preserves almost all downstream accuracy.

The Heavy-Hitter Oracle (H₂O) greedily maintains (i) all recent tokens and (ii) the set of highest-output-score tokens, using accumulated attention weights as selection criteria. The algorithm provides approximation guarantees relative to the true optimal, and at inference time, increases throughput by up to 3×–29× (depending on LLM and engine) with negligible loss in output quality (Zhang et al., 2023).

5. Quantitative Results and Comparative Analysis

Key findings across domains are organized as follows:

Context / Method	Oracle Performance Gain	Principle
LLM: KV enrichment	+3.92pp at fixed cache (Qwen3-0.6B)	Prefill on exemplars, enrich via slice
Hardware: Parrot IL	+16pp over LRU (SPEC)	Imitate Belady, ranking loss + LSTM
LLM: H₂O kv-cache	Up to 29× throughput, ≈0 loss	Retain heavy-hitters (accum. attn)

• LLM cache enrichment can recover most few-shot gains purely through semantics, without increased cache length (Fu et al., 3 Oct 2025).
• In memory systems, Parrot approaches within $\sim$25\% of Belady's optimal, outperforming all heuristics or prior learned baselines (Liu et al., 2020).
• Heavy-hitter retention sharply reduces memory and boosts inference throughput with almost no task-specific degradation (Zhang et al., 2023).

6. Implications, Limitations, and Future Directions

Cache enrichment oracles demonstrate that (a) semantic or structural cache augmentation can yield disproportionately large improvements at fixed budget, (b) these improvements originate from specific, context-sensitive enhancements (layer- or item-selective), and (c) oracles inspire practical, albeit approximate, policy designs (e.g., learnable gates in C2C, imitation-learned replacements in Parrot, greedy submodular heavy-hitter retention in H₂O).

Limitations include:

• Real systems must address inference and model-size overhead (notably for hardware learned policies).
• Submodular heavy-hitter oracles presuppose non-uniform attention; their advantage vanishes under uniform or adversarial patterns.
• Multilevel cache hierarchies and reward-delayed environments furnish open benchmarks for further oracle-guided RL/IL paper.

These oracles collectively provide upper bounds and design blueprints for advanced cache management in transformers, system hardware, and hybrid multi-model ensembles. Practical adoption continues to evolve as models, workloads, and computational regimes expand.