Epiphany-Aware KV Cache Eviction Without the Attention Matrix

Abstract: As reasoning models emit chains of thought tens of thousands of tokens long, KV cache increasingly becomes a deployment bottleneck. Existing cache eviction methods rank tokens by attention weight, which is a noisy importance proxy in long reasoning traces, and prohibits the use of fused kernels in production inference by forcing the model to materialize the attention matrix. In this work, we instead score tokens with a metric we term the epiphany score: the change in the model's internal representation, read directly from the forward pass with no attention matrix and negligible extra state. Our resulting cache eviction method, EpiKV, requires no training, classifier, or custom kernel, and can be used directly in FlashAttention inference stacks unchanged -- scaling to a 16x longer feasible context than attention-based scoring. upper-mid layers negatively) and remove a positional trend with a causal rolling z-score. At a 4096-token cache EpiKV reaches 72% on MATH-500, matching the strongest attention-based baseline (ThinKV 71%, H2O 67%); a lag-normalized KV variant reaches 37% on AIME-2024 at 8192 tokens against the best of them (33%), at up to 2.8x the speed.

• The paper introduces the EpiKV method which computes 'epiphany scores' from hidden state changes, bypassing noisy attention matrices.
• It demonstrates a two-band architecture where mid-layer dynamics correlate differently with token importance, enhancing cache eviction decisions.
• Experiments show that EpiKV scales inference to 65k tokens with up to 2.8× speed gains and competitive accuracy versus attention-based methods.

Epiphany-Aware KV Cache Eviction Without the Attention Matrix

Motivation and Problem Setting

Transformer LLMs designed for advanced reasoning tasks must maintain extremely long context windows, leading to rapid growth in key-value (KV) cache memory. Tasks such as mathematical problem solving with lengthy chains of thought exacerbate this bottleneck, as the generation of $10^4$ to $10^5$ tokens dominates device memory, restricting concurrent request capacity and scaling. Traditional approaches to KV cache eviction rely on attention-based scoring—ranking tokens by attention weights. However, attention weights are empirically noisy proxies for token importance in long-range reasoning, and, critically, their extraction is fundamentally incompatible with modern fused attention kernels like FlashAttention, since it requires materializing the full $n \times n$ attention matrix.

This paper introduces an alternative: EpiKV, which discards reliance on attention weights and instead computes a token importance signal based on internal model dynamics at generation time, compatible with production inference stacks that leverage fused kernels.

Methodology: Epiphany Scores and Two-Band Anatomy

The core contribution is the definition of an "epiphany score," computed as the change in the model's hidden state at the token level, extracted directly from the forward pass without reference to the attention matrix. For a given layer $l$ and token position $t$, the per-token hidden state change is measured as $g_l(t) = \lVert h_l(t) - h_l(t-1) \rVert_2$, where $h_l(t)$ is the hidden state output by the transformer at position $t$.

Through empirical correlation studies with counterfactual token-importance labels created via occlusion experiments, the authors uncover a two-band architecture:

• Band A (layers 7–13): Positive correlation; larger changes indicate more important tokens.
• Band B (layers 18–25): Negative correlation; larger changes indicate less important tokens.

The net importance score is constructed as

$s(t) = \bar{g}_{10}(t) - \bar{g}_{21}(t)$

with $\bar{g}_l(t)$ representing a rolling mean of $10^5$0 over a trailing window.

Figure 1: Per-layer Spearman $10^5$1 between rolling-64 hidden-state change and counterfactual importance; positive correlation in Band A (7–13), negative in Band B (18–25).

The raw score exhibits a strong positional trend, potentially biasing eviction toward early or late tokens regardless of content importance. To address this, the authors implement a causal rolling $10^5$2-score normalization, transforming absolute magnitude into local deviation, yielding the final EpiKV method.

Figure 2: Peak GPU memory usage per forward pass vs. context length. Attention-based methods (eager) exhaust the GPU at 8192 tokens; EpiKV, which does not require the attention matrix, scales to 65,536 tokens—a $10^5$3 gain.

Experimental Evaluation

Benchmarks and Policies

Experiments are conducted on DeepSeek-R1-Distill-LLaMA-8B over two primary datasets: MATH-500 (long mathematical reasoning traces) and AIME-2024 (even longer traces, higher cache pressure). EpiKV is directly compared to strong attention-based baselines such as ThinKV and H2O, as well as attention-matrix-free competitors like Lag-KV.

• FA2-compatible methods: Only leverage cached KV and hidden states, allowing operation within FlashAttention/production inference stacks.
• Attention-based methods: Require materializing the attention matrix, incompatible with FlashAttention.

Accuracy vs. Cache Budget

On MATH-500, at 4096-token cache budget, EpiKV achieves 72% accuracy, matching or exceeding all baselines—ThinKV (71%), H2O (67%). On AIME-2024 at 8192 tokens, the lag-normalized method reaches 37% versus the best attention-based at 33%. Under tight budgets, attention-based and hybrid methods sometimes outperform FA2-compatible policies, but EpiKV attains at least parity at the deployment-relevant budgets.

Figure 3: Accuracy versus cache budget on MATH-500. FA2-compatible methods match or exceed the strongest attention-based baselines.

Throughput and GPU Memory

EpiKV enables significant engineering advantages:

• Speed: At 8192 tokens on AIME-2024, lag-normalized KV achieves a wall-clock time per problem up to 2.8$10^5$4 faster than attention-based eviction, owing to fully fused kernel compatibility.
• Memory scaling: Only EpiKV sustains context lengths up to 65k tokens on a single 80GB GPU. In contrast, attention-based methods hit the memory wall at 8k tokens.

  Figure 4: Accuracy vs. wall-clock time per problem on AIME-2024 at 8192-token budget. FA2-compatible methods (green) lie on the accuracy–speed Pareto frontier; lag-KV is both most accurate and fastest.

  

  Figure 5: Maximum concurrent requests on an 80GB GPU vs. context length. Without eviction, capacity collapses as trace length increases; with a fixed budget via eviction, concurrent capacity remains flat.

Theoretical and Practical Implications

The "two-band" signature aligns with findings from mechanistic interpretability, which localize factual recall and feature routing to mid-network layers. The observation that importance-tracking shifts with dataset/task difficulty further supports the hypothesis that model computation is adaptively routed by problem complexity.

Practically, EpiKV delivers a plug-and-play replacement for attention-based eviction in production-serving stacks (e.g., vLLM, TGI, SGLang), requiring no retraining or custom kernels. This yields a significant boost in scalable inference for long-context LLMs without regressions on key benchmarks.

Limitations and Future Directions

• Results are centered on a single model family (DeepSeek-R1-Distill-LLaMA-8B); generalization to other architectures and scales is not demonstrated.
• At tightest cache budgets, hybrid and attention-based methods can outperform pure FA2-compatible approaches, indicating potential for future hybridization using non-attention cues.
• The method requires setting band locations per task/dataset regime; automatic adaptation could further streamline deployment.
• Latency and memory are measured per-example on single GPUs; further work should rigorously validate system-level throughput and multi-GPU scaling.

Chunk-level scoring, per-layer budgets, and quantization approaches offer orthogonal opportunities for continued improvement and stacking.

Conclusion

EpiKV introduces a KV cache eviction method fully compatible with fused attention kernels, leveraging internal model dynamics to identify and preserve important tokens throughout long reasoning traces. Its empirical performance at equal or greater accuracy than attention-based baselines, alongside substantial engineering gains in speed and memory efficiency, makes it an attractive and production-viable approach for scaling LLM inference to ultra-long contexts.