EpiKV: Epiphany-Aware KV Cache Eviction

• EpiKV is a KV-cache eviction policy that uses hidden state changes to compute token importance without materializing attention matrices.
• It addresses the linear growth challenge in long reasoning traces by scoring tokens based on their representational shifts, enhancing efficiency.
• The method operates online and causally within standard FlashAttention stacks, offering scalable, training-free inference improvements.

Searching arXiv for the primary paper and named related work to ground citations. Search query: "^^^^^^^^1ti:\1 KV Cache Eviction Without the Attention Matrix\"" EpiKV is a KV-cache eviction policy for long-reasoning LLM inference that replaces attention-derived token importance with a directly observable signal from the model’s hidden states. Introduced in "Epiphany-Aware KV Cache Eviction Without the Attention Matrix" (&&&^^^^1ti:\1^^^^&&&), the method scores tokens by the extent to which they coincide with a shift in internal computation, rather than by how much subsequent tokens attend to them. The paper positions this design against two constraints that become acute in long chains of thought: the linear growth of KV cache with sequence length, and the incompatibility between attention-matrix materialization and fused inference paths such as FlashAttention. EpiKV therefore operates as a causal, online, attention-free policy that requires no training, classifier, or custom kernel, while remaining usable in standard FlashAttention inference stacks unchanged (&&&^^^^1ti:\1^^^^&&&).

1. Problem setting and motivation

The immediate problem addressed by EpiKV is KV-cache growth during reasoning-oriented decoding. In long reasoning traces, chains of thought may extend to tens of thousands of tokens, and the KV cache grows linearly with sequence length. Under such conditions, cache memory becomes the dominant bottleneck at decode time, so practical systems retain only a budget of PRESERVED_PLACEHOLDER_^^^^1ti:\1^^^^ tokens and evict the remainder (&&&^^^^1ti:\1^^^^&&&).

The paper argues that existing reasoning-oriented eviction methods, including H2O, ThinKV, and RaaS, mostly rank tokens by attention-derived importance. It identifies two independent deficiencies in that approach. First, attention is treated as a noisy proxy for importance in reasoning traces: attention sinks and filler tokens may receive high attention despite limited semantic importance, whereas important milestone tokens may cease to attract high attention after they are generated even though they remain load-bearing for later reasoning. Second, attention-based scoring is architecturally expensive because it requires materializing the $n \times n$ attention matrix, which breaks the memory efficiency of fused attention implementations and forces eager execution when output_attentions=True is enabled (&&&^^^^1ti:\1^^^^&&&).

The deployment implication is explicit. On an 8^^^^1ti:\1^^^^GB A1^^^^1ti:\1ti:\1^^^^, the paper reports that an eager forward pass with attention weights runs out of memory at 8192 tokens, whereas the same model in FlashAttention mode scales to 65,536 tokens. This is the basis for the claim that EpiKV can scale to a 16x longer feasible context than attention-based scoring in production-style inference settings (&&&^^^^1ti:\1^^^^&&&).

2. Epiphany score and representational signal

EpiKV replaces attention weights with what the paper terms the epiphany score: a per-token measure of change in the model’s residual stream or hidden state. For layer $l$ and decode position $t$, with hidden state $h_l(t)$, the layerwise change is defined as

$g_l(t)=\lVert h_l(t)-h_l(t-1)\rVert_2.$

The intended interpretation is that tokens associated with large representational shifts are more likely to mark consequential reasoning steps: an intermediate conclusion, a transition from exploration to commitment, or another structurally important point in the computation (&&&^^^^1ti:\1^^^^&&&).

A central empirical finding is a two-band anatomy in a 32-layer reasoning model. In layers 7–13, high $g_l(t)$ is positively correlated with token importance. In layers 18–25, high $g_l(t)$ is negatively correlated with token importance. The method therefore does not treat all hidden-state change as uniformly informative. Instead, it combines a positive signal from mid-layers with a negative signal from upper-mid layers. A simple combined score is

$s(t)=\bar{g}_{10}(t)-\bar{g}_{21}(t),$

where $\bar{g}_l(t)$ is a rolling mean over a trailing window of $n \times n$^^^^1ti:\1^^^^ tokens (&&&^^^^1ti:\1^^^^&&&).

This formulation is consequential because it is computed directly from the standard forward pass: hidden states are read through the normal output_hidden_states interface, differences across consecutive decode positions are computed online, and the attention matrix is never materialized. A common misconception is that cache eviction for reasoning necessarily depends on attention maps; the paper’s central counterexample is that representation change alone can serve as the ranking signal (&&&^^^^1ti:\1^^^^&&&).

3. Causal online eviction algorithm

The full EpiKV procedure is a causal, online token-ranking policy. First, for each generated token $n \times n$1 and each layer $n \times n$2, it computes the hidden-state delta $n \times n$3. Second, it smooths this signal over a causal trailing window of size $n \times n$4, producing $n \times n$5. The paper states that this smoothing improves correlation substantially relative to the raw signal (&&&^^^^1ti:\1^^^^&&&).

Third, it selects the two empirically relevant layer bands: Band A, layers 7–13, with positive correlation to importance; and Band B, layers 18–25, with negative correlation. The primary score uses representative layers near the centers of these bands, especially layers 1^^^^1ti:\1^^^^ and 21. Fourth, it combines them as $n \times n$6 (&&&^^^^1ti:\1^^^^&&&).

A further methodological result is that raw hidden-state change has a monotonic positional trend within a trace. In effect, the raw score can track position as much as content, which would confound token ranking. To remove this positional drift, the paper introduces a causal rolling z-score,

$n \times n$7

where $n \times n$8 and $n \times n$9 are the trailing-window mean and standard deviation, and $l$^^^^1ti:\1^^^^ is a small stabilizer. The final EpiKV score is then

$l$1

At each decode step, the policy preserves prompt or prefill tokens, preserves a trailing recency window, scores the remaining cached positions, keeps the top-$l$2 tokens by score, and evicts the rest permanently. Because each token’s score is frozen when the token is generated, the method remains fully online and causal (&&&^^^^1ti:\1^^^^&&&).

4. Systems compatibility and inference-stack implications

A defining property of EpiKV is that it does not require output_attentions=True, attention-matrix materialization, custom attention kernels, training, or an auxiliary classifier. The only required signals are cached K/V tensors, which are already resident, and hidden states exposed by the standard inference interface. This makes the method compatible with standard fused inference stacks such as FlashAttention, vLLM, TGI, and SGLang without modifying the FlashAttention kernel itself (&&&^^^^1ti:\1^^^^&&&).

The paper describes an implementation through HuggingFace DynamicCache and records several engineering fixes needed for correctness. Keep masks had to be moved to the correct device in multi-GPU settings. After eviction, the cache had to be rebuilt so that _seen_tokens matched the retained cache length. Otherwise, causal masking would be off by one. These are presented as cache-management details rather than kernel modifications, reinforcing the paper’s claim that the method drops into existing FlashAttention-based serving stacks without architectural surgery (&&&^^^^1ti:\1^^^^&&&).

This suggests a more precise distinction between EpiKV and attention-based baselines than a purely algorithmic comparison would capture. The contrast is not only between two token-importance signals; it is between a policy that preserves fused-kernel execution and one that can force a reversion to eager attention, thereby reinstating the quadratic memory behavior that long-context deployment is trying to avoid (&&&^^^^1ti:\1^^^^&&&).

5. Empirical results and benchmark comparisons

The reported MATH-5^^^^1ti:\1ti:\1^^^^ result without eviction is 75%. EpiKV reaches ^^^^1ti:\1^^^^% at a 512-token cache, 27% at 1^^^^1ti:\1^^^^24, 49% at 2^^^^1ti:\1^^^^48, and 72% at 4^^^^1ti:\1^^^^96. At the 4^^^^1ti:\1^^^^96-token budget, ThinKV is reported at 71% and H2O at 67%, so EpiKV is at parity with, or slightly above, the strongest attention-based baseline at that budget (&&&^^^^1ti:\1^^^^&&&).

Setting	Reported result	Comparison
MATH-5^^^^1ti:\1ti:\1^^^^, no eviction	75%	Reference
MATH-5^^^^1ti:\1ti:\1^^^^, EpiKV, 4^^^^1ti:\1^^^^96	72%	ThinKV 71%, H2O 67%
MATH-5^^^^1ti:\1ti:\1^^^^, EpiKV, 2^^^^1ti:\1^^^^48	49%	Budget-sensitive regime
MATH-5^^^^1ti:\1ti:\1^^^^, EpiKV, 1^^^^1ti:\1^^^^24	27%	Small-budget degradation
MATH-5^^^^1ti:\1ti:\1^^^^, EpiKV, 512	^^^^1ti:\1^^^^%	Small-budget degradation

The H2O failure mode is described as especially severe on reasoning traces. On MATH-5^^^^1ti:\1ti:\1^^^^, H2O reaches 67% at 4^^^^1ti:\1^^^^96, 49% at 2^^^^1ti:\1^^^^48, 5% at 1^^^^1ti:\1^^^^24, and 1% at 512. The paper further reports collapsed generation rather than merely incorrect answers: at 1^^^^1ti:\1^^^^24, H2O produces no generated answer on 93/1^^^^1ti:\1ti:\1^^^^ problems; at 2^^^^1ti:\1^^^^48, on 48/1^^^^1ti:\1ti:\1^^^^; and at 4^^^^1ti:\1^^^^96, on 27/1^^^^1ti:\1ti:\1^^^^. At 512, it often emits unstructured text with no extractable answer (&&&^^^^1ti:\1^^^^&&&).

AIME-2^^^^1ti:\1^^^^24 is presented as a harder and longer-trace benchmark. The no-eviction score is 43.3%. EpiKV reports ^^^^1ti:\1^^^^.^^^^1ti:\1 at 512, 1^^^^1ti:\1^^^^24, and 2^^^^1ti:\1^^^^48; 16.7% at 4^^^^1ti:\1^^^^96; and 33.3% at 8192. The paper also reports an attention-free variant, Lag-KV, at 2^^^^1ti:\1^^^^.^^^^1ti:\1 for 4^^^^1ti:\1^^^^96 and 36.7% for 8192, compared with ThinKV at 6.7%, 2^^^^1ti:\1^^^^.^^^^1ti:\1 and 3^^^^1ti:\1^^^^.^^^^1ti:\1 for 2^^^^1ti:\1^^^^48, 4^^^^1ti:\1^^^^96, and 8192 respectively, and H2O at 2^^^^1ti:\1^^^^.^^^^1ti:\1 and 33.3% for 4^^^^1ti:\1^^^^96 and 8192 (&&&^^^^1ti:\1^^^^&&&).

Setting	Reported result	Comparison
AIME-2^^^^1ti:\1^^^^24, no eviction	43.3%	Reference
AIME-2^^^^1ti:\1^^^^24, EpiKV, 8192	33.3%	H2O 33.3%, ThinKV 3^^^^1ti:\1^^^^.^^^^1ti:\1
AIME-2^^^^1ti:\1^^^^24, Lag-KV, 8192	36.7%	Best reported attention-free variant
AIME-2^^^^1ti:\1^^^^24, EpiKV, 4^^^^1ti:\1^^^^96	16.7%	ThinKV 2^^^^1ti:\1^^^^.^^^^1ti:\1 H2O 2^^^^1ti:\1^^^^.^^^^1ti:\1

The paper also reports runtime comparisons at 8192 on AIME-2^^^^1ti:\1^^^^24: Lag-KV takes 44^^^^1ti:\1^^^^.5s per problem, ThinKV 721.^^^^1ti:\1^^^^s, and RaaS 1238.7s. This is the basis for the claim of up to 2.8x speed and about 1.6x relative to ThinKV. The text is careful not to claim universal dominance across all budgets; it instead emphasizes deployable budgets, where EpiKV or related attention-free variants are parity-or-better while preserving FlashAttention compatibility (&&&^^^^1ti:\1^^^^&&&).

6. Limitations, assumptions, and interpretation

The paper is explicit about its limits. Evaluation is shown on a single model family, DeepSeek-R1-Distill-LLaMA-8B, and cross-architecture transfer is not tested. The results are budget-sensitive, and no single FlashAttention-2-compatible method wins at every budget. AIME-2^^^^1ti:\1^^^^24 contains only 3^^^^1ti:\1^^^^ problems, so the paper treats its results as directional and notes that the confidence intervals for AIME span zero. Throughput measurements are single-GPU, single-example measurements, with some results described as projections or microbenchmarks rather than full batched deployment measurements (&&&^^^^1ti:\1^^^^&&&).

The selected layer bands are also not presented as universal constants. The paper states that important layers shift with task difficulty and that, on GSM8K, the anatomy moves toward earlier layers. Layers 1^^^^1ti:\1^^^^ and 21 are therefore regime-specific hyperparameters rather than architecture-independent invariants. The method also assumes access to hidden states through standard inference APIs, a causal decode setting in which token importance can be fixed online at generation time, and a policy in which prompt or prefill tokens and a recency window are always preserved (&&&^^^^1ti:\1^^^^&&&).

A plausible implication is that EpiKV should be understood less as a closed-form universal recipe than as a specific design pattern for attention-free cache eviction: rank tokens by locally normalized representational change, preserve structural tokens, and keep the policy causal and production-compatible. Within that framing, its main contribution is to redefine token importance in long reasoning traces as a property of internal state transition rather than explicit attention allocation (&&&^^^^1ti:\1^^^^&&&).