KV Cache Eviction in Transformer LLMs

• KV Cache Eviction is a technique that manages memory growth in transformer LLMs by selectively evicting intermediate key-value tensors.
• It employs methods like mixed-precision quantization and adaptive allocation to balance performance with storage efficiency.
• Advanced strategies incorporate compensation mechanisms and hardware-aware optimizations to preserve context integrity.

Key-Value (KV) cache eviction is a critical memory management problem in the deployment and inference of transformer-based LLMs, especially as sequence lengths, batch sizes, and model scales increase. The KV cache holds the intermediate key and value tensors associated with all previous tokens, dramatically accelerating autoregressive generation by avoiding redundant computation. However, the cache’s linear growth with respect to sequence length and batch size often leads it to dwarf even the model’s own parameter memory footprint—creating a principal bottleneck in both throughput and scalability. Consequently, a rich array of algorithms has emerged to efficiently evict, compress, or quantize the KV cache while striving to minimize adverse effects on performance, context integrity, and reliability.

1. Motivations, Risks, and Challenges of KV Cache Eviction

The transformer architecture’s reliance on the KV cache introduces grave memory inefficiencies as context length expands, with some deployments observing the cache exceeding model parameter storage. While discarding (evicting) the least “important” KV pairs has become a common approach, empirical evidence demonstrates that naive or aggressive eviction frequently triggers critical failures—loss of system prompt memory leading to safety breaches, hallucinations, and context loss. Even theoretically optimal oracle eviction strategies exhibit significant degradation as more KV pairs are dropped, highlighting intrinsic risks when informative context is permanently removed (Yang et al., 28 Feb 2024).

A major challenge is that importance signals typically used for scoring (e.g., attention accumulations or derived key/query statistics) are often positionally biased, model-specific, or fail to anticipate the shifting relevance of tokens during decoding and multi-turn interaction. These signal limitations motivate new methods that preserve as much information as possible from evicted tokens, adapt precision, or re-inject context information through compensation.

2. Mixed Precision and Hybrid Retention-Compression Methods

The catastrophic context loss caused by hard eviction can be substantially mitigated by retaining even a lossy, quantized version of discarded KV entries. The MiKV approach (Yang et al., 28 Feb 2024) therefore compresses the entire KV cache using importance-aware mixed-precision quantization: critical tokens are maintained in high precision (e.g., FP16), while less important KV pairs are down-quantized (e.g., INT2/INT4). Formally, the quantization is performed per token using

$$\widehat{x} = \alpha \left\lfloor \frac{x-\beta}{\alpha} \right\rceil + \beta, \quad \text{where } \alpha = \frac{\max(x) - \min(x)}{2^N - 1}, \quad \beta = \min(x).$$

Critically, channel-wise outliers in query/key distributions are dynamically rebalanced by scaling each head/channel with

$$b_{lhc} = \sqrt{ \frac{\max|q_{lhc}|}{\max|k_{lhc}|} }$$

before quantization, ensuring that low-bit precision KV pairs still preserve essential signal. This hybrid retention-compression secures state-of-the-art performance and circumvents the abrupt accuracy drops associated with abrupt “all-or-nothing” eviction.

Quantitative experiments (e.g., on line retrieval, MMLU, HumanEval, GSM8k) show that retaining even INT3/INT4 representations of discarded KV entries recovers near-full accuracy, with overall KV cache size compressed to 20–25% of original without substantive performance loss.

3. Adaptive Allocation and Fine-Grained Budgeting

Uniformly distributing the cache budget over layers or attention heads—an approach typical in earlier works (e.g., SnapKV, PyramidKV)—wastes resources on some heads that are either highly concentrated or relatively irrelevant. Ada-KV (Feng et al., 16 Jul 2024) analytically models the upper bound on generation loss caused by cache eviction as

$$D = 2hC - 2C \sum_{i=1}^{h} \sum_{j \in \text{retained}} A^{ij},$$

and demonstrates that minimizing $D$ (subject to a global KV budget $B$) is equivalent to maximizing the sum of retained attention weights. The optimal (loss-minimizing) budget allocation is thus head-dependent:

1. Concatenate all attention weights across heads and select the top $B$ indices overall.
2. Assign the per-head budget $B^*_i$ as the count of top indices within each head.

Plug-and-play integration of this adaptive strategy with previous eviction baselines (e.g., Ada-SnapKV, Ada-Pyramid) reduces the theoretical upper bound on quality loss, yielding empirical improvements in open-ended language understanding and retrieval tasks, especially under tight memory budgets.

4. Specialized Head- and Task-Aware Compression Schemes

Not all attention heads contribute equally to global context retention. RazorAttention (Tang et al., 22 Jul 2024) observes that only a very small fraction of “retrieval heads” reliably attend to all context tokens, whereas most are strictly local and replicate the “attention sink” effect. In this design, the full cache is maintained for retrieval heads, while all others are truncated to recent tokens; remote tokens in these non-retrieval heads are replaced with “compensation tokens”—aggregates

$$\hat{k} = \frac{1}{N_d} \sum_{m \in \mathcal{D}} k_m, \qquad \hat{v} = \frac{1}{N_d} \sum_{m \in \mathcal{D}} v_m,$$

and attention outputs are correspondingly weighted. This architecture achieves >70% KV cache compression with minimal accuracy loss and, crucially, is compatible with high-performance attention kernels such as FlashAttention due to its head-wise rather than per-token importance logic.

Other methods (e.g., CAKE (Qin et al., 16 Mar 2025)) further introduce layer preference-aware “cake slicing” for cache. Preferences are computed using both spatial attention entropy and temporal variance:

$$P = \mathcal{H}^{1/\tau_1} \, \mathcal{V}^{1/\tau_2}, \qquad B_\ell = \frac{P_\ell}{\sum_k P_k} B_{\text{total}},$$

enabling adaptive, globally coordinated cache allocation that dynamically responds to both input structure and layer-specific information flow.

5. Theory, Limitations, and Pitfalls of Static and Attention-Based Heuristics

Purely attention-score based eviction, whether accumulated over history (as in StreamingLLM, H2O, TOVA, SnapKV, PyramidKV, Scissorhands), or computed from explicit local or block-wise attention, suffers several bias sources:

• Monotonic position bias, resulting in systematic retention of early (prefix) tokens even when their contextual utility wanes (Gu et al., 4 Jun 2025).
• Incompatibility with FlashAttention and other efficient attention kernels, as materializing large attention matrices is prohibitive for long (e.g., >100K token) contexts (Tang et al., 22 Jul 2024).
• Failure to anticipate “token importance recurrence” (where crucial tokens become salient only later in decoding, as revealed in LazyEviction (Zhang et al., 19 Jun 2025) and in various chain-of-thought reasoning tasks).

To mitigate these weaknesses, methods such as AhaKV (Gu et al., 4 Jun 2025) propose adaptive holistic attention—where eviction scoring restricts attention accumulation to a fixed recent window, and applies “step gain softmax” scaling,

$\lambda = \sqrt{ 2 \log(i/k) / d },$

selected to maintain expected attention entropy, thereby ensuring late tokens are not disproportionately discounted. Moreover, injecting value vector priors (e.g., scaling by L₂ norm, filtered and renormalized) further balances eviction scores.

6. Efficient, Query-Consistent, and Hardware-Aware Eviction Algorithms

Query consistency is a persistent challenge: tokens judged unimportant during input-prefill can become critical in decoding—misaligned selection thus degrades model performance. The Lookahead Q-Cache (LAQ) approach (Wang et al., 24 May 2025) addresses this by generating pseudo lookahead queries in a low-cost pre-decoding phase, using these Q-Cache samples to more accurately estimate future token importance:

$$M_{\text{ahead}} = \operatorname{ArgSort} \left( \sum_{q_i \in Q} q_i K^\top \right)[:B]$$

where Q is the set of lookahead queries. This mechanism yields significant accuracy benefits in long-context and retrieval evaluations under tight cache budgets.

For hardware-constrained or secure inference scenarios, additional constraints apply. HashEvict (Liu et al., 13 Dec 2024) computes “pre-attention” eviction by using binarized locality-sensitive hashes as lightweight surrogates for true attention, enabling O(1) cache updates per step with minimal resource cost. MPCache (Zeng et al., 12 Jan 2025) designs its algorithms for secure multi-party computation by combining one-pass static eviction (to remove always-unimportant tokens) with a dynamic query-aware selection stage and multiple communication/min-computation optimizations (e.g., approximate similarity, cluster-level pruning, cross-layer index sharing).

Voting-based algorithms (Wang et al., 1 Jul 2025) consider each token as a “voter” on eviction and aggregate votes across tokens using adaptive thresholds, prioritizing cache updates that best balance hardware simplicity with retention of semantically significant KV pairs, and exploiting efficient, reconfigurable hardware arrays to maintain low-latency inference.

7. Advanced Compensation, Calibration, and Multi-Model Collaboration

To address phenomena such as “saliency shift” (where the tokens’ importance changes over the course of decoding), recent methods deploy auxiliary compensation and calibration strategies. SmallKV (Zhao et al., 3 Aug 2025) uses a small LLM (SLM) as a proxy to supply attention patterns for tokens deemed marginally important in the main LLM, maintaining critical context even under aggressive compression. Attention matching between the full KV cache of the SLM and the (progressively evicted) LLM is exploited to approximate marginal token attention and losslessly substitute for evicted keys.

CaliDrop (Su et al., 26 Jul 2025) proposes a speculative calibration mechanism whereby offloaded (evicted) tokens are not discarded outright; instead, their previously computed attention contributions are stored and conditionally re-used during future decoding if the cosine similarity between current and historical queries exceeds a tunable threshold. If similarity falls below a lower threshold, full recalculation can be triggered, trading modest recomputation cost for substantial accuracy improvements compared to pure hard-eviction.

Methods such as LazyEviction (Zhang et al., 19 Jun 2025) integrate observation window-based lagged eviction and recurrence interval tracking, retaining tokens based on their maximum observed recurrence interval—a design crucial for multi-step reasoning where knowledge must “recur” throughout long chains of attention.

Summary Table: Selected KV Cache Eviction Strategies

Method	Principle	Key Innovations
MiKV (Yang et al., 28 Feb 2024)	Mixed-precision quantization	Retains evicted KV pairs in INT2/INT3/INT4; outlier-aware balancing
Ada-KV (Feng et al., 16 Jul 2024)	Adaptive head-wise budgeting	Minimize upper bound on attention-output loss; plug-in to Top-K
RazorAttention (Tang et al., 22 Jul 2024)	Retrieval head/full vs. truncated cache	Compensation tokens; head-specific mechanics; FlashAttention-compatible
CAKE (Qin et al., 16 Mar 2025)	Layer preference/cascading allocation	Spatial+temporal attention indicators, dynamic reallocation
KeyDiff (Park et al., 21 Apr 2025)	Key similarity to anchor vector	Training-free, no attention computation, FlashAttention friendly
AhaKV (Gu et al., 4 Jun 2025)	Adaptive holistic attention/value prior	Adaptive λ in softmax, recent-window summation, value-based scaling
SmallKV (Zhao et al., 3 Aug 2025)	SLM attention compensation	Multi-model attention matching, marginal token substitution
CaliDrop (Su et al., 26 Jul 2025)	Calibration of evicted tokens	Historical query matching, conditional offloaded KV reuse
LazyEviction (Zhang et al., 19 Jun 2025)	Recurrence interval tracking	Windowed eviction, maximum recurrence-based retention

Concluding Remarks

State-of-the-art research on KV cache eviction demonstrates that naïve token dropping—based solely on historic attention statistics—risks catastrophic performance loss due to hallucination, context loss, or safety violations. Hybrid methods that compress, quantize, or otherwise transform “evicted” KV pairs prove substantially more robust, especially when combined with head/layer-adaptivity, key-similarity driven logic, and compensation schemes that proactively utilize auxiliary models or historical computations. Hardware and systems-level integration (FlashAttention, MPC, edge accelerators) has become integral, pushing methods toward explicit hardware-awareness and pre-attention logic. Open problems remain, particularly around task-specific or input-adaptive KV retention, integration with future attention and quantization kernels, and the balance between aggressive compression and context integrity under real-world, multi-turn, or multi-query workflows.