LAVa: Dynamic KV Cache Eviction

• The paper introduces a unified strategy for KV-cache compression that minimizes information loss by dynamically allocating budgets across Transformer layers and heads.
• It utilizes a theoretically informed score based on layer attention output loss and value norms to guide token eviction.
• Empirical results demonstrate up to 9× faster decoding and robust performance on benchmarks like LongBench and Needle-In-A-Haystack, enhancing both extraction and generation tasks.

Searching arXiv for the specific LAVa paper and the KV-cache compression methods it is compared against. {"query":"arXiv (Shen et al., 11 Sep 2025) LAVa Layer-wise KV Cache Eviction with Dynamic Budget Allocation"} {"query":"H2O KV cache compression arXiv SnapKV PyramidKV AdaKV CAKE VATP"} LAVa denotes “Layer-wise KV Cache Eviction with Dynamic Budget Allocation”, a cache-compression method for long-context LLM inference that frames KV-cache eviction as the minimization of information loss in Transformer residual streams rather than as a collection of heuristics (Shen et al., 11 Sep 2025). Its defining contribution is a unified, training-free strategy that allocates cache budgets dynamically across both heads and layers, using a theoretically motivated score derived from layer attention output loss. In the reported experiments, LAVa is evaluated on LongBench, Needle-In-A-Haystack, Ruler, and InfiniteBench, where it is described as consistently maintaining top performance across task types while reducing memory pressure and accelerating decoding (Shen et al., 11 Sep 2025).

1. Problem setting and motivation

KV cache is commonly used to accelerate LLM inference with long contexts, yet its high memory demand creates a direct pressure for cache compression (Shen et al., 11 Sep 2025). The central problem addressed by LAVa is that existing compression methods are described as largely heuristic and as lacking dynamic budget allocation, especially when the cache budget must be divided non-uniformly across attention heads and Transformer layers (Shen et al., 11 Sep 2025).

LAVa treats cache compression as an optimization problem over a global cache budget $\mathbb{B}$:

$$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$

subject to

$$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$

Here $\mathcal{I}_{l,h}[i]$ is a binary mask indicating whether token $i$ is kept in head $h$ of layer $l$, while $\mathcal{B}_{l,h}$ and $\mathcal{B}_l$ denote the budgets assigned to a head and a layer, respectively (Shen et al., 11 Sep 2025). A further constraint is that recent tokens within a window of size $w$ are always retained (Shen et al., 11 Sep 2025). The information loss term $$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$0 is described as the cross-entropy between the original logits and those computed with the compressed cache (Shen et al., 11 Sep 2025).

This formulation is significant because it shifts the design of cache eviction from hand-crafted retention rules toward an explicit loss-minimization objective. A plausible implication is that the method aims to make budget assignment a first-class optimization variable rather than a post hoc engineering choice.

2. Layer attention output loss as the organizing principle

Instead of approximating the global logit loss directly, LAVa focuses on layer attention output loss (Shen et al., 11 Sep 2025). The paper states the following relation:

$$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$1

where $$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$2 is the layer attention output with the full cache and $$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$3 is the corresponding output after eviction (Shen et al., 11 Sep 2025). This localizes the compression objective to the level of attention-output perturbation.

The main theoretical device is an upper bound for the $$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$4 layer output loss:

$$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$5

where $$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$6 is the attention weight of token $$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$7 for head $$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$8 at step $$\min_{\mathcal{I},\mathcal{B}} \mathcal{P}(x^{1\ldots N}_1, \mathcal{I}, \mathcal{B})$$9, $$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$0 is the maximum value norm for that head, and $$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$1 is a constant (Shen et al., 11 Sep 2025). In the paper’s interpretation, this bound directly links eviction loss to both attention mass and value magnitude.

This is the conceptual core of LAVa. Existing methods are often compared through attention-only or rule-based criteria, whereas LAVa uses a bound that incorporates value norms as well (Shen et al., 11 Sep 2025). That makes it possible to compare entries across heads, which is necessary if head budgets are to be allocated dynamically rather than uniformly.

3. Token scoring and dynamic head budgets

From the loss bound, LAVa derives a score for each token, head, and layer:

$$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$2

This score combines recent attention scores over a window of size $$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$3 with the maximum value norm for the head (Shen et al., 11 Sep 2025). The score is explicitly intended to measure cross-head, cross-layer importance and thereby permit comparison of entries that older heuristics typically rank only within a head (Shen et al., 11 Sep 2025).

Dynamic head budgeting is then performed by flattening all scores $$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$4 across heads within a layer and evicting the least important entries first, up to the layer budget $$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$5 (Shen et al., 11 Sep 2025). The result is a non-uniform allocation of per-head cache capacity that adapts at each decoding step.

The paper emphasizes that this head-level adaptivity is especially important for extraction tasks (Shen et al., 11 Sep 2025). In the reported ablations, disabling dynamic head budget sharply reduces performance on extraction tasks, while retaining the full dynamic mechanism preserves strong results across task families (Shen et al., 11 Sep 2025). This suggests that extractive workloads depend sensitively on a small subset of heads whose retention needs differ from the layer average.

4. Dynamic layer budgets

LAVa also allocates the global budget dynamically across layers by measuring uncertainty in the score distribution for each layer (Shen et al., 11 Sep 2025). The procedure first normalizes scores into probabilities,

$$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$6

then computes layer entropy,

$$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$7

and finally sets the layer budgets proportionally,

$$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$8

The stated interpretation is that layers with greater uncertainty receive more budget (Shen et al., 11 Sep 2025). In other words, when the score distribution is less concentrated and eviction choices are less clear, the method preserves more entries.

This layer-level mechanism is reported to be crucial for generation tasks, including code completion (Shen et al., 11 Sep 2025). The experiments specifically identify a task-dependent asymmetry: dynamic layer budgets are crucial for generation tasks, whereas dynamic head budgets play a key role in extraction tasks (Shen et al., 11 Sep 2025). That distinction is one of the paper’s main empirical insights, and it motivates the claim that a fully dynamic method is preferable to strategies that adapt only one axis of the cache.

5. Empirical performance and benchmark behavior

LAVa is evaluated on LongBench, Needle-In-A-Haystack, Ruler, and InfiniteBench (Shen et al., 11 Sep 2025). The abstract states that these experiments demonstrate its superiority, and the detailed summary adds that LAVa outperforms all baselines on LongBench across various budgets, with especially strong gains at lower budgets, while also delivering the best or equal-best retrieval accuracy on Needle-In-A-Haystack among open methods (Shen et al., 11 Sep 2025). On Ruler and InfiniteBench, it is described as consistently delivering higher average performance than prior pruning and budgeting methods (Shen et al., 11 Sep 2025).

A prominent systems result is that LAVa yields $$\sum_{i \in [N]} \mathcal{I}_{l,h}[i] = \mathcal{B}_{l,h}; \qquad \sum_{h \in [H]} \mathcal{B}_{l,h} = \mathcal{B}_l; \qquad \sum_{l \in [L]} \mathcal{B}_l = \mathbb{B}.$$9 faster decoding for 128K-token sequences relative to a full-cache FlashAttention-2 baseline (Shen et al., 11 Sep 2025). This is paired with a memory tradeoff: the method requires slightly more memory than single-layer, single-head methods because it stores value norms, but it remains much more efficient than a full cache or complex hybrid methods such as CAKE and PyramidKV (Shen et al., 11 Sep 2025).

The ablation studies are used to isolate the contributions of its components. The score function itself provides large gains even with fixed budgets; removing dynamic head budgets harms extraction tasks; removing dynamic layer budgets most harms generation tasks; and combining both mechanisms yields the strongest overall robustness across task types (Shen et al., 11 Sep 2025). Within the reported evidence, this supports the paper’s description of LAVa as a fully dynamic compression method rather than a single-metric eviction rule.

6. Relation to prior KV-cache compression methods and naming ambiguity

The paper positions LAVa against several earlier cache-compression strategies. It contrasts LAVa with heuristic-based methods such as H2O and SnapKV, with fixed rule-based layer budgeting such as PyramidKV, with dynamic layer budgeting methods such as CAKE, with per-head adaptive budgeting methods such as AdaKV, and with value-aware token scoring methods such as VATP (Shen et al., 11 Sep 2025). The claimed distinction is that LAVa is the first unified strategy for cache eviction and dynamic budget allocation that does not rely on training or on combining multiple strategies (Shen et al., 11 Sep 2025).

This comparative framing also clarifies a common source of confusion: the string “LAVA” or “Lava” appears in multiple unrelated literatures. Examples in the provided corpus include “LAVA: Language Audio Vision Alignment for Contrastive Video Pre-Training” (Gurram et al., 2022), “LAVA: Label-efficient Visual Learning and Adaptation” (Nassar et al., 2022), “LAVA: Long-horizon Visual Action based Food Acquisition” (Bhaskar et al., 2024), and several works tied to the neuromorphic Lava software framework (Snyder et al., 2023, Snyder et al., 2024, Snyder et al., 2024). In the LLM-inference literature, however, “LAVa” specifically denotes the layer-wise KV-cache eviction method introduced in 2025 (Shen et al., 11 Sep 2025).

That naming ambiguity matters because the capitalization is not incidental. In this context, the lowercase terminal “a” identifies a particular contribution in cache compression, not the broader set of papers using “LAVA” as an acronym in multimodal learning, robotics, traffic analytics, or planetary science (Shen et al., 11 Sep 2025).

7. Significance and implications

LAVa’s importance lies in combining three elements that earlier methods often separated: a formal objective based on information loss in residual streams, a score that compares entries across heads using both attention and value magnitude, and a dynamic allocator that redistributes the budget across layers according to uncertainty (Shen et al., 11 Sep 2025). The result is a unified, training-free framework for cache eviction that is intended to remain effective across heterogeneous workloads.

The strongest empirical interpretation advanced by the paper is that cache compression is not governed by a single universal retention policy. Instead, the relevant budget axis depends on the task family: layer adaptivity is decisive for generation, and head adaptivity is decisive for extraction (Shen et al., 11 Sep 2025). This suggests that future KV-cache research will likely continue to move away from uniform or fixed-budget schemes.

At the same time, the paper does not present LAVa as costless. The method incurs a modest memory overhead relative to single-layer, single-head approaches because of value-norm storage (Shen et al., 11 Sep 2025). Even so, within the reported benchmark regime it is presented as a favorable tradeoff, pairing substantial speed gains with top or near-top task performance under constrained cache budgets (Shen et al., 11 Sep 2025).