CompressKV: Efficient KV Cache Compression

• CompressKV is a KV-cache compression framework for GQA-based LLMs that uses Semantic Retrieval Heads to retain semantically important tokens.
• It employs a head-function-aware token selection strategy to mitigate biases from traditional methods that favor boundary tokens.
• Empirical evaluations show that CompressKV maintains over 99% of full-cache performance with significantly reduced memory usage.

Searching arXiv for papers on CompressKV and closely related KV-cache compression methods. In contemporary transformer inference, “CompressKV” denotes the reduction of the key–value cache footprint during autoregressive decoding, a problem that becomes acute because KV storage grows linearly with context length and model depth. In the narrower, paper-specific sense, CompressKV is a KV-cache compression framework for Grouped Query Attention (GQA)-based LLMs that selects retained tokens using Semantic Retrieval Heads (SRHs) and allocates per-layer KV budgets from offline estimates of layer-wise eviction error (Lin et al., 23 Jun 2026). The method was introduced to address a recurring limitation of prior eviction schemes: aggregating attention over all heads can cause Streaming Heads to dominate token scoring, which in turn retains mostly boundary tokens and evicts semantically important mid-context evidence (Lin et al., 4 Aug 2025).

1. Problem formulation and architectural setting

In a decoder-only transformer, each generated token attends to all previous tokens through a per-layer KV cache. For a sequence of length $T$, memory grows linearly with sequence length for each layer, and long-context inference is therefore constrained by both KV storage and decode-time access cost (Lin et al., 23 Jun 2026). This issue is especially salient in GQA-based models such as LLaMA-3.1, Mistral, and Qwen2.5, where many query heads share a smaller number of key–value groups (Lin et al., 23 Jun 2026).

CompressKV is positioned within the eviction family of KV compression methods. In these methods, the cache is reduced by dropping tokens whose KV entries are judged “unimportant.” Existing approaches such as StreamingLLM, SnapKV, PyramidKV, CAKE, HeadKV, and AdaKV typically compute token importance by summing or averaging attention over all heads or all groups, then retain top-ranked tokens under a fixed or adaptively allocated memory budget (Lin et al., 23 Jun 2026). The central critique advanced by CompressKV is that this procedure is head-agnostic: it assumes that all heads are equally informative, even though different heads implement distinct functions such as streaming, retrieval, and reasoning (Lin et al., 4 Aug 2025).

The method therefore recasts KV eviction as a head-function-aware selection problem. Rather than letting all heads vote on token importance, it first identifies the subset of heads that reliably focus on answer spans and semantically relevant context, then uses only those heads to determine which tokens survive compression (Lin et al., 23 Jun 2026).

2. Semantic Retrieval Heads

The defining construct of CompressKV is the Semantic Retrieval Head. SRHs are heads that place substantial attention mass over the answer span and surrounding semantic context when the model is generating correct answers (Lin et al., 23 Jun 2026). They are intended to capture both boundary-token retrieval and semantically important interior evidence, and are therefore broader than earlier notions of “retrieval heads” that depend on top-1 or top-$k$ attention peaks on exact answer tokens (Lin et al., 4 Aug 2025).

CompressKV identifies SRHs offline using a span-aggregation score. For each head $h$,

$$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$

where $N$ is the number of generation steps, $y_t$ is the generated token at step $t$, $\mathcal{A}$ is the set of answer-span positions, and $a_{t,j}^{(h)}$ is the attention weight from the current query to token position $j$ in head $k$0 (Lin et al., 23 Jun 2026). This score credits heads that distribute attention over the entire answer span and nearby semantics, rather than only heads whose single largest attention weight lands exactly on an answer token (Lin et al., 4 Aug 2025).

This distinction matters empirically. Traditional retrieval-head scoring can assign many early-layer heads zero importance, whereas SRH scoring identifies non-zero but meaningful importance across more layers (Lin et al., 4 Aug 2025). Causal masking experiments further support the distinction: masking top-$k$1 traditional retrieval heads on Needle-in-a-Haystack causes a small performance drop of about $k$2, whereas masking top-$k$3 SRHs causes a drop of $k$4; masking top-$k$5 or top-$k$6 SRHs produces drops of $k$7 and $k$8, respectively (Lin et al., 23 Jun 2026). This suggests that SRHs are more causally central to long-context retrieval than peak-based retrieval heads.

For token selection, CompressKV chooses the top-$k$9 SRHs in each layer, with default $h$0 (Lin et al., 23 Jun 2026). This choice is supported by ablation: on Mistral-7B at budget 256, top-$h$1 heads per layer yields the best result, while top-$h$2 underperforms and larger sets such as top-$h$3 introduce noise (Lin et al., 23 Jun 2026).

3. Token ranking and layer-adaptive budget allocation

After SRHs are identified, CompressKV computes token importance during prefilling using a SnapKV-style pooling procedure, but restricted to SRHs only. For each selected SRH in layer $h$4, the attention matrix over the prompt is collected during prefill and converted into a one-dimensional token-score vector using window_size = 8 and kernel_size = 5 (Lin et al., 23 Jun 2026). These per-head vectors are then averaged: $h$5 and the top $h$6 tokens in that layer are retained (Lin et al., 23 Jun 2026). The resulting retained-token index set is shared by all heads and all head groups in the layer.

This design preserves the structured KV layout expected by standard GQA implementations, while preventing streaming-head dominance in the token-scoring step. Qualitative attention analyses in the paper show that streaming heads sharply attend to first and last tokens, whereas semantic retrieval heads focus on answer tokens and semantically related interior words such as “eat” and “a thing” around “sandwich” (Lin et al., 23 Jun 2026). CompressKV uses this difference operationally: only SRHs are allowed to define importance.

A second component determines how many tokens each layer may keep. CompressKV measures a per-layer eviction error by comparing full-cache and compressed-cache attention-block outputs: $h$7

$h$8

then aggregates relative perturbation across decoding steps: $h$9 Layers with larger $$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$0 are treated as more sensitive to KV compression and therefore receive larger budgets (Lin et al., 23 Jun 2026).

Given a total budget $$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$1, a minimum budget $$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$2, and a maximum budget $$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$3, the allocation proceeds by first assigning $$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$4 tokens to every layer, then distributing the remaining capacity proportionally to $$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$5, while clipping each layer to $$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$6 (Lin et al., 23 Jun 2026). In the reported configuration, $$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$7 and $$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$8 (Lin et al., 23 Jun 2026). Because this calculation is done offline, the method avoids the online entropy- or variance-based heuristics used by several earlier schemes.

4. Empirical behavior

CompressKV is evaluated on LongBench and Needle-in-a-Haystack across GQA-based models including LLaMA-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Qwen2.5-14B-Instruct, and Qwen2.5-32B-Instruct (Lin et al., 23 Jun 2026). The reported comparisons use greedy decoding and match baselines under the same average per-layer KV budgets (Lin et al., 23 Jun 2026).

On LongBench, the method consistently improves the accuracy–budget trade-off. For LLaMA-3.1-8B, the full-cache average is $$S_{\mathrm{SRH}(h)} = \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},$$9. At 256 tokens per layer, CompressKV reaches $N$0, compared with $N$1 for SnapKV, $N$2 for PyramidKV, $N$3 for CAKE, $N$4 for HeadKV, and $N$5 for AdaKV (Lin et al., 23 Jun 2026). At 1024 tokens per layer, it reaches $N$6, slightly above the strongest baselines (Lin et al., 23 Jun 2026). Similar patterns hold for Mistral-7B, Qwen2.5-14B, and Qwen2.5-32B (Lin et al., 23 Jun 2026). The paper states that CompressKV maintains over 99% of full-cache performance with only 19% KV budget, and retains 97% of full QA performance with 3% KV budget on LongBench (Lin et al., 23 Jun 2026).

On Needle-in-a-Haystack, CompressKV outperforms StreamingLLM, SnapKV, PyramidKV, CAKE, HeadKV, and AdaKV at every tested budget (Lin et al., 23 Jun 2026). For LLaMA-3.1-8B, it reaches near-lossless performance at 2048 tokens per layer, corresponding to roughly 5% of the full cache, and retains about 90% of original accuracy with only 256 tokens per layer, corresponding to about 0.7% of the full KV capacity (Lin et al., 23 Jun 2026). The abstract summarizes this as 90% accuracy with just 0.7% KV storage on Needle-in-a-Haystack (Lin et al., 23 Jun 2026).

Ablations isolate the contributions of the two main components. On Mistral-7B with budget 256, the SnapKV baseline scores $N$7; replacing its token-selection stage with SRH-guided selection increases this to $N$8, and adding layer-aware budget allocation yields $N$9 (Lin et al., 23 Jun 2026). This supports the interpretation that SRH-based token ranking and error-aware layer allocation are complementary rather than redundant.

5. Relation to the broader KV-compression literature

CompressKV belongs to a broader family of methods that attempt to reduce the cost of long-context inference by operating on the KV cache. Within that space, it is most closely aligned with eviction-based methods, but it is distinguished by its explicit treatment of head heterogeneity and by its offline layer-allocation mechanism (Lin et al., 23 Jun 2026).

Several adjacent lines of work pursue other structural choices. KVCompose keeps the cache fully structured for standard inference engines while allowing head-specific token selection through “composite tokens,” and assigns budgets across layers using a global allocation mechanism over attention-derived composite scores (Akulov et al., 5 Sep 2025). CommonKV attacks cross-layer redundancy through adjacent parameter sharing, SVD-based latent KV representations, and adaptive budget allocation based on cosine similarity, with reported compatibility with quantization and eviction methods (Wang et al., 22 Aug 2025). VQKV uses residual simple vector quantization to compress keys and values into integer indices, reporting an 82.8% compression ratio on LLaMA3.1-8B while retaining 98.6% of baseline LongBench performance and enabling 4.3× longer generation length on the same memory footprint (Wang et al., 17 Mar 2026). LeanKV unifies heterogeneous key/value quantization with token pruning and an on-GPU memory manager, achieving 2.7× to 5.7× KV compression with near-lossless accuracy and 1.9× to 5.4× throughput gains (Zhang et al., 2024).

This broader landscape suggests that “CompressKV” can denote both a specific SRH-based framework and, more generally, the class of techniques that reduce KV memory by eviction, quantization, projection, or cross-layer sharing. A plausible implication is that these approaches are partly orthogonal. The CompressKV papers themselves explicitly discuss compatibility with MInference, XAttention, KIVI, HeadKV, and AdaKV-style head-level allocation (Lin et al., 23 Jun 2026). That is consistent with the wider literature, in which CommonKV is described as orthogonal to quantization and eviction (Wang et al., 22 Aug 2025), and VQKV addresses a different axis—vector quantization rather than token retention (Wang et al., 17 Mar 2026).

6. Integration, scope, and limitations

CompressKV is designed for standard decoder-only inference pipelines on GQA-based LLMs. It does not require retraining or fine-tuning: offline work consists of SRH identification on a calibration dataset and layer-wise error estimation on representative long-context tasks, after which online inference compresses the cache during prefill and continues decoding with the reduced cache (Lin et al., 23 Jun 2026). Operationally, it fits naturally between prefill and decode: per layer, SRH attention is pooled, tokens are ranked, the top $y_t$0 indices are retained, and all other KV entries are evicted (Lin et al., 4 Aug 2025).

The method’s scope is also explicit. It is designed for GQA-based models and relies on head-function heterogeneity being meaningful and sufficiently stable to support offline SRH selection (Lin et al., 23 Jun 2026). The policy is static once computed: SRH sets and per-layer budget vectors are fixed per model and budget level rather than recomputed per prompt (Lin et al., 23 Jun 2026). The evaluation concentrates on LongBench and Needle-in-a-Haystack, which emphasize retrieval-heavy and question-answering behavior, so broader task coverage remains limited in the available evidence (Lin et al., 23 Jun 2026).

These limitations do not reduce the conceptual significance of the method. CompressKV’s main contribution is to turn mechanistic observations about attention heads into a concrete systems primitive: token retention is no longer a pooled statistic over all heads, but a selection process driven by the heads that best preserve answer-span and semantic-context retrieval (Lin et al., 4 Aug 2025). In that sense, CompressKV occupies a specific and influential position within KV-cache compression research: it is an eviction framework whose distinguishing claim is not merely that some tokens matter more than others, but that some heads know which tokens matter before generation (Lin et al., 4 Aug 2025).