---
title: 'CompressKV: Efficient KV Cache Compression'
url: https://www.emergentmind.com/topics/compresskv
type: topic
---

# CompressKV: Efficient KV Cache Compression

Searching arXiv for recent papers on CompressKV and closely related KV-cache compression methods.
In contemporary transformer inference, “CompressKV” denotes the reduction of the key–value cache footprint during autoregressive decoding, a problem that becomes acute because KV storage grows linearly with context length and model depth. In the narrower, paper-specific sense, CompressKV is a KV-cache compression framework for Grouped Query Attention (GQA)-based large language models that selects retained tokens using **Semantic Retrieval Heads (SRHs)** and allocates per-layer KV budgets from offline estimates of layer-wise eviction error [2606.24467]. The method was introduced to address a recurring limitation of prior eviction schemes: aggregating attention over all heads can cause **Streaming Heads** to dominate token scoring, which in turn retains mostly boundary tokens and evicts semantically important mid-context evidence [2508.02401].

## 1. Problem formulation and architectural setting

In a decoder-only transformer, each generated token attends to all previous tokens through a per-layer KV cache. For a sequence of length \(T\), memory grows linearly with sequence length for each layer, and long-context inference is therefore constrained by both KV storage and decode-time access cost [2606.24467]. This issue is especially salient in GQA-based models such as LLaMA-3.1, Mistral, and Qwen2.5, where many query heads share a smaller number of key–value groups [2606.24467].

CompressKV is positioned within the eviction family of KV compression methods. In these methods, the cache is reduced by dropping tokens whose KV entries are judged “unimportant.” Existing approaches such as StreamingLLM, SnapKV, PyramidKV, CAKE, HeadKV, and AdaKV typically compute token importance by summing or averaging attention over all heads or all groups, then retain top-ranked tokens under a fixed or adaptively allocated memory budget [2606.24467]. The central critique advanced by CompressKV is that this procedure is **head-agnostic**: it assumes that all heads are equally informative, even though different heads implement distinct functions such as streaming, retrieval, and reasoning [2508.02401].

The method therefore recasts KV eviction as a head-function-aware selection problem. Rather than letting all heads vote on token importance, it first identifies the subset of heads that reliably focus on answer spans and semantically relevant context, then uses only those heads to determine which tokens survive compression [2606.24467].

## 2. Semantic Retrieval Heads

The defining construct of CompressKV is the **Semantic Retrieval Head**. SRHs are heads that place substantial attention mass over the answer span and surrounding semantic context when the model is generating correct answers [2606.24467]. They are intended to capture both boundary-token retrieval and semantically important interior evidence, and are therefore broader than earlier notions of “retrieval heads” that depend on top-1 or top-\(k\) attention peaks on exact answer tokens [2508.02401].

CompressKV identifies SRHs offline using a span-aggregation score. For each head \(h\),
\[
S_{\mathrm{SRH}(h)}
= \sum_{t=1}^{N} \mathbf{1}_{\{y_t \in \mathcal{A}\}} \sum_{j \in \mathcal{A}} a_{t,j}^{(h)},
\]
where \(N\) is the number of generation steps, \(y_t\) is the generated token at step \(t\), \(\mathcal{A}\) is the set of answer-span positions, and \(a_{t,j}^{(h)}\) is the attention weight from the current query to token position \(j\) in head \(h\) [2606.24467]. This score credits heads that distribute attention over the entire answer span and nearby semantics, rather than only heads whose single largest attention weight lands exactly on an answer token [2508.02401].

This distinction matters empirically. Traditional retrieval-head scoring can assign many early-layer heads zero importance, whereas SRH scoring identifies non-zero but meaningful importance across more layers [2508.02401]. Causal masking experiments further support the distinction: masking top-\(10\) traditional retrieval heads on Needle-in-a-Haystack causes a small performance drop of about \(1.02\), whereas masking top-\(10\) SRHs causes a drop of \(24.55\); masking top-\(20\) or top-\(30\) SRHs produces drops of \(72.56\) and \(73.81\), respectively [2606.24467]. This suggests that SRHs are more causally central to long-context retrieval than peak-based retrieval heads.

For token selection, CompressKV chooses the top-\(k\) SRHs in each layer, with default \(k=4\) [2606.24467]. This choice is supported by ablation: on Mistral-7B at budget 256, top-\(4\) heads per layer yields the best result, while top-\(2\) underperforms and larger sets such as top-\(24\) introduce noise [2606.24467].

## 3. Token ranking and layer-adaptive budget allocation

After SRHs are identified, CompressKV computes token importance during prefilling using a SnapKV-style pooling procedure, but restricted to SRHs only. For each selected SRH in layer \(l\), the attention matrix over the prompt is collected during prefill and converted into a one-dimensional token-score vector using `window_size = 8` and `kernel_size = 5` [2606.24467]. These per-head vectors are then averaged:
\[
s_j^{(l)} = \frac{1}{k} \sum_{h \in \text{Top-}k\ \text{SRHs}} s_j^{(h)},
\]
and the top \(B_l\) tokens in that layer are retained [2606.24467]. The resulting retained-token index set is shared by all heads and all head groups in the layer.

This design preserves the structured KV layout expected by standard GQA implementations, while preventing streaming-head dominance in the token-scoring step. Qualitative attention analyses in the paper show that streaming heads sharply attend to first and last tokens, whereas semantic retrieval heads focus on answer tokens and semantically related interior words such as “eat” and “a thing” around “sandwich” [2606.24467]. CompressKV uses this difference operationally: only SRHs are allowed to define importance.

A second component determines **how many** tokens each layer may keep. CompressKV measures a per-layer eviction error by comparing full-cache and compressed-cache attention-block outputs:
\[
\mathbf{O}^l_{\text{full},t}
= \mathbf{W}^l_{O}\,\mathrm{Attn}\!\left(
\mathbf{Q}^l_t,\ \mathbf{K}^l_{\text{full}},\ \mathbf{V}^l_{\text{full}}
\right),
\]
\[
\mathbf{O}^l_{\text{comp},t}
= \mathbf{W}^l_{O}\,\mathrm{Attn}\!\left(
\mathbf{Q}^l_t,\ \mathbf{K}^l_{\text{comp}},\ \mathbf{V}^l_{\text{comp}}
\right),
\]
then aggregates relative perturbation across decoding steps:
\[
e^{(l)} =
\sum_{t=1}^{T}
\frac{\|\Delta \mathbf{O}^{l}_{t}\|_{F}}
{\|\mathbf{O}^{l}_{\mathrm{full},t}\|_{F} + \epsilon},
\qquad
\tilde{e}^{(l)} =
\frac{e^{(l)}}{\sum_k e^{(k)}}.
\]
Layers with larger \(\tilde{e}^{(l)}\) are treated as more sensitive to KV compression and therefore receive larger budgets [2606.24467].

Given a total budget \(B_{\text{total}}\), a minimum budget \(m\), and a maximum budget \(M\), the allocation proceeds by first assigning \(m\) tokens to every layer, then distributing the remaining capacity proportionally to \(\tilde{e}^{(l)}\), while clipping each layer to \(M\) [2606.24467]. In the reported configuration, \(m = 32\) and \(M = 3 \times B_{\text{per-layer}}\) [2606.24467]. Because this calculation is done offline, the method avoids the online entropy- or variance-based heuristics used by several earlier schemes.

## 4. Empirical behavior

CompressKV is evaluated on LongBench and Needle-in-a-Haystack across GQA-based models including LLaMA-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Qwen2.5-14B-Instruct, and Qwen2.5-32B-Instruct [2606.24467]. The reported comparisons use greedy decoding and match baselines under the same average per-layer KV budgets [2606.24467].

On **LongBench**, the method consistently improves the accuracy–budget trade-off. For LLaMA-3.1-8B, the full-cache average is \(49.08\). At **256 tokens per layer**, CompressKV reaches \(46.71\), compared with \(45.21\) for SnapKV, \(44.36\) for PyramidKV, \(46.30\) for CAKE, \(44.11\) for HeadKV, and \(44.45\) for AdaKV [2606.24467]. At **1024 tokens per layer**, it reaches \(48.24\), slightly above the strongest baselines [2606.24467]. Similar patterns hold for Mistral-7B, Qwen2.5-14B, and Qwen2.5-32B [2606.24467]. The paper states that CompressKV maintains **over 99%** of full-cache performance with only **19%** KV budget, and retains **97%** of full QA performance with **3%** KV budget on LongBench [2606.24467].

On **Needle-in-a-Haystack**, CompressKV outperforms StreamingLLM, SnapKV, PyramidKV, CAKE, HeadKV, and AdaKV at every tested budget [2606.24467]. For LLaMA-3.1-8B, it reaches near-lossless performance at **2048 tokens per layer**, corresponding to roughly **5%** of the full cache, and retains about **90%** of original accuracy with only **256 tokens per layer**, corresponding to about **0.7%** of the full KV capacity [2606.24467]. The abstract summarizes this as **90% accuracy with just 0.7% KV storage** on Needle-in-a-Haystack [2606.24467].

Ablations isolate the contributions of the two main components. On Mistral-7B with budget 256, the SnapKV baseline scores \(43.76\); replacing its token-selection stage with SRH-guided selection increases this to \(44.96\), and adding layer-aware budget allocation yields \(45.43\) [2606.24467]. This supports the interpretation that SRH-based token ranking and error-aware layer allocation are complementary rather than redundant.

## 5. Relation to the broader KV-compression literature

CompressKV belongs to a broader family of methods that attempt to reduce the cost of long-context inference by operating on the KV cache. Within that space, it is most closely aligned with **eviction-based** methods, but it is distinguished by its explicit treatment of head heterogeneity and by its offline layer-allocation mechanism [2606.24467].

Several adjacent lines of work pursue other structural choices. **KVCompose** keeps the cache fully structured for standard inference engines while allowing head-specific token selection through “composite tokens,” and assigns budgets across layers using a global allocation mechanism over attention-derived composite scores [2509.05165]. **CommonKV** attacks cross-layer redundancy through adjacent parameter sharing, SVD-based latent KV representations, and adaptive budget allocation based on cosine similarity, with reported compatibility with quantization and eviction methods [2508.16134]. **VQKV** uses residual simple vector quantization to compress keys and values into integer indices, reporting an **82.8%** compression ratio on LLaMA3.1-8B while retaining **98.6%** of baseline LongBench performance and enabling **4.3×** longer generation length on the same memory footprint [2603.16435]. **LeanKV** unifies heterogeneous key/value quantization with token pruning and an on-GPU memory manager, achieving **2.7× to 5.7×** KV compression with near-lossless accuracy and **1.9× to 5.4×** throughput gains [2412.03131].

This broader landscape suggests that “CompressKV” can denote both a specific SRH-based framework and, more generally, the class of techniques that reduce KV memory by eviction, quantization, projection, or cross-layer sharing. A plausible implication is that these approaches are partly orthogonal. The CompressKV papers themselves explicitly discuss compatibility with MInference, XAttention, KIVI, HeadKV, and AdaKV-style head-level allocation [2606.24467]. That is consistent with the wider literature, in which CommonKV is described as orthogonal to quantization and eviction [2508.16134], and VQKV addresses a different axis—vector quantization rather than token retention [2603.16435].

## 6. Integration, scope, and limitations

CompressKV is designed for standard decoder-only inference pipelines on GQA-based LLMs. It does not require retraining or fine-tuning: offline work consists of SRH identification on a calibration dataset and layer-wise error estimation on representative long-context tasks, after which online inference compresses the cache during prefill and continues decoding with the reduced cache [2606.24467]. Operationally, it fits naturally between prefill and decode: per layer, SRH attention is pooled, tokens are ranked, the top \(B_l\) indices are retained, and all other KV entries are evicted [2508.02401].

The method’s scope is also explicit. It is designed for GQA-based models and relies on head-function heterogeneity being meaningful and sufficiently stable to support offline SRH selection [2606.24467]. The policy is static once computed: SRH sets and per-layer budget vectors are fixed per model and budget level rather than recomputed per prompt [2606.24467]. The evaluation concentrates on LongBench and Needle-in-a-Haystack, which emphasize retrieval-heavy and question-answering behavior, so broader task coverage remains limited in the available evidence [2606.24467].

These limitations do not reduce the conceptual significance of the method. CompressKV’s main contribution is to turn mechanistic observations about attention heads into a concrete systems primitive: token retention is no longer a pooled statistic over all heads, but a selection process driven by the heads that best preserve answer-span and semantic-context retrieval [2508.02401]. In that sense, CompressKV occupies a specific and influential position within KV-cache compression research: it is an eviction framework whose distinguishing claim is not merely that some tokens matter more than others, but that **some heads know which tokens matter before generation** [2508.02401].

Source: https://www.emergentmind.com/topics/compresskv