CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

Abstract: Long-context LLM inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs. These methods ignore the different functionalities of attention heads, leading to the eviction of critical tokens and thus degrading the performance of LLMs. To address this issue, we propose CompressKV, a resource-efficient KV-cache compression framework for GQA-based LLMs. Instead of aggregating attention scores from all heads, CompressKV identifies Semantic Retrieval Heads (SRHs) that capture both the initial and final tokens of a prompt and semantically important mid-context evidence, and uses them to select tokens whose KV pairs should be retained. Furthermore, CompressKV allocates cache budgets across layers according to offline estimates of layer-wise eviction error. Experiments on LongBench and Needle-in-a-Haystack show that CompressKV consistently outperforms existing KV-cache eviction methods across memory budgets. Notably, it preserves over 97\% of full-cache performance using only 3\% of the KV cache on LongBench question-answering tasks and achieves 90\% accuracy with just 0.7\% KV storage on Needle-in-a-Haystack. These results demonstrate an improved resource--performance trade-off for long-context LLM inference. Our code is publicly available at: https://github.com/TUDa-HWAI/CompressKV

• The paper introduces Semantic Retrieval Heads (SRHs) to guide token retention in KV-cache compression for LLM inference.
• It employs offline error-aware layer-adaptive allocation to optimize cache retention while maintaining high performance.
• Experimental results across multiple LLMs show significant compression ratios, achieving up to 90% accuracy with extreme memory reduction.

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

Motivation and Problem Definition

The acceleration of LLMs toward extended context sizes has introduced prohibitive memory and latency bottlenecks, largely attributable to the linear scaling of key-value (KV) cache storage during autoregressive decoding. Predominant solutions employ token-level KV-cache eviction strategies, often aggregating attention scores across heads within GQA groups, neglecting head functionality heterogeneity. This head-agnostic approach inadvertently results in the eviction of mid-sequence, semantically critical tokens, thereby degrading model inference fidelity. CompressKV directly addresses this by introducing Semantic Retrieval Heads (SRHs), which are identified to guide attention-aware token retention and budget allocation.

Figure 1: Streaming heads’ dominance in token eviction causes crucial mid-context evidence to be lost, undermining inference quality.

Semantic Retrieval Head Identification and Token Selection

CompressKV departs from the conventional copy-and-paste or top-k hit criteria for retrieval heads by quantifying heads based on aggregated attention mass across the entire answer span. SRHs are credited not only for sharp, peak-driven attention but for distributing semantic attention around relevant spans. This span-aggregation enables robust capture of tokens that contribute to reasoning and context, even if not exhibiting strict peak behavior. The method evaluates heads offline using calibration datasets and computes each head’s SRH score as the total attention assigned to answer-span tokens during correct generations.

Figure 2: SRHs aggregate attention over the answer span, overcoming the copy-and-paste limitations of traditional retrieval head selection.

Token selection is then driven by the top SRH(s) per layer: tokens retaining high aggregated attention from these heads are kept in the cache, while others are evicted, ensuring mid-sequence evidence is not lost. All heads within each layer share preserved token indices determined by SRH-driven aggregation, as visualized below.

Figure 3: CompressKV identifies SRHs that exhibit distributed, semantically meaningful attention patterns across answer spans.

Error-Aware Layer-Adaptive Cache Allocation

To optimize cache retention under constrained budgets, CompressKV computes layer-wise "eviction error": the Frobenius norm of the difference between attention-block outputs using compressed versus full KV cache. Budgets are then allocated proportionally to these offline error estimates, ensuring layers implicated in retrieval and reasoning retain more tokens, while less critical layers receive minimal allocation. The procedure is entirely offline, eliminating runtime profiling overhead. Minimum and maximum per-layer allocations ensure full layer utilization while avoiding budget concentration.

Experimental Results and Numerical Evidence

CompressKV was evaluated across multiple GQA-based LLMs (Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Qwen2.5-14B/32B-Instruct) on LongBench (16 diverse subtasks) and Needle-in-a-Haystack (NIAH) retrieval benchmarks. It consistently outperforms prior cache eviction baselines (StreamingLLM, SnapKV, PyramidKV, CAKE, HeadKV, AdaKV) for equivalent budgets. Notably, CompressKV preserves over 97% of full-cache performance using merely 3% of the KV cache on LongBench QA tasks and achieves 90% accuracy on NIAH with just 0.7% cache retention—a strong numerical claim indicating superior resource/performance trade-off.

Figure 4: CompressKV yields the highest average LongBench scores across KV-cache budgets, with pronounced gains under tight memory settings.

Figure 5: On NIAH, CompressKV maintains high retrieval performance even at extreme cache compression ratios.

The ablation studies underscore SRHs’ essentiality: masking even a small subset of SRHs causes steep retrieval accuracy drops, while traditional retrieval heads induce minor degradation. Component analysis demonstrates that both SRH-driven token selection and layer-adaptive allocation offer cumulative accuracy improvements.

Figure 6: End-to-end inference latency and memory usage remain nearly flat for eviction-based methods under increasing context sizes, contrasting with linear growth for full cache baselines.

Orthogonality and Integration

CompressKV is fully orthogonal to contemporary LLM efficiency methods. When combined with prefilling-stage accelerators (e.g., MInference, XAttention) or KV-cache quantization (KIVI), it synergistically reduces runtime and memory usage. Head-level allocation (HeadKV/AdaKV) integrations consistently yield further accuracy gains, especially notable under stringent KV budgets, validating CompressKV as an adaptable module atop existing infrastructure.

Figure 7: Integrating CompressKV with prefilling and quantization techniques further enhances memory savings without major accuracy sacrifice.

Figure 8: Head-level allocation integrations with CompressKV produce measurable increases in benchmark scores under constrained memory budgets.

Practical and Theoretical Implications

CompressKV demonstrates that retrieval-aware token selection and error-driven layer adaptation are key for sustainable, high-fidelity long-context inference on resource-constrained hardware. The span-aggregation standard for SRH identification generalizes across model families and context lengths, mitigating limitations seen in prior, head-agnostic methods. The approach is particularly suited for GQA-based architectures, offering robust, transferable improvements for ad-hoc compression and hybrid efficiency stacks.

On a theoretical level, the SRH mechanism provides a principled paradigm for attention head differentiation and semantic evidence localization in transformer models, enabling further study of head-level functional partitioning and cache compression dynamics. The offline layer-error estimation both decouples allocation from runtime profiling and opens avenues for model-agnostic cache compression policies.

Conclusion

CompressKV establishes a new benchmark for resource-efficient long-context inference in GQA-based LLMs. By leveraging Semantic Retrieval Heads and offline error-aware layer-wise allocation, it achieves significant compression ratios with minimal performance degradation, validated across multiple models and benchmarks. Its compatibility with auxiliary efficiency techniques and strict numerical gains render it a preferred solution for sustainable LLM deployment under hardware constraints. The SRH-driven paradigm also offers a promising direction for future research on attention specialization and adaptive cache management.