C$^2$KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

Abstract: Long-context inference is central to modern LLM applications such as retrieval-augmented generation and multi-document reasoning. To mitigate the growing inference cost, recent work has explored key-value (KV) cache reuse to reduce redundant prefill computation. However, existing reuse methods primarily focus on computation savings and overlook a critical bottleneck in long-context LLM serving: the cost of storing and accessing large KV caches. While KV compression appears to be a natural complement, naively combining compression with non-prefix KV reuse often leads to severe accuracy degradation. In this work, we propose C$<sup>2$KV, a unified framework for non-prefix KV reuse that jointly optimizes KV extraction and inference-time concatenation. C$<sup>2$KV learns a composable and compressed KV cache manifold that is explicitly designed to be position-agnostic. Our approach introduces a lightweight sidecar Extractor with learnable compression tokens and a structured attention flow, enabling modular KV representations that can be flexibly reused and concatenated without modifying the frozen base model. We further employ a compression-concatenation co-training strategy to align extraction-time representations with their downstream reuse behavior. Extensive experiments across multiple long-context benchmarks and model families demonstrate that C$<sup>2$KV significantly reduces KV cache storage and transfer costs, achieving up to 17$\times$ inference speedup under long contexts, while preserving generation quality.

• The paper introduces C²KV, a sidecar extractor that uses structured one-way attention and concatenation-aware training to create context-independent KV caches compressed by 4×–16×.
• The method delivers up to 17× faster inference, largely flattens decode-latency growth through 128k-token contexts, and often matches or exceeds full-context and reuse baselines on LongBench.
• The results show that compression must be designed for composability: naive 4× compression causes severe accuracy collapse, while C²KV maintains competitive retrieval quality without blending or selective recomputation.

Motivation and problem statement

Long-context in-context learning workloads such as retrieval-augmented generation (RAG), agent memory, and few-shot prompting inject multiple long documents into the model input, making LLM inference expensive along three axes simultaneously: $O(n^2)$ prefill compute for KV cache construction, $O(L)$ memory capacity for cache storage, and $O(L)$ bandwidth for cache access during decoding. The paper's central observation is that existing non-prefix KV reuse research has concentrated almost exclusively on the first axis—reducing redundant prefill computation—while the dominant bottleneck in practical serving has shifted to KV cache storage and transfer. Because caches are typically held outside high-bandwidth accelerator memory and loaded on demand, the DRAM-to-HBM transfer of uncompressed KV states becomes the primary latency factor as context length grows.

The natural remedy—combining KV compression with non-prefix reuse—is shown to fail when applied naively. The authors demonstrate that stacking a generic compression method (SnapKV at 4×) on top of any existing reuse strategy causes severe accuracy collapse: on HotpotQA with Llama3.1-8B, naive reuse drops from 0.3973 to 0.0547, Block-Attention from 0.4271 to 0.0743, EPIC from 0.4363 to 0.0897, and CacheBlend from 0.2970 to 0.0532. The root cause is that standard compressed KV representations are context-coupled and position-dependent, hence not composable across contexts.

Limitations of prior approaches

The paper frames existing methods within an encode–blend–infer abstraction and identifies two failure modes. Training-free selective recomputation methods (CacheBlend, EPIC, KVShare) suffer from an inherent KV deviation gap: unrecomputed tokens carry states computed under a different context, and this deviation grows monotonically as the recomputation ratio decreases, creating a rigid latency–accuracy trade-off. Training-based methods (Block-Attention, KVLink) modify attention structure to natively support disjoint blocks but incur capability loss—the paper reports that a Llama3.1-8B model fine-tuned with Block-Attention loses up to 10.4% on LongBench—and require costly per-model fine-tuning that hinders adaptation to rapidly iterating base models. Neither category addresses storage or bandwidth pressure.

The C²KV framework

C²KV augments a frozen base LLM with a lightweight sidecar module, the C² Extractor, which encodes each document independently into a compressed, position-agnostic KV representation at a $k{:}1$ ratio. Three components are central:

C² Tokens as memory slots. For each block of $k$ document tokens, one learnable C² Token (all sharing a single embedding) is introduced, giving $\lceil n/k \rceil$ compressed slots per document. Per-layer trainable QKV projection heads operate exclusively on these tokens; the base model's frozen projections handle original tokens. Trainable parameters amount to roughly 10% of the base model for Qwen3-4B and Llama3.1-8B. A residual variant adds mean-pooled hidden states of each token block to its C² Token at the first layer.

Structured Information Flow (SIF). A block-structured attention mask enforces three constraints: original tokens attend only causally among themselves (their hidden states remain bit-identical to the frozen base model); each C² Token attends only to its own block plus a sink block; and C² Tokens attend causally to preceding C² Tokens for document-level accumulation. This asymmetry distinguishes C² Tokens from anchor-token compression: information flows one-way from original tokens into latent KV carriers rather than through bidirectional semantic bottlenecks, so extracted KVs are context-independent by construction. Ablations confirm this: replacing SIF with bidirectional anchor tokens degrades Qasper F1 from 0.3755 to 0.2750 and GovReport from 0.2967 to 0.1280, while relaxing block-locality (Info-Leakage or Global-Info variants) also consistently hurts accuracy.

Compression–concatenation co-training. Training uses only a standard autoregressive SFT loss on answers conditioned on concatenated multi-document KV caches, with no auxiliary reconstruction or alignment loss. Because supervision occurs after concatenation and document order/count vary across samples, the extractor is forced to produce "merge-ready" representations valid under arbitrary compositions. At inference time, retrieved segments receive fresh positional embeddings via RoPE re-rotation and are concatenated directly—no blending or recomputation—reducing TTFT to a load-only operation.

Evaluation results

Experiments cover Qwen3-4B-Instruct-2507, Llama-3.1-8B-Instruct, and Qwen2.5-7B-Instruct on LongBench tasks, trained on 120k samples from HotpotQA, 2WikiMultiHopQA, and LongMagpie for a single epoch.

The headline system result is up to 17× inference speedup over existing approaches under long contexts. On the TTFT–accuracy trade-off across seven LongBench tasks, C²KV occupies the favorable upper-left region: full recompute achieves the accuracy ceiling but prohibitive TTFT, while EPIC and CacheBlend reduce TTFT with substantial accuracy instability. Under 4× compression, C²KV frequently matches or exceeds baselines despite compressing more aggressively—for example, on Llama3.1-8B it reaches 0.4828 on HotpotQA versus 0.5343 for full recompute, and 0.4477 on 2WikiMQA versus 0.4018, outperforming all reuse baselines on the latter.

Decode-time behavior is equally important given the paper's bandwidth thesis: as context length scales from 16k to 128k tokens, full-length caches show near-linear per-token latency growth, whereas C²KV largely flattens this curve, showing only mild increase even at 128k. On RULER with 4× compression, retrieval performance remains stable from 4k to 64k context lengths across answer depths. Scaling to Qwen3-14B preserves these trends, with C²KV-4x-Res matching or exceeding the full-context baseline on MuSiQue, SAMSum, and MultiNews.

A notable robustness result concerns compression-ratio generalization: a single extractor trained with ratios randomly sampled from {4×, 8×, 16×} performs well at inference-time budgets including the unseen 10× setting, and dynamic-ratio training consistently improves over fixed-ratio training. Degradation from 4× to 16× is gradual—for instance, WikiMQA on Llama3.1-8B holds at 0.4298 at 16× versus 0.4018 for the uncompressed full-context baseline.

Limitations and open questions

The paper concedes two limitations explicitly. First, the framework assumes reusable content is identified and extracted offline; online or incremental KV extraction remains unaddressed, which limits applicability to dynamically growing corpora. Second, compression is uniform across documents and layers; content-aware or adaptive budget allocation that preserves composability is left open. Two further caveats bear on interpretation: the TTFT protocol excludes offline extraction cost and system-prompt/query prefill, so end-to-end savings depend on high cache hit rates in the serving workload; and GSM8K results show some variance, with C²KV at 4× trailing EPIC on several models (e.g., 0.6175 vs. 0.7319 on Llama3.1-8B), suggesting few-shot arithmetic reasoning is more sensitive to compression than retrieval-style QA.

Conclusion

C²KV reframes non-prefix KV cache reuse around the storage and bandwidth bottleneck rather than prefill computation alone. Its contribution is a jointly optimized design—a frozen-base sidecar extractor, structured one-way attention flow, and concatenation-aware training—that yields KV representations which are simultaneously compressed and composable. Empirically, it delivers up to 17× speedup, flattens decode-latency scaling to 128k contexts, and sustains competitive accuracy at 4×–16× compression where naive compression-plus-reuse combinations collapse. The approach demonstrates that composability must be designed into the representation itself rather than approximated post hoc.