KV Cache in Autoregressive Transformers

• KV Cache is a mechanism in autoregressive transformers that stores past attention keys and values to enable efficient token decoding without recomputation.
• The memory footprint of KV Cache grows linearly with sequence length, batch size, and model depth, impacting latency and throughput in large-scale deployments.
• Advanced management techniques—such as eviction, merging, quantization, and retrieval-based compression—optimize KV Cache usage and maintain generation quality.

A Key-Value Cache (KV Cache) in an autoregressive transformer is the stored collection of past attention keys and values across all layers and heads, reused during decoding so that each new token attends to prior context without recomputing the entire prefix. In standard notation, a layer forms $Q = XW_Q$, $K = XW_K$, and $V = XW_V$, and decode-time attention uses the current query against cached $K,V$ from all earlier tokens (Staniszewski et al., 3 Nov 2025). This mechanism is indispensable for practical LLM inference, but its memory footprint grows linearly with context length, batch size, and model depth, making KV management a central systems problem in long-context and multi-turn serving (Yuan et al., 3 Mar 2025).

1. Formal definition and role in autoregressive decoding

In a decoder-only transformer, each layer and head produces, for every token, a key vector and a value vector. For a sequence of length $T$, one formulation writes

$$Q \in \mathbb{R}^{T \times h \times d_{\text{head}}},\quad K \in \mathbb{R}^{T \times h \times d_{\text{head}}},\quad V \in \mathbb{R}^{T \times h \times d_{\text{head}}},$$

with per-layer, per-head slices $K_{\ell,j}\in\mathbb{R}^{T\times d_{\text{head}}}$ and $V_{\ell,j}\in\mathbb{R}^{T\times d_{\text{head}}}$ (Staniszewski et al., 3 Nov 2025). A simpler single-head exposition writes the attention output at step $t$ as

$$\mathbf{o}_t = \mathrm{softmax}\left(\frac{\mathbf{q}_t^\top \mathbf{K}_t}{\sqrt{d}}\right)^\top \mathbf{V}_t,$$

where $K = XW_K$0 and $K = XW_K$1 (Yuan et al., 3 Mar 2025).

The cache is populated in two phases. During prefill, the model processes the prompt once and stores keys and values for all prompt tokens. During decoding, each new token contributes only one new query, key, and value, and attention is computed against the stored cache plus the new entry (Staniszewski et al., 3 Nov 2025). This converts “recompute all past tokens every step” into “compute one token’s K/V every step,” dramatically reducing compute and latency (Staniszewski et al., 3 Nov 2025).

The raw memory cost of a 16-bit KV cache for a model with $K = XW_K$2 layers, $K = XW_K$3 heads, head dimension $K = XW_K$4, and sequence length $K = XW_K$5 is

$K = XW_K$6

where the factor $K = XW_K$7 is “2 floats (K+V) × 2 bytes per 16-bit float” (Staniszewski et al., 3 Nov 2025). Equivalent formulations in the broader literature write KV memory as $K = XW_K$8 or $K = XW_K$9, depending on notation and whether $V = XW_V$0 is suppressed (Yuan et al., 3 Mar 2025).

2. Memory scaling, bandwidth pressure, and the serving bottleneck

KV memory grows linearly with processed tokens, but in practice it often dominates runtime memory. Concrete 16-bit KV sizes per 1K tokens reported for several decoder-only models are as follows (Staniszewski et al., 3 Nov 2025):

Model	KV size / 1K tokens
Qwen 2.5 R1 1.5B	28 MiB
Qwen 2.5 R1 7B	56 MiB
Llama 3.1 8B	128 MiB
Llama 3.3 70B Instruct	320 MiB
Mistral NeMo 12B	160 MiB
MN-Minitron 8B	160 MiB

At 16k context, Llama 3.1 8B has on the order of $V = XW_V$1 GB of KV cache per sequence (Staniszewski et al., 3 Nov 2025). In another measurement, Llama-2-7B with batch size 8 and sequence length 4K uses about 34.36 GB of full KV cache; Llama-2-13B uses 53.69 GB; Llama-2-70B with GQA uses 17.18 GB; and Mistral-7B with GQA uses 8.59 GB (Yang et al., 2024). The KVCrush study gives an even larger deployment-scale example: for OPT-175B, batch size $V = XW_V$2, and sequence length $V = XW_V$3, the KV cache is roughly 4608 GB, while model weights are around 325 GB (Jha et al., 24 Feb 2025).

This memory footprint affects both latency and throughput. GPU HBM is limited, so per-request caches of gigabytes restrict batch size, number of concurrent users, and maximum context length (Staniszewski et al., 3 Nov 2025). In large-scale serving, frameworks such as vLLM keep KV blocks and reuse them when prompts share prefixes, but this yields huge on-GPU “hot” caches, “warm/cold” tiers on CPU DRAM and NVMe, and expensive transfers between prefill and decode nodes (Staniszewski et al., 3 Nov 2025). The same pressure appears in multimodal models: for LLaVA-1.6-34B with batch size 4, 5 images per prompt, and 2K visual tokens per image, the KV cache for visual tokens alone requires about 110 GB of HBM (Tu et al., 2024).

A recurring systems dilemma follows directly. When memory is tight, one must choose between keeping more KV caches on-GPU, offloading caches to CPU or disk, or discarding and recomputing them later. The first improves latency but reduces concurrency; the second saves HBM but incurs bandwidth and latency costs; the third eliminates storage overhead but increases TTFT and compute (Staniszewski et al., 3 Nov 2025).

3. Reuse, staleness, and multi-tenant KV cache management

KV caches are not only ephemeral decode-time buffers. In multi-turn chat and iterative workflows, they become reusable state. A conversation can be formalized as

$V = XW_V$4

where each new turn appends additional user or system tokens to the existing text. If the KV cache for the previous conversation prefix is still available, the next turn needs only to compute keys and values for the newly appended tokens rather than rerunning prefill over the entire history (Staniszewski et al., 3 Nov 2025).

This is especially useful in iterative code editing and long-document question answering, where many turns share a long, stable prefix (Staniszewski et al., 3 Nov 2025). A stale KV cache is one that corresponds to an older conversation turn or prefix, is not currently being decoded on, but may be needed soon if the session resumes. Stale caches occupy GPU or CPU memory while inactive, creating a latency–throughput trade-off: keeping them hot reduces TTFT, but lowers effective capacity for active requests (Staniszewski et al., 3 Nov 2025).

Beyond single-session reuse, recent systems pursue cross-request reuse. KVShare targets multi-user services in which many prompts are semantically similar but not identical. It stores past prompts and KV caches in a vector database, retrieves similar prior requests via GTE/mGTE embeddings, constructs edit operations with a DELTA Tree, and recomputes only placeholder positions through PartialAttention (Yang et al., 17 Mar 2025). On real conversation datasets, up to 36% of conversations have at least one semantically similar prompt under cosine distance $V = XW_V$5, and on derived similar-conversation subsets, at least 60% of WildChat-Similar requests and over 50% of ShareGPT-Similar requests achieve token-level KV hit rates of at least 90% (Yang et al., 17 Mar 2025). Multi-task experiments on Qwen2.5-7B, Llama3.1-8B, and Yi1.5-9B report TTFT reduction by up to $V = XW_V$6, throughput increase by $V = XW_V$7 compared to full KV recompute, and a 20.38% accuracy boost over SOTA methods (Yang et al., 17 Mar 2025).

These results establish a broader view of KV caches as reusable service-state objects, not merely transient tensors. This suggests that compression, indexing, and scheduling are inseparable from the semantics of prefix reuse.

4. Eviction, merging, and token-selection policies

A major class of KV optimization methods reduces memory by retaining only a subset of tokens. The simplest strategies are positional: StreamingLLM and related methods keep an initial segment and a recent sliding window, evicting everything else (Yuan et al., 3 Mar 2025). Attention-based strategies use token importance derived from attention statistics. H2O uses cumulative attention; TOVA uses the last-step attention score; Scissorhands binarizes cumulative importance; SnapKV and PyramidKV use attention-based token selection with different allocation heuristics (Yuan et al., 3 Mar 2025).

These methods are effective but contentious because they delete information. “No Token Left Behind” reports that eviction can induce safety breaches, hallucinations, and context loss, and shows on a line retrieval task that retaining evicted KV pairs in low precision substantially recovers the incurred degradation (Yang et al., 2024). Its mixed-precision KV cache keeps important tokens at higher precision and low-importance tokens in low precision rather than deleting them. For Llama-2-7B, batch 8, sequence length 4K, MiKV reduces full KV from 34.36 GB to 8.59 GB at 25% cache with MMLU 43.9% versus 44.0% for full cache, and to 6.87 GB at 20% cache with MMLU 42.7% (Yang et al., 2024).

Merging-based methods try to preserve information that eviction would destroy. WeightedKV starts from an empirical asymmetry: on LLaMA-2-7B, singular values for keys drop quickly to near zero, while values have a much heavier tail, suggesting stronger low-rank structure in keys than in values (Yuan et al., 3 Mar 2025). Its mechanism discards keys of less important tokens while merging their values into neighboring tokens by a convex combination weighted by historical average attention scores (Yuan et al., 3 Mar 2025). KeepKV argues that existing merging methods still perturb outputs because the merged token’s attention mass is strictly less than the total pre-merge attention mass, a failure mode it calls Attention Sag (Tian et al., 14 Apr 2025). It introduces Electoral Votes and Zero Inference-Perturbation Merging, proving zero perturbation at the current step and reporting more than $V = XW_V$8 throughput while keeping superior generation quality even with 10% KV cache budgets (Tian et al., 14 Apr 2025).

Other work makes token selection more diversity-aware. KVCrush represents each token by a binary “head-behavior” signature, groups similar signatures, and keeps representative tokens rather than only high-attention tokens. It reports $V = XW_V$9 LongBench KV cache reduction with less than 1% accuracy drop and less than 0.5% total inference latency (Jha et al., 24 Feb 2025). GraphKV replaces static top-$K,V$0 selection with graph-based decay-signal propagation on token similarity graphs, and reports that it outperforms the suboptimal KNorm method by 45.88% and achieves approximately 3% improvement over SnapKV and PyramidKV with KV size 512 on LLaMA-8B (Li et al., 30 Aug 2025).

A recurring misconception is that token eviction is only a ranking problem. These studies collectively indicate that selection is also a redundancy problem: high-score tokens are often highly similar, and preserving a diverse subset can matter as much as preserving the highest scores.

5. Quantization, transform coding, sparse representation, and retrieval-based compression

A second major class of methods compresses the representation of each retained KV entry rather than deleting tokens. One line treats KV channels as strongly inter-dependent. “KV Cache is 1 Bit Per Channel” estimates that the joint entropy of multiple key/value channels grows more slowly than the sum of their marginal entropies, and proposes Coupled Quantization (CQ), which quantizes channel groups jointly rather than scalarwise (Zhang et al., 2024). CQ preserves model quality with KV cache quantized down to 1-bit, and configurations such as CQ-8c8b correspond to 1 bit per scalar channel through vector codebooks rather than 1-bit scalar quantizers (Zhang et al., 2024).

CSR instead represents KV vectors as sparse combinations of dictionary atoms. It stores sparse indices and coefficients instead of dense vectors, uses Matching Pursuit, and learns an offline dictionary with NeuralDict (Zhang et al., 2024). On LLaMA3-8B with head dimension 128, CSR with $K,V$1 corresponds to an effective 1-bit KV cache, and its LongBench results are comparable to state-of-the-art KV quantization algorithms while remaining usable in ultra-low-bit regimes (Zhang et al., 2024).

CSKV compresses along the channel dimension rather than the token dimension. Motivated by singular value analysis of KV tensors, it factorizes $K,V$2 and $K,V$3 into low-rank products, stores low-dimensional features, and augments them with a window-based full-precision branch (Wang et al., 2024). It reports 80% KV memory reduction while maintaining long-context capability, and up to 95% compression when combined with 4-bit quantization via QAT (Wang et al., 2024).

KVTC frames KV compression explicitly as transform coding. It combines PCA-based feature decorrelation, adaptive quantization, and DEFLATE entropy coding, with calibration performed once per model and compression setting (Staniszewski et al., 3 Nov 2025). Across Llama 3, Mistral NeMo, and R1-Qwen 2.5, it achieves up to $K,V$4 compression while maintaining reasoning and long-context accuracy, and $K,V$5 or higher for specific use cases (Staniszewski et al., 3 Nov 2025). For Mistral NeMo 12B at 8k context, recompute TTFT is 3098 ms versus 380 ms for KVTC decompression (Staniszewski et al., 3 Nov 2025).

PQCache reformulates selective attention as a maximum inner-product search problem over cached keys and uses product quantization to index them (Zhang et al., 2024). It performs approximate MIPS with PQ codes and centroids, then fetches exact K/V for top-$K,V$6 tokens, and reports 4.60% score improvement over existing methods on InfiniteBench together with low system latency in both prefilling and decoding (Zhang et al., 2024). KVComp combines LLM-aware quantization with GPU Huffman coding, reporting on average 47% and up to 83% higher memory reduction rate than existing methods with little or no model accuracy degradation, and decompression throughput exceeding 400 GB/s for keys in fused kernels (Jiang et al., 30 Aug 2025).

These approaches differ sharply in mechanism—vector quantization, sparse coding, channel shrinking, transform coding, entropy coding, ANN retrieval—but converge on the same empirical point: KV tensors are not unstructured arrays. They exhibit channel coupling, low-rank structure, cross-head and cross-layer redundancy, or compressible symbol distributions.

6. Layer-adaptive, modality-aware, and streaming variants

In multimodal and streaming settings, KV cache behavior departs from the text-only case. PrefixKV studies large vision-LLMs and argues that existing methods generally overlook the distinct importance distributions of KV vectors across layers by keeping the same cache size for each layer (Wang et al., 2024). It reframes layer-wise retention as a global prefix configuration search over cumulative priority curves and reports up to $K,V$7 throughput speedup for LLaVA-1.5-7B at 20% compression with batch size 16, while remaining close to full-cache PPL and ROUGE on LLaVA-Description and MM-Vet (Wang et al., 2024).

VL-Cache is explicitly modality-aware. It observes that VLM attention exhibits distinct visual and text token sparsity patterns, then uses post-vision attention to allocate layer-wise budgets and score tokens (Tu et al., 2024). Retaining only 10% of the KV cache achieves accuracy comparable to full cache, while generating 100 tokens can be accelerated by up to $K,V$8 end-to-end and decoding by up to $K,V$9, with 90% KV memory reduction in GPU (Tu et al., 2024).

Streaming video introduces a different pathology: continuously constructing new KV entries while evicting old ones can contaminate representations of recent inputs. DSCache addresses this with a cumulative past KV cache and a separate instant cache built from a recent feature buffer, plus position-agnostic encoding so that RoPE can be reapplied at use time (Pang et al., 3 May 2026). On Streaming Video QA benchmarks, it reports an average 2.5% accuracy gain over prior methods, while keeping fixed memory in unbounded streams (Pang et al., 3 May 2026).

These results indicate that KV cache management depends not only on memory size but also on token type, layer role, and temporal regime. In VLMs, visual and textual tokens should not be scored identically. In streaming video, cache construction itself becomes a modeling problem.

7. Theoretical unification, limitations, and open directions

Much of the KV literature is heuristic, but recent work has started to supply explicit objectives. CapKV reinterprets eviction through the Information Bottleneck principle under a linear-Gaussian surrogate of attention, deriving a closed-form mutual information objective for the effective information capacity of a retained KV subset (Yang et al., 28 Apr 2026). It argues that many existing eviction rules can be viewed as approximations to the same capacity-maximization principle, and introduces a capacity-aware method based on a log-determinant approximation with statistical leverage scores (Yang et al., 28 Apr 2026). This provides a theoretical counterpoint to purely empirical top-$T$0 heuristics.

At the same time, current methods retain notable constraints. KVTC notes limitations in calibration realism, scale beyond 70B, imperfect correlation between Frobenius reconstruction error and task accuracy, and non-optimized compression kernels (Staniszewski et al., 3 Nov 2025). PrefixKV depends on attention as a proxy for importance and requires offline configuration estimation, even if only a small number of samples suffices in its experiments (Wang et al., 2024). CSR identifies runtime overhead from Matching Pursuit and de-sparsification, and more generally the engineering complexity of dictionary management and online updates (Zhang et al., 2024). KeepKV relies on the locality of future attention scores for EMA prediction and on merging sufficiently similar candidates to keep perturbation bounded (Tian et al., 14 Apr 2025). DSCache proves positional equivalence for RoPE, but not for arbitrary positional schemes (Pang et al., 3 May 2026).

The field is therefore converging on a few broad directions already stated explicitly across the literature: online inference in compressed domains, combining compression with eviction or sparsity, learned transforms or autoencoders instead of fixed PCA or codebooks, adaptive per-layer or per-head compression, and better theoretical objectives for cache retention (Staniszewski et al., 3 Nov 2025). This suggests that future KV cache systems will be hybrid: partly codec, partly scheduler, and partly memory model, with the cache treated as a structured representation whose storage, movement, and approximation are all first-class design variables.