---
title: KV Cache in Autoregressive Transformers
url: https://www.emergentmind.com/topics/key-value-cache-kv-cache
type: topic
---

# KV Cache in Autoregressive Transformers

A **Key-Value Cache (KV Cache)** in an autoregressive transformer is the stored collection of past attention **keys** and **values** across all layers and heads, reused during decoding so that each new token attends to prior context without recomputing the entire prefix. In standard notation, a layer forms \(Q = XW_Q\), \(K = XW_K\), and \(V = XW_V\), and decode-time attention uses the current query against cached \(K,V\) from all earlier tokens [2511.01815]. This mechanism is indispensable for practical large language model inference, but its memory footprint grows linearly with context length, batch size, and model depth, making KV management a central systems problem in long-context and multi-turn serving [2503.01330].

## 1. Formal definition and role in autoregressive decoding

In a decoder-only transformer, each layer and head produces, for every token, a key vector and a value vector. For a sequence of length \(T\), one formulation writes
\[
Q \in \mathbb{R}^{T \times h \times d_{\text{head}}},\quad
K \in \mathbb{R}^{T \times h \times d_{\text{head}}},\quad
V \in \mathbb{R}^{T \times h \times d_{\text{head}}},
\]
with per-layer, per-head slices \(K_{\ell,j}\in\mathbb{R}^{T\times d_{\text{head}}}\) and \(V_{\ell,j}\in\mathbb{R}^{T\times d_{\text{head}}}\) [2511.01815]. A simpler single-head exposition writes the attention output at step \(t\) as
\[
\mathbf{o}_t = \mathrm{softmax}\left(\frac{\mathbf{q}_t^\top \mathbf{K}_t}{\sqrt{d}}\right)^\top \mathbf{V}_t,
\]
where \(\mathbf{K}_t=[\mathbf{k}_1,\dots,\mathbf{k}_t]\) and \(\mathbf{V}_t=[\mathbf{v}_1,\dots,\mathbf{v}_t]\) [2503.01330].

The cache is populated in two phases. During **prefill**, the model processes the prompt once and stores keys and values for all prompt tokens. During **decoding**, each new token contributes only one new query, key, and value, and attention is computed against the stored cache plus the new entry [2511.01815]. This converts “recompute all past tokens every step” into “compute one token’s K/V every step,” dramatically reducing compute and latency [2511.01815].

The raw memory cost of a 16-bit KV cache for a model with \(L\) layers, \(h\) heads, head dimension \(d_{\text{head}}\), and sequence length \(t\) is
\[
4\,L\,h\,d_{\text{head}}\,t \quad \text{bytes},
\]
where the factor \(4\) is “2 floats (K+V) × 2 bytes per 16-bit float” [2511.01815]. Equivalent formulations in the broader literature write KV memory as \(O(L\cdot H\cdot T\cdot d_{\text{head}})\) or \(O(LNd)\), depending on notation and whether \(H d_{\text{head}} = d\) is suppressed [2503.01330].

## 2. Memory scaling, bandwidth pressure, and the serving bottleneck

KV memory grows linearly with processed tokens, but in practice it often dominates runtime memory. Concrete 16-bit KV sizes per 1K tokens reported for several decoder-only models are as follows [2511.01815]:

| Model | KV size / 1K tokens |
|---|---:|
| Qwen 2.5 R1 1.5B | 28 MiB |
| Qwen 2.5 R1 7B | 56 MiB |
| Llama 3.1 8B | 128 MiB |
| Llama 3.3 70B Instruct | 320 MiB |
| Mistral NeMo 12B | 160 MiB |
| MN-Minitron 8B | 160 MiB |

At 16k context, Llama 3.1 8B has on the order of \(\sim 2\) GB of KV cache per sequence [2511.01815]. In another measurement, Llama-2-7B with batch size 8 and sequence length 4K uses about 34.36 GB of full KV cache; Llama-2-13B uses 53.69 GB; Llama-2-70B with GQA uses 17.18 GB; and Mistral-7B with GQA uses 8.59 GB [2402.18096]. The KVCrush study gives an even larger deployment-scale example: for OPT-175B, batch size \(128\), and sequence length \(8{,}000\), the KV cache is roughly 4608 GB, while model weights are around 325 GB [2503.00022].

This memory footprint affects both latency and throughput. GPU HBM is limited, so per-request caches of gigabytes restrict batch size, number of concurrent users, and maximum context length [2511.01815]. In large-scale serving, frameworks such as vLLM keep KV blocks and reuse them when prompts share prefixes, but this yields huge on-GPU “hot” caches, “warm/cold” tiers on CPU DRAM and NVMe, and expensive transfers between prefill and decode nodes [2511.01815]. The same pressure appears in multimodal models: for LLaVA-1.6-34B with batch size 4, 5 images per prompt, and 2K visual tokens per image, the KV cache for visual tokens alone requires about 110 GB of HBM [2410.23317].

A recurring systems dilemma follows directly. When memory is tight, one must choose between keeping more KV caches on-GPU, offloading caches to CPU or disk, or discarding and recomputing them later. The first improves latency but reduces concurrency; the second saves HBM but incurs bandwidth and latency costs; the third eliminates storage overhead but increases TTFT and compute [2511.01815].

## 3. Reuse, staleness, and multi-tenant KV cache management

KV caches are not only ephemeral decode-time buffers. In multi-turn chat and iterative workflows, they become reusable state. A conversation can be formalized as
\[
\mathcal{C} = ((x_0, y_0), (x_1, y_1), \dots),
\]
where each new turn appends additional user or system tokens to the existing text. If the KV cache for the previous conversation prefix is still available, the next turn needs only to compute keys and values for the newly appended tokens rather than rerunning prefill over the entire history [2511.01815].

This is especially useful in iterative code editing and long-document question answering, where many turns share a long, stable prefix [2511.01815]. A **stale** KV cache is one that corresponds to an older conversation turn or prefix, is not currently being decoded on, but may be needed soon if the session resumes. Stale caches occupy GPU or CPU memory while inactive, creating a latency–throughput trade-off: keeping them hot reduces TTFT, but lowers effective capacity for active requests [2511.01815].

Beyond single-session reuse, recent systems pursue **cross-request** reuse. KVShare targets multi-user services in which many prompts are semantically similar but not identical. It stores past prompts and KV caches in a vector database, retrieves similar prior requests via GTE/mGTE embeddings, constructs edit operations with a DELTA Tree, and recomputes only placeholder positions through PartialAttention [2503.16525]. On real conversation datasets, up to 36% of conversations have at least one semantically similar prompt under cosine distance \(\le 0.05\), and on derived similar-conversation subsets, at least 60% of WildChat-Similar requests and over 50% of ShareGPT-Similar requests achieve token-level KV hit rates of at least 90% [2503.16525]. Multi-task experiments on Qwen2.5-7B, Llama3.1-8B, and Yi1.5-9B report TTFT reduction by up to \(9.39\times\), throughput increase by \(1.2\times\) compared to full KV recompute, and a 20.38% accuracy boost over SOTA methods [2503.16525].

These results establish a broader view of KV caches as reusable service-state objects, not merely transient tensors. This suggests that compression, indexing, and scheduling are inseparable from the semantics of prefix reuse.

## 4. Eviction, merging, and token-selection policies

A major class of KV optimization methods reduces memory by retaining only a subset of tokens. The simplest strategies are positional: StreamingLLM and related methods keep an initial segment and a recent sliding window, evicting everything else [2503.01330]. Attention-based strategies use token importance derived from attention statistics. H2O uses cumulative attention; TOVA uses the last-step attention score; Scissorhands binarizes cumulative importance; SnapKV and PyramidKV use attention-based token selection with different allocation heuristics [2503.01330].

These methods are effective but contentious because they delete information. “No Token Left Behind” reports that eviction can induce safety breaches, hallucinations, and context loss, and shows on a line retrieval task that retaining evicted KV pairs in low precision substantially recovers the incurred degradation [2402.18096]. Its mixed-precision KV cache keeps important tokens at higher precision and low-importance tokens in low precision rather than deleting them. For Llama-2-7B, batch 8, sequence length 4K, MiKV reduces full KV from 34.36 GB to 8.59 GB at 25% cache with MMLU 43.9% versus 44.0% for full cache, and to 6.87 GB at 20% cache with MMLU 42.7% [2402.18096].

Merging-based methods try to preserve information that eviction would destroy. WeightedKV starts from an empirical asymmetry: on LLaMA-2-7B, singular values for keys drop quickly to near zero, while values have a much heavier tail, suggesting stronger low-rank structure in keys than in values [2503.01330]. Its mechanism discards keys of less important tokens while merging their values into neighboring tokens by a convex combination weighted by historical average attention scores [2503.01330]. KeepKV argues that existing merging methods still perturb outputs because the merged token’s attention mass is strictly less than the total pre-merge attention mass, a failure mode it calls **Attention Sag** [2504.09936]. It introduces **Electoral Votes** and **Zero Inference-Perturbation Merging**, proving zero perturbation at the current step and reporting more than \(2\times\) throughput while keeping superior generation quality even with 10% KV cache budgets [2504.09936].

Other work makes token selection more diversity-aware. KVCrush represents each token by a binary “head-behavior” signature, groups similar signatures, and keeps representative tokens rather than only high-attention tokens. It reports \(4\times\) LongBench KV cache reduction with less than 1% accuracy drop and less than 0.5% total inference latency [2503.00022]. GraphKV replaces static top-\(k\) selection with graph-based decay-signal propagation on token similarity graphs, and reports that it outperforms the suboptimal KNorm method by 45.88% and achieves approximately 3% improvement over SnapKV and PyramidKV with KV size 512 on LLaMA-8B [2509.00388].

A recurring misconception is that token eviction is only a ranking problem. These studies collectively indicate that selection is also a redundancy problem: high-score tokens are often highly similar, and preserving a diverse subset can matter as much as preserving the highest scores.

## 5. Quantization, transform coding, sparse representation, and retrieval-based compression

A second major class of methods compresses the representation of each retained KV entry rather than deleting tokens. One line treats KV channels as strongly inter-dependent. “KV Cache is 1 Bit Per Channel” estimates that the joint entropy of multiple key/value channels grows more slowly than the sum of their marginal entropies, and proposes **Coupled Quantization (CQ)**, which quantizes channel groups jointly rather than scalarwise [2405.03917]. CQ preserves model quality with KV cache quantized down to 1-bit, and configurations such as CQ-8c8b correspond to 1 bit per scalar channel through vector codebooks rather than 1-bit scalar quantizers [2405.03917].

CSR instead represents KV vectors as sparse combinations of dictionary atoms. It stores sparse indices and coefficients instead of dense vectors, uses Matching Pursuit, and learns an offline dictionary with NeuralDict [2412.11741]. On LLaMA3-8B with head dimension 128, CSR with \(s=4,s_n=1\) corresponds to an effective 1-bit KV cache, and its LongBench results are comparable to state-of-the-art KV quantization algorithms while remaining usable in ultra-low-bit regimes [2412.11741].

CSKV compresses along the channel dimension rather than the token dimension. Motivated by singular value analysis of KV tensors, it factorizes \(W_K\) and \(W_V\) into low-rank products, stores low-dimensional features, and augments them with a window-based full-precision branch [2409.10593]. It reports 80% KV memory reduction while maintaining long-context capability, and up to 95% compression when combined with 4-bit quantization via QAT [2409.10593].

KVTC frames KV compression explicitly as transform coding. It combines PCA-based feature decorrelation, adaptive quantization, and DEFLATE entropy coding, with calibration performed once per model and compression setting [2511.01815]. Across Llama 3, Mistral NeMo, and R1-Qwen 2.5, it achieves up to \(20\times\) compression while maintaining reasoning and long-context accuracy, and \(40\times\) or higher for specific use cases [2511.01815]. For Mistral NeMo 12B at 8k context, recompute TTFT is 3098 ms versus 380 ms for KVTC decompression [2511.01815].

PQCache reformulates selective attention as a maximum inner-product search problem over cached keys and uses product quantization to index them [2407.12820]. It performs approximate MIPS with PQ codes and centroids, then fetches exact K/V for top-\(k\) tokens, and reports 4.60% score improvement over existing methods on InfiniteBench together with low system latency in both prefilling and decoding [2407.12820]. KVComp combines LLM-aware quantization with GPU Huffman coding, reporting on average 47% and up to 83% higher memory reduction rate than existing methods with little or no model accuracy degradation, and decompression throughput exceeding 400 GB/s for keys in fused kernels [2509.00579].

These approaches differ sharply in mechanism—vector quantization, sparse coding, channel shrinking, transform coding, entropy coding, ANN retrieval—but converge on the same empirical point: KV tensors are not unstructured arrays. They exhibit channel coupling, low-rank structure, cross-head and cross-layer redundancy, or compressible symbol distributions.

## 6. Layer-adaptive, modality-aware, and streaming variants

In multimodal and streaming settings, KV cache behavior departs from the text-only case. PrefixKV studies large vision-language models and argues that existing methods generally overlook the distinct importance distributions of KV vectors across layers by keeping the same cache size for each layer [2412.03409]. It reframes layer-wise retention as a global prefix configuration search over cumulative priority curves and reports up to \(1.8\times\) throughput speedup for LLaVA-1.5-7B at 20% compression with batch size 16, while remaining close to full-cache PPL and ROUGE on LLaVA-Description and MM-Vet [2412.03409].

VL-Cache is explicitly modality-aware. It observes that VLM attention exhibits distinct visual and text token sparsity patterns, then uses post-vision attention to allocate layer-wise budgets and score tokens [2410.23317]. Retaining only 10% of the KV cache achieves accuracy comparable to full cache, while generating 100 tokens can be accelerated by up to \(2.33\times\) end-to-end and decoding by up to \(7.08\times\), with 90% KV memory reduction in GPU [2410.23317].

Streaming video introduces a different pathology: continuously constructing new KV entries while evicting old ones can contaminate representations of recent inputs. DSCache addresses this with a **cumulative past KV cache** and a separate **instant cache** built from a recent feature buffer, plus position-agnostic encoding so that RoPE can be reapplied at use time [2605.01858]. On Streaming Video QA benchmarks, it reports an average 2.5% accuracy gain over prior methods, while keeping fixed memory in unbounded streams [2605.01858].

These results indicate that KV cache management depends not only on memory size but also on token type, layer role, and temporal regime. In VLMs, visual and textual tokens should not be scored identically. In streaming video, cache construction itself becomes a modeling problem.

## 7. Theoretical unification, limitations, and open directions

Much of the KV literature is heuristic, but recent work has started to supply explicit objectives. CapKV reinterprets eviction through the Information Bottleneck principle under a linear-Gaussian surrogate of attention, deriving a closed-form mutual information objective for the effective information capacity of a retained KV subset [2604.25975]. It argues that many existing eviction rules can be viewed as approximations to the same capacity-maximization principle, and introduces a capacity-aware method based on a log-determinant approximation with statistical leverage scores [2604.25975]. This provides a theoretical counterpoint to purely empirical top-\(k\) heuristics.

At the same time, current methods retain notable constraints. KVTC notes limitations in calibration realism, scale beyond 70B, imperfect correlation between Frobenius reconstruction error and task accuracy, and non-optimized compression kernels [2511.01815]. PrefixKV depends on attention as a proxy for importance and requires offline configuration estimation, even if only a small number of samples suffices in its experiments [2412.03409]. CSR identifies runtime overhead from Matching Pursuit and de-sparsification, and more generally the engineering complexity of dictionary management and online updates [2412.11741]. KeepKV relies on the locality of future attention scores for EMA prediction and on merging sufficiently similar candidates to keep perturbation bounded [2504.09936]. DSCache proves positional equivalence for RoPE, but not for arbitrary positional schemes [2605.01858].

The field is therefore converging on a few broad directions already stated explicitly across the literature: **online inference in compressed domains**, **combining compression with eviction or sparsity**, **learned transforms or autoencoders instead of fixed PCA or codebooks**, **adaptive per-layer or per-head compression**, and **better theoretical objectives for cache retention** [2511.01815]. This suggests that future KV cache systems will be hybrid: partly codec, partly scheduler, and partly memory model, with the cache treated as a structured representation whose storage, movement, and approximation are all first-class design variables.

Source: https://www.emergentmind.com/topics/key-value-cache-kv-cache