--- title: KV Caching in Transformers url: https://www.emergentmind.com/topics/kv-caching type: topic --- # KV Caching in Transformers Key–Value (KV) caching is a central optimization enabling efficient inference in transformer-based large language models (LLMs), vision-language models, and generative autoregressive architectures more broadly. By storing the intermediate key ($K$) and value ($V$) projections of previously seen tokens, this technique allows self-attention to be computed with linear rather than quadratic complexity, and forms the basis for a diverse ecosystem of algorithmic and systems-level innovations in memory scalability, cache compression, and high-throughput deployment. ## 1. Fundamentals of KV Caching in Transformer Architectures In the canonical transformer decoder, at each decoding step $t$, newly produced token hidden states are projected to keys and values, and appended to layerwise caches $K \in \mathbb{R}^{T \times d}$, $V \in \mathbb{R}^{T \times d}$, with $T$ denoting total context length and $d$ the per-head hidden size. At every step, queries $Q$ are scored against all past keys: $Q K^\top \in \mathbb{R}^{1 \times T}$, followed by a softmax and a value-weighted sum. Naively storing all $K,V$ pairs scales both memory and compute quadratically with sequence length, rendering inference memory-bound and potentially intractable for long contexts, especially in high-resolution or multi-modal scenarios (e.g., GUI agents, document understanding, image generation) [2510.00536]. KV caching amortizes attention computation, ensuring that, after initial prefill, each new token’s attention cost is $O(T d)$, and total attention cost over $T$ steps per head is $O(T^2 d)$. However, as context windows increase, unbounded growth of KV caches presents severe challenges for memory usage, latency, and throughput [2410.03065, 2506.02634]. ## 2. Compression and Eviction Algorithms for Efficient KV Caching To mitigate ballooning memory demand from long contexts, contemporary research has developed advanced KV cache eviction and compression mechanisms: - **Layer-aware compression:** GUI-KV [2510.00536] underscores the importance of adapting cache budgets per layer; in GUI agents, empirically uniform attention sparsity motivates a flat per-layer allocation, outperforming non-uniform or pyramidal schemes previously used for natural images. - **Cake-slicing optimization:** CAKE [2503.12491] formalizes cache allocation as a utility-maximizing problem over layer-specific preference scores, derived from the entropy (spatial dispersion) and variance (temporal shift) of recent attention patterns. Allocation proceeds via a cascading, monotonic eviction procedure, ensuring tight adherence to memory constraints and theoretically optimal per-layer slice selection. - **Personalized per-layer schedules:** XKV [2412.05896] demonstrates that cache-value importance varies dramatically per layer, and models this as a discrete knapsack problem. A greedy, heap-based allocation achieves optimal per-layer compression, yielding $61.6\%$ memory reduction and $2.1\times$ efficiency gains over static uniform methods. - **Redundancy-aware token selection:** R-KV [2505.24133] and KVCrush [2503.00022] incorporate both attention-based importance and redundancy (cosine or Hamming similarities in key space), with R-KV in particular pruning highly redundant tokens in chain-of-thought reasoning, enabling over $90\%$ cache reduction with negligible accuracy loss. - **Sparse token and window attention:** ALISA [2403.17312] introduces Sparse Window Attention (SWA), retaining only a union of locally recent and globally most-attended tokens in the cache, dramatically shrinking KV footprint (to $\sim 20\%$ of baseline) while maintaining throughput and accuracy in long-sequence autoregressive settings. - **Semantic compression:** SentenceKV [2504.00970] restructures the token-level KV cache into semantically-aggregated sentence blocks, storing compact sentence embeddings on GPU while offloading less critical token KVs to CPU. Decoding retrieves only semantically-relevant tokens, preserving accuracy at high compression levels. - **Scale-adaptive policies for multi-scale architectures:** AMS-KV [2511.16047] in visual autoregressive models retains only condensed “global” and recency-based “local” tokens at each scale, guided by inter-scale similarity. This dramatically reduces cache and attention cost in coarse-to-fine image synthesis, with up to $84.8\%$ KV memory reduction. ## 3. Quantization and Mixed-Precision Strategies Quantizing keys and values in the cache is an orthogonal—often synergistic—approach to footprint reduction: - **Joint channel quantization:** Coupled Quantization (CQ) [2405.03917] exploits inter-channel dependencies, jointly encoding groups of $c$ channels with $b$ bits ($b/c$ bits/channel), enabling effective quantization down to $1$ bit/channel (16$\times$ compression) with competitive accuracy. - **Dynamic channelwise precision boost:** The Kitty system [2511.18643] ranks key channels by sensitivity (magnitude heuristic) and selectively boosts the top $f$ fraction to $4$ bits, with the remainder quantized to $2$ bits. By maintaining a unified page-centric memory layout and coalesced access (Triton kernels), Kitty preserves the full $8\times$ memory advantage of 2-bit quantization while nearly eliminating the accuracy gap ($<1\%$ drop on challenging reasoning and code tasks). - **Block and value-specific strategies:** Offloading non-critical value vectors (KCache [2404.18057]) and windowed quantization (Q-Buffer, Sinks in Kitty) further reduce cache cost outside the main GPU memory. ## 4. Systems, Caching Layers, and Distributed Inference Modern LLM serving systems integrate KV caching with sophisticated paged, distributed, and offloaded designs: - **Page-chunked KV layers:** LMCache [2510.09665] introduces a cache-aware, connector-driven layer that exposes KV caches as first-class data structures. Batched, chunked I/O, asynchronous compute–I/O pipelining, and modular connectors enable cross-query cache sharing, prefill–decode disaggregation, and robust cache migration, yielding up to $15\times$ throughput boosts at enterprise scale. - **Workload-aware cache management:** Analysis of cloud traces [2506.02634] reveals that real-world KV reuse patterns are highly skewed, mainly driven by single-turn requests. Probabilistic workload-aware eviction, based on empirical future reuse probability per request type, outperforms classical LRU/LFU/FIFO policies by $1.5$–$23.9$ pp in hit rate, reducing TTFT by $28$–$42\%$. - **Multi-tenant and cross-agent reuse:** KVShare [2503.16525] and KVCOMM [2510.12872] generalize prefix cache sharing to multi-tenant and multi-agent deployments. Dual-Stage High Deviation (DHD) algorithms identify token-level cache recomputation requirements via embedding-level and edit distance analyses, orchestrating selection and partial recomputation under tight accuracy constraints. KVCOMM's anchor-pool mechanism aligns KV offsets across diverging agent contexts, achieving $70\%$ cross-agent reuse and $6$–$8\times$ TTFT speedup in collaborative LLM systems. - **Bidirectional compute–I/O scheduling:** Cake [2410.03065] and LMCache [2510.09665] recognize that for long prefixes, computing or loading a KV cache from storage are complementary; dynamic “meeting in the middle" algorithms minimize prefill latency, adapting to GPU and I/O availability. ## 5. Domain- and Hardware-Specific KV Caching Specialized applications necessitate further adaptation: - **GUI-KV [2510.00536]:** For GUI agents with highly redundant visual inputs, spatial saliency scoring (residual $\ell_2$-norms) and temporal redundancy projection (QR subspace overlap across frames) enable plug-and-play, training-free cache pruning that not only reduces decoding FLOPs by nearly 40% but actually improves step accuracy over the full-cache baseline. - **Edge devices and memory hierarchy:** The Kelle system [2510.16040] co-designs cache and on-chip memory, leveraging eDRAM’s density with two-dimensional adaptive refresh, selective recomputation, and fine-grained attention-driven eviction. Exploiting the volatility/importance trade-off in bit positions and tokens yields $3.94\times$ speedup and $4.46\times$ energy savings on LLaMA2-7B relative to SRAM-only baselines, demonstrating that modest (N’ ≈ 128–512) transient KV budgets with recomputation and adaptive scheduling achieve near-cloud accuracy on resource-constrained hardware. - **Diffusion LMs:** In bidirectional diffusion language models, KV caching is challenging due to non-monotonic token unmasking. FreeCache [2505.21467] introduces block-wise KV approximation: once a block is “clean,” its projections are reused in all further steps, reducing dominant $\mathcal{O}(L^2)$ costs to $\mathcal{O}(LB)$, yielding $2$–$5\times$ speedup with $<2\%$ downstream accuracy penalty. ## 6. Empirical Results and Comparative Evaluations The empirical impact of KV caching and its variants is substantial across inference regimes, domains, and workloads: | Method/System | Peak Memory Reduction | Throughput Speedup | Accuracy Drop | Context/Task | Reference | |--------------------|----------------------|--------------------|--------------|-----------------------|-------------------| | GUI-KV | 38.9% FLOP↓ | – | $+4.1\%$ | GUI-AgentNetBench | [2510.00536] | | CAKE | $>95\%$ ($3.2\%$ cache) | $>10\times$ | $<0.3$pts | LongBench (128K ctx) | [2503.12491] | | R-KV | 90% | $6.6\times$ | $<1\%$ | Reasoning/Math | [2505.24133] | | XKV | 61.6% | $2.1\times$ | negligible | LongBench | [2412.05896] | | LMCache | up to $15\times$ | $2.1$–$4.1\times$ | $<0.5\%$ | Enterprise LLM | [2510.09665] | | Kitty | $8\times$ | $2.1$–$4.1\times$ | $<1\%$ | Reasoning/Code LLM | [2511.18643] | | Kelle (eDRAM) | $4.5\times$ energy | $3.94\times$ | negligible | Edge LLM | [2510.16040] | | SentenceKV | 30–40% mem savings | – | $<0.5\%$ | PG-19, LongBench | [2504.00970] | | KVShare | – | $1.2\times$ | $+0.8$ BLEU | Multi-tenant LLM | [2503.16525] | | FreeCache (DLM) | – | $2$–$5\times$ | $1$–$2\%$ | Diffusion LMs | [2505.21467] | These results show state-of-the-art cache compression, lossless or near-lossless accuracy on demanding long-context and reasoning tasks, and system-level throughput or TTFT improvements from $2\times$ to $15\times$ over naive or non-caching baselines. ## 7. Outlook and Future Directions KV caching has evolved from a basic memory–compute tradeoff mechanism into a rich research area intersecting cache-aware algorithms, distributed and edge deployment, quantization, redundancy elimination, and content-aware policy design. Current directions include: - Head- and block-level adaptive allocation beyond the layer granularity [2503.12491, 2510.00536] - Tighter integration of hardware features (e.g., memory hierarchy, adaptive refresh, eDRAM/DRAM interplay [2510.16040]) - Plug-and-play composition of compression, quantization, and workflow-aware policies for multi-agent and retrieval-augmented scenarios [2510.12872, 2507.07400] - Extension to non-autoregressive (diffusion, bidirectional, multi-scale) models [2505.21467, 2511.16047] - End-to-end optimization jointly considering cache layout, kernel fusion, I/O scheduling, and pipeline orchestration [2510.09665, 2410.03065] - Fully semantic or hierarchical compression schemes (sentence/paragraph/graph-level) that retain contextual and task-relevant information without reverting to fine-grained token-level heuristics [2504.00970] In sum, KV caching constitutes a critical substrate for scalable, high-performance inference across the expanding landscape of generative AI, and advances in cache-adaptive algorithms and systems will continue to play a pivotal role in realizing the potential of long-context and collaborative foundation models. Source: https://www.emergentmind.com/topics/kv-caching