KV-Context Augmentation in Transformers

• KV-context augmentation is a family of methods that manipulate the transformer key-value cache to optimize long-context inference by modifying retention, summarization, and recomputation.
• It encompasses various techniques including internal memory injection, selection and merging, retrieval-based recomputation, and learned compressibility to balance efficiency with performance.
• Practical implementations show significant improvements in decoding speed and memory usage in multimodal models and long-context applications, underscoring its potential in transformer systems.

to=arxiv_search.search 彩神争霸官方ונים 北京赛车怎么json {"query":"KV cache compression augmentation long context transformer multimodal training-free retrieval arXiv", "max_results": 10, "sort_by": "relevance"}{"query":"KV cache compression augmentation long context transformer multimodal training-free retrieval arXiv", "max_results": 10, "sort_by": "relevance"} gives tool error: tool call failed for arxiv_search.search

to=arxiv_search.search 下载彩神争霸 天天彩票提现json {"query":"KV cache compression long context LLM", "max_results": 5}{"query":"KV cache compression long context LLM", "max_results": 5} gives tool error: tool call failed for arxiv_search.search

KV-context augmentation denotes a family of methods that manipulate the transformer key–value cache as the primary locus of context management, rather than treating context only as an input-token sequence. In the literature surveyed here, this includes changing which KV entries are retained, how they are summarized or merged, when they are recomputed, and how additional latent memory is injected into attention. The common motivation is that long-context inference is increasingly constrained by KV-cache memory and decode-time access costs that scale linearly with prefix length, especially in multimodal models and multi-turn serving settings (Gelberg et al., 7 May 2026, Wan et al., 2024, Tang et al., 3 Jan 2026).

1. Scope, formal setting, and method families

In a standard decoder stack, prompt tokens produce per-layer keys and values,

$$\mathbf{K}=\mathbf{X}\mathbf{W}_K,\qquad \mathbf{V}=\mathbf{X}\mathbf{W}_V,$$

and at decoding step $t$,

$$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$

Because the cache stores all past tokens, its size grows linearly with prompt length and generated length, and attention repeatedly reads the cache during decoding (Wan et al., 2024). Several works therefore treat the KV cache itself as a manipulable representation of context rather than a passive by-product of autoregressive inference (Xiao et al., 19 Feb 2025, Hu et al., 13 Jun 2025).

Within this framing, KV-context augmentation can be organized into a small number of recurrent patterns. Some methods inject new KV memory that was not present in the raw token stream; others restructure or compress the existing cache; still others retrieve or recompute only the subset of KV states that are expected to influence the current query. A plausible implication is that the field has converged on the KV cache as a shared abstraction spanning compression, retrieval, memory hierarchies, and external knowledge integration.

Pattern	Representative methods	KV operation
Internal memory injection	KV-Embedding, RACC, AtlasKV	prepend or inject extra KV slots
Selection and recomposition	LOOK-M, MEDA, Chelsea, MosaicKV, SeKV	retain, merge, cluster, reconstruct
Retrieval and recomputation	ActQKV, InfoFlow KV, cascading KV	retrieve or recompute a subset of KV states
Learned compressibility	KV-Distill, KV-CAT	train models or adapters for more compressible KV

This typology is descriptive rather than taxonomic. Several systems combine multiple patterns in one pipeline, such as selection plus merging, or external KV memory plus hierarchical retrieval (Wan et al., 2024, Wan et al., 24 Feb 2025, Huang et al., 20 Oct 2025).

2. Injecting additional memory into attention

A direct form of KV-context augmentation adds latent memory to attention without extending the visible token sequence. KV-Embedding does this by taking the final-token KV pair at selected layers and re-routing it as a prepended prefix,

$$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$

so that every position can attend to a sequence-level summary within a single forward pass. The method further adds an attention bias $b$ to the re-routed prefix slot and reports that $b=1.0$ is a robust optimum around 1.0 for Mistral-7B. On MTEB, KV-Embedding reaches $0.4937$ on Qwen3-4B, $0.5341$ on Mistral-7B-Instruct, and $0.5270$ on Llama-3.1-8B-Instruct, outperforming training-free baselines such as Echo, PromptEOL, and Token Prepending (Tang et al., 3 Jan 2026).

RACC applies the same general principle to knowledge-based visual question answering, but the injected memory is not derived from the final token of the same sequence. Instead, a hyperMLLM compresses retrieved documents and the image–question pair into soft-prompt representations, aggregates them, and then projects the aggregated vector into additional KV cache for each layer of a frozen baseMLLM. This yields a compact modulation in the form of KV cache rather than long retrieved text. RACC reports $63.92\%$ on OK-VQA and reduces inference latency by $t$0 relative to RAVQA-v2; with pre-saved compressed prompts on WIT, evaluation time drops from $t$1 s to $t$2 s and disk space from $t$3 MB to $t$4 MB (Weng et al., 2024).

AtlasKV generalizes KV injection to billion-scale knowledge graphs. KG2KV rewrites triples into query–key–value strings, embeds them, and projects them into attention-space KVs, while HiKVP builds a three-level hierarchy so that per-token knowledge access scales with $t$5 rather than $t$6. The attention output is decomposed into a knowledge term and a sequence term,

$t$7

AtlasKV reports less than $t$8 GB VRAM for $t$9B triples and uses no external retrievers, long context priors, or retraining when adapting to new knowledge (Huang et al., 20 Oct 2025).

A common misconception is that such methods are merely alternative prompting strategies. The surveyed systems operate inside the attention mechanism: they modify the set of keys and values that queries see, often without adding any new visible tokens at all. KV-Embedding explicitly shows that simply removing the causal mask is harmful: on Qwen3-4B, “Last Token + Bi-Attn” averages $$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$0, “Last Token + Causal Attn” $$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$1, and KV-Embedding $$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$2; on Mistral-7B the corresponding figures are $$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$3, $$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$4, and $$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$5 (Tang et al., 3 Jan 2026).

3. Selection, merging, clustering, and multi-resolution caches

A second major line of work augments context by reorganizing the existing cache into a more informative compressed representation. In multimodal models, LOOK-M is explicitly motivated by the observation that during prompt prefill the model prioritizes more textual attention over image features. It boosts text-token importance by setting

$$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$6

keeps a recent window and top-$$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$7 important tokens, and then merges pruned image-token KV pairs into retained ones using averaged, pivotal, or weighted merging. At a $$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$8 KV budget on LLaVA-v1.5-7B, “LOOK-M (TP + P-Merge)” matches or exceeds full cache across almost all MileBench subtasks; at $$\mathbf{q}_t=\mathbf{x}_t\mathbf{W}_Q,\qquad \mathbf{x}_{t,\mathrm{out}}=\operatorname{Softmax}\!\left(\mathbf{q}_t\mathbf{K}^{\top}/\sqrt{D}\right)\mathbf{V}.$$9 KV budget it remains close to full cache while H2O, SnapKV, and RoCo collapse on several tasks, including NH $$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$0 for those text-only baselines (Wan et al., 2024).

MEDA makes the same multimodal argument at the layer level. It allocates layer-wise KV budget from cross-modal attention entropy,

$$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$1

then selects high-attention KVs and merges the rest into conserved tokens. MEDA reports up to $$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$2 KV cache memory reduction and $$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$3 times faster decoding, while maintaining or enhancing performance on long multi-image and long-video tasks (Wan et al., 24 Feb 2025).

Chelsea replaces explicit eviction with online clustering. It divides the sequence into chunks, uses Chunked Soft Matching with an alternating partition inside each chunk, and merges each cluster into a centroid. Its approximate attention uses cluster degrees $$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$4 through

$$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$5

Chelsea achieves up to $$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$6 KV cache memory reduction, up to $$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$7 faster decoding, and up to $$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$8 lower end-to-end latency while maintaining performance close to full KV on LongBench (Hu et al., 13 Jun 2025).

More aggressive cache restructuring appears in MosaicKV, Spherical KV, SeKV, and the cascading KV cache. MosaicKV performs dynamic two-dimensional compression across both sequence and channel dimensions, selecting important elements per KV vector and compression strategies per segment; on an H800 it reports up to $$\tilde{\mathbf{K}}_{\le i}^{(l)}=[\,\mathbf{k}_n^{(l)}\Vert \mathbf{K}_{\le i}^{(l)}\,],\qquad \tilde{\mathbf{V}}_{\le i}^{(l)}=[\,\mathbf{v}_n^{(l)}\Vert \mathbf{V}_{\le i}^{(l)}\,],$$9 attention speedup, $b$0 lower decode latency, $b$1 higher throughput, $b$2 lower memory usage, and only $b$3 average accuracy loss on LongBench and RULER (Qiang et al., 1 Jul 2026). Spherical KV stores keys in a spherical parameterization and computes logits directly from angle codes without reconstructing dense keys; under matched quality it reports throughput gains of $b$4 and effective KV bytes/token reductions of $b$5 as context length grows from $b$6K to $b$7K (Chauhan et al., 13 May 2026). SeKV organizes the cache into entropy-guided semantic spans with summaries on GPU and low-rank SVD bases on CPU, and then zooms in on query-relevant spans during decoding; it improves over the strongest semantic compression baseline by $b$8 on average and reduces GPU memory by $b$9 versus full KV caching at $b=1.0$0K context (Abaskohi et al., 30 Jun 2026). The cascading KV cache, by contrast, uses hierarchically cascaded sub-caches and attention-score EMA to retain exponentially older history within fixed cache size; it reports better passkey retrieval accuracy at $b=1.0$1M tokens after four doublings of a $b=1.0$2K cache and $b=1.0$3 lower prefill latency than FlashAttention on $b=1.0$4M tokens (Willette et al., 2024).

These methods collectively challenge the misconception that compression is equivalent to irreversible loss. In LOOK-M, MEDA, Chelsea, MosaicKV, and SeKV, compression is often a change of representation: clustered centroids, merged pseudo-tokens, low-rank reconstructions, or span summaries can preserve or later recover information that a flat eviction policy would simply discard (Wan et al., 2024, Wan et al., 24 Feb 2025, Hu et al., 13 Jun 2025, Abaskohi et al., 30 Jun 2026).

4. Query-conditioned retrieval and selective recomputation

A third family augments context by deciding, at runtime, which latent states deserve access rather than by keeping a uniformly compressed cache. ActQKV formulates KV retrieval as a vector-retrieval problem over the cache. In prefill, it builds an activation-aware probe-query from token queries $b=1.0$5 using Activation Bias,

$b=1.0$6

and retrieves the most similar historical KV chunks. During decoding it allocates layer-wise KV cut-offs using entropy of similarity scores. On Long-Bench with a $b=1.0$7K KV budget, ActQKV reaches $b=1.0$8 average versus $b=1.0$9 for full context and $0.4937$0 for TSLLM; on $0.4937$1-Bench it reaches $0.4937$2, exceeding TSLLM at $0.4937$3 while using $0.4937$4K rather than $0.4937$5K KV (Xiao et al., 19 Feb 2025).

InfoFlow KV addresses a different query-conditioned problem: precomputed document KVs in retrieval-augmented long-context QA do not preserve global causal dependencies or inference-consistent RoPE geometry. It therefore scores tokens by prompt-to-context attention norm,

$0.4937$6

selects top-$0.4937$7 tokens for recomputation, reconstructs global positional assignments, and optionally reorders chunks before recomputation. The paper reports consistent gains over CacheBlend and EPIC under comparable efficiency budgets, and in multi-GPU sequence-parallel settings reports $0.4937$8 speedup at $0.4937$9K and $0.5341$0 speedup at $0.5341$1K versus a single-GPU baseline (Teng et al., 5 Mar 2026).

This query-conditioned view also clarifies a recurrent empirical pattern: more resident KV is not always better than better-chosen KV. ActQKV exceeds full-context Long-Bench average with only about $0.5341$2 of the full KV cache, and LOOK-M at $0.5341$3 KV budget can match or exceed the full cache on many multimodal subtasks (Xiao et al., 19 Feb 2025, Wan et al., 2024). This suggests that the cache is often noisy as well as large.

5. Learned compressibility, distillation, and robustness

Several papers move from post-hoc manipulation to training the model or an adaptor so that useful information is easier to store in fewer KV states. KV-Distill is the most explicit formulation of learned KV-context compression. It treats compressed and uncompressed caches as a student–teacher pair, selects top-$0.5341$4 tokens with a learned scorer, routes selected tokens through conditional rsLoRA adapters, and trains with a symmetric distillation objective

$0.5341$5

with $0.5341$6. On SQuAD, Llama-3 8B drops from $0.5341$7 to $0.5341$8 at $0.5341$9 KV and $0.5270$0 at $0.5270$1 KV, while question-independent H$0.5270$2 collapses to $0.5270$3 and $0.5270$4 at the same budgets; on GovReport, additional domain finetuning preserves Rouge-L almost fully down to $0.5270$5 KV (Chari et al., 13 Mar 2025).

Training can also target the backbone itself. “Training Transformers for KV Cache Compressibility” formalizes KV compressibility as a property of learned representations rather than of context alone, states that almost any sequence-to-vector function admits both highly compressible and inherently non-compressible transformer implementations, and proposes KV-Compression Aware Training (KV-CAT), a continued pretraining procedure with train-time KV sparsification that masks KV slots during training. The abstract reports improved quality–budget tradeoffs of downstream compression methods across retrieval, long-context question answering, and perplexity-based evaluation of compressed-prefix continuation (Gelberg et al., 7 May 2026).

A related but distinct question is whether long-context fine-tuning improves robustness under downstream compression. In “Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context LLMs,” GRPO-based objectives yield in-domain gains of up to $0.5270$6 points over the base model, but out-of-domain generalization remains task dependent; under RetrievalAttention compression, the base model shows an average drop of $0.5270$7, while the best reported drop is $0.5270$8 for the ID+C objective (Molfese et al., 26 Jan 2026). This is weaker than the near-lossless regime of KV-Distill, but it is evidence that KV robustness can be shaped by training signals rather than only by inference policies.

A plausible synthesis is that the field is splitting into two regimes: post-hoc cache engineering for frozen models, and representation-level training so that post-hoc engineering works better. The available evidence supports both, but with different operating points and implementation burdens (Chari et al., 13 Mar 2025, Gelberg et al., 7 May 2026, Molfese et al., 26 Jan 2026).

6. Serving systems, quantization, and deployment constraints

The most deployment-oriented papers argue that KV-context augmentation is not only a modeling problem but also a systems problem. Unified KV Pooling studies long-context serving with KV offloading to host memory and SSDs and identifies two bottlenecks: serialized retrieval through host memory and SSD, and SSD retrieval spending $0.5270$9 of its time in the kernel filesystem rather than device access. It introduces unified KV pooling across multiple host-memory modules and SSDs, plus KV-passthrough via SPDK. Across LLaMA 3.1-8B, GPT-OSS-20B, and Qwen3-30B-A3B, it reduces TTFT in long contexts by about $63.92\%$0 over state-of-the-art techniques, brings all models under $63.92\%$1 s TTFT at $63.92\%$2K context, and reduces blocked I/O time by up to $63.92\%$3 (Kang et al., 10 Jun 2026).

UltraQuant addresses the same bottleneck from the opposite direction: reduce bytes per KV sufficiently that long prefixes remain resident. It stores FP4 KV tensors with UE8M0 group scales and uses native scaled-MFMA support on AMD CDNA4. On a long-context, multi-turn agentic workload, UltraQuant cuts P50 TTFT by $63.92\%$4 in the cache-pressured late rounds, $63.92\%$5 across all rounds, and raises output throughput by $63.92\%$6 over an FP8 KV baseline (Chakrabarti et al., 18 Jun 2026).

These results matter because many algorithmic KV methods assume that a reduced or reorganized cache automatically translates into lower latency. The surveyed systems work shows that this is not guaranteed. Spherical KV explicitly argues that compression methods often leave the critical-path bottleneck only partially addressed when compressed states are reconstructed into dense vectors during decoding, and therefore computes attention logits directly from angle-domain codes (Chauhan et al., 13 May 2026). MosaicKV likewise emphasizes that fine-grained sparsity and compression-management overhead can offset the gains from compression, motivating packed formats and heterogeneous CPU–GPU pipelines (Qiang et al., 1 Jul 2026). Unified KV Pooling shows that even when the algorithmic working set is small enough, the storage stack can dominate latency if the retrieval path is poorly designed (Kang et al., 10 Jun 2026).

A final misconception is that multimodal KV compression can be solved by directly importing text-only heuristics. Both LOOK-M and MEDA argue the opposite: image and video tokens dominate sequence length, cross-modal interactions vary sharply across layers, and text-prior or cross-modal-entropy signals materially improve performance relative to text-only baselines (Wan et al., 2024, Wan et al., 24 Feb 2025). This suggests that “KV-context augmentation” is not a single technique but a design space whose concrete instantiation depends on modality mix, serving regime, and whether the desired outcome is compression, retrieval, reconstruction, or memory injection.