KV Collector Mechanisms in AI Inference

• KV Collector is a set of inference-time mechanisms that select, compress, and retrieve key-value cache entries to maintain attention fidelity while reducing storage and latency.
• These methods employ diverse retention strategies, including token eviction, quantization, and dynamic retrieval across text, vision, and multimodal domains.
• KV Collector approaches balance quality and efficiency by modeling token importance through attention, value, and risk factors tailored to specific system demands.

“KV Collector” denotes a class of inference-time mechanisms that select, preserve, compress, reuse, or retrieve the subset of Key-Value (KV) cache entries judged most consequential for downstream attention. In recent work, the term spans long-context LLMs, multimodal LLMs (MLLMs), vision-language large models (VLLMs), graphical-user-interface agents, and next-scale visual autoregressive (VAR) transformers. The unifying problem is that KV caches grow with sequence length, scale count, resolution, or interaction steps, and can dominate both memory footprint and decode-time bandwidth; the unifying objective is to keep the attention computation faithful while reducing storage, latency, or prefill cost (Xu et al., 20 Nov 2025, Wang et al., 17 Mar 2026, Roy et al., 7 Dec 2025, Jiang et al., 30 Dec 2025, Akulov et al., 5 Sep 2025, Jiang et al., 29 Oct 2025, Cederlund et al., 14 May 2026, Han et al., 1 Jun 2026).

1. Problem setting and system role

KV collection arises because cached keys and values are indispensable for autoregressive decoding, yet their storage cost scales with model depth, hidden size, context length, and batch size. One explicit formulation gives

$$\text{KV\_Cache\_Size} = 2 \times P \times N_{\text{layers}} \times d_{\text{model}} \times L_{\text{seq}} \times B,$$

with the factor of $2$ accounting for both $K$ and $V$ (Roy et al., 7 Dec 2025). The same paper reports that, for GPT-2 Medium with $N_{\text{layers}}=24$, $d_{\text{model}}=1024$, FP16 storage, sequence length $2048$, and batch size $8$, the KV cache is about 1.61 GB, larger than the model’s ~690 MB FP16 weight footprint (Roy et al., 7 Dec 2025). KVLinC gives a related serving-side example: for Llama-3-8B at 8k context and batch size 16, the KV cache alone consumes about 16 GB (Saxena et al., 6 Oct 2025).

The pressure becomes sharper outside plain text decoding. In GUI VLM agents, UI-TARS-1.5-7B consumes 76 GB of GPU memory on merely five screenshots (Han et al., 1 Jun 2026). In VAR image generation, cache growth is tied not only to token count but also to the scale-by-scale generation schedule; HeatKV explicitly describes the resulting KV growth as superlinearly dependent on image resolution (Cederlund et al., 14 May 2026). In retrieval-augmented generation, InfoFlow KV identifies a different failure mode: chunk-level caches are precomputed under chunk-local causal masks and chunk-local RoPE positions, but inference consumes a single globally ordered sequence, so cached states computed in isolation do not fully restore cross-chunk information flow (Teng et al., 5 Mar 2026).

A concise view of the design space is as follows.

Paradigm	Representative method	Core retention principle
Eviction or selection	ReST-KV, CurDKV, CompilerKV, KVCompose	Keep tokens whose removal or omission most harms output
Compression or encoding	VQKV, KV-CAR, PackKV, KVLinC	Store a smaller surrogate for each KV state
Retrieval or recomputation	CTkvr, InfoFlow KV	Materialize only the subset needed for the current query
Domain-specific cache shaping	AMS-KV, FlashCache, PureKV, HeatKV, STaR-KV	Use modality or scale structure to guide retention

This taxonomy suggests that “collection” is not restricted to token eviction. In the current literature it also includes compressed surrogates, dynamically recalled subsets, and modality-aware retention policies.

2. Importance estimation and token retention

A large subset of KV collectors ranks cached entries by an importance proxy, then retains a budgeted subset. The simplest proxies use attention weights; the more recent methods replace or augment attention with output-aware, value-aware, risk-aware, or structure-aware signals.

ReST-KV formulates eviction as an output-reconstruction problem. For a KV pair $n$, its indicator is the layer-output discrepancy caused by removing that pair, and the paper derives an approximation

$$\mathbf{I}_t[n] = \frac{\mathbf{A}_t[n]}{1-\mathbf{A}_t[n]} \left\| \operatorname{MHA}\!\left(\mathbf{x}_t,\langle \mathbf{K}_T,\mathbf{V}_T \rangle \right) - \mathbf{v}_n \mathbf{W}_O \right\|_2.$$

This explicitly models attention redistribution after eviction rather than relying on raw attention magnitudes alone. ReST-KV then smooths the score temporally with EMA and spatially with an adaptive shifted window (An et al., 9 May 2026). CurDKV reaches a similar conclusion from a different route: it argues that KV compression should preserve $2$0, not merely $2$1, and therefore scores tokens using leverage-style signals from both keys and values, with combined score $2$2 (Sengupta et al., 18 Sep 2025). CompilerKV further treats token retention as a prefill-only decision problem subject to prompt-dependent risk. It stabilizes token utility with cumulative attention and normalized value magnitude, reweights heads by an offline-learned Head Heterogeneity Table, and gates retention using a Risk-Adaptive Threshold derived from attention entropy and local perplexity (Yang et al., 9 Feb 2026).

Other methods retain attention as the primary observable but alter how it is aggregated. KVCompose estimates head- and layer-specific token importance from task or context attention, lets each head rank tokens independently, and then aligns those head-specific selections into “composite tokens” so that the compressed cache remains tensor-compatible with standard inference engines (Akulov et al., 5 Sep 2025). PureKV uses lower-layer cumulative attention over non-recent tokens,

$2$3

and weights it by value norms to form $2$4; high-layer importance is then estimated without high-layer attention matrices via $2$5, which preserves compatibility with FlashAttention and sparse attention kernels (Jiang et al., 29 Oct 2025). STaR-KV starts from a recent-query attention score $2$6, then multiplies it by a subspace-aware mutual-information prior $2$7, a temporal discount $2$8 for visual history, and an entropy-derived temperature transformation $2$9, thereby replacing a fixed top-$K$0 cutoff with a distribution-aware ranking (Han et al., 1 Jun 2026).

These systems share a common shift: importance is no longer treated as a scalar directly read off a single attention map. It is instead reconstructed from the expected contribution of a token to output fidelity, long-range risk, head specialization, temporal stability, or modality-specific structure.

3. Compression as collection: storing surrogates instead of full KV states

A second line of work collects KV information by replacing full-precision states with smaller representations. In this regime, the collector is the encoder, quantizer, or packer that determines which aspects of each KV vector are preserved.

VQKV treats each key or value vector as a structured object to be encoded by Residual Simple Vector Quantization. For $K$1, the method iteratively selects codewords $K$2 and reconstructs

$K$3

The cache stores codebook indices rather than floating-point tensors, keeps an initial uncompressed segment $K$4 and a recent local window $K$5, and reconstructs only the slice needed for the current decoding step. On LLaMA3.1-8B, the paper uses $K$6, $K$7, $K$8, $K$9, yielding 82.8% compression; it reports retention of 98.6% of baseline performance on LongBench and 4.3x longer generation length on the same memory footprint (Wang et al., 17 Mar 2026).

KV-CAR combines two orthogonal forms of reduction. First, a layer-wise autoencoder compresses each $V$0 and $V$1 vector from dimension $V$2 to $V$3 before caching and reconstructs it at retrieval time. Second, an L1-based similarity test identifies adjacent-layer heads whose KV tensors can be safely reused. The framework is described as architecture-agnostic, and across GPT-2 and TinyLLaMA evaluations it achieves up to 47.85 percent KV cache memory reduction with minimal reported quality loss (Roy et al., 7 Dec 2025). PackKV, by contrast, is a systems-oriented collector: it applies quantization, encode-aware repacking, bit-packing, seamless appending, and computation-aware decompression. Its defining property is that decompression is fused into matrix-vector multiplication, so there is no global-memory “decompress-then-compute” stage. At matched accuracy relative to KIVI, PackKV reports average memory-reduction improvements of 153.2% for the $V$4 cache and 179.6% for the $V$5 cache, with average throughput gains of approximately +75.6% for K and +171.6% for V across A100 and RTX Pro 6000 GPUs (Jiang et al., 30 Dec 2025).

KVLinC targets the extreme low-bit regime. It quantizes the cache to 2-bit form with group size 128, uses raw channel-wise quantization for keys and Hadamard-rotated token-wise quantization for values, and adds lightweight linear correction adapters to compensate for key-induced attention error. The corrected attention output is expressed as a ratio whose numerator and denominator each add a learned correction term $V$6, and the recurrent correction states make the overhead independent of context length (Saxena et al., 6 Oct 2025). The paper reports up to 2.55× faster inference than the Flash Attention baseline and up to 3.5× larger batch size (Saxena et al., 6 Oct 2025).

Taken together, these methods treat collection as rate–distortion control. Instead of deciding which tokens survive, they decide which information within each token’s KV state survives.

4. Retrieval and recomputation collectors

Some collectors do not permanently retain a fixed compressed subset. They reconstruct or retrieve the currently relevant subset on demand.

InfoFlow KV addresses long-context RAG, where per-document caches are prefetched independently and therefore miss global causal dependencies. Its selector is prompt-conditioned attention mass

$V$7

computed only after reconstructing inference-consistent global positions $V$8 and $V$9 for context and prompt tokens. The recomputation set is then $N_{\text{layers}}=24$0 (Teng et al., 5 Mar 2026). The paper shows that GLOBAL RoPE geometry is superior to mismatched alternatives such as HL-HP or TL-TP, and reports that on 4 NVIDIA H100 GPUs the proposed method reduces TTFT from 707.8 ms to 427.6 ms at 16K, and from 2350.1 ms to 914.0 ms at 32K, relative to ring attention (Teng et al., 5 Mar 2026). The method also proposes chunk reordering: independently scored chunks are moved closer to the prompt, and a second selection pass is then run under the final global layout (Teng et al., 5 Mar 2026).

CTkvr replaces full-cache attention with a two-stage retrieval scheme. During prefilling, it selects the last $N_{\text{layers}}=24$1 query vectors as centroids, computes top-$N_{\text{layers}}=24$2 key IDs for each centroid, and stores a lightweight centroid-grained index $N_{\text{layers}}=24$3. During decoding, it first recalls candidate keys via centroid similarity and then reranks at token granularity. The design is motivated by the observation that adjacent queries under RoPE are highly similar and share most of their top-$N_{\text{layers}}=24$4 KV entries (Lu et al., 17 Dec 2025). The paper reports less than 1% accuracy degradation across multiple benchmarks, up to 3× throughput on Llama-3-8B, up to 4× throughput on Yi-9B, and at 96K context supports batch sizes up to 10× larger on Llama-3-8B and 20× larger on Yi-9B (Lu et al., 17 Dec 2025). It also reports index construction up to 50× faster than HNSW, 1000× faster than IVF, and 10000× faster than KMeans (Lu et al., 17 Dec 2025).

These two systems exemplify a distinct collector semantics. They do not merely shrink a static cache; they restore or retrieve the portion of the cache that can still causally influence the current answer.

5. Domain-specific collectors in visual, multimodal, and multi-scale generation

In multimodal and visual settings, the utility of KV states is shaped by additional structure: image scales, temporal redundancy, spatial specialization, and low-frequency backbones. Several recent collectors therefore replace generic recency heuristics with domain-specific policies.

AMS-KV is tailored to next-scale VAR. It begins from three observations: attending to tokens from local scales significantly contributes to generation quality; allocating a small amount of memory for the coarsest scales, termed condensed scales, stabilizes multi-scale image generation; and strong KV similarity across finer scales is predominantly observed in cache-efficient layers, whereas cache-demanding layers exhibit weaker inter-scale similarity (Xu et al., 20 Nov 2025). The method permanently keeps early condensed scales and adaptively flushes later scales according to inter-scale similarity and layer role. An ablation over condensed scales shows FID 2.48 / IS 298.35 with $N_{\text{layers}}=24$5, FID 2.16 / IS 333.04 with $N_{\text{layers}}=24$6, FID 2.09 / IS 333.51 with $N_{\text{layers}}=24$7, versus FID 2.07 / IS 336.15 for the baseline (Xu et al., 20 Nov 2025). Across the full model, AMS-KV reports KV-cache reduction of up to 84.83%, self-attention latency reduction of 60.48%, and stable scaling from batch size 128 to 256 when the baseline VAR-d30 encounters out-of-memory failures at 128 (Xu et al., 20 Nov 2025). For Infinity-2B, it also reports 31.4% reduction in total memory usage and better reconstruction fidelity than SWA and STA, with PSNR 21.92, SSIM 0.669, and LPIPS 0.112 (Xu et al., 20 Nov 2025).

HeatKV also targets VAR, but allocates memory at the granularity of a head-scale pair. Using a small offline calibration set of 10 calibration prompts, it computes the Scale-dependent Cumulative Attention Score

$N_{\text{layers}}=24$8

ranks head-scale dependencies, and compiles a static pruning schedule under a fixed budget (Cederlund et al., 14 May 2026). On Infinity-2B, HeatKV reports at 20% budget: 7800 MB KV cache, FID 1.02, LPIPS 0.044, PSNR 28.43; at 10% budget: FID 1.67, LPIPS 0.073, PSNR 25.40; and at 4% budget: FID 2.70, LPIPS 0.127, PSNR 22.14 (Cederlund et al., 14 May 2026). The paper states that HeatKV at 10% budget outperforms the strongest baseline at 20%, which underlies its claim of approximately 2× better compression efficiency (Cederlund et al., 14 May 2026).

FlashCache revisits multimodal KV compression from the perspective of KV-matrix distribution. It observes that frequency-domain energy is concentrated in low frequencies, extracts a low-pass-filtered Base KV via DCT/IDCT, defines Outlier KVs by the deviation score

$N_{\text{layers}}=24$9

and allocates per-layer budgets according to the high-frequency energy ratio $d_{\text{model}}=1024$0 (Yang et al., 20 Nov 2025). The result is an attention-score-free framework compatible with FlashAttention, reporting up to 1.69 times faster decoding with 80% lower KV memory usage while maintaining task performance (Yang et al., 20 Nov 2025). PureKV, designed for VLLMs, couples cache compression with Spatial-Temporal Sparse Attention. It ranks non-recent tokens by lower-layer cumulative attention weighted with value norms, retains a recent window of size $d_{\text{model}}=1024$1, and selects the top $d_{\text{model}}=1024$2 historical tokens; its ST-SpAttn module keeps only first-frame and intra-frame or previous-frame links to purify spatial noise and temporal redundancy (Jiang et al., 29 Oct 2025). The paper reports 5.0 times KV cache compression and 3.16 times prefill acceleration, with Qwen2.5-VL-7B achieving average 0.5429 at 20% budget and 0.5360 at 10% on MVBench (Jiang et al., 29 Oct 2025).

STaR-KV specializes further to GUI VLM agents. It rejects two structural assumptions—single shared saliency maps and fixed top-$d_{\text{model}}=1024$3 cutoffs—and instead calibrates token scores along three axes: online mutual information for subspace-aware spatial weighting, a temporal stability discount for redundant visual history, and Adaptive Entropy-Based Sharpening with temperature

$d_{\text{model}}=1024$4

On UI-TARS-1.5-7B, STaR-KV reports average accuracy 49.94 at 40% budget versus 49.75 for full cache, and 47.31 at 20% budget; on OpenCUA-7B it reports 50.04 at 40% and 47.01 at 20% (Han et al., 1 Jun 2026). At a 20% KV-cache budget, it cuts peak GPU memory by nearly 40% relative to full cache, with ScreenSpot-Pro memory dropping from 37.36 GB to 22.97 GB (Han et al., 1 Jun 2026).

The common implication is that modality structure changes what “important KV” means. Early image scales, high-frequency outliers, persistent GUI subspaces, and temporally local video cues are all treated as first-class retention signals.

6. Compatibility, calibration, and recurring trade-offs

A major point of divergence among KV collectors is compatibility with modern inference stacks. Several methods are explicitly designed to avoid dependence on full attention-score matrices. FlashCache is attention-score-free and therefore compatible with FlashAttention (Yang et al., 20 Nov 2025). PureKV estimates high-layer importance from lower-layer statistics so that high layers can still use FlashAttention or sparse attention (Jiang et al., 29 Oct 2025). CurDKV is also presented as compatible with FlashAttention and Grouped Query Attention because it scores keys and values directly rather than requiring explicit attention matrices (Sengupta et al., 18 Sep 2025). KVCompose prioritizes a structured dense tensor layout specifically to remain usable in vLLM and Hugging Face Transformers without custom sparse kernels (Akulov et al., 5 Sep 2025).

A second trade-off concerns whether a method is training-free, calibration-based, or training-aware. VQKV is training-free at inference time but requires offline codebook training on about 10M tokens from OpenWebText (Wang et al., 17 Mar 2026). HeatKV uses 10 calibration prompts to build a static pruning schedule and reports that using 1, 2, 5, or 10 calibration prompts yields nearly identical MS-COCO performance around FID 1.66–1.67, LPIPS 0.073, and PSNR 25.38–25.40 at 10% budget on Infinity-2B (Cederlund et al., 14 May 2026). CompilerKV pushes calibration furthest: it compiles a Head Heterogeneity Table and a Risk Threshold Table offline from about 50K long-context prompts, using a 20×4 bin grid and Conservative Q-Learning, then deploys them in a strictly prefill-only, one-shot selection regime (Yang et al., 9 Feb 2026). KV-CAR is more training-aware still, as its layer-wise autoencoders must be trained and its reuse patterns analyzed or fine-tuned (Roy et al., 7 Dec 2025).

A third trade-off is systems complexity. PackKV, CTkvr, and KVLinC achieve their reported gains partly by co-designing kernels and memory layout. PackKV relies on block-independent compression and fused decompression–matvec kernels (Jiang et al., 30 Dec 2025). CTkvr uses CPU–GPU co-execution, custom CUDA kernels, and multistream overlap (Lu et al., 17 Dec 2025). KVLinC employs a custom Triton attention kernel for quantized KV decoding (Saxena et al., 6 Oct 2025). By contrast, methods such as KVCompose, FlashCache, and CompilerKV place greater emphasis on drop-in deployment under standard pipelines (Akulov et al., 5 Sep 2025, Yang et al., 20 Nov 2025, Yang et al., 9 Feb 2026).

Finally, virtually all collectors expose a quality–efficiency frontier rather than an unqualified improvement. VQKV identifies decoding efficiency as its main limitation because vector quantization still introduces extra compute (Wang et al., 17 Mar 2026). CurDKV reports higher prefill latency—at 128K tokens, from roughly 10 s to around 14–15 s at high compression—while reducing generation time to under 5–6 s (Sengupta et al., 18 Sep 2025). ReST-KV adds about 2% more TTFT than SnapKV, but at 128k context reports ~36.0% reduction in peak memory and 10.61× decoding latency reduction (An et al., 9 May 2026). CompilerKV is explicitly prefill-only, so its one-shot retention decisions are irreversible during decoding (Yang et al., 9 Feb 2026). STaR-KV notes that its evaluation covers two 7B-scale open-source GUI VLMs and four benchmarks, marking a bounded empirical scope (Han et al., 1 Jun 2026).

Across these variants, a stable pattern emerges. The strongest recent KV collectors do not treat the cache as a uniform token stream. They model heterogeneity across heads, layers, scales, subspaces, prompts, or frequency bands, and they evaluate success by how well compressed, retrieved, or selectively retained KV states preserve the behavior of the full attention computation.