Compressed KV-Cache Distillation

• Compressed KV-cache distillation is a set of strategies that reduce LLM memory usage by compressing key–value caches through quantization, low-rank projection, and learned distillation.
• It employs adaptive precision allocation and redundancy-aware methods to preserve downstream performance while achieving up to 90% memory reduction.
• These techniques address memory and throughput bottlenecks in long-context generation and high-demand scenarios, enabling scalable and cost-effective AI inference.

Compressed KV-cache distillation refers to the suite of algorithmic and representational strategies for reducing the memory footprint of the key–value (KV) cache in LLMs, while maintaining minimal impact on downstream performance. The KV cache holds the keys and values generated by the model for past tokens at each layer, supporting efficient autoregressive inference but scaling linearly with sequence length and model depth. This creates severe memory and computational bottlenecks in long-context generation, resource-constrained deployment, and high-throughput scenarios. Recent works have introduced a range of quantization, redundancy reduction, low-rank projection, and learned distillation schemes that enable aggressive KV cache compression—facilitating both latency and throughput improvements in LLM serving pipelines.

1. Quantization-based Compressed KV-Cache

Early and state-of-the-art approaches exploit adaptive quantization, predicated on the observation that the key and value caches exhibit different sensitivity to quantization noise. In particular, the key cache highly influences the softmax attention mechanism—errors here tend to be amplified, whereas value cache errors propagate more linearly. “QAQ: Quality Adaptive Quantization for LLM KV Cache” (Dong et al., 7 Mar 2024) formalizes this with layer-wise derivative analyses and derives distinct upper bounds for quantization error in keys and values:

• Key cache quantization constraints, e.g.,

$$2\{\mathrm{K}\}_t \leq \frac{1}{\sum_{d=1}^D Q_d^2} \log\left(\frac{T^3}{T-1} 2\{\mathrm{S}\}_{\max} + 1\right)$$

• Value cache quantization constraints, e.g.,

$\{\mathrm{V}\}_t \leq \frac{1}{\sqrt{T}S_t}$

In combination with outlier-aware quantization—storing extreme values in sparse full-precision format—QAQ achieves up to a 10$\times$ compression ratio with negligible performance degradation, outperforming baseline eviction and quantization methods by up to 1.8$\times$ in lossless settings. An “attention window” mechanism also ensures tokens whose attention scores spike late in context are not prematurely over-compressed.

Additionally, matrix decomposition methods such as DecoQuant (Liu et al., 21 May 2024) employ tensor (e.g., Matrix Product Operator) decompositions to relocate outliers into small local tensors. Low-bit quantization is then applied to the bulk, outlier-depleted tensor, with outlier tensors left at higher precision, achieving up to 75% memory reduction with minimal quality loss. Efficient dequantization kernels can fuse the quantization reversal directly into inner attention operations, minimizing computation and data movement.

Numerous contemporary works reinforce the imperatives of precision allocation. For example, LeanKV (Zhang et al., 4 Dec 2024) adopts “Hetero-KV” quantization—allocating higher bitwidth to keys (e.g., 8 bits) and lower to values (e.g., 4 bits)—supported by theoretical analysis and in contrast to uniform quantization. This leverages the fact that keys, via the softmax normalization and denominator effect, are more critical than values. Spectral analysis further validates that key matrices possess systematically higher singular values (spectral and Frobenius norms) than value matrices, requiring more gentle quantization ("Quantize What Counts" (Hariri et al., 20 Feb 2025)).

Method	Key innovation	Typical Ratio	Accuracy loss
QAQ	Sensitivity-aware quantization	$\sim$10x	negligible
DecoQuant	MPO-decomposed quantization	4x–8x	negligible
LeanKV	Hetero-KV bit allocation	2.7–5.7x	near-lossless
Spectral-KV	Spectral gap-guided bit alloc.	2x–4x	negligible

2. Low-Rank, Depthwise, and Chunkwise Decomposition

In addition to quantization, several works target the structure of the KV cache to minimize redundancy, especially in the channel (hidden dimension), depth (layer), and semantic axes.

Low-rank projection strategies compress the hidden feature dimension of key/value matrices using learned or post-hoc projections:

• Palu (Chang et al., 30 Jul 2024) introduces group-head low-rank decomposition (G-LRD) with Fisher information based rank allocation. Submatrices spanning multiple heads are jointly decomposed, and quantization is applied to the latent compressed space, attaining $>$91% memory savings when combined.
• MatryoshkaKV (Lin et al., 16 Oct 2024) shows simple PCA projections degrade LLM output when r decreases, due to nonlinearity in attention; it therefore proposes knowledge-distilled, trainable orthogonal projections, trained under a “Matryoshka” (nesting) strategy that randomizes r at each step, enforcing column-wise importance. Adaptively allocating per-layer/per-head rank, it achieves over 90% performance (vs. uncompressed) at 60% compression.
• CLLA (Yang et al., 20 Oct 2024) moves further by learning low-dim latent vectors ($C$-dimensional, $C \ll ND_h$) shared across layers. Each layer recovers large-dimension K and V via projection, and quantization is performed on the latent. CLLA-quant compresses to 2.1% of the original KV cache size on benchmarks with no quality loss.

Across-layer (depth dimension) redundancy is also exploited. MiniCache (Liu et al., 23 May 2024) observes middle-to-deep layer KV caches are similar for adjacent layers and merges them using spherical linear interpolation (SLERP) in the direction space. Distinct activations are unmerged via a retention threshold. Experiments report up to 5× compression and 5× throughput improvement.

Chunk-wise semantic compression extends the unit of compression from tokens to groups of contiguous tokens (“chunks”), as in ChunkKV (Liu et al., 1 Feb 2025). By scoring and pruning chunks based on aggregated attention, semantic units are retained, preserving context coherence under aggressive compression. Layer-wise index reuse allows chunk selection indices to be shared across layers, reducing computational overhead and yielding up to 26.5% throughput gains.

3. Redundancy and Importance-Aware Compression

Recent approaches stress the importance of considering both the importance (influence on generation) and redundancy (semantic similarity) of context tokens. R-KV (Cai et al., 30 May 2025) introduces a dual-score selection: an attention-based importance metric and a semantic redundancy metric (computed as softmax-normalized cosine similarity among key vectors). Joint consideration enables the method to compress to as little as 10% of the original KV cache with nearly perfect accuracy on chain-of-thought reasoning, far outperforming token-level or attention-only baselines. Notably, the method maintains $>$100% of full-cache performance (i.e., slight improvements attributed to de-noising of redundant traces) in some settings.

KaVa (Kuzina et al., 2 Oct 2025) extends this notion for latent reasoning. The teacher’s explicit reasoning trace is compressed using a redundancy–importance-aware eviction module; the resulting abstracted KV cache supervises a continuous latent reasoning student who aligns its internal KV trajectory directly to the compressed teacher cache using a matching loss:

$$\mathcal{L}_\mathrm{KV} = \frac{1}{2M}[ \| \text{sg}(\tilde{K}_t) - K_s \|_p^p + \| \text{sg}(\tilde{V}_t) - V_s \|_p^p ]$$

where sg$(\cdot)$ is the stop-gradient operator. This compressed KV-cache supervision bridges the performance–efficiency gap between chain-of-thought and latent models.

4. Dynamic, Adaptive, and Lossless Compression Policies

Leading research indicates the optimal compression ratio should not be static but instead task-, sequence-, and input-adaptive.

• DBudgetKV (Ni et al., 24 Feb 2025) introduces a per-input, attention-norm-based stopping rule. Tokens are greedily pruned by decreasing importance until the cumulative Frobenius norm of attention drops by more than 1% (relative to the full cache). This dynamic allocation enables up to 85% compression on simple tasks and lower compression on harder cases (e.g., math), always targeting “full-cache performance.”
• KeepKV (Tian et al., 14 Apr 2025) develops a merging approach that maintains zero perturbation in the output attention. Via its Electoral Votes mechanism—the vote count $p_i$ records the merged history of every KV pair—softmax attention is scaled so that output distribution remains consistent before and after merging, theoretically eliminating accuracy loss. ZIP-Merging further ensures merged vector scaling preserves previous stepwise influence.
• Batch-Max (Metel et al., 7 Dec 2024) addresses the practical context in which prefilling and decoding phases utilize different cache sizes. By compressing KV cache during both phases using an average attention–eviction rule, substantially higher throughput is achieved at large batch sizes, with accuracy within 2.2% of full KV even in aggressive regimes.

Several methods, e.g., AQUA-KV (Shutova et al., 31 Jan 2025), use one-shot calibration to exploit inter-layer predictability. Here, linear regressors predict the next layer’s KV vectors, only quantizing the unpredictable (residual) component, enabling near-lossless performance at 2–2.5 bits/value.

5. System-level and Streaming Implementations

Producing significant computational and memory savings without introducing throughput bottlenecks requires coordinated system/hardware co-design.

• KVComp (Jiang et al., 30 Aug 2025) combines fine-grained quantization (blockwise for keys; tokenwise for values) with GPU-resident Huffman entropy encoding. During inference, decompression is fused with matrix–vector multiplication kernels, avoiding global memory transfers and exceeding 400 GB/s throughput for keys. The tiling and fusion strategy ensures hardware efficiency and scalability for long-context and batch scenarios.
• FastKV (Jo et al., 3 Feb 2025) separates early-layer full-context propagation from token-selective later-layer continuation, improving time-to-first-token by 1.97× and throughput by 4.82× versus baseline, with sub-1% accuracy loss.
• Streaming and online frameworks (e.g., InfiniPot-V (Kim et al., 18 Jun 2025), BalanceKV (Han et al., 11 Feb 2025)) introduce continual, input-length-independent memory budgets for streaming data (e.g., video). InfiniPot-V uses temporal redundancy and value-norm ranking to select representative tokens within a strict memory cap, sustaining real-time video understanding with $>$90% memory reduction and accuracy matching full-cache models. BalanceKV uses discrepancy theory and vector balancing to select token subsets that provably $\epsilon$–approximate softmax attention under sublinear space, suitable for arbitrary streaming contexts.

6. Trainable and Distillation-based Compression

Empirically, learned compression via knowledge distillation or context selection yields superior worst-case performance. KV-Distill (Chari et al., 13 Mar 2025) formulates compression as a student–teacher problem, using a KL-type divergence between next-token distributions predicted by compressed and uncompressed caches. Both forward and reverse KL divergences are combined to balance mean- and mode-seeking behavior:

$$L(\theta) = \lambda \cdot D(p || q_\theta) + (1 - \lambda) \cdot D(q_\theta || p)$$

with $\lambda > 0.5$ to favor mean-matching. Token selection is performed via importance scores, retaining the top-$k$ tokens using a non-differentiable selection matrix, and gradients are back-propagated using attention decay based on estimated importance. This approach achieves near-lossless compression—maintaining up to 99% of performance with over 99% length reduction in summarization, extractive QA, and multi-document comprehension tasks—while being adaptable to various model sizes and families.

7. Impact, Limitations, and Future Directions

Compressed KV-cache distillation underpins the ability of LLMs to handle long contexts in memory-bounded environments (edge devices, cloud batch serving, streaming video), and to efficiently scale throughput in high-demand settings. Strong empirical evidence suggests that combined strategies—leveraging quantization precision, redundancy-aware selection, dynamic budget allocation, and hardware-optimized kernels—enable aggressive compression (often exceeding 90%) with minimal (<1%) or even zero accuracy loss on challenging downstream tasks.

Nevertheless, there remain open challenges:

• Theoretical guarantees for losslessness are only partial, with most methods relying on empirical validation.
• Hardware-specific optimizations may not generalize across all accelerators.
• For domain adaptation and extremely long context lengths (hundreds of thousands of tokens), the scalability of some methods awaits further in-depth validation.
• Further integration with global model compression, activation quantization, and multi-modal contexts (e.g., vision or audio in streaming settings) remains an active area of research.

Compressed KV-cache distillation thus encompasses a growing set of algorithmic, representational, and system-level innovations designed to remove the memory bottleneck for long-context and high-throughput LLM inference, setting the foundation for more scalable, cost-effective, and versatile AI systems.