---
title: Constant-Memory KV Cache Methods
url: https://www.emergentmind.com/topics/constant-memory-kv-cache
type: topic
---

# Constant-Memory KV Cache Methods

Constant-memory KV cache refers to techniques for bounding the GPU or system memory footprint of the key-value (KV) cache used in Transformer-based language models during inference, such that memory usage remains effectively independent of sequence length or input context size. This problem is motivated by the observation that, in standard autoregressive inference, the KV cache grows linearly with the number of input and generated tokens and often dominates memory consumption, representing a severe bottleneck particularly for long-context and high-throughput inference. Recent research has produced a diversity of algorithmic, systems, and architectural approaches that tightly compress, prune, or re-architect the KV cache to provide constant or nearly constant memory usage, with minimal (or zero) accuracy trade-off even at aggressive compression rates.

## 1. Motivation and Challenges in KV Cache Compression

The KV cache is central to efficient transformer inference, enabling O(1) or O(n) complexity for sequential token generation by storing per-token key and value activations from every attention layer. However, for long sequences, accumulated KV pairs consume substantial memory—often exceeding the model weights themselves—substantially limiting achievable context length or batch size on accelerators with fixed memory budgets [2402.18096, 2412.03131, 2412.11741]. Traditional mitigation strategies—such as discarding past tokens, static quantization, or attention window sliding—introduce context loss, hallucinations, and degraded accuracy, with especially pronounced deficits in safety-critical or context-sensitive applications [2402.18096]. The core challenge is to decouple the cache size from the context length while maintaining high-fidelity context for all relevant tasks.

## 2. Mixed-Precision Quantization and Importance-Aware Retention

Several recent methodologies address the constant-memory goal using mixed-precision quantization and importance-aware token retention. MiKV [2402.18096] employs an adaptive scheme where critical KV pairs (as determined by attention-based importance scoring) are stored at high precision (e.g., FP16 or 8-bit), while unimportant pairs—typically destined for eviction—are retained at lower precision (e.g., INT4 or INT2). This approach ensures that even "evicted" tokens continue to convey partial contextual information, leading to minimal quality degradation. The quantization is performed via asymmetric per-token formulas, with careful outlier handling using per-channel balancing:
$$
\hat{x} = I(x) = \alpha \left\lfloor\frac{x - \beta}{\alpha}\right\rfloor + \beta,
\quad \alpha = \frac{\max(x) - \min(x)}{2^N-1},\;\; \beta = \min(x)
$$
This regime bounds total cache growth, enabling constant memory operation at compression ratios as low as 20–25% of the full cache size, with accuracy and contextuality nearly indistinguishable from uncompressed baselines.

LeanKV [2412.03131] further refines this by introducing Hetero-KV quantization (allocating higher precision to keys than values) and per-head adaptive sparsity, while integrating on-GPU memory management to support fine-grained mixed-precision layouts, thereby offering $2.7 \times$ to $5.7 \times$ compression at near-lossless performance.

## 3. Personalized and Adaptive Cache Budgeting

Constant-memory can also be attained via dynamic, non-uniform allocation of cache resources according to layer- or head-specific metrics. XKV [2412.05896] observes pronounced variance in the retention importance of KV pairs across network layers—a phenomenon quantified using per-layer attention score vectors and the metric
$$
R_i = \frac{\sum\,\text{Topk}(n_i, w_i)}{\sum\,w_i} \times 100\%
$$
where $n_i$ is the retained KV pair count for layer $i$, and $w_i$ its attention vector. By formulating cache allocation as a combinatorial optimization, XKV assigns personalized retention budgets per layer, solving via a greedy allocation that maximizes overall contextual retention under a global memory constraint. The result is a mean $61.6\%$ reduction in KV cache size and up to a $5.2\times$ throughput boost on long-context benchmarks.

BaKlaVa [2502.13176] generalizes this approach to head-level granularity, using a one-time attention-profile driven budget search that reallocates saved memory from low-importance heads/layers to high-importance ones, yielding up to a $70\%$ compression ratio without baseline performance loss.

CAKE [2503.12491] introduces a "cake-slicing" methodology, assigning cache budgets in proportion to each layer’s preference
$$
\mathcal{P} = \mathcal{H}^{1/\tau_1} \mathcal{V}^{1/\tau_2}
$$
where $\mathcal{H}$ and $\mathcal{V}$ are the attention entropy and variance (spatial and temporal dispersion). Guided by these metrics, CAKE adaptively slices the global memory budget and performs cascading eviction while using a novel mean-variance eviction indicator to support dynamic, context-sensitive retention.

## 4. Sparse and Approximate Representations

A distinct approach to constant-memory caching transforms the dense KV tensor into sparse or compressed forms. CSR [2412.11741] constructs a sparse representation using Matching Pursuit over a learned dictionary $D$:
$$
x \approx D \cdot r(x, D, s)
$$
with $r$ constrained to $s$ non-zero coefficients, learned using NeuralDict. Only indices and coefficients are stored, reducing memory to as little as $\sim 1$ bit/channel, approaching the theoretical compressibility limit for practical context lengths. Inference reconstructs approximate keys and values on demand, preserving accuracy at extreme compression rates.

Streaming and subsampling-based approximations, exemplified by BalanceKV [2502.07861], use discrepancy theory to subsample a balanced subset of key-value states via recursive streaming merge-and-reduce:
$$
\|z - \mathrm{Attn}(q, K, V)\|_2 \leq \epsilon \|\mathrm{softmax}(Kq/\sqrt{d})\|_2 \|V\|_F
$$
Such techniques rigorously bound the approximation error while capping memory usage sublinearly in sequence length.

## 5. Token Selection, Merging, and Residual Techniques

A spectrum of methods compresses the KV cache by contextually selecting and aggregating tokens. MorphKV [2503.00979] maintains a constant-size cache by dynamically partitioning tokens into a fixed window of recent tokens and a selection of correlated distant tokens ranked using recent attention scores:
$$
G_{i+1} = \mathrm{Top}_C(F_i) \cup \{\text{R recent tokens}\}
$$
This adaptive correlation-aware retention avoids early-token bias, achieving average savings over $50\%$ without context degradation.

ZeroMerge [2503.10714] (ZSMerge) combines head-level token importance budgeting, residual merging (token fusion by dot-product compatibility and momentum), and compensated attention scoring:
$$
\tilde{a}_t(T) = \exp(q^\top k_t / V_d + \alpha \log w_t)
$$
enabling memory reductions by $20\times$ ($5\%$ of original) at constant performance and improved throughput.

KeepKV [2504.09936] eliminates the “attention sag” of convex merging by recording merging history (electoral votes) and applying zero-perturbation merging with adaptive scaling:
$$
o_t = \frac{\sum_i p_i s_i^t v_i}{\sum_i p_i s_i^t}
$$
The merge weights are proportional to prior attention and votes, yielding exact output preservation at each step.

GraphKV [2509.00388] advances the field by recasting token retention as a graph signal-propagation problem, penalizing redundancy through iterative cosine-similarity decay:
$$
s_j^{(t)} = s_j^{(t-1)} \prod_{o_i \in N(o_j)} (1 - e_{ij})
$$
Retained tokens are both high-importance and contextually diverse, with demonstrable gains under constant-memory budgets.

## 6. Quantization and Coding Optimized for KV Cache

Quantization-specific strategies, such as NQKV [2505.16210], exploit the normal-like distribution of KV cache blocks to perform information-theoretically optimal, blockwise quantile quantization in 4 bits ("NF4"):
$$
I_K = \mathrm{quantize}_{NF4}(X_K)
$$
These methods, combined with blockwise dequantization on inference, realize $4\times$ memory reductions with sub-percent accuracy loss, allowing for $2\times$ batch size or $4\times$ longer context within the same memory constraints.

KVComp [2509.00579] extends this direction by coupling error-controlled, token-wise quantization and GPU-optimized Huffman encoding. Critical architectural co-designs—such as cache-resident decompression and fused decoding-matrix multiplication—support up to $83\%$ higher memory reduction and outperform standard attention kernels in total throughput.

## 7. Advanced System and Architectural Adaptations

Some solutions achieve a constant-memory profile not (only) by compressing the KV cache, but by systemic or architectural memory management. MIRAGE [2507.11507] dynamically repurposes preallocated GPU parameter memory (from inactive models or temporarily unused layers) for KV cache, with a layer selection protocol ensuring forced reallocation does not impede computation:
$$
T_T \times N \leq T_\mathrm{Compute}
$$
This strategy, optimized for high-bandwidth interconnects (e.g., NVIDIA GH200), yields up to $82.5\%$ reduction in latency and $86.7\%$ higher throughput in multi-tenant LLM serving.

## 8. Plug-and-Play and Orthogonal Methods

Recent innovations emphasize modularity: KVCrush [2503.00022] introduces a binary, head-behavior-based token representation and low-overhead Hamming grouping, operating in conjunction with other approaches such as mixed precision. CommonKV [2508.16134] employs SVD-based cross-layer parameter sharing to align latent representations, with adaptive budget allocation based on cache similarity, and is orthogonal to quantization and eviction strategies; their integration achieves up to $98\%$ compression with limited loss.

Other frameworks, like PQCache [2407.12820], apply product quantization for keys in a two-phase scheme (prefill and decoding), supporting approximate MIPS for token selection and overlapped CPU–GPU operation; or KVzip [2505.23416], which uses LLM-driven context reconstruction as a universal, query-agnostic importance metric for retention, with scoring performed in constant-memory chunked sweeps.

## Conclusion

Constant-memory KV cache is an active field of research at the intersection of quantization, pruning, selection, approximation, and systems engineering. No single method dominates: mixed-precision, sparse coding, adaptive selection, and architectural strategies can be co-designed or composited for diverse model deployments. Current methods demonstrate the feasibility of reducing KV cache memory usage by up to $20\times$ or more, with minimal context or accuracy loss—enabling long-context and batch-intensive LLM inference on commodity hardware, and representing a foundational advance in the tractability and scalability of transformer models for natural language and vision.

Source: https://www.emergentmind.com/topics/constant-memory-kv-cache