---
title: 'KeepKV: Adaptive Key-Value Cache Merging'
url: https://www.emergentmind.com/topics/keepkv
type: topic
---

# KeepKV: Adaptive Key-Value Cache Merging

KeepKV is an adaptive key-value (KV) cache merging method for efficient large language model inference that is designed to eliminate output perturbation while preserving performance under strict memory constraints. It was introduced in response to two persistent difficulties in long-context decoding: irreversible information loss in eviction-based KV compression, and attention inconsistency in earlier merging-based methods. Its central mechanisms are Electoral Votes, which record merging history and adjust attention scores accordingly, and Zero Inference-Perturbation Merging, which constructs merged KV entries so that the current-step attention output is preserved exactly [2504.09936].

## 1. KV cache compression setting and the motivation for KeepKV

In a standard transformer decoder, the query, key, and value for token \(x_t\) are
\[
q_t = x_t W_q,\quad k_t = x_t W_k,\quad v_t = x_t W_v,
\]
with \(W_q,W_k,W_v\in\mathbb{R}^{d\times d}\). During prefilling for a prompt of length \(L\),
\[
K_L = X_L W_k = [k_1,\dots,k_L],\quad V_L = X_L W_v = [v_1,\dots,v_L],
\]
and during decoding the cache grows by concatenation,
\[
K_t = [K_{t-1}, k_t],\quad V_t = [V_{t-1}, v_t].
\]
At step \(t\), the attention output is
\[
A^t = \mathrm{softmax}\!\left(\frac{q_t K_t^\top}{\sqrt{d}}\right),\quad
o_t = \sum_{i=1}^t A_i^t v_i
= \frac{\sum_{i=1}^t s_i^t v_i}{\sum_{i=1}^t s_i^t},
\]
where \(s_i^t = \exp\!\left(\frac{q_t k_i}{\sqrt{d}}\right)\). Because the cache stores keys and values for all prior tokens across all layers and heads, its memory grows with the number of layers, heads, hidden dimension, sequence length, and batch size. A concrete example given for LLaMA-3-70B, batch size 128, and context 8K is roughly \(320\) GB of KV memory, making KV compression a necessary component of efficient inference [2504.09936].

KeepKV is situated against two families of prior methods. Eviction-based schemes reduce the number of cached tokens using attention scores, position heuristics, or layer/head budgets, but once a KV entry is removed its information is gone. Merging-based schemes attempt to preserve more information by folding discarded KV pairs into retained ones, yet the paper argues that existing merging rules alter attention distributions and therefore perturb model outputs. This framing is the foundation of KeepKV: it is not simply a token selection rule, but a merging framework whose stated design goal is to compress the cache while avoiding merge-induced output shifts [2504.09936].

## 2. Eviction loss, convex merging, and the attention-sag problem

The paper formalizes the failure mode of plain eviction by removing one KV pair \((k_e,v_e)\). The resulting output becomes
\[
o'_t = \frac{\sum_{i\neq e} s_i^t v_i}{\sum_{i\neq e} s_i^t}
      = \frac{1}{1-A_e^t}\bigl(o_t - A_e^t v_e\bigr),
\]
so the perturbation is directly governed by the evicted token’s attention mass \(A_e^t\). This makes clear why small individual removals can still accumulate into severe degradation, especially when a token that was previously unimportant becomes salient later. The paper connects this to information loss, degraded quality, hallucinations, and inconsistency in long-context tasks [2504.09936].

A common misconception is that merging automatically avoids this problem. KeepKV argues that prior convex merging does not preserve attention correctly. If an evicted pair \((k_e,v_e)\) is merged into a carrier \((k_c,v_c)\) via
\[
k_r = w_e k_e + w_c k_c,\quad v_r = w_e v_e + w_c v_c,\quad w_e+w_c=1,
\]
then the merged attention score \(A_r'^t\) is strictly less than the sum of the original attention scores:
\[
A_r'^t < A_e^t + A_c^t.
\]
This is the paper’s “Attention Sag theorem.” The intuition is that several distinct tokens, each with its own score, are collapsed into one entry with a single score, and the exponential-softmax normalization prevents the merged token from inheriting the full original mass. The resulting redistribution raises the relative weight of unmerged entries and yields nonzero output perturbation for almost all realistic configurations. KeepKV therefore treats attention consistency, rather than mere retention of averaged content, as the central requirement for KV merging [2504.09936].

## 3. Electoral Votes and attention with merge history

KeepKV introduces an integer vote count \(p_i\) for each KV entry. Initially every token has \(p_i=1\). When several entries are merged, their votes are summed, so the merged entry represents multiple original tokens. Attention is then redefined as
\[
o_t = \frac{\sum_{i=1}^{t} p_i s_i^t v_i}{\sum_{i=1}^{t} p_i s_i^t}.
\]
Under this formulation, a merged entry with vote count \(p_r\) behaves as if \(p_r\) identical copies of that KV pair were present in the cache. The mechanism is intended to record merge history explicitly rather than discarding it [2504.09936].

For a merged group \(K'_e\), zero perturbation at the current step requires preservation of both the numerator contribution and the denominator mass:
\[
\sum_{i\in K'_e} p_i s_i^t v_i = p_r s_r^t v_r,\quad
\sum_{i\in K'_e} p_i s_i^t = p_r s_r^t.
\]
These constraints show why Electoral Votes are necessary. Without the multiplicative factor \(p_r\), one merged token cannot generally match the total attention mass previously distributed over multiple entries. With votes, the merged token can be made equivalent to several identical entries in the attention computation, which enables the subsequent zero-perturbation construction. This mechanism distinguishes KeepKV from earlier merging schemes that treated the merged token as an ordinary single entry [2504.09936].

## 4. Zero Inference-Perturbation Merging

The Zero Inference-Perturbation Merging rule, abbreviated ZIP, chooses the merged key, value, and vote count so that the current-step output is unchanged. Let \(K'_e\) denote the set of indices being merged and define
\[
w_i = p_i s_i^t,\quad W = \sum_{i\in K'_e} w_i.
\]
KeepKV sets
\[
v_r = \frac{\sum_{i\in K'_e} p_i s_i^t v_i}{\sum_{i\in K'_e} p_i s_i^t},
\qquad
p_r = \sum_{i\in K'_e} p_i.
\]
The merged key is chosen so that the new attention score satisfies the denominator constraint, yielding
\[
k_r = \frac{\left(\sum_{i\in K'_e} p_i s_i^t k_i\right)
\ln\left(\frac{\sum_{i\in K'_e} p_i s_i^t}{\sum_{i\in K'_e} p_i}\right)}
{\sum_{i\in K'_e} p_i s_i^t \ln s_i^t}.
\]
For the two-entry case \(K'_e=\{e,c\}\), these expressions reduce to the corresponding weighted-average form with \(w_e = p_e s_e^t\) and \(w_c = p_c s_c^t\) [2504.09936].

The paper’s zero inference-perturbation theorem states that with Electoral Votes and ZIP, the merged attention output at the current step satisfies
\[
\|o'_t - o_t\| = 0.
\]
The significance is precise: KeepKV does not merely claim better empirical fidelity, but exact equality of the attention output at the merge step. This should not be conflated with a guarantee over all future steps. The exact result is local to the step at which ZIP is applied; the multi-step case is handled separately through attention prediction and an explicit perturbation bound [2504.09936].

## 5. Adaptive operation during decoding and bounded future perturbation

KeepKV is designed to operate on top of a base cache budgeting policy rather than replacing token selection outright. In the workflow described in the paper, a policy such as PyramidInfer first proposes which tokens would be evicted under a target budget. For each evicted token, KeepKV searches among retained entries for a merge target with the highest cosine similarity,
\[
\cos\theta(k_e,k_c)=\frac{\langle k_e,k_c\rangle}{\|k_e\|\|k_c\|},
\]
and applies ZIP only if the similarity exceeds a threshold \(T=0.8\). This makes the method adaptive: it merges only when there is a sufficiently compatible retained carrier, and otherwise falls back to eviction-like behavior for that token. The method is model-agnostic at the KV tensor level and does not require retraining or fine-tuning [2504.09936].

Because ZIP depends on attention scores at the merge step, KeepKV extends the construction to future decoding using an exponential moving average predictor. For token \(i\), the estimated score is
\[
\hat{s}_i^t = \frac{S_i^t}{1-\alpha^t},
\]
with
\[
S_i^t =
\begin{cases}
\sum_{k=t-w}^{t}(1-\alpha)\alpha^{t-k}s_i^k, & t=L,\\[4pt]
\alpha S_i^{t-1} + (1-\alpha)s_i^t, & t>L.
\end{cases}
\]
Replacing \(s_i^{t'}\) by \(\hat{s}_i^{t'}\) in the ZIP formulas yields the predictive merging rule used beyond the immediate step. If the relative prediction error satisfies
\[
\left|1-\frac{\hat{s}_i^{t'}}{s_i^{t'}}\right|\le \epsilon,\quad \epsilon<1,
\]
and if value differences inside the merged set are bounded by \(\gamma\), then the future output perturbation \(\Theta_{t'}=\|o_{t'}-o'_{t'}\|\) is bounded by
\[
\Theta_{t'} < \frac{2\epsilon(1+\epsilon)\gamma}{(1-\epsilon)^2}.
\]
The paper additionally states that if either \(\epsilon=0\) or the merged KV pairs are identical, then \(\Theta_{t'}=0\). This establishes an explicit distinction between exact current-step preservation and bounded multi-step degradation [2504.09936].

## 6. Empirical results, engineering properties, and position within KV-cache research

KeepKV is evaluated on OPT, LLaMA-2, LLaMA-3, and Mistral, with question answering tasks such as COPA, MathQA, and OpenBookQA, summarization tasks such as XSUM and CNN/DailyMail, and long-context settings including LongBench. Using cache allocation from PyramidInfer and hyperparameters including merge threshold \(T=0.8\) and \(\beta=1.2\), the paper reports that KeepKV consistently outperforms eviction-based baselines such as StreamingLLM, H2O, and PyramidInfer, as well as prior merging methods such as CaM and D2O. At a 5% budget, its performance is reported as closest to the full-cache baseline, and the abstract states that it enhances inference throughput by more than \(2\times\) while keeping superior generation quality even with 10% KV cache budgets [2504.09936].

Within the broader KV-compression literature, KeepKV occupies the branch centered on merge consistency rather than purely on token scoring or quantization. Related work has emphasized other axes of the problem: KVCompose uses attention-guided, layer-adaptive composite tokens and a global allocation mechanism while remaining compatible with standard inference pipelines [2509.05165]; CurDKV selects keys and values using leverage scores derived from approximated CUR decomposition and argues that attention-score approximation does not guarantee output preservation [2509.15038]; CompressKV identifies Semantic Retrieval Heads and adds a layer-adaptive KV cache allocation strategy for GQA-based models [2508.02401]; LeanKV unifies heterogeneous quantization and pruning and reports \(2.7\times\) to \(5.7\times\) KV compression with near-lossless accuracy [2412.03131]; MiKV retains evicted KV pairs in low precision while keeping important KV pairs in high precision [2402.18096]. This pattern suggests that KeepKV’s distinctive contribution is not simply compression ratio, but a formal treatment of attention consistency under merging: it targets the elimination of merge-induced output perturbation at the step of compression and a bounded error profile thereafter.

Source: https://www.emergentmind.com/topics/keepkv