---
title: 'TRIM-KV: Retention-Driven KV Cache Eviction'
url: https://www.emergentmind.com/topics/retention-driven-kv-cache-eviction-trim-kv
type: topic
---

# TRIM-KV: Retention-Driven KV Cache Eviction

Searching arXiv for TRIM-KV and closely related KV cache eviction papers to ground the article in current literature.
Retention-driven KV cache eviction denotes a class of memory-bounded inference methods for autoregressive transformers in which the system explicitly decides which cached key-value pairs should remain resident over time, rather than relying solely on recency or on instantaneous attention to a small observation window. TRIM-KV, introduced in "Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs," is a learned formulation of this idea: each token is assigned an intrinsic retention score at creation time, that score decays over time, and eviction removes the token with the smallest current retained strength when a fixed cache budget is exceeded [2512.03324]. The method is positioned against quantization, offloading, and heuristic attention-based eviction, with the central claim that long-term utility is better modeled as token durability than as recently observed attention.

## 1. Problem formulation and conceptual shift

The practical motivation is the standard long-horizon inference bottleneck. In decoder-only large language models, the KV cache grows linearly with sequence length, while attention cost grows quadratically. Under long-context or long-generation workloads, memory and bandwidth become the dominant systems constraints, and a fixed memory budget \(M\) forces permanent eviction decisions [2512.03324].

TRIM-KV is distinguished by the point at which importance is estimated. Conventional heuristic eviction methods such as StreamingLLM, H2O, SnapKV, and related approaches typically infer importance from recency or from later attention-derived signals. TRIM-KV instead learns token importance at creation time from the token’s own contextual embedding and the layer/head in which it appears. The paper describes this as a **retention gate**, which predicts a scalar \(\beta \in [0,1]\) representing token-specific memory strength. This replaces the question “which token was recently attended?” with “how durable is this token likely to be?” [2512.03324].

The paper first expresses memory-bounded attention with binary retention indicators:
\[
o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,
\]
subject to the monotonicity condition
\[
\alpha_{ti} \ge \alpha_{t+1,i}, \quad \forall i,t,
\]
and the budgeted objective
\[
\min_{\alpha} L_{\mathrm{base}}(o_t'; o_t)
\quad \text{s.t.} \quad
\sum_{i=1}^t \alpha_{ti} \le M.
\]
This formulation makes explicit that eviction is not merely sparse attention. Sparse attention decides what to read from a still-complete cache; retention-driven eviction decides what survives in memory at all. That distinction later becomes central in streaming-oriented work such as Nexus Sampling, which treats irreversible survival under a fixed budget as the defining structural property of the problem [2606.23961].

## 2. Retention gate, temporal decay, and eviction rule

TRIM-KV assigns every token a scalar retention score when the token is created. Rather than converting that score directly into a hard keep/drop decision, it uses an exponential decay rule:
\[
\bar{\alpha}_{ti} = \beta_i^{\,t-i}.
\]
If \(\beta_i \approx 1\), the token decays slowly; if \(\beta_i\) is small, its retained strength vanishes quickly. The paper explicitly rejects a sigmoid-over-eviction-time parameterization as too flat and difficult to optimize, and adopts exponential decay as the operative retention curve [2512.03324].

The differentiable retention-gated attention rule is
\[
q_t = W_Q x_t,\quad k_t = W_K x_t,\quad v_t = W_V x_t,\quad \beta_t = g(x_t),
\]
\[
o_t = \sum_{i=1}^{t} \frac{\exp\!\left(\beta_i^{\,t-i} q_t^\top k_i\right)}{\sum_{j=1}^{t}\exp\!\left(\beta_j^{\,t-j} q_t^\top k_j\right)} v_i.
\]
The gate \(g\) is lightweight, with two forms reported:
\[
g(x)=\sigma(W_\beta x + b), \quad W_\beta \in \mathbb{R}^{1\times d},
\]
or
\[
g(x)=\sigma(\mathrm{MLP}(x)+b).
\]
If all \(\beta_t=1\), the formulation reduces to standard attention.

At inference time, TRIM-KV maintains a fixed-size cache. A new token is appended, and if the cache size exceeds \(M\), eviction is deterministic:
\[
j_{\mathrm{evic}} = \arg\min_{j \in S_t} \left\{\beta_j^{t-j} \mid j \in S_t\right\}.
\]
The token with the smallest current retention score is removed. The policy is therefore monotone and non-retrieval-based: once evicted, a token does not return [2512.03324].

This design is best understood as intrinsic-utility scoring rather than query-conditioned scoring. Related work makes the contrast explicit. Judge Q improves attention-based prefill eviction by replacing the usual last-prefill-window queries with learned soft-token queries that better approximate future decoded-token attention [2509.10798]. TRIM-KV, by contrast, does not attempt to approximate future query attention at inference time; it learns a durable scalar memory strength in advance.

## 3. Training objective and optimization regime

TRIM-KV is trained as a distillation-based plug-in over a frozen pretrained language model. During training, the attention blocks are replaced with retention-gated attention, but the backbone model weights are not updated. Only the retention gates are fine-tuned [2512.03324].

The quality objective combines forward KL distillation with next-token prediction:
\[
\mathcal{L}_{\mathrm{quality}} =
D_{\mathrm{KL}}\!\big(p(\cdot|x)\,\|\,q_\theta(\cdot|x)\big)
+
\mathbb{E}_{(x,y)}\!\left[-\log q_\theta(y|x)\right].
\]
Here \(p\) is the frozen pretrained LLM and \(q_\theta\) is the retention-gated model. Capacity is enforced by a hinge-like penalty:
\[
L_{\mathrm{cap}} =
\frac{1}{T(T-M)}
\sum_{t=1}^{T}
\max\left\{0,\sum_{i=1}^{t}\beta_i^{\,t-i}-M\right\}.
\]
The full objective is
\[
\min_{\theta}\; L_{\mathrm{quality}}+\lambda_{\mathrm{cap}}L_{\mathrm{cap}}.
\]
The paper also defines the analytical sparsity measure
\[
1-\frac{2}{T(T+1)}\sum_{i<t}\beta_i^{\,t-i}.
\]

In the main long-generation setup, training uses OpenR1-MATH-220k with \(\lambda_{\mathrm{cap}}=1.0\), memory capacity \(M=512\), a single-hidden-layer MLP gate with hidden size 512, and bias initialized to a large positive value such as \(b=8.0\), so the model begins with almost no forgetting. For long-context experiments, the training mixture is SynthLong, BookSum, and Buddhi, with maximum sequence length up to 128K and, in one configuration, \(M=4096\). Additional implementation details reported for the math setup are learning rate \(2\times 10^{-4}\), weight decay \(0.01\), batch size 1 per GPU, and gradient accumulation 4 [2512.03324].

The ablations attribute substantial importance to all three terms. Removing forward KL hurts performance; removing next-token prediction also hurts; removing the capacity loss causes a sharp drop. The paper further reports that an MLP gate outperforms a simple linear projection, that a large positive initialization bias is important for stable training, and that training memory \(M\) should be near the intended deployment budget [2512.03324].

## 4. Inference-time behavior, overhead, and reported empirical performance

At deployment, TRIM-KV computes one additional scalar retention score per token and stores it alongside the KV states. The reported storage cost is approximately \(1/d_h\) relative to the KV states, and the paper describes this as negligible in practice. Unlike R-KV, it does not store queries. For RoPE-based models, it caches post-rotated keys, making eviction orthogonal to positional encoding [2512.03324].

The method is intended to be lightweight at inference. The decoding procedure is: project the current token to \(q_t,k_t,v_t\); compute \(\beta_t=g(x_t)\); append the token to the cache; run attention over the current cache; and, if the cache exceeds \(M\), evict the cached token with minimum \(\beta_j^{t-j}\). On a single H200 GPU, the paper reports that at 32K context and batch 4, TRIM-KV reaches **130.48 tok/s** and **31.39 s** decode time, compared with **68.44 tok/s** and **59.84 s** for FullKV, and **124.67 tok/s** and **33.00 s** for SnapKV. At 16K context and batch 8, TRIM-KV reaches **279.90 tok/s**, compared with **138.97 tok/s** for FullKV and **244.60 tok/s** for SnapKV [2512.03324].

Empirically, the paper reports strong results across several evaluation regimes. On GSM8K, MATH-500, and AIME24, TRIM-KV is reported as consistently best among eviction methods and better than learnable retrieval baselines; at the same budget it yields a **198% relative improvement** over attention-guided eviction baselines such as R-KV and SnapKV on the math suite, and compared with SeerAttn-R it achieves a **58.4% pass@1 gain** at the same budget. In some settings, including Qwen3-4B on AIME24, it surpasses the full KV-cache baseline [2512.03324].

On LongProc, the method again outperforms all eviction baselines and sometimes exceeds FullKV at tight budgets. On LongMemEval, the paper states that it can match the performance of a full cache while using only **25% of the KV budget**; the reported overall accuracies are **49.4** for Full KV at 131072, **48.2** for TRIM-KV at 32768, **42.6** at 16384, and **30.2** at 4096. On LongBench in a chunked-prefill setting, the average relative change is **0.00** for Full KV, **-4.82%** for LocRet, and **-0.64%** for TRIM-KV. On LongBench-V2, the paper reports **28.79** average accuracy for Full KV and **30.68** for TRIM-KV, corresponding to **+6.56%** relative improvement [2512.03324].

These latter results motivate a recurrent claim in the paper: selective retention can behave as a form of regularization by suppressing noise from uninformative tokens, rather than functioning purely as a lossy compression mechanism.

## 5. Emergent retention structure, interpretability, and limitations

A notable feature of TRIM-KV is that the learned retention scores are analyzed not only as control signals for memory, but also as probes into head- and layer-specific function. The paper reports that the model naturally recovers several hand-designed heuristics without explicit programming: high retention for sink tokens, sliding-window-like behavior in early layers, A-shaped patterns, and gist compression through preserved punctuation, especially periods [2512.03324].

The qualitative examples are specific. In an AIME24 case study, high retention is assigned to task-relevant tokens such as “ometer,” “shop,” “walk,” and “minutes,” while whitespace and punctuation receive low retention except where punctuation appears to play a summarizing role. The paper also states that later layers are typically sparser and more specialized than earlier layers. Example heads are described as focusing on problem statements, instructions, chain-of-thought prompts, mathematical symbols, operators, numbers, or general-purpose coherence tokens. A reported hypothesis is that some heads retain period tokens as implicit gist tokens [2512.03324].

This interpretability angle should not be conflated with a guarantee of universal superiority. The paper is explicit about several limitations. The backbone LLM remains frozen during training. Inference still uses standard attention; only eviction is learned. Existing KV-cache and FlashAttention implementations assume uniform sequence lengths across heads, so fully efficient per-head variable-length caches are deferred to future work. Retrieval-heavy or otherwise incompressible contexts remain difficult for all eviction methods, including TRIM-KV. The method also uses a fixed budget rather than fully adaptive budget allocation [2512.03324].

A common misconception is therefore that retention-driven eviction simply learns a better version of recent-attention scoring. TRIM-KV does not do that. Its score is not a post hoc estimate of which tokens a recent query found salient, but a forecast of how long a token should remain useful. Another misconception is that reported gains over FullKV imply that larger caches are intrinsically harmful. The paper’s phrasing is narrower: selective forgetting can serve as regularization in some settings, not that eviction universally dominates full-cache inference.

## 6. Position within the broader retention-driven KV eviction literature

TRIM-KV sits within a rapidly expanding literature on retention-driven KV cache management, but its mechanism is distinct from several adjacent lines of work. Judge Q addresses the myopia of last-prefill-window attention by appending learnable soft tokens whose attention is trained to align with actual response-token attention; at inference it replaces the query source for attention-based scoring while leaving top-\(k\) pruning logic unchanged [2509.10798]. This suggests a learned-query alternative to heuristic prefill scoring, whereas TRIM-KV removes the need for query-conditioned eviction scores at inference altogether.

Nexus Sampling treats the problem as streaming, irreversible retention under a fixed budget. It replaces deterministic top-\(K\) survival with Nexus scoring plus weighted reservoir sampling, and at **80% KV cache eviction** it is reported to match dense attention within **1%** on LongBench while using **up to 10× smaller per-sequence cache memory** [2606.23961]. A plausible implication is that TRIM-KV and Nexus Sampling emphasize different failure modes: TRIM-KV targets intrinsic long-term utility at token creation, while Nexus Sampling targets long-run survival of subtly important tokens whose per-step scores fluctuate.

Other methods alter the retention signal rather than the survival rule. KVP frames eviction as reinforcement learning over rankings, using lightweight per-head RL agents trained on future-attention rewards computed from offline traces [2602.10238]. CapKV reframes eviction through an Information Bottleneck objective and approximates a log-determinant capacity term with leverage scores, interpreting many prior heuristics as approximations to a common capacity-maximization principle [2604.25975]. ReST-KV replaces raw attention heuristics with layer-wise output reconstruction and spatial-temporal smoothing, reporting **+2.58%** on LongBench and **15.2%** on RULER over prior baselines [2605.08840]. DefensiveKV argues that the underlying stability assumption of mean-aggregated importance is fragile, and replaces mean aggregation with worst-case-risk aggregation; under **20% cache size**, it reports generation-quality-loss reductions of **2.3×** for DefensiveKV and **4.3×** for Layer-DefensiveKV versus the strongest baseline [2510.13334].

There are also workload-specific extensions of the retention-driven viewpoint. IntentKV is designed for multi-turn agent inference, where old evidence can become relevant again and prefix-cache composability matters. It uses a session-level QueryMemory, a learned residual scorer, and slot-map redirection to a sentinel dead slot, and at an **8k KV budget** it reports almost no accuracy drop relative to full cache while sharply reducing peak request tokens and raw KV reads [2606.09916]. GraphKV, by contrast, is a graph-based refinement layer that starts from existing token scores and propagates a decay signal over a key-similarity graph to reduce redundancy in the retained set [2509.00388]. HashEvict takes a markedly different route: it is pre-attention and query-local, using locality-sensitive hashing and Hamming distance instead of attention-based or learned retention signals, and reports **30%–70% KV cache compression** with high performance across several task types [2412.16187].

Taken together, these works make clear that “retention-driven KV cache eviction” is not a single algorithmic template but a family of approaches organized around a common systems constraint: under a fixed memory budget, the cache must preserve what will matter later, not merely what was easy to score now. TRIM-KV’s specific contribution to that family is to make retention an intrinsic, learned property of tokens at birth, expressed through a decaying scalar memory strength rather than through live query-conditioned heuristics [2512.03324].

Source: https://www.emergentmind.com/topics/retention-driven-kv-cache-eviction-trim-kv