TRIM-KV: Retention-Driven KV Cache Eviction

• The paper introduces TRIM-KV, a learned token retention mechanism that predicts token durability for efficient KV cache eviction in transformer models.
• It leverages an intrinsic retention gate with exponential decay to prioritize long-term utility over recency, optimizing performance under strict memory constraints.
• Empirical evaluations demonstrate improvements in inference speed and task accuracy compared to conventional heuristic and attention-based eviction methods.

Searching arXiv for TRIM-KV and closely related KV cache eviction papers to ground the article in current literature. Retention-driven KV cache eviction denotes a class of memory-bounded inference methods for autoregressive transformers in which the system explicitly decides which cached key-value pairs should remain resident over time, rather than relying solely on recency or on instantaneous attention to a small observation window. TRIM-KV, introduced in "Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs," is a learned formulation of this idea: each token is assigned an intrinsic retention score at creation time, that score decays over time, and eviction removes the token with the smallest current retained strength when a fixed cache budget is exceeded (Bui et al., 3 Dec 2025). The method is positioned against quantization, offloading, and heuristic attention-based eviction, with the central claim that long-term utility is better modeled as token durability than as recently observed attention.

1. Problem formulation and conceptual shift

The practical motivation is the standard long-horizon inference bottleneck. In decoder-only LLMs, the KV cache grows linearly with sequence length, while attention cost grows quadratically. Under long-context or long-generation workloads, memory and bandwidth become the dominant systems constraints, and a fixed memory budget $M$ forces permanent eviction decisions (Bui et al., 3 Dec 2025).

TRIM-KV is distinguished by the point at which importance is estimated. Conventional heuristic eviction methods such as StreamingLLM, H2O, SnapKV, and related approaches typically infer importance from recency or from later attention-derived signals. TRIM-KV instead learns token importance at creation time from the token’s own contextual embedding and the layer/head in which it appears. The paper describes this as a retention gate, which predicts a scalar $\beta \in [0,1]$ representing token-specific memory strength. This replaces the question “which token was recently attended?” with “how durable is this token likely to be?” (Bui et al., 3 Dec 2025).

The paper first expresses memory-bounded attention with binary retention indicators: $$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$ subject to the monotonicity condition

$\alpha_{ti} \ge \alpha_{t+1,i}, \quad \forall i,t,$

and the budgeted objective

$$\min_{\alpha} L_{\mathrm{base}}(o_t'; o_t) \quad \text{s.t.} \quad \sum_{i=1}^t \alpha_{ti} \le M.$$

This formulation makes explicit that eviction is not merely sparse attention. Sparse attention decides what to read from a still-complete cache; retention-driven eviction decides what survives in memory at all. That distinction later becomes central in streaming-oriented work such as Nexus Sampling, which treats irreversible survival under a fixed budget as the defining structural property of the problem (Duong et al., 22 Jun 2026).

2. Retention gate, temporal decay, and eviction rule

TRIM-KV assigns every token a scalar retention score when the token is created. Rather than converting that score directly into a hard keep/drop decision, it uses an exponential decay rule: $\bar{\alpha}_{ti} = \beta_i^{\,t-i}.$ If $\beta_i \approx 1$, the token decays slowly; if $\beta_i$ is small, its retained strength vanishes quickly. The paper explicitly rejects a sigmoid-over-eviction-time parameterization as too flat and difficult to optimize, and adopts exponential decay as the operative retention curve (Bui et al., 3 Dec 2025).

The differentiable retention-gated attention rule is

$$q_t = W_Q x_t,\quad k_t = W_K x_t,\quad v_t = W_V x_t,\quad \beta_t = g(x_t),$$

$$o_t = \sum_{i=1}^{t} \frac{\exp\!\left(\beta_i^{\,t-i} q_t^\top k_i\right)}{\sum_{j=1}^{t}\exp\!\left(\beta_j^{\,t-j} q_t^\top k_j\right)} v_i.$$

The gate $\beta \in [0,1]$0 is lightweight, with two forms reported: $\beta \in [0,1]$1 or

$\beta \in [0,1]$2

If all $\beta \in [0,1]$3, the formulation reduces to standard attention.

At inference time, TRIM-KV maintains a fixed-size cache. A new token is appended, and if the cache size exceeds $\beta \in [0,1]$4, eviction is deterministic: $\beta \in [0,1]$5 The token with the smallest current retention score is removed. The policy is therefore monotone and non-retrieval-based: once evicted, a token does not return (Bui et al., 3 Dec 2025).

This design is best understood as intrinsic-utility scoring rather than query-conditioned scoring. Related work makes the contrast explicit. Judge Q improves attention-based prefill eviction by replacing the usual last-prefill-window queries with learned soft-token queries that better approximate future decoded-token attention (Liu et al., 13 Sep 2025). TRIM-KV, by contrast, does not attempt to approximate future query attention at inference time; it learns a durable scalar memory strength in advance.

3. Training objective and optimization regime

TRIM-KV is trained as a distillation-based plug-in over a frozen pretrained LLM. During training, the attention blocks are replaced with retention-gated attention, but the backbone model weights are not updated. Only the retention gates are fine-tuned (Bui et al., 3 Dec 2025).

The quality objective combines forward KL distillation with next-token prediction: $\beta \in [0,1]$6 Here $\beta \in [0,1]$7 is the frozen pretrained LLM and $\beta \in [0,1]$8 is the retention-gated model. Capacity is enforced by a hinge-like penalty: $\beta \in [0,1]$9 The full objective is

$$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$0

The paper also defines the analytical sparsity measure

$$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$1

In the main long-generation setup, training uses OpenR1-MATH-220k with $$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$2, memory capacity $$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$3, a single-hidden-layer MLP gate with hidden size 512, and bias initialized to a large positive value such as $$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$4, so the model begins with almost no forgetting. For long-context experiments, the training mixture is SynthLong, BookSum, and Buddhi, with maximum sequence length up to 128K and, in one configuration, $$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$5. Additional implementation details reported for the math setup are learning rate $$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$6, weight decay $$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$7, batch size 1 per GPU, and gradient accumulation 4 (Bui et al., 3 Dec 2025).

The ablations attribute substantial importance to all three terms. Removing forward KL hurts performance; removing next-token prediction also hurts; removing the capacity loss causes a sharp drop. The paper further reports that an MLP gate outperforms a simple linear projection, that a large positive initialization bias is important for stable training, and that training memory $$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$8 should be near the intended deployment budget (Bui et al., 3 Dec 2025).

4. Inference-time behavior, overhead, and reported empirical performance

At deployment, TRIM-KV computes one additional scalar retention score per token and stores it alongside the KV states. The reported storage cost is approximately $$o_t' = \sum_{i=1}^{t} \frac{\exp\!\left(\alpha_{ti} q_t^\top k_i\right)}{\sum_{j=1}^{t} \exp\!\left(\alpha_{tj} q_t^\top k_j\right)} v_i,$$9 relative to the KV states, and the paper describes this as negligible in practice. Unlike R-KV, it does not store queries. For RoPE-based models, it caches post-rotated keys, making eviction orthogonal to positional encoding (Bui et al., 3 Dec 2025).

The method is intended to be lightweight at inference. The decoding procedure is: project the current token to $\alpha_{ti} \ge \alpha_{t+1,i}, \quad \forall i,t,$0; compute $\alpha_{ti} \ge \alpha_{t+1,i}, \quad \forall i,t,$1; append the token to the cache; run attention over the current cache; and, if the cache exceeds $\alpha_{ti} \ge \alpha_{t+1,i}, \quad \forall i,t,$2, evict the cached token with minimum $\alpha_{ti} \ge \alpha_{t+1,i}, \quad \forall i,t,$3. On a single H200 GPU, the paper reports that at 32K context and batch 4, TRIM-KV reaches 130.48 tok/s and 31.39 s decode time, compared with 68.44 tok/s and 59.84 s for FullKV, and 124.67 tok/s and 33.00 s for SnapKV. At 16K context and batch 8, TRIM-KV reaches 279.90 tok/s, compared with 138.97 tok/s for FullKV and 244.60 tok/s for SnapKV (Bui et al., 3 Dec 2025).

Empirically, the paper reports strong results across several evaluation regimes. On GSM8K, MATH-500, and AIME24, TRIM-KV is reported as consistently best among eviction methods and better than learnable retrieval baselines; at the same budget it yields a 198% relative improvement over attention-guided eviction baselines such as R-KV and SnapKV on the math suite, and compared with SeerAttn-R it achieves a 58.4% pass@1 gain at the same budget. In some settings, including Qwen3-4B on AIME24, it surpasses the full KV-cache baseline (Bui et al., 3 Dec 2025).

On LongProc, the method again outperforms all eviction baselines and sometimes exceeds FullKV at tight budgets. On LongMemEval, the paper states that it can match the performance of a full cache while using only 25% of the KV budget; the reported overall accuracies are 49.4 for Full KV at 131072, 48.2 for TRIM-KV at 32768, 42.6 at 16384, and 30.2 at 4096. On LongBench in a chunked-prefill setting, the average relative change is 0.00 for Full KV, -4.82% for LocRet, and -0.64% for TRIM-KV. On LongBench-V2, the paper reports 28.79 average accuracy for Full KV and 30.68 for TRIM-KV, corresponding to +6.56% relative improvement (Bui et al., 3 Dec 2025).

These latter results motivate a recurrent claim in the paper: selective retention can behave as a form of regularization by suppressing noise from uninformative tokens, rather than functioning purely as a lossy compression mechanism.

5. Emergent retention structure, interpretability, and limitations

A notable feature of TRIM-KV is that the learned retention scores are analyzed not only as control signals for memory, but also as probes into head- and layer-specific function. The paper reports that the model naturally recovers several hand-designed heuristics without explicit programming: high retention for sink tokens, sliding-window-like behavior in early layers, A-shaped patterns, and gist compression through preserved punctuation, especially periods (Bui et al., 3 Dec 2025).

The qualitative examples are specific. In an AIME24 case study, high retention is assigned to task-relevant tokens such as “ometer,” “shop,” “walk,” and “minutes,” while whitespace and punctuation receive low retention except where punctuation appears to play a summarizing role. The paper also states that later layers are typically sparser and more specialized than earlier layers. Example heads are described as focusing on problem statements, instructions, chain-of-thought prompts, mathematical symbols, operators, numbers, or general-purpose coherence tokens. A reported hypothesis is that some heads retain period tokens as implicit gist tokens (Bui et al., 3 Dec 2025).

This interpretability angle should not be conflated with a guarantee of universal superiority. The paper is explicit about several limitations. The backbone LLM remains frozen during training. Inference still uses standard attention; only eviction is learned. Existing KV-cache and FlashAttention implementations assume uniform sequence lengths across heads, so fully efficient per-head variable-length caches are deferred to future work. Retrieval-heavy or otherwise incompressible contexts remain difficult for all eviction methods, including TRIM-KV. The method also uses a fixed budget rather than fully adaptive budget allocation (Bui et al., 3 Dec 2025).

A common misconception is therefore that retention-driven eviction simply learns a better version of recent-attention scoring. TRIM-KV does not do that. Its score is not a post hoc estimate of which tokens a recent query found salient, but a forecast of how long a token should remain useful. Another misconception is that reported gains over FullKV imply that larger caches are intrinsically harmful. The paper’s phrasing is narrower: selective forgetting can serve as regularization in some settings, not that eviction universally dominates full-cache inference.

6. Position within the broader retention-driven KV eviction literature

TRIM-KV sits within a rapidly expanding literature on retention-driven KV cache management, but its mechanism is distinct from several adjacent lines of work. Judge Q addresses the myopia of last-prefill-window attention by appending learnable soft tokens whose attention is trained to align with actual response-token attention; at inference it replaces the query source for attention-based scoring while leaving top-$\alpha_{ti} \ge \alpha_{t+1,i}, \quad \forall i,t,$4 pruning logic unchanged (Liu et al., 13 Sep 2025). This suggests a learned-query alternative to heuristic prefill scoring, whereas TRIM-KV removes the need for query-conditioned eviction scores at inference altogether.

Nexus Sampling treats the problem as streaming, irreversible retention under a fixed budget. It replaces deterministic top-$\alpha_{ti} \ge \alpha_{t+1,i}, \quad \forall i,t,$5 survival with Nexus scoring plus weighted reservoir sampling, and at 80% KV cache eviction it is reported to match dense attention within 1% on LongBench while using up to 10× smaller per-sequence cache memory (Duong et al., 22 Jun 2026). A plausible implication is that TRIM-KV and Nexus Sampling emphasize different failure modes: TRIM-KV targets intrinsic long-term utility at token creation, while Nexus Sampling targets long-run survival of subtly important tokens whose per-step scores fluctuate.

Other methods alter the retention signal rather than the survival rule. KVP frames eviction as reinforcement learning over rankings, using lightweight per-head RL agents trained on future-attention rewards computed from offline traces (Moschella et al., 10 Feb 2026). CapKV reframes eviction through an Information Bottleneck objective and approximates a log-determinant capacity term with leverage scores, interpreting many prior heuristics as approximations to a common capacity-maximization principle (Yang et al., 28 Apr 2026). ReST-KV replaces raw attention heuristics with layer-wise output reconstruction and spatial-temporal smoothing, reporting +2.58% on LongBench and 15.2% on RULER over prior baselines (An et al., 9 May 2026). DefensiveKV argues that the underlying stability assumption of mean-aggregated importance is fragile, and replaces mean aggregation with worst-case-risk aggregation; under 20% cache size, it reports generation-quality-loss reductions of 2.3× for DefensiveKV and 4.3× for Layer-DefensiveKV versus the strongest baseline (Feng et al., 15 Oct 2025).

There are also workload-specific extensions of the retention-driven viewpoint. IntentKV is designed for multi-turn agent inference, where old evidence can become relevant again and prefix-cache composability matters. It uses a session-level QueryMemory, a learned residual scorer, and slot-map redirection to a sentinel dead slot, and at an 8k KV budget it reports almost no accuracy drop relative to full cache while sharply reducing peak request tokens and raw KV reads (Li et al., 6 Jun 2026). GraphKV, by contrast, is a graph-based refinement layer that starts from existing token scores and propagates a decay signal over a key-similarity graph to reduce redundancy in the retained set (Li et al., 30 Aug 2025). HashEvict takes a markedly different route: it is pre-attention and query-local, using locality-sensitive hashing and Hamming distance instead of attention-based or learned retention signals, and reports 30%–70% KV cache compression with high performance across several task types (Liu et al., 2024).

Taken together, these works make clear that “retention-driven KV cache eviction” is not a single algorithmic template but a family of approaches organized around a common systems constraint: under a fixed memory budget, the cache must preserve what will matter later, not merely what was easy to score now. TRIM-KV’s specific contribution to that family is to make retention an intrinsic, learned property of tokens at birth, expressed through a decaying scalar memory strength rather than through live query-conditioned heuristics (Bui et al., 3 Dec 2025).