Learnable Token Eviction

• Learnable token eviction is a method that replaces fixed heuristics with adaptive retention policies driven by downstream loss and future utility.
• It is applied in diverse domains such as whole-slide image analysis and LLM inference to reduce computational complexity through selective token retention.
• Recent approaches employ techniques like top-k gating, exponential decay, and attention-flow integration to maintain essential information while compressing context.

Learnable token eviction denotes the optimization of policies or parameters that decide which tokens, key-value entries, visual patches, or other discrete units of context are retained under a fixed budget, rather than relying on fixed recency rules, local attention heuristics, or uniform sampling. Recent work uses the idea in several settings: gigapixel whole-slide image reasoning, where trainable sparsification replaces training-free patch reduction; decoder-only LLM inference, where KV-cache entries are ranked or gated for bounded-memory decoding; and long-horizon agent memory, where proactive, query-blind retention preserves load-bearing details before the future query exists (Chen et al., 7 Jun 2026, Liao et al., 29 Nov 2025, Lia et al., 18 Jun 2026).

1. Definition and problem setting

The central problem is budgeted retention under sequence growth. In gigapixel whole-slide images, a single slide commonly yields $N > 10^5$ non-overlapping patches, so dense tokenization induces quadratic self-attention complexity $O(N^2)$ and a long visual prefix, while decisive pathology can occupy less than $1\%$ of the tissue (Chen et al., 7 Jun 2026). In autoregressive LLMs, KV cache memory grows linearly with prompt or decode length, and time per decoding step also grows linearly with context length for standard attention, which constrains throughput, batch size, and long-context inference (Luo et al., 21 May 2026). In prefix caching, GPU memory is scarce and cached blocks compete for reuse; in long-horizon agents, eviction is proactive and query-blind, so dropping an exact identifier or path can break later execution even if a summary remains semantically plausible (Fang et al., 12 May 2026, Lia et al., 18 Jun 2026).

Across these settings, learnable eviction replaces fixed heuristics with objectives tied to downstream loss, teacher attention, future utility, or eviction feedback. The retained object varies by system: prompt tokens in Judge Q, KV entries in TRIM-KV and KVP, KV blocks in SAECache, and full units of history in LRE (Liu et al., 13 Sep 2025, Bui et al., 3 Dec 2025, Moschella et al., 10 Feb 2026, Fang et al., 12 May 2026, Lia et al., 18 Jun 2026). This suggests a broader operational definition in which “token” often stands for the natural retention unit of the architecture.

2. Core algorithmic patterns

A recurring design pattern is to decouple scoring from hard retention. In whole-slide image reasoning, a scorer $S_\theta$ produces per-token scores $s$, z-score normalization yields $\hat{s}$, and a temperature-scaled Soft Top-K operator computes $\alpha = \Phi_K(\hat{s}/\tau)$ with $\sum_i \alpha_i \approx K$ during training. SparseLearn then replaces discrete dropping with a variance-preserving gate,

$$\tilde{x}_i = \sqrt{\alpha_i}\,x_i + \sqrt{1-\alpha_i}\,\epsilon_i,$$

followed by a diagonal-attention denoiser; at inference, the training module is removed and deterministic Hard Top-K keeps only the highest-scoring tokens (Chen et al., 7 Jun 2026). In Attention-Gate, a lightweight module inserted before each self-attention layer emits per-token, per-head, per-layer probabilities $p_{l,h,t}$ and binary flags

$O(N^2)$0

which mask attention columns and determine which K/V states are stored; training uses a Straight-Through Estimator with a sparsity regularizer (Zeng et al., 2024).

Other systems learn a scalar retention law directly. TRIM-KV computes a per-token, per-head intrinsic importance $O(N^2)$1 at creation time and defines the effective retention score at future step $O(N^2)$2 as

$O(N^2)$3

so eviction under a budget becomes repeated removal of the smallest decayed score (Bui et al., 3 Dec 2025). LRE uses a few-kilobyte, CPU-only, language-model-free logistic regressor,

$O(N^2)$4

and selects older units under a budget by maximizing total keep-probability subject to a knapsack constraint, while an always-retained recent window is kept outside the budget (Lia et al., 18 Jun 2026).

A second recurring pattern is compensation for discarded information. Meta-Soft appends synthesized soft probes, estimates importance, evicts under a budget, and then redistributes evicted semantics into retained values through attention-flow integration (Luo et al., 21 May 2026). IndexMem couples a learned indexer with a latent memory module that compresses evicted tokens into an online-updated state and adds an explicit residual readout to the retained-cache attention output (Yang et al., 25 May 2026). These mechanisms differ from pure deletion and aim to reduce irreversible forgetting.

3. Decoder-side KV-cache eviction in LLMs

One line of work learns proxy queries during prefill. Judge Q appends a list of learnable soft tokens to the prompt, trains only their embeddings, and aligns their attention-to-prompt map with the attention map of actual decoded response tokens. At inference, the averaged soft-token attention $O(N^2)$5 becomes an importance distribution, and top-$O(N^2)$6 prompt tokens are retained in the KV cache. Under the same eviction budget, the method reports approximately 1 point improvement on LongBench and over 3 points on RULER relative to existing eviction approaches (Liu et al., 13 Sep 2025). Meta-Soft generalizes this idea by synthesizing prompt-adaptive soft probes from a learnable orthogonal meta-library with Gumbel-Softmax selection, then preserving dropped context by attention-flow redistribution; on LongBench it preserves $O(N^2)$7–$O(N^2)$8 of Full-KV performance for $O(N^2)$9 and improves end-to-end latency by $1\%$0–$1\%$1 versus Full KV (Luo et al., 21 May 2026).

A second line learns dynamic retention or adapts the base model to sparse masks. Attention-Gate injects a global-context gating module before each self-attention layer and reports mean eviction of $1\%$2 in continual pre-training and $1\%$3–$1\%$4 in supervised fine-tuning while maintaining or improving task accuracy (Zeng et al., 2024). G-KV, by contrast, does not learn a separate neural scoring network for eviction itself; it uses a decayed global attention score that aggregates intermittent importance across windows and makes the method learnable through RL-Sparse or distillation so that the masked policy used at inference is also optimized during training. Under a 512-token budget on AMC 23, G-KV outperforms baselines by nearly $1\%$5 pass@1 and yields up to $1\%$6 throughput gains for DeepSeek-Qwen-7B and up to $1\%$7 for DeepSeek-LLaMA-8B (Liao et al., 29 Nov 2025).

A third line predicts future utility more explicitly. KVP trains lightweight per-head RL agents on pre-computed traces using only keys, values, and positions, parameterizes a Plackett–Luce ranking policy, and optimizes a budget-agnostic reward that aggregates future-utility loss across all budgets; the added prefill overhead is about $1\%$8, with zero decoding overhead (Moschella et al., 10 Feb 2026). ForesightKV first constructs Golden Eviction traces from future attention scores, then distills them with pairwise ranking loss and refines the policy with GRPO focused on loss spikes in low-entropy tokens; on Qwen3-4B at 32K tokens it reports a $1\%$9 throughput gain at $S_\theta$0 relative to full cache (Dong et al., 3 Feb 2026). TRIM-KV instead learns each token’s intrinsic importance at creation time, applies exponential decay, and evicts the minimum decayed score under the memory bound; across reasoning, procedural generation, conversational memory, and long-context benchmarks it consistently outperforms heuristic eviction and learnable retrieval baselines in low-memory regimes, and on 32K context with 1K generation achieves approximately $S_\theta$1 decoding throughput versus full KV (Bui et al., 3 Dec 2025).

IndexMem adds a different ingredient: a learned importance indexer trained by KL distillation from backbone attention logits, together with a latent memory that compensates the missing residual caused by eviction. On RULER it reports gains of up to $S_\theta$2 points under aggressive eviction and more stable Needle-in-a-Haystack retrieval than heuristic baselines (Yang et al., 25 May 2026).

4. Vision and multimodal token sparsification

In whole-slide image reasoning, learnable token eviction is formulated as trainable sparsification rather than heuristic patch pruning. A scorer between a frozen visual encoder and the vision-to-language projector assigns importance to patch features, Soft Top-K produces a continuous budgeted weighting during training, SparseLearn uses a variance-preserving noise gate and a diagonal-attention denoiser, and inference discards the training path entirely and retains only the highest-scoring $S_\theta$3 tokens. With $S_\theta$4, the method compresses the visual sequence to roughly $S_\theta$5 of its original length on large slides, achieves a $S_\theta$6 average reduction and up to $S_\theta$7 dynamic compression, and reports $S_\theta$8 overall accuracy on SlideBench (TCGA), $S_\theta$9 on SlideBench (BCNB), and $s$0 on WSI-VQA* (Chen et al., 7 Jun 2026).

Hybrid linear-attention models use the same idea to counter recurrent-state forgetfulness. In laLTE, a lightweight grouped 1D CNN predicts per-token, per-head retention scores from pre-RoPE keys and values, combined with sliding-window attention and attention sinks in an “A-plus-column” sparsity pattern. The method maintains a fixed-capacity out-of-window cache per head, with thresholded retention and top-$s$1 replacement, preserving constant per-step time and space. With $s$2 per head and observed average retained count around $s$3, laLTE improves retrieval-intensive benchmarks relative to pure Gated DeltaNet and GDN+SWA; at 0.4B scale it reports $s$4 on RULER single needle-in-a-haystack versus $s$5 for GDN and $s$6 on EVAPORATE versus $s$7 for GDN (He et al., 23 Oct 2025).

A neighboring but distinct line is learnable token merging. LTM-Transformer does not evict tokens; instead it learns a Softmax-normalized mask $s$8 and forms merged tokens $s$9, motivated by a separable variational upper bound on an Information Bottleneck objective. The distinction matters: merging preserves information through aggregation, whereas eviction removes tokens or KV entries under a retention policy (Wang et al., 2024).

5. Prefix caches and long-horizon memory

In serving systems, the retained object may be a prefix-cache block rather than a token. SAECache treats not all tokens as equally worth caching and exploits differences among system prompts, user queries, tool outputs, model responses, and chain-of-thought traces. It combines a multi-queue architecture, semantic-aware token weighting learned online through eviction feedback, and fully adaptive online updates of log-normal timing parameters, position-decay power, queue weights, and meta-parameters. The paper reports that token types exhibit up to $\hat{s}$0 variation in reuse rates and that SAECache yields $\hat{s}$1–$\hat{s}$2 TTFT improvement over production-style baselines across heterogeneous workloads (Fang et al., 12 May 2026).

In agent memory, the same retention logic is pushed outside the model. LRE is a few-kilobyte, CPU-only, language-model-free scorer that operates on units of history rather than tokens: one action-observation step in agents, or one turn or session in conversational settings. The system scores each unit causally, selects under a budget by greedy value density, and emits retained history verbatim so that exact identifiers survive. On AppWorld with $\hat{s}$3 tokens and the last $\hat{s}$4 units always kept, it achieves $\hat{s}$5 task goal completion versus $\hat{s}$6 for keep-all while using zero compressor calls and a lower peak prompt; on LoCoMo it reaches $\hat{s}$7 macro-AUC in supervised form and achieves the best deployable token-F1 at $\hat{s}$8 retention while reading about $\hat{s}$9 fewer tokens than full context (Lia et al., 18 Jun 2026). This suggests that learnable eviction is not restricted to neural attention internals; it also names deployable external policies for proactive fidelity preservation.

6. Distinctions, limitations, and open directions

The literature makes several distinctions that are often blurred. First, learnability does not always mean a separate neural eviction network. In G-KV, the scoring function itself is a decayed aggregation of local attention, while learnability enters through post-training of the masked policy via RL-Sparse or distillation (Liao et al., 29 Nov 2025). Second, learnable eviction is not identical to token compression in general: LTM merges tokens, CAOTE is training-free and ranks eviction candidates by attention-output error using both attention scores and value vectors, and neither method is a standard learned retention policy in the KV-cache sense (Wang et al., 2024, Goel et al., 18 Apr 2025). Third, eviction need not imply irrevocable deletion; Meta-Soft’s attention-flow integration and IndexMem’s latent residual memory are explicit attempts to conserve evicted semantics rather than merely discard them (Luo et al., 21 May 2026, Yang et al., 25 May 2026).

Failure modes are equally consistent across domains. Learnable whole-slide eviction can underweight macroscopic patterns or diffuse margins because scoring is patch-local and the budget $\alpha = \Phi_K(\hat{s}/\tau)$0 is fixed (Chen et al., 7 Jun 2026). Prompt-adaptive probe methods can mis-rank rare but crucial tokens under domain shift or adversarial prompts, and value fusion can induce semantic drift when too many dropped tokens route into a small retained set (Luo et al., 21 May 2026). Even strong learned retention gates struggle on incompressible retrieval-heavy tasks, where the context genuinely resists bounded-cache approximation (Bui et al., 3 Dec 2025). Agent settings add a different constraint: because eviction is proactive and query-blind, some future dependencies are only weakly expressed in the prefix, so no scorer can guarantee recovery of every load-bearing detail (Lia et al., 18 Jun 2026).

Open directions already identified in the literature include dynamic budgets, per-layer and per-head allocation, integration with streaming attention or sliding caches, hybridization with quantization, low-rank projection, token merging, and head pruning, and more end-to-end training of the backbone under the same sparse regime used at inference (Chen et al., 7 Jun 2026, Luo et al., 21 May 2026, Yang et al., 25 May 2026, Bui et al., 3 Dec 2025). This suggests a field moving away from purely local attention proxies toward budget-aware retention models with explicit training objectives, explicit separation between training-time relaxation and inference-time hard selection, and, increasingly, explicit mechanisms for preserving what eviction would otherwise destroy.