---
title: Learnable Token Eviction
url: https://www.emergentmind.com/topics/learnable-token-eviction
type: topic
---

# Learnable Token Eviction

Learnable token eviction denotes the optimization of policies or parameters that decide which tokens, key-value entries, visual patches, or other discrete units of context are retained under a fixed budget, rather than relying on fixed recency rules, local attention heuristics, or uniform sampling. Recent work uses the idea in several settings: gigapixel whole-slide image reasoning, where trainable sparsification replaces training-free patch reduction; decoder-only LLM inference, where KV-cache entries are ranked or gated for bounded-memory decoding; and long-horizon agent memory, where proactive, query-blind retention preserves load-bearing details before the future query exists [2606.08641] [2512.00504] [2606.20954].

## 1. Definition and problem setting

The central problem is budgeted retention under sequence growth. In gigapixel whole-slide images, a single slide commonly yields $N > 10^5$ non-overlapping patches, so dense tokenization induces quadratic self-attention complexity $O(N^2)$ and a long visual prefix, while decisive pathology can occupy less than $1\%$ of the tissue [2606.08641]. In autoregressive LLMs, KV cache memory grows linearly with prompt or decode length, and time per decoding step also grows linearly with context length for standard attention, which constrains throughput, batch size, and long-context inference [2605.22337]. In prefix caching, GPU memory is scarce and cached blocks compete for reuse; in long-horizon agents, eviction is proactive and query-blind, so dropping an exact identifier or path can break later execution even if a summary remains semantically plausible [2605.18825] [2606.20954].

Across these settings, learnable eviction replaces fixed heuristics with objectives tied to downstream loss, teacher attention, future utility, or eviction feedback. The retained object varies by system: prompt tokens in Judge Q, KV entries in TRIM-KV and KVP, KV blocks in SAECache, and full units of history in LRE [2509.10798] [2512.03324] [2602.10238] [2605.18825] [2606.20954]. This suggests a broader operational definition in which “token” often stands for the natural retention unit of the architecture.

## 2. Core algorithmic patterns

A recurring design pattern is to decouple scoring from hard retention. In whole-slide image reasoning, a scorer $S_\theta$ produces per-token scores $s$, z-score normalization yields $\hat{s}$, and a temperature-scaled Soft Top-K operator computes $\alpha = \Phi_K(\hat{s}/\tau)$ with $\sum_i \alpha_i \approx K$ during training. SparseLearn then replaces discrete dropping with a variance-preserving gate,
$$
\tilde{x}_i = \sqrt{\alpha_i}\,x_i + \sqrt{1-\alpha_i}\,\epsilon_i,
$$
followed by a diagonal-attention denoiser; at inference, the training module is removed and deterministic Hard Top-K keeps only the highest-scoring tokens [2606.08641]. In Attention-Gate, a lightweight module inserted before each self-attention layer emits per-token, per-head, per-layer probabilities $p_{l,h,t}$ and binary flags
$$
m_{l,h,t} = \mathbb{1}[p_{l,h,t} \ge \tau],
$$
which mask attention columns and determine which K/V states are stored; training uses a Straight-Through Estimator with a sparsity regularizer [2410.12876].

Other systems learn a scalar retention law directly. TRIM-KV computes a per-token, per-head intrinsic importance $\beta_{l,h}(i) \in [0,1]$ at creation time and defines the effective retention score at future step $t$ as
$$
r_{l,h}(i \to t) = \beta_{l,h}(i)^{t-i},
$$
so eviction under a budget becomes repeated removal of the smallest decayed score [2512.03324]. LRE uses a few-kilobyte, CPU-only, language-model-free logistic regressor,
$$
p_i = \sigma(\theta^\top \phi(u_i, u_{\le i})),
$$
and selects older units under a budget by maximizing total keep-probability subject to a knapsack constraint, while an always-retained recent window is kept outside the budget [2606.20954].

A second recurring pattern is compensation for discarded information. Meta-Soft appends synthesized soft probes, estimates importance, evicts under a budget, and then redistributes evicted semantics into retained values through attention-flow integration [2605.22337]. IndexMem couples a learned indexer with a latent memory module that compresses evicted tokens into an online-updated state and adds an explicit residual readout to the retained-cache attention output [2605.25475]. These mechanisms differ from pure deletion and aim to reduce irreversible forgetting.

## 3. Decoder-side KV-cache eviction in LLMs

One line of work learns proxy queries during prefill. Judge Q appends a list of learnable soft tokens to the prompt, trains only their embeddings, and aligns their attention-to-prompt map with the attention map of actual decoded response tokens. At inference, the averaged soft-token attention $A_{\text{soft}}$ becomes an importance distribution, and top-$B$ prompt tokens are retained in the KV cache. Under the same eviction budget, the method reports approximately 1 point improvement on LongBench and over 3 points on RULER relative to existing eviction approaches [2509.10798]. Meta-Soft generalizes this idea by synthesizing prompt-adaptive soft probes from a learnable orthogonal meta-library with Gumbel-Softmax selection, then preserving dropped context by attention-flow redistribution; on LongBench it preserves $92.9$–$97.2\%$ of Full-KV performance for $B \in \{128,256\}$ and improves end-to-end latency by $1.2\times$–$10.5\times$ versus Full KV [2605.22337].

A second line learns dynamic retention or adapts the base model to sparse masks. Attention-Gate injects a global-context gating module before each self-attention layer and reports mean eviction of $51.87\%$ in continual pre-training and $55.49$–$60.00\%$ in supervised fine-tuning while maintaining or improving task accuracy [2410.12876]. G-KV, by contrast, does not learn a separate neural scoring network for eviction itself; it uses a decayed global attention score that aggregates intermittent importance across windows and makes the method learnable through RL-Sparse or distillation so that the masked policy used at inference is also optimized during training. Under a 512-token budget on AMC 23, G-KV outperforms baselines by nearly $20\%$ pass@1 and yields up to $12.18\times$ throughput gains for DeepSeek-Qwen-7B and up to $19.7\times$ for DeepSeek-LLaMA-8B [2512.00504].

A third line predicts future utility more explicitly. KVP trains lightweight per-head RL agents on pre-computed traces using only keys, values, and positions, parameterizes a Plackett–Luce ranking policy, and optimizes a budget-agnostic reward that aggregates future-utility loss across all budgets; the added prefill overhead is about $1\%$, with zero decoding overhead [2602.10238]. ForesightKV first constructs Golden Eviction traces from future attention scores, then distills them with pairwise ranking loss and refines the policy with GRPO focused on loss spikes in low-entropy tokens; on Qwen3-4B at 32K tokens it reports a $9.79\times$ throughput gain at $B=1\text{K}$ relative to full cache [2602.03203]. TRIM-KV instead learns each token’s intrinsic importance at creation time, applies exponential decay, and evicts the minimum decayed score under the memory bound; across reasoning, procedural generation, conversational memory, and long-context benchmarks it consistently outperforms heuristic eviction and learnable retrieval baselines in low-memory regimes, and on 32K context with 1K generation achieves approximately $2\times$ decoding throughput versus full KV [2512.03324].

IndexMem adds a different ingredient: a learned importance indexer trained by KL distillation from backbone attention logits, together with a latent memory that compensates the missing residual caused by eviction. On RULER it reports gains of up to $25$ points under aggressive eviction and more stable Needle-in-a-Haystack retrieval than heuristic baselines [2605.25475].

## 4. Vision and multimodal token sparsification

In whole-slide image reasoning, learnable token eviction is formulated as trainable sparsification rather than heuristic patch pruning. A scorer between a frozen visual encoder and the vision-to-language projector assigns importance to patch features, Soft Top-K produces a continuous budgeted weighting during training, SparseLearn uses a variance-preserving noise gate and a diagonal-attention denoiser, and inference discards the training path entirely and retains only the highest-scoring $32$ tokens. With $K=32$, the method compresses the visual sequence to roughly $0.78\%$ of its original length on large slides, achieves a $58\times$ average reduction and up to $128\times$ dynamic compression, and reports $73.32\%$ overall accuracy on SlideBench (TCGA), $56.89\%$ on SlideBench (BCNB), and $60.76\%$ on WSI-VQA* [2606.08641].

Hybrid linear-attention models use the same idea to counter recurrent-state forgetfulness. In laLTE, a lightweight grouped 1D CNN predicts per-token, per-head retention scores from pre-RoPE keys and values, combined with sliding-window attention and attention sinks in an “A-plus-column” sparsity pattern. The method maintains a fixed-capacity out-of-window cache per head, with thresholded retention and top-$b$ replacement, preserving constant per-step time and space. With $b=512$ per head and observed average retained count around $256$, laLTE improves retrieval-intensive benchmarks relative to pure Gated DeltaNet and GDN+SWA; at 0.4B scale it reports $63.3\%$ on RULER single needle-in-a-haystack versus $51.7\%$ for GDN and $25.26$ on EVAPORATE versus $19.47$ for GDN [2510.20787].

A neighboring but distinct line is learnable token merging. LTM-Transformer does not evict tokens; instead it learns a Softmax-normalized mask $G$ and forms merged tokens $\tilde{Z}(G)=G^\top Z$, motivated by a separable variational upper bound on an Information Bottleneck objective. The distinction matters: merging preserves information through aggregation, whereas eviction removes tokens or KV entries under a retention policy [2407.15219].

## 5. Prefix caches and long-horizon memory

In serving systems, the retained object may be a prefix-cache block rather than a token. SAECache treats not all tokens as equally worth caching and exploits differences among system prompts, user queries, tool outputs, model responses, and chain-of-thought traces. It combines a multi-queue architecture, semantic-aware token weighting learned online through eviction feedback, and fully adaptive online updates of log-normal timing parameters, position-decay power, queue weights, and meta-parameters. The paper reports that token types exhibit up to $756\times$ variation in reuse rates and that SAECache yields $1.4\times$–$2.7\times$ TTFT improvement over production-style baselines across heterogeneous workloads [2605.18825].

In agent memory, the same retention logic is pushed outside the model. LRE is a few-kilobyte, CPU-only, language-model-free scorer that operates on units of history rather than tokens: one action-observation step in agents, or one turn or session in conversational settings. The system scores each unit causally, selects under a budget by greedy value density, and emits retained history verbatim so that exact identifiers survive. On AppWorld with $B=2048$ tokens and the last $5$ units always kept, it achieves $41.1\%$ task goal completion versus $44.0\%$ for keep-all while using zero compressor calls and a lower peak prompt; on LoCoMo it reaches $0.829$ macro-AUC in supervised form and achieves the best deployable token-F1 at $20\%$ retention while reading about $68\%$ fewer tokens than full context [2606.20954]. This suggests that learnable eviction is not restricted to neural attention internals; it also names deployable external policies for proactive fidelity preservation.

## 6. Distinctions, limitations, and open directions

The literature makes several distinctions that are often blurred. First, learnability does not always mean a separate neural eviction network. In G-KV, the scoring function itself is a decayed aggregation of local attention, while learnability enters through post-training of the masked policy via RL-Sparse or distillation [2512.00504]. Second, learnable eviction is not identical to token compression in general: LTM merges tokens, CAOTE is training-free and ranks eviction candidates by attention-output error using both attention scores and value vectors, and neither method is a standard learned retention policy in the KV-cache sense [2407.15219] [2504.14051]. Third, eviction need not imply irrevocable deletion; Meta-Soft’s attention-flow integration and IndexMem’s latent residual memory are explicit attempts to conserve evicted semantics rather than merely discard them [2605.22337] [2605.25475].

Failure modes are equally consistent across domains. Learnable whole-slide eviction can underweight macroscopic patterns or diffuse margins because scoring is patch-local and the budget $K$ is fixed [2606.08641]. Prompt-adaptive probe methods can mis-rank rare but crucial tokens under domain shift or adversarial prompts, and value fusion can induce semantic drift when too many dropped tokens route into a small retained set [2605.22337]. Even strong learned retention gates struggle on incompressible retrieval-heavy tasks, where the context genuinely resists bounded-cache approximation [2512.03324]. Agent settings add a different constraint: because eviction is proactive and query-blind, some future dependencies are only weakly expressed in the prefix, so no scorer can guarantee recovery of every load-bearing detail [2606.20954].

Open directions already identified in the literature include dynamic budgets, per-layer and per-head allocation, integration with streaming attention or sliding caches, hybridization with quantization, low-rank projection, token merging, and head pruning, and more end-to-end training of the backbone under the same sparse regime used at inference [2606.08641] [2605.22337] [2605.25475] [2512.03324]. This suggests a field moving away from purely local attention proxies toward budget-aware retention models with explicit training objectives, explicit separation between training-time relaxation and inference-time hard selection, and, increasingly, explicit mechanisms for preserving what eviction would otherwise destroy.

Source: https://www.emergentmind.com/topics/learnable-token-eviction