Semantic Cache Eviction

• Semantic cache eviction is a set of cache management policies that evict entries based on estimated future utility rather than mere recency.
• It utilizes semantic signals such as future attention, output contribution, token roles, and structural metadata to preserve key information.
• These methods deliver practical gains by reducing cache size, lowering GPU memory usage, and accelerating decoding in long-context scenarios.

Semantic cache eviction is the class of cache-management policies that evict entries according to estimated future usefulness, predictive information, semantic role, or downstream service impact rather than according to recency alone. In the recent literature, the term is most developed for transformer key–value (KV) caches under long-context inference, but closely related formulations also appear in prefix-cache management across requests and in tiered cache networks. Across these settings, the central shift is from local heuristics such as LRU, sliding windows, or raw attention magnitude toward eviction rules that attempt to preserve future decoding utility, long-horizon semantic anchors, structurally important prompt regions, or weighted downstream demand (Tang et al., 9 Feb 2026, Moschella et al., 10 Feb 2026, Yang et al., 28 Apr 2026, Fang et al., 12 May 2026, Alpay et al., 20 Jun 2026).

1. Conceptual scope and problem setting

The immediate technical motivation is that KV caching is indispensable for long-context transformer inference, yet the cache grows linearly with sequence length while effective attention cost and memory traffic scale with the amount of retained context. For long contexts, the KV cache becomes a primary bottleneck in GPU memory and latency, which makes eviction or compression necessary (Tang et al., 9 Feb 2026, Yang et al., 28 Apr 2026). In agentic workloads this pressure is amplified by heterogeneous traces containing system instructions, plans, user turns, retrieved documents, tool outputs, and scratchpad reasoning, all of which exhibit different reuse patterns (Matam et al., 12 Jul 2026).

“Semantic” in this literature does not denote a single metric. It instead denotes a family of objectives that try to preserve what matters for future computation. Some papers define this through future attention or future output contribution; some through information capacity; some through semantic role labels or topic structure; and some through explicit downstream service connectivity. A common negative diagnosis is that conventional policies treat cache entries too uniformly. In prefix caches, for example, different token types exhibit up to 756x variation in reuse rates, yet policies such as LRU largely treat cached blocks uniformly (Fang et al., 12 May 2026). In KV eviction, multiple papers argue that recency, accumulated attention, or fixed per-head budgets ignore head heterogeneity, inter-head interactions, or semantic structure already present in the prompt or orchestrator (Tang et al., 9 Feb 2026, Mai et al., 8 May 2026, Matam et al., 12 Jul 2026).

Context	Cached object	Semantic signal
Long-context LLM inference	KV entries or KV pages	Future attention, output contribution, information capacity, region role
Prefix caching across requests	Prefix KV blocks	Token type, session reuse, structural reuse
Tiered cache networks	Aligned storage blocks	Weighted downstream demand cut, service continuity

This broader usage implies that semantic cache eviction is not restricted to token-level importance ranking. It also includes global budget allocation across layers or heads, multi-queue replacement across request sessions, and graph-based eviction that minimizes service cuts rather than local miss surrogates (Qin et al., 16 Mar 2025, Wu et al., 25 Feb 2026, Alpay et al., 20 Jun 2026).

2. Formalizations of semantic utility

A major line of work defines semantic utility through future contribution. LU-KV introduces Oracle Importance for token $j$ in layer $l$, head $h$ as the maximum future contribution of that token to the output representation over a decoding horizon: $$I_{l,h,j}=\max_{k\in\{1,\dots,K_{\max}\}}\left\|A_{l,h,k,j}\,v_{l,h,j}W^{(e,h)}\right\| .$$ This definition is explicitly future-oriented: a token is important if it contributes significantly to the output at some future step, even if its prefill-time attention was only moderate (Tang et al., 9 Feb 2026).

KVP uses a related but simpler supervision target: the future utility of token $x_i$ is its cumulative future attention from later tokens,

$\mathrm{Imp}(x_i)=\sum_{j=n+1}^{n+f} A(x_i,x_j),$

and eviction is reframed as learning a budget-agnostic ranking whose top-$b$ tokens maximize reward across all budgets (Moschella et al., 10 Feb 2026). ForesightKV goes further by constructing a “Golden Eviction” oracle from future attention blocks and then training a scorer to imitate the resulting long-term ranking under a pairwise ranking loss, later refining it with GRPO on sequence-level rewards focused on low-entropy tokens with large loss increases (Dong et al., 3 Feb 2026).

A second line defines semantic utility through information preservation. CapKV reformulates eviction under a linear–Gaussian surrogate of attention and derives the mutual-information objective

$$I(q;Y\mid Z_C)=\frac{1}{2}\log\det\!\left(I+\Sigma_{\mathrm{noise}}^{-1}U_CK_C\Lambda_QK_C^\top U_C^\top\right),$$

so eviction becomes a capacity-maximization problem: retain the subset of KV pairs that preserves the most information about future queries in the output (Yang et al., 28 Apr 2026). This is an information-bottleneck view of semantic eviction.

A third line makes the objective output-aware rather than attention-aware. LaProx starts from the actual layer computation $Y=A(VW_O)$ and derives a token importance score proportional to

$p_i \propto \|A[:,i]\|_2 \cdot \|(VW_O)[i,:]\|_2 .$

The semantic claim here is that tokens should be judged by their contribution to the layer output, not by attention weights alone, because values and output projection materially affect the representation actually propagated forward (Mai et al., 8 May 2026).

A fourth line defines semantics via region or role priors rather than latent geometric surrogates. MemDecay assigns each prompt region a base priority $l$0 and decay rate $l$1, then combines this with an attention-derived term: $l$2 This turns semantic role labels such as system, plan, retrieval, or scratchpad into explicit retention priors (Matam et al., 12 Jul 2026). SAECache applies the same general principle to prefix caches: token type, session structure, and structural position become factors in a global retention score for shared KV blocks (Fang et al., 12 May 2026).

These formalizations differ in mechanism, but they agree on a common point: cache entries should be priced by future consequence, not merely by past access count or instantaneous score magnitude.

3. Methodological families

Recent methods fall into several recurring families.

Global utility allocation and offline profiling: LU-KV treats eviction as a global combinatorial allocation problem over heads. It keeps the intra-head metric $l$3 fixed, estimates per-head oracle loss curves offline, relaxes them with isotonic regression / convex-hull fitting, and then uses a marginal-utility greedy solver that is optimal for the relaxed separable diminishing-returns objective. The result is a static lookup table $l$4 that maps a global compression ratio to per-head local compression ratios (Tang et al., 9 Feb 2026). CAKE performs a related but layer-level allocation by defining preference scores from recent-window attention entropy and variance, then assigning

$l$5

with a cascading implementation that preserves the final allocation under bounded prefilling memory (Qin et al., 16 Mar 2025).

Learned ranking or control policies: KVP trains lightweight per-head RL agents using only keys, values, and positions, with a Plackett–Luce ranking policy and a reward defined over all budgets (Moschella et al., 10 Feb 2026). ForesightKV uses supervised distillation from a future-attention oracle and then RL fine-tuning for long reasoning traces (Dong et al., 3 Feb 2026). Attention-Gate inserts learned modules before self-attention and outputs per-layer, per-head binary retention flags conditioned on the full context, allowing the model to learn which KV states can be discarded (Zeng et al., 2024). IndexMem trains an indexer to imitate the model’s own attention patterns and supplements hard eviction with a latent memory that stores compressed residual information from evicted tokens (Yang et al., 25 May 2026).

Information-theoretic and output-aware scoring: CapKV interprets many heuristics as approximations to the same log-determinant capacity objective and scores tokens with a leverage-like quantity

$l$6

where the weight $l$7 encodes query relevance and $l$8 is a value-induced direction (Yang et al., 28 Apr 2026). LaProx, by contrast, uses output-aware matrix approximation and globally comparable token scores based on $l$9 (Mai et al., 8 May 2026).

Defensive and stochastic streaming policies: Nexus Sampling argues that deterministic top-$h$0 in a streaming regime is brittle because a single below-cutoff step becomes an irreversible decision. It adds a bridge-aware “Nexus score” to direct attention and then uses weighted reservoir sampling, giving every positive-score token a positive survival probability over long streams (Duong et al., 22 Jun 2026). DefensiveKV and Layer-DefensiveKV instead target fragility in the aggregation step: they replace mean aggregation with a worst-case-oriented rule $h$1, yielding substantial robustness gains under extreme compression (Feng et al., 15 Oct 2025).

Role-, region-, and topic-aware cache management: MemDecay uses orchestrator-provided prompt regions and calibrated decay rates (Matam et al., 12 Jul 2026). SAECache uses a multi-queue architecture for prefix caches, with separate queues for evict-first, structural, chat, and agentic blocks, and learns token-type weights online from miss-after-eviction feedback (Fang et al., 12 May 2026). RAC generalizes the same idea beyond KV-specific settings by combining Topical Prevalence and Topic Structural Importance for online semantic replacement under long reuse distances (Wu et al., 25 Feb 2026).

Segmentation and adaptive block-size methods: SABlock addresses the mismatch between token-level flexibility and block-level coherence by first performing semantic segmentation, then applying segment-guided token scoring, and finally choosing segment-specific compression block sizes under a budget-driven search (Chen et al., 26 Oct 2025).

A plausible implication is that “semantic cache eviction” is best regarded as a design space rather than a single algorithmic recipe: future utility can be inferred from attention, values, output projections, region metadata, topic structure, or explicit service graphs.

4. Runtime realization and systems behavior

A distinguishing systems question is whether semantic decisions are made offline and reused, once after prefill, or continuously during decoding.

LU-KV is explicitly task-agnostic at inference time: all global optimization is distilled offline into head-wise ratios $h$2, and runtime consists only of table lookup, per-head scoring by the chosen metric, and top-$h$3 selection. The lookup and budget calculation are $h$4 and are performed once per compression event (Tang et al., 9 Feb 2026). KVP likewise performs all learned work between prefill and decoding: on Qwen‑7B it adds about 0.15 GFLOPs/token in prefill, taking total prefill from 14.00 GFLOPs/token to 14.15 GFLOPs/token, and incurs zero overhead during autoregressive decoding because compression is applied only once after prefill (Moschella et al., 10 Feb 2026).

Other methods operate continuously. Nexus Sampling is designed for fixed-budget streaming eviction and recomputes block weights from recent queries at each eviction step (Duong et al., 22 Jun 2026). ForesightKV evicts every $h$5 tokens during long reasoning traces, preserving the most recent $h$6 KV pairs and resampling among older candidates with a learned scorer (Dong et al., 3 Feb 2026). MemDecay updates attention EWMAs and decay clocks at observation steps, then evicts the lowest-scoring non-pinned pages under a fixed page budget (Matam et al., 12 Jul 2026). Prefix-cache methods such as SAECache act across requests rather than within a single generation: cache eviction is triggered by GPU prefix-memory pressure, and the victim score is computed from queue weight, token-type weight, local survival priority, and elapsed time (Fang et al., 12 May 2026).

The runtime state that semantic methods maintain differs accordingly. Learned token-ranking methods maintain per-head or per-layer scorers; structural methods maintain page scores, topic statistics, or queue-local timers; and graph-theoretic methods maintain reachability or demand-cut state. In the tiered-network setting of service-cut certificates, semantic eviction is implemented as an all-candidate downstream impact computation $h$7 together with independently checkable reachability certificates, so the victim is the admissible aligned block with minimum weighted demand cut (Alpay et al., 20 Jun 2026).

Several papers emphasize that semantic eviction is operationally useful only if its overhead is modest. CapKV reports a per-eviction complexity of $h$8 per head and states that overhead is similar to other methods (Yang et al., 28 Apr 2026). CAKE reports over 10x speedup in decoding latency compared to full cache at 128K context with FlashAttention‑2 (Qin et al., 16 Mar 2025). SABlock reports, under a fixed cache budget of 1,024, a 46.28% reduction in peak memory usage and up to 9.5x faster decoding on a 128K context length (Chen et al., 26 Oct 2025). ForesightKV reports throughput speedups up to 9.79x at 32K with a 1K budget on Qwen3‑4B (Dong et al., 3 Feb 2026).

5. Empirical patterns and observed gains

Across benchmarks, semantic eviction is consistently most valuable under aggressive compression, retrieval-heavy tasks, reasoning traces, and heterogeneous prompt structures.

LU-KV reports that, on LongBench and RULER, it achieves an 80% reduction in KV cache size with minimal performance degradation, while also reducing inference latency and GPU memory footprint (Tang et al., 9 Feb 2026). At the same 80% compression, LU-KV on RULER‑16K with the SnapKV metric yields 69.98 on Mistral‑7B, compared with 37.48 for AdaKV and 29.53 for Uniform; on Qwen2.5‑32B it reaches 80.47 versus 44.04 for AdaKV (Tang et al., 9 Feb 2026). KVP reports that it significantly outperforms baselines on RULER and OASST2‑4k and also generalizes zero-shot to LongBench, BOOLQ, and ARC, despite using no queries or attention scores as inputs at inference time (Moschella et al., 10 Feb 2026).

Information-theoretic and output-aware methods report similar patterns. CapKV shows high Spearman correlations between simplified capacity measures and downstream performance, specifically 0.75–0.85+ with very small $h$9-values, and consistently outperforms EA, KeyDiff, Knorm, SnapKV, and Sink across LongBench compression ratios (Yang et al., 28 Apr 2026). LaProx reports that on LongBench at a 128-token budget, roughly 5% of full KV, it reduces accuracy loss by up to 2× relative to strong baselines and maintains model performance with only 5% of the KV cache across LongBench and Needle-In-A-Haystack configurations (Mai et al., 8 May 2026).

Segmentation- and memory-augmented methods show especially strong retrieval behavior. SABlock reports 99.9% retrieval accuracy on NIAH with only 96 KV entries, nearly matching a full-cache baseline that retains up to 8K entries (Chen et al., 26 Oct 2025). IndexMem reports improvements on RULER of up to 25 points under aggressive eviction, markedly more stable Needle-in-a-Haystack retrieval, and superior LongBench compression curves relative to heuristic policies (Yang et al., 25 May 2026).

Reasoning-oriented learned policies show the same theme. ForesightKV reports that on AIME2024, Qwen3‑4B with a 1K budget reaches 54.5 pass@1, while R‑KV at 2K reaches 44.8; the smaller semantic budget outperforms the larger heuristic one (Dong et al., 3 Feb 2026). On reasoning traces, the paper attributes much of the gain to preserving low-entropy factual tokens whose loss spikes under naive eviction (Dong et al., 3 Feb 2026).

Semantic role information is especially effective for preserving invariants. MemDecay reports that attention lifetimes differ by an order of magnitude across regions, with system-token half-lives range from 148 to 189 decoding steps, compared with 14 to 16 for scratchpad tokens (Matam et al., 12 Jul 2026). With pinning enabled, MemDecay preserves system-region facts at full-cache accuracy in every setting, whereas no baseline preserves more than 13 of 24 (Matam et al., 12 Jul 2026). By contrast, the same paper also finds that accumulated-attention retention performs better on unpinned content, indicating that structural semantics alone is not yet sufficient (Matam et al., 12 Jul 2026).

Robustness to worst-case failures is another recurring empirical theme. DefensiveKV and Layer-DefensiveKV report reducing generation quality loss by 2.3x and 4.3x respectively versus the strongest baseline under a 20% cache size (Feng et al., 15 Oct 2025). Nexus Sampling reports that at 80% KV cache eviction it matches dense attention within 1% on LongBench while improving retrieval-heavy tasks and achieving up to 10x smaller per-sequence cache memory (Duong et al., 22 Jun 2026).

These results suggest a broad regularity: semantics-aware methods tend to help most when importance is sparse, delayed, or heterogeneous across tokens, heads, regions, or sessions.

6. Limitations, recurring controversies, and open directions

A central controversy concerns whether attention is importance. Multiple papers argue that raw or accumulated attention is only a local proxy and can be biased or miscalibrated. AhaKV shows that accumulated attention under causal masking has a positional bias in expectation and therefore over-retains early tokens, motivating recent-window accumulation, entropy-controlled softmax scaling, and a value-based prior (Gu et al., 4 Jun 2025). LU-KV argues that score magnitudes are not comparable across heads because heads differ in predictive fidelity, so uniform or raw cross-head top-$$I_{l,h,j}=\max_{k\in\{1,\dots,K_{\max}\}}\left\|A_{l,h,k,j}\,v_{l,h,j}W^{(e,h)}\right\| .$$0 can misallocate budget (Tang et al., 9 Feb 2026). LaProx and CapKV make a related point in different language: values, output projection, and query statistics matter, so attention alone is not a faithful measure of semantic contribution (Mai et al., 8 May 2026, Yang et al., 28 Apr 2026).

Another controversy is query-agnostic versus query-aware semantics. Task-agnostic profiles and query-free learned scorers are attractive for production settings where the final query is unknown at compression time, but they necessarily optimize expected future utility rather than instance-specific relevance (Tang et al., 9 Feb 2026, Moschella et al., 10 Feb 2026). Several papers explicitly note this trade-off. LU-KV relies on structural stability across tasks and may need recalibration if model fine-tuning changes head semantics (Tang et al., 9 Feb 2026). KVP is model-specific and may require retraining for different architectures or domains (Moschella et al., 10 Feb 2026). ForesightKV similarly depends on oracle traces from a training distribution of long reasoning outputs (Dong et al., 3 Feb 2026).

Streaming policies introduce a different trade-off between determinism and survivability. Nexus Sampling argues that deterministic top-$$I_{l,h,j}=\max_{k\in\{1,\dots,K_{\max}\}}\left\|A_{l,h,k,j}\,v_{l,h,j}W^{(e,h)}\right\| .$$1 is structurally fragile in continuous streams, while weighted reservoir sampling better preserves subtly important bridge tokens; the cost is stochastic variance across runs (Duong et al., 22 Jun 2026). DefensiveKV identifies fragility in mean aggregation itself and reframes robustness as worst-case risk control, but this conservative stance may over-retain entries when importance is genuinely stable (Feng et al., 15 Oct 2025).

Region- and role-aware methods depend on metadata quality. MemDecay assumes reliable orchestrator region labels and reports that attention-score normalization is the main limitation of its current formulation (Matam et al., 12 Jul 2026). SAECache likewise depends on meaningful token-type labels and queue routing; it reports that fixed-parameter alternatives can degrade by up to 2.7x under workload mismatch, which motivates fully adaptive online updates (Fang et al., 12 May 2026).

Theoretical abstractions also impose limits. CapKV’s mutual-information objective rests on a linear–Gaussian surrogate of attention and approximates $$I_{l,h,j}=\max_{k\in\{1,\dots,K_{\max}\}}\left\|A_{l,h,k,j}\,v_{l,h,j}W^{(e,h)}\right\| .$$2, so it is not an exact model of transformer behavior (Yang et al., 28 Apr 2026). Service-cut formulations make the downstream semantics explicit, but minimum aligned block actions are NP-complete even with equal block size, which places a hard limit on exact action-optimal semantic reclamation in general tiered networks (Alpay et al., 20 Jun 2026).

Open directions recur across papers. These include better oracle definitions based on logit-level or loss-level perturbation (Tang et al., 9 Feb 2026, Dong et al., 3 Feb 2026), task- or domain-specific semantic profiles (Tang et al., 9 Feb 2026), richer query statistics beyond a mean query vector (Yang et al., 28 Apr 2026), head- or group-specific variants for MQA and GQA (Tang et al., 9 Feb 2026), joint use with quantization or KV merging (Qin et al., 16 Mar 2025, Dong et al., 3 Feb 2026), multimodal or non-KV caches (Yang et al., 28 Apr 2026, Moschella et al., 10 Feb 2026), and tiered retention schemes that degrade precision or location before irreversible eviction (Matam et al., 12 Jul 2026). A plausible synthesis is that future semantic cache eviction will be hybrid: structural priors, learned future-utility models, attention-derived evidence, and systems constraints will be combined rather than treated as competing alternatives.