---
title: Sparse Attention with Token Eviction
url: https://www.emergentmind.com/topics/sparse-attention-with-token-eviction
type: topic
---

# Sparse Attention with Token Eviction

Searching arXiv for recent papers on sparse attention with token eviction to ground the article in the current literature.
Sparse attention with token eviction refers to attention mechanisms that selectively retain a small subset of tokens or cached key-value (KV) pairs for attention computation while dropping, releasing, compressing, or bypassing the remainder. In contemporary long-context inference, this design appears in query-aware top-$k$ retrieval from KV caches, dynamic top-$p$ or cumulative-mass selection, one-time post-prefill cache compression, compress-attend-decompress layers, and hardware-aligned sparse kernels. Across these variants, the common objective is to replace full-cache attention with budgeted access that preserves accuracy comparable to full attention while reducing compute, memory, or bandwidth demands [2502.06766] [2602.03216] [2503.08879].

## 1. Core formulation

In autoregressive decoding, the canonical setup consists of a new query vector and a growing cache of past keys and values. A representative dynamic sparse formulation attaches a lightweight indexer with $H_i$ heads and projection dimension $d_i$, computes per-token scores
$$
S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),
$$
selects
$$
\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},
$$
and performs standard attention only on the selected subset,
$$
\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.
$$
Over an entire sequence, this incurs only $O(k\!\cdot\! t)$ bandwidth per token instead of $O(t^2)$ [2603.13430].

A related long-context design maintains two disjoint pools per layer: a prefilled cache stored off-GPU, $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$, and a small on-GPU window of recently generated tokens, $K_{\mathrm{gen},\ell},V_{\mathrm{gen},\ell} \in \mathbb{R}^{M\times D}$. With query $q$, it selects only the $k$ most relevant keys from the large cache via
$$
I_k(q) = \arg \mathrm{top}_k(s/\sqrt{D}),
$$
computes sparse attention on those entries, computes full attention on the recent window, and forms
$$
\mathrm{Out} = \mathrm{Out}_{\mathrm{cache}} + \mathrm{Out}_{\mathrm{gen}}.
$$
In this setting, full attention per new token has compute $O(LND)$ and GPU memory $O(LND)$, whereas $k$-attention stores only the generated window and retrieved values on GPU, with total GPU memory $O(L(W+k)D)$ [2502.06766].

These formulations make the essential point precise: token eviction is not a single algorithm, but a budgeting principle. The budget may be expressed as a fixed token count $k$, a retained fraction, a cumulative attention mass target, or a cache capacity $B$, but the operational effect is the same: only a subset of past tokens participates in the expensive attention path.

## 2. Selection and eviction criteria

The most direct criterion is fixed-budget top-$k$. This appears in dynamic sparse attention indexers, in long-context $k$-attention, and in one-time cache compression methods. SAGE-KV, for example, performs a single pass of top-$k$ selection immediately after prefill, allocating the token budget $B$ into sink, recent, and selected tokens, then forming a reduced cache
$$
C^\ell = \mathrm{Concat}( P^\ell_{1:S},\; P^\ell_{S + I^\ell_\star},\; P^\ell_{S+E+1:N-1},\; P^\ell_N ),
$$
so that subsequent decoding uses $O(d\!\cdot\! B)$ per layer instead of $O(d\!\cdot\! N)$ [2503.08879].

A second class replaces fixed budgets with cumulative-mass constraints. Tactic chooses the smallest subset $S$ such that
$$
\sum_{i\in S} A_i \ge \tau \sum_{i=1}^N A_i = \tau,
$$
thereby selecting tokens by a target fraction of total attention mass rather than a fixed count [2502.12216]. RTPurbo similarly uses dynamic top-$p$: for each retrieval head it selects the smallest set $S_h(m)$ such that
$$
\sum_{n\in S_h(m)} \exp s_h(m,n) \ge p_{\mathrm{top}} \cdot \sum_{n=1}^T \exp s_h(m,n),
$$
with the size $|S_h(m)|$ varying by query [2605.16928]. In video DiTs, cumulative-energy filtering derives a decision score
$$
t_i^j = \hat m_i - \mathrm{LSE}_i^{(j-1)},
$$
and skips a tile whenever $t_i^j < \lambda$, with the threshold tied to a recall constraint rather than a fixed token count [2606.16317].

A third class uses proxy signals that are cheaper than full attention. FASA identifies head-specific “dominant” frequency-chunks in RoPE, computes token-importance prediction only on those chunks, and then performs focused full-dimensional attention on the retained subset [2602.03152]. STS uses a smaller draft model inside speculative decoding as a predictor of which tokens and heads matter for a larger target model, constructing a binary mask before the target attention kernels run [2605.15508].

A fourth class defines eviction directly in terms of output perturbation or recurrence rather than score magnitude. CAOTE derives a closed-form eviction score
$$
c_j = \frac{\alpha_j}{1-\alpha_j}\,\|O - v_j\|_2,
$$
and evicts the token with the smallest $c_j$, thereby combining the query-key alignment $\alpha_j$ with the value-vector deviation $\|v_j-O\|_2$ [2504.14051]. LazyEviction introduces Token Importance Recurrence and uses MRI-centric heuristics,
$$
I_t[i] =
\begin{cases}
H1_t[i] + H2_t[i], & \text{if } MRI_t[i] \neq 0,\\
H1_t[i], & \text{if } MRI_t[i] = 0,
\end{cases}
$$
to preserve tokens that may regain importance after several decoding steps [2506.15969].

Finally, some systems learn eviction policies explicitly. In laLTE, each head computes retention scores $r_{j,h}\in(0,1)$, binarizes them at $\tau=0.5$, and attends only to tokens in
$$
I_{i,h} = \{ j \le i : [j \le s] \vee [m_{j,h}=1] \vee [j \ge i-w] \},
$$
thereby combining sink tokens, a sliding window, and a learnable retained cache [2510.20787].

## 3. Irreversibility, reconstruction, and head/layer dynamics

A central issue in token eviction is whether the decision is permanent. Token Sparse Attention was introduced partly in response to methods that “permanently evict tokens at specific layers,” because token importance shifts significantly across layers and different heads prioritize different tokens [2602.03216]. TSA therefore compresses per-head $Q,K,V$ to a reduced token set, performs dense attention on that compact subset, and then decompresses the output back to the original sequence by scattering the attended vectors into an $L\times d$ buffer and adding the residual connection. Unselected positions remain zero before the residual, so tokens skipped in one layer remain available in the next [2602.03216].

The empirical motivation for this interleaved design is explicit. TSA reports only $\sim 50\%$ overlap in top-1% tokens between layers separated by $>4$, and uses this observation to argue that “permanent eviction in early layers cannot capture these dynamics” [2602.03216]. The same concern appears in LazyEviction, which reports that $>95\%$ of tokens in reasoning tasks exhibit $MRI > 1$, meaning that a large majority of tokens regain high attention after at least one intervening step [2506.15969]. These observations directly challenge the assumption that low current attention implies long-term irrelevance.

Other methods preserve recoverability by explicit rebuilding. ADORE keeps a fixed-size KV cache, evicts the lowest-score token in the cache, but also selects $R$ high-score tokens from a released-token pool and rebuilds their KV states in parallel, so that discarded tokens “may become essential for future decoding” [2407.02328]. This makes token release and token reconstruction part of the same mechanism rather than opposite design choices.

By contrast, some systems apply logical rather than physical eviction. TidalDecode states that it never physically deletes K-V rows from the cache; the cache remains size $n$. What is “evicted” is the act of loading and multiplying with all $n$ tokens: sparse layers only load the selected $k \ll n$ rows into registers, and the selected positions are reused across many consecutive layers through position-persistent sparse attention [2410.05076]. This suggests that token eviction spans a spectrum from permanent cache deletion to temporary attention bypass to inactive storage.

## 4. Systems and hardware implications

Sparse attention with token eviction is often presented as an algorithmic reduction from $O(N)$ or $O(N^2)$ to budgeted access, but several papers show that the systems bottleneck is not eliminated automatically. Dynamic sparse attention is a clear example. Its token-dependent top-$k$ pattern fragments the KV working set, and only $\sim 50$–$60\%$ of the top-$k$ entries carry over from one token to the next. Measured persistence is only $1.82$ on average with $\sigma=2.86$, “new lookups” per step average $0.55\!\cdot\!k$, and naïve DSA on H200 drops HBM utilization from $36\%$ to $7.3\%$ and SM utilization from $23.5\%$ to $10.9\%$ [2603.13430].

The proposed architectural remedy is a last-level cache reservation system. The LL cache is subdivided into a normal region and a fully-associative, token-granular reserved region, controlled by KV cache metadata and an LRU controller. When a token index in $\Omega_t$ hits the reserved region, it is served from L2; on a miss, a 64 B key+value pair is fetched from HBM into the reserved region, and tokens are evicted by LRU if capacity is exceeded. With a small reserved capacity, the system keeps “hot” top-$k$ tokens in L2 for the next $1$–$2$ steps [2603.13430]. The reported effect is substantial: without reservation, decode is $1.87\times$ slower than an ideal contiguous gather; reserving 10 MB reduces slowdown to $1.50\times$, and reserving 20 MB reduces it to $1.15\times$ [2603.13430].

A different systems strategy is kernel fusion and deep integration with existing dense kernels. TSA is “fully compatible with dense attention implementations, including Flash Attention,” because compression and decompression happen outside the attention kernel and the compacted $Q,K,V$ tensors are dense and contiguous in memory [2602.03216]. Cumulative-energy filtering for video DiTs is even more tightly integrated with Flash Attention: the method reuses the running maximum and log-sum-exp already maintained by the fused tilewise softmax, so “no extra pre-masking, sorting or scanning is needed” [2606.16317]. NSA likewise frames sparse attention as a hardware-aligned design problem, using blockwise access, coalesced HBM loads, and custom Triton kernels to sustain arithmetic intensity while selecting only a small subset of tokens [2502.11089].

Speculative pipelines add yet another systems dimension. STS uses the draft model’s attention scores as a mask that is “known in advance,” which allows mask generation to overlap with target computation and enables KV prefetching from CPU memory [2605.15508]. In this setting, token eviction is coupled not only to attention reduction but also to asynchronous scheduling.

## 5. Empirical regimes and application domains

In long-context autoregressive LLM inference, token eviction has been used to push sequence length far beyond conventional dense-attention budgets. One $k$-attention system reports inference on context windows up to 1M tokens using approximately 16GB of GPU RAM; by attending to less than 2% of input tokens, it achieves over 95% of model performance on RULER, AlpacaEval, and Open LLM Leaderboard, with 3–5× throughput gains on long contexts such as 100k+ [2502.06766]. Tactic reports up to $7.29\times$ decode attention speedup and an overall $1.58\times$ end-to-end inference speedup [2502.12216]. RTPurbo reports up to a $9.36\times$ prefill speedup at 1M context and about a $2.01\times$ decode speedup, while adapting a full-attention model within a few hundred training steps [2605.16928]. STS reports a $2.67\times$ speedup at approximately $90\%$ sparsity on NarrativeQA, with negligible accuracy degradation compared to dense attention [2605.15508].

Interleaved and native sparse approaches emphasize accuracy-latency trade-offs rather than cache compression alone. TSA reports up to $\times 3.23$ attention speedup at 128K context with less than 1% accuracy degradation, and shows that the method can be composed with existing sparse kernels such as FlexPrefill [2602.03216]. NSA reports that its dynamic hierarchical sparse attention uses only $5$–$10\%$ of tokens for attention in practice, while matching or exceeding full attention on general benchmarks, long-context tasks, and instruction-based reasoning [2502.11089]. TidalDecode, which freezes token sets across many layers and refreshes them at a small number of token-selection layers, reports up to $2.1\times$ end-to-end decoding speed-up over full attention [2410.05076].

Reasoning workloads expose a different operating point. LazyEviction reports KV-cache reduction of $50\%{\sim}70\%$ while maintaining comparable accuracy, motivated by the recurrence of previously important tokens in chain-of-thought traces [2506.15969]. FASA reports nearly $100\%$ of full-KV performance on LongBench-V1 when only keeping 256 tokens, and a $2.56\times$ speedup using just $18.9\%$ of the cache on AIME24 [2602.03152]. SAGE-KV reports $4\times$ higher memory efficiency than StreamLLM and $2\times$ higher memory efficiency than Quest while maintaining accuracy comparable to full attention on LongBench [2503.08879].

The same design pattern has also spread beyond decoder-only LLMs. In diffusion LLMs, Sparse-dLLM combines sparse attention with delayed bidirectional sparse caching and reports up to $10\times$ higher throughput than vanilla dLLMs, with comparable performance and similar peak memory costs [2508.02558]. In video generation, cumulative-energy filtering on Wan 2.2 raises sparsity from $61.42\%$ to $82\%$ with a VBench metric drop of less than $5\%$, producing a $1.61\times$ increase in computational efficiency [2606.16317]. In multimodal LLMs, ZipR1 reduces the token ratio of Qwen2/2.5-VL from $80\%$ to $25\%$ with a minimal accuracy reduction on 13 image and video benchmarks [2504.18579]. In streaming visual geometry transformers, Evict3R reduces peak memory from 18.63 GB to 9.39 GB on 7-Scenes with long sequences while accuracy and completeness drop by only 0.003 [2509.17650]. In spiking transformers, SparseSpikformer combines token and weight pruning and cuts down GFLOPs by $20\%$ while maintaining the accuracy of the original model [2311.08806].

## 6. Limitations, misconceptions, and open directions

A recurring misconception is that sparse attention with token eviction is only a question of picking fewer tokens. Multiple papers argue that the decisive issue is *which* tokens can be omitted without breaking retrieval, reasoning, or output fidelity. Tactic explicitly criticizes fixed token budgets for overlooking variations in importance across heads, layers, and contexts [2502.12216]. TSA argues that permanent eviction may “retain irrelevant tokens or rely on irreversible early decisions despite the layer-/head-wise dynamics of token importance” [2602.03216]. LazyEviction shows that periodic reactivation of previously unimportant tokens is common in long reasoning, so purely myopic rules can cause “knowledge discontinuity” [2506.15969].

Another misconception is that training-free methods fully solve the prefill problem. Long-context $k$-attention notes that pre-fill still requires either large-GPU precomputation or distributed schemes such as Ring Attention to build the initial KV cache [2502.06766]. RTPurbo still retains the full KV cache for retrieval heads during the prefill stage [2605.16928]. This suggests that sparse decode and sparse prefill remain distinct engineering problems even when both are described as token eviction.

Value-aware and recovery-aware methods also expose open algorithmic questions. CAOTE is greedy: it optimizes the immediate output error of single-token or blockwise eviction, but does not optimize the joint error of removing multiple tokens simultaneously [2504.14051]. ADORE requires controller training and top-$K$ label collection from a full-attention model, adding engineering complexity [2407.02328]. laLTE preserves constant $O(1)$ time and space complexity of linear attention, but its learnable eviction policy still depends on end-to-end training and specialized kernels [2510.20787].

Current papers make several future directions explicit. DSA work proposes “future architectural and algorithmic exploration to improve serving of DSA on modern inference platforms” [2603.13430]. RTPurbo identifies fully sparse prefill and adaptive head routing as open extensions [2605.16928]. CAOTE points to multi-token combinatorial selection and adaptive norm choice [2504.14051]. FASA emphasizes that a one-time, head-specific calibration of dominant frequency-chunks generalizes across tasks, suggesting that more structure may remain to be extracted from positional encodings and head specialization [2602.03152]. A plausible implication is that the next generation of token-eviction systems will combine query-aware selection, recoverability, and hardware-aware serving rather than treating sparsity as a masking rule in isolation.

Source: https://www.emergentmind.com/topics/sparse-attention-with-token-eviction