Sparse Attention with Token Eviction

• Sparse attention with token eviction is a mechanism that selectively retains a limited subset of tokens from the cache, reducing quadratic compute and memory demands in transformer models.
• It employs diverse selection criteria—including fixed-budget top-k, cumulative mass thresholds, and proxy signals—to dynamically assess token importance and decide which tokens to keep or evict.
• Empirical results demonstrate significant speedups (up to 9×) and memory efficiency improvements while maintaining accuracy comparable to full attention, especially in long-context inference scenarios.

Searching arXiv for papers on sparse attention with token eviction to ground the article in the current literature. Sparse attention with token eviction refers to attention mechanisms that selectively retain a small subset of tokens or cached key-value (KV) pairs for attention computation while dropping, releasing, compressing, or bypassing the remainder. In contemporary long-context inference, this design appears in query-aware top-$k$ retrieval from KV caches, dynamic top-$p$ or cumulative-mass selection, one-time post-prefill cache compression, compress-attend-decompress layers, and hardware-aligned sparse kernels. Across these variants, the common objective is to replace full-cache attention with budgeted access that preserves accuracy comparable to full attention while reducing compute, memory, or bandwidth demands (Synk et al., 10 Feb 2025, Jo et al., 3 Feb 2026, Wang et al., 11 Mar 2025).

1. Core formulation

In autoregressive decoding, the canonical setup consists of a new query vector and a growing cache of past keys and values. A representative dynamic sparse formulation attaches a lightweight indexer with $H_i$ heads and projection dimension $d_i$, computes per-token scores

$$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$

selects

$\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$

and performs standard attention only on the selected subset,

$$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$

Over an entire sequence, this incurs only $O(k\!\cdot\! t)$ bandwidth per token instead of $O(t^2)$ (Levy, 13 Mar 2026).

A related long-context design maintains two disjoint pools per layer: a prefilled cache stored off-GPU, $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$, and a small on-GPU window of recently generated tokens, $p$0. With query $p$1, it selects only the $p$2 most relevant keys from the large cache via

$p$3

computes sparse attention on those entries, computes full attention on the recent window, and forms

$p$4

In this setting, full attention per new token has compute $p$5 and GPU memory $p$6, whereas $p$7-attention stores only the generated window and retrieved values on GPU, with total GPU memory $p$8 (Synk et al., 10 Feb 2025).

These formulations make the essential point precise: token eviction is not a single algorithm, but a budgeting principle. The budget may be expressed as a fixed token count $p$9, a retained fraction, a cumulative attention mass target, or a cache capacity $H_i$0, but the operational effect is the same: only a subset of past tokens participates in the expensive attention path.

2. Selection and eviction criteria

The most direct criterion is fixed-budget top-$H_i$1. This appears in dynamic sparse attention indexers, in long-context $H_i$2-attention, and in one-time cache compression methods. SAGE-KV, for example, performs a single pass of top-$H_i$3 selection immediately after prefill, allocating the token budget $H_i$4 into sink, recent, and selected tokens, then forming a reduced cache

$H_i$5

so that subsequent decoding uses $H_i$6 per layer instead of $H_i$7 (Wang et al., 11 Mar 2025).

A second class replaces fixed budgets with cumulative-mass constraints. Tactic chooses the smallest subset $H_i$8 such that

$H_i$9

thereby selecting tokens by a target fraction of total attention mass rather than a fixed count (Zhu et al., 17 Feb 2025). RTPurbo similarly uses dynamic top-$d_i$0: for each retrieval head it selects the smallest set $d_i$1 such that

$d_i$2

with the size $d_i$3 varying by query (Zhou et al., 16 May 2026). In video DiTs, cumulative-energy filtering derives a decision score

$d_i$4

and skips a tile whenever $d_i$5, with the threshold tied to a recall constraint rather than a fixed token count (Li et al., 15 Jun 2026).

A third class uses proxy signals that are cheaper than full attention. FASA identifies head-specific “dominant” frequency-chunks in RoPE, computes token-importance prediction only on those chunks, and then performs focused full-dimensional attention on the retained subset (Wang et al., 3 Feb 2026). STS uses a smaller draft model inside speculative decoding as a predictor of which tokens and heads matter for a larger target model, constructing a binary mask before the target attention kernels run (Xu et al., 15 May 2026).

A fourth class defines eviction directly in terms of output perturbation or recurrence rather than score magnitude. CAOTE derives a closed-form eviction score

$d_i$6

and evicts the token with the smallest $d_i$7, thereby combining the query-key alignment $d_i$8 with the value-vector deviation $d_i$9 (Goel et al., 18 Apr 2025). LazyEviction introduces Token Importance Recurrence and uses MRI-centric heuristics,

$$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$0

to preserve tokens that may regain importance after several decoding steps (Zhang et al., 19 Jun 2025).

Finally, some systems learn eviction policies explicitly. In laLTE, each head computes retention scores $$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$1, binarizes them at $$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$2, and attends only to tokens in

$$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$3

thereby combining sink tokens, a sliding window, and a learnable retained cache (He et al., 23 Oct 2025).

3. Irreversibility, reconstruction, and head/layer dynamics

A central issue in token eviction is whether the decision is permanent. Token Sparse Attention was introduced partly in response to methods that “permanently evict tokens at specific layers,” because token importance shifts significantly across layers and different heads prioritize different tokens (Jo et al., 3 Feb 2026). TSA therefore compresses per-head $$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$4 to a reduced token set, performs dense attention on that compact subset, and then decompresses the output back to the original sequence by scattering the attended vectors into an $$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$5 buffer and adding the residual connection. Unselected positions remain zero before the residual, so tokens skipped in one layer remain available in the next (Jo et al., 3 Feb 2026).

The empirical motivation for this interleaved design is explicit. TSA reports only $$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$6 overlap in top-1% tokens between layers separated by $$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$7, and uses this observation to argue that “permanent eviction in early layers cannot capture these dynamics” (Jo et al., 3 Feb 2026). The same concern appears in LazyEviction, which reports that $$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$8 of tokens in reasoning tasks exhibit $$S_{t,s} = \sum_{j=1}^{H_i} w_{t,j}^{(i)} \cdot \mathrm{ReLU}( q_{t,j}^{(i)} \cdot k_s^{(i)} ),$$9, meaning that a large majority of tokens regain high attention after at least one intervening step (Zhang et al., 19 Jun 2025). These observations directly challenge the assumption that low current attention implies long-term irrelevance.

Other methods preserve recoverability by explicit rebuilding. ADORE keeps a fixed-size KV cache, evicts the lowest-score token in the cache, but also selects $\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$0 high-score tokens from a released-token pool and rebuilds their KV states in parallel, so that discarded tokens “may become essential for future decoding” (Zhang et al., 2024). This makes token release and token reconstruction part of the same mechanism rather than opposite design choices.

By contrast, some systems apply logical rather than physical eviction. TidalDecode states that it never physically deletes K-V rows from the cache; the cache remains size $\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$1. What is “evicted” is the act of loading and multiplying with all $\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$2 tokens: sparse layers only load the selected $\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$3 rows into registers, and the selected positions are reused across many consecutive layers through position-persistent sparse attention (Yang et al., 2024). This suggests that token eviction spans a spectrum from permanent cache deletion to temporary attention bypass to inactive storage.

4. Systems and hardware implications

Sparse attention with token eviction is often presented as an algorithmic reduction from $\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$4 or $\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$5 to budgeted access, but several papers show that the systems bottleneck is not eliminated automatically. Dynamic sparse attention is a clear example. Its token-dependent top-$\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$6 pattern fragments the KV working set, and only $\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$7–$\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$8 of the top-$\Omega_t = \mathrm{Top}\text{-}k_s \; S_{t,s},$9 entries carry over from one token to the next. Measured persistence is only $$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$0 on average with $$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$1, “new lookups” per step average $$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$2, and naïve DSA on H200 drops HBM utilization from $$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$3 to $$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$4 and SM utilization from $$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$5 to $$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$6 (Levy, 13 Mar 2026).

The proposed architectural remedy is a last-level cache reservation system. The LL cache is subdivided into a normal region and a fully-associative, token-granular reserved region, controlled by KV cache metadata and an LRU controller. When a token index in $$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$7 hits the reserved region, it is served from L2; on a miss, a 64 B key+value pair is fetched from HBM into the reserved region, and tokens are evicted by LRU if capacity is exceeded. With a small reserved capacity, the system keeps “hot” top-$$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$8 tokens in L2 for the next $$\mathrm{Attention}_t = \mathrm{softmax}( q_t \cdot K_{\Omega_t}^\top/\sqrt{d_k} ) V_{\Omega_t}.$$9–$O(k\!\cdot\! t)$0 steps (Levy, 13 Mar 2026). The reported effect is substantial: without reservation, decode is $O(k\!\cdot\! t)$1 slower than an ideal contiguous gather; reserving 10 MB reduces slowdown to $O(k\!\cdot\! t)$2, and reserving 20 MB reduces it to $O(k\!\cdot\! t)$3 (Levy, 13 Mar 2026).

A different systems strategy is kernel fusion and deep integration with existing dense kernels. TSA is “fully compatible with dense attention implementations, including Flash Attention,” because compression and decompression happen outside the attention kernel and the compacted $O(k\!\cdot\! t)$4 tensors are dense and contiguous in memory (Jo et al., 3 Feb 2026). Cumulative-energy filtering for video DiTs is even more tightly integrated with Flash Attention: the method reuses the running maximum and log-sum-exp already maintained by the fused tilewise softmax, so “no extra pre-masking, sorting or scanning is needed” (Li et al., 15 Jun 2026). NSA likewise frames sparse attention as a hardware-aligned design problem, using blockwise access, coalesced HBM loads, and custom Triton kernels to sustain arithmetic intensity while selecting only a small subset of tokens (Yuan et al., 16 Feb 2025).

Speculative pipelines add yet another systems dimension. STS uses the draft model’s attention scores as a mask that is “known in advance,” which allows mask generation to overlap with target computation and enables KV prefetching from CPU memory (Xu et al., 15 May 2026). In this setting, token eviction is coupled not only to attention reduction but also to asynchronous scheduling.

5. Empirical regimes and application domains

In long-context autoregressive LLM inference, token eviction has been used to push sequence length far beyond conventional dense-attention budgets. One $O(k\!\cdot\! t)$5-attention system reports inference on context windows up to 1M tokens using approximately 16GB of GPU RAM; by attending to less than 2% of input tokens, it achieves over 95% of model performance on RULER, AlpacaEval, and Open LLM Leaderboard, with 3–5× throughput gains on long contexts such as 100k+ (Synk et al., 10 Feb 2025). Tactic reports up to $O(k\!\cdot\! t)$6 decode attention speedup and an overall $O(k\!\cdot\! t)$7 end-to-end inference speedup (Zhu et al., 17 Feb 2025). RTPurbo reports up to a $O(k\!\cdot\! t)$8 prefill speedup at 1M context and about a $O(k\!\cdot\! t)$9 decode speedup, while adapting a full-attention model within a few hundred training steps (Zhou et al., 16 May 2026). STS reports a $O(t^2)$0 speedup at approximately $O(t^2)$1 sparsity on NarrativeQA, with negligible accuracy degradation compared to dense attention (Xu et al., 15 May 2026).

Interleaved and native sparse approaches emphasize accuracy-latency trade-offs rather than cache compression alone. TSA reports up to $O(t^2)$2 attention speedup at 128K context with less than 1% accuracy degradation, and shows that the method can be composed with existing sparse kernels such as FlexPrefill (Jo et al., 3 Feb 2026). NSA reports that its dynamic hierarchical sparse attention uses only $O(t^2)$3–$O(t^2)$4 of tokens for attention in practice, while matching or exceeding full attention on general benchmarks, long-context tasks, and instruction-based reasoning (Yuan et al., 16 Feb 2025). TidalDecode, which freezes token sets across many layers and refreshes them at a small number of token-selection layers, reports up to $O(t^2)$5 end-to-end decoding speed-up over full attention (Yang et al., 2024).

Reasoning workloads expose a different operating point. LazyEviction reports KV-cache reduction of $O(t^2)$6 while maintaining comparable accuracy, motivated by the recurrence of previously important tokens in chain-of-thought traces (Zhang et al., 19 Jun 2025). FASA reports nearly $O(t^2)$7 of full-KV performance on LongBench-V1 when only keeping 256 tokens, and a $O(t^2)$8 speedup using just $O(t^2)$9 of the cache on AIME24 (Wang et al., 3 Feb 2026). SAGE-KV reports $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$0 higher memory efficiency than StreamLLM and $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$1 higher memory efficiency than Quest while maintaining accuracy comparable to full attention on LongBench (Wang et al., 11 Mar 2025).

The same design pattern has also spread beyond decoder-only LLMs. In diffusion LLMs, Sparse-dLLM combines sparse attention with delayed bidirectional sparse caching and reports up to $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$2 higher throughput than vanilla dLLMs, with comparable performance and similar peak memory costs (Song et al., 4 Aug 2025). In video generation, cumulative-energy filtering on Wan 2.2 raises sparsity from $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$3 to $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$4 with a VBench metric drop of less than $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$5, producing a $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$6 increase in computational efficiency (Li et al., 15 Jun 2026). In multimodal LLMs, ZipR1 reduces the token ratio of Qwen2/2.5-VL from $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$7 to $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$8 with a minimal accuracy reduction on 13 image and video benchmarks (Chen et al., 23 Apr 2025). In streaming visual geometry transformers, Evict3R reduces peak memory from 18.63 GB to 9.39 GB on 7-Scenes with long sequences while accuracy and completeness drop by only 0.003 (Mahdi et al., 22 Sep 2025). In spiking transformers, SparseSpikformer combines token and weight pruning and cuts down GFLOPs by $K_\ell,V_\ell \in \mathbb{R}^{N\times D}$9 while maintaining the accuracy of the original model (Liu et al., 2023).

6. Limitations, misconceptions, and open directions

A recurring misconception is that sparse attention with token eviction is only a question of picking fewer tokens. Multiple papers argue that the decisive issue is which tokens can be omitted without breaking retrieval, reasoning, or output fidelity. Tactic explicitly criticizes fixed token budgets for overlooking variations in importance across heads, layers, and contexts (Zhu et al., 17 Feb 2025). TSA argues that permanent eviction may “retain irrelevant tokens or rely on irreversible early decisions despite the layer-/head-wise dynamics of token importance” (Jo et al., 3 Feb 2026). LazyEviction shows that periodic reactivation of previously unimportant tokens is common in long reasoning, so purely myopic rules can cause “knowledge discontinuity” (Zhang et al., 19 Jun 2025).

Another misconception is that training-free methods fully solve the prefill problem. Long-context $p$00-attention notes that pre-fill still requires either large-GPU precomputation or distributed schemes such as Ring Attention to build the initial KV cache (Synk et al., 10 Feb 2025). RTPurbo still retains the full KV cache for retrieval heads during the prefill stage (Zhou et al., 16 May 2026). This suggests that sparse decode and sparse prefill remain distinct engineering problems even when both are described as token eviction.

Value-aware and recovery-aware methods also expose open algorithmic questions. CAOTE is greedy: it optimizes the immediate output error of single-token or blockwise eviction, but does not optimize the joint error of removing multiple tokens simultaneously (Goel et al., 18 Apr 2025). ADORE requires controller training and top-$p$01 label collection from a full-attention model, adding engineering complexity (Zhang et al., 2024). laLTE preserves constant $p$02 time and space complexity of linear attention, but its learnable eviction policy still depends on end-to-end training and specialized kernels (He et al., 23 Oct 2025).

Current papers make several future directions explicit. DSA work proposes “future architectural and algorithmic exploration to improve serving of DSA on modern inference platforms” (Levy, 13 Mar 2026). RTPurbo identifies fully sparse prefill and adaptive head routing as open extensions (Zhou et al., 16 May 2026). CAOTE points to multi-token combinatorial selection and adaptive norm choice (Goel et al., 18 Apr 2025). FASA emphasizes that a one-time, head-specific calibration of dominant frequency-chunks generalizes across tasks, suggesting that more structure may remain to be extracted from positional encodings and head specialization (Wang et al., 3 Feb 2026). A plausible implication is that the next generation of token-eviction systems will combine query-aware selection, recoverability, and hardware-aware serving rather than treating sparsity as a masking rule in isolation.