Inference-Time Decode Sparsity

• Inference-time decode sparsity is a method that selectively executes operations during token generation to cut compute and memory costs in autoregressive models.
• It employs techniques like KV-cache pruning, sparse attention, and dynamic skipping to optimize performance, especially for long reasoning chains.
• Empirical results show significant latency and throughput improvements, with certain methods achieving up to 2–6× speedups without compromising accuracy.

Inference-time decode sparsity denotes a family of methods that reduce the compute and memory cost of autoregressive generation by executing only a selected subset of the operations that would be performed by dense decoding. In LLMs and related sequence generators, the immediate target is usually the decode stage rather than prefill, because the model has already consumed the prompt and must then generate tokens one by one while the KV cache grows with sequence length. In long reasoning traces, this regime can dominate end-to-end cost: one line of work reports that reasoning tasks can make the decode phase account for 99% of job-completion time, with standard dense decoding incurring $O(N)$ time and $O(N)$ memory per request as the cache grows to length $N$ (Hu et al., 16 Feb 2025). Across the literature, decode sparsity is instantiated through KV-cache pruning, sparse attention over selected context blocks or tokens, activation- and channel-level sparsification in FFN and MoE components, structured removal of rows and columns from weights, and dynamic skipping or reuse of residual computations during test-time generation (Hu et al., 16 Feb 2025, Wang et al., 26 Dec 2025, Svirsky et al., 9 Feb 2026, Ji et al., 13 May 2026).

1. Problem formulation and computational regime

The canonical decode-sparsity setting is token-by-token autoregressive generation after a prompt has been prefed into the model. In this setting, each newly generated token attends to prior tokens stored in the KV cache, so dense attention requires per-step access to all previous context. The simplified asymptotic description used in several papers is $O(N)$ time and $O(N)$ memory per decode step, with total decode latency growing quadratically over a long generation because each step attends to all preceding tokens (Hu et al., 16 Feb 2025). This cost profile is especially acute for mathematical and programmatic reasoning, where the model may produce thousands of intermediate tokens before emitting a final answer (Hu et al., 16 Feb 2025).

Several papers emphasize that decode is not merely compute-heavy but frequently memory-bound. AsyncSpade frames long chain-of-thought test-time scaling as a KV-cache growth problem that amplifies HBM–SRAM I/O pressure and degrades time per output token, while Mustafar argues that KV-cache size is the major bottleneck in decode performance because attention becomes dominated by memory-bound operations over Keys and Values (Luo et al., 8 Oct 2025, Joo et al., 28 May 2025). In hardware-oriented work on sparse block-diagonal LLMs, decode is described as inherently memory-bound on conventional Von-Neumann architectures, with repeated weight movement rather than arithmetic dominating cost (Lima et al., 13 Oct 2025).

The same logic extends beyond ordinary one-token decoding. In speculative decoding, the verification stage can become the dominant bottleneck because the target model must score multiple drafted tokens in parallel, forcing repeated attention over a large KV cache as well as dense FFN and, in MoE models, expert routing (Wang et al., 26 Dec 2025). Decode sparsity therefore refers not only to pruning context access in the standard autoregressive loop, but also to sparsifying the verification, cache-management, and model-execution subroutines that arise in accelerated inference pipelines.

2. Methodological dimensions of decode sparsity

A central distinction in the literature is between static and dynamic sparsity. “Block-wise Dynamic Sparseness” defines dynamic sparseness as input-dependent omission of parts of a matrix-vector multiplication $Wh$, in contrast to static sparseness, which permanently zeroes out selected positions in weight matrices (Hadifar et al., 2020). The dynamic method introduces a gating function $\mathcal{G}(h,\theta,\varsigma)$, applies a top-$k$ operator to choose active entries or blocks, and computes $(\mathcal{G}(h,\theta,\varsigma)\odot W)h$. The block-wise formulation is specifically motivated by inference efficiency, because coarse blocks avoid the indexing overhead and irregular memory access associated with unstructured sparsity (Hadifar et al., 2020).

Later LLM work generalizes this idea to contextual sparsity. “Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time” hypothesizes that for a particular input, only a small input-dependent subset of attention heads and MLP parameters is needed to produce approximately the same output as the dense model, and proposes on-the-fly prediction of these active subsets without retraining the base LLM (Liu et al., 2023). The paper reports around 80% sparsity in attention heads and 95% sparsity in MLP neurons on OPT-175B for a given input, with over $2\times$ latency reduction relative to FasterTransformer and over $O(N)$0 relative to Hugging Face, without compromising model quality (Liu et al., 2023).

A second axis concerns what is being sparsified. Some methods sparsify the context dimension by selecting tokens, blocks, or pages from the KV cache; others sparsify activations within FFNs or MoE routing; others compress the base model itself. FineGates belongs to the latter category: it learns binary stochastic gates over entire rows and columns of pretrained weight matrices, producing a structured sparse subnetwork that can be permanently compressed for inference (Svirsky et al., 9 Feb 2026). This is distinct from low-rank adaptation, because the dense base model is not left intact at runtime. The paper repeatedly emphasizes that structured row/column sparsity is what makes decode-time wall-clock gains plausible, whereas unstructured sparsity often fails to map efficiently to standard dense kernels (Svirsky et al., 9 Feb 2026).

A third axis concerns whether sparsity is training-free. Attention-value pruning in transformers offers a training-free example: small attention values $O(N)$1 are replaced by zero at inference, and the remaining values can be quantized to a few discrete levels. On question answering and sentiment-analysis tasks with BERT and RoBERTa, the paper reports that nearly 80% of attention values can be pruned with minimal loss, and that pruning plus 3-bit log-scale quantization retains strong QA accuracy without retraining (Ji et al., 2021). Although this work is not about autoregressive long-decode LLM serving, it established an early inference-time sparsification template: exploit redundancy in already-trained attention computations directly at application time (Ji et al., 2021).

Taken together, these works suggest that “decode sparsity” is not a single algorithmic primitive. It is a broader inference paradigm in which the dense forward pass is treated as an upper bound on required computation rather than a mandatory execution pattern.

3. KV-cache and context sparsity in long reasoning decode

Long reasoning decode has motivated a specialized line of work on sparse KV-cache retention and context selection. “RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning” argues that prior sparse KV-cache methods fail to match the attention dynamics of long chain-of-thought reasoning, where token importance is neither purely recent nor permanently persistent (Hu et al., 16 Feb 2025). Its main empirical observation is a decode-stage “waterfall pattern”: milestone tokens, analogous to lemmas in a proof, receive high attention for a period and then fade permanently. This is contrasted with phoenix tokens, which can become important again after a long interval and are said to occur often in the short prefill prompt (Hu et al., 16 Feb 2025).

RaaS operationalizes this observation with timestamped retention. During each decoding step, if a token’s attention exceeds a threshold $O(N)$2, or equivalently if it lies among the top $O(N)$3 of attention scores in that step, it receives the latest timestamp. When the cache is full, the token or page with the oldest timestamp is evicted. The system retains all prefill tokens without eviction, because reasoning prompts are usually short and phoenix tokens are concentrated there. To match paged inference engines, the implementation is page-based with page_size = 16, uses the same representative selection method as Quest for fairness, and updates timestamps and evictions interleaved with ordinary self-attention so that overhead is negligible (Hu et al., 16 Feb 2025). The resulting claim is high accuracy with $O(N)$4 time and $O(N)$5 memory, in contrast to Quest’s $O(N)$6 time but $O(N)$7 memory (Hu et al., 16 Feb 2025).

The empirical evaluation in RaaS is narrowly focused and technically important. On GSM8k, MATH500, and AIME with Marco-o1, Qwen2.5-Math 7B, Mistral Math 7B, and DeepScaleR 1.5B, the paper reports that RaaS matches Quest in accuracy and latency while exhibiting constant-memory behavior once decode length exceeds the cache budget, and that a cache budget of 1024 tokens is generally enough to match Dense accuracy (Hu et al., 16 Feb 2025). A micro-benchmark further argues that discarding milestone tokens can force re-reasoning and even cause the model to get stuck (Hu et al., 16 Feb 2025).

Mustafar extends KV-cache sparsity in a different direction by promoting unstructured pruning rather than structured token or page eviction. The method prunes Key and Value caches at inference time with per-token magnitude-based unstructured sparsity, keeps the most recent 32 tokens dense, and stores pruned caches in a bitmap-based compressed format. A custom sparse attention kernel then loads compressed tiles, decompresses on chip, and computes directly over the compressed representation (Joo et al., 28 May 2025). The paper reports sparsity levels up to 70% without compromising accuracy or requiring fine-tuning, compressed KV size down to about 45% of dense inference at 70% sparsity, and increased tokens/sec throughput of up to $O(N)$8 compared to dense inference (Joo et al., 28 May 2025).

A broader empirical perspective appears in “Inference Time Context Sparsity: Illusion or Opportunity?”, which studies 20 models across five model families under decode-time sparse attention. Its primary method is exact oracle top-$O(N)$9 selection over the context, complemented by vAttention stochastic selection in selected experiments. The reported trend is that many modern LLMs, especially larger and hybrid models, remain remarkably robust even at aggressive sparsity levels such as $N$0, across retrieval, multi-hop QA, mathematical reasoning, and agentic coding tasks (Joshi et al., 22 May 2026). This does not identify a single optimal eviction policy, but it does support the proposition that context sparsity is not restricted to a narrow class of long-reasoning benchmarks (Joshi et al., 22 May 2026).

4. Sparse verification, speculative decoding, and test-time scaling

Speculative decoding changes where sparsity is most useful. In the standard formulation, a draft model proposes $N$1, the target model computes $N$2, and proposed tokens are accepted sequentially with probability $N$3, reverting to resampling from $N$4 upon rejection (Wang et al., 26 Dec 2025). “Accelerate Speculative Decoding with Sparse Computation in Verification” argues that this setup makes verification, not token proposal, the dominant computational bottleneck, especially for long contexts and MoE targets (Wang et al., 26 Dec 2025).

The proposed sparse verification framework jointly sparsifies attention, FFN, and MoE computation during verification. For attention, the KV cache is partitioned into blocks by head and token span, scored using the first draft token’s query, and only top-$N$5 blocks are retained, while earliest and latest blocks are always preserved because of attention sink and locality effects. For FFNs, channels with $N$6 are treated as inactive and removed from the dominant up- and down-projections. For MoE, the method calibrates layer-specific thresholds that allow skipping up to $N$7 of the $N$8 selected experts per token while preserving high-weight experts (Wang et al., 26 Dec 2025). The paper also identifies two forms of redundancy unique to speculative verification: inter-draft-token redundancy, where different draft tokens retrieve highly overlapping KV blocks, and inter-layer redundancy, where adjacent deeper layers exhibit similar retrieval masks. These observations motivate reuse of the first draft token’s retrieval for later draft tokens and reuse of anchor-layer masks for nearby non-anchor layers (Wang et al., 26 Dec 2025).

Empirically, the paper reports favorable efficiency–accuracy trade-offs on summarization, QA, code completion, and mathematical reasoning, with stable acceptance length under moderate sparsity. In the hybrid sparse configuration, acceptance lengths remain close to the strict baseline, such as 2.42 vs. 2.44 on GovReport and 2.70 vs. 2.71 on 2WikiMQA (Wang et al., 26 Dec 2025). This matters because speculative decoding speed depends jointly on verification cost and the number of draft tokens accepted per step.

Test-time scaling in long chain-of-thought generation introduces a related but distinct systems problem: query-aware sparse decoding itself can become a bottleneck if KV selection must wait for the current query. “AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding” addresses this by predicting the next-token query state from a short window of recent queries, using a lightweight temporal-regressive module based on ridge regression, and overlapping KV filtering with forward inference in a disaggregated two-rank pipeline (Luo et al., 8 Oct 2025). The method claims to eliminate the sequential dependence of prior query-aware sparse decoding without sacrificing model performance, and reports over 20% TPOT reduction relative to Quest and at least 50% relative to full attention on Qwen3-8B and Qwen3-32B while matching or surpassing their accuracy on AIME-24/25, GPQA-Diamond, and MATH-500 (Luo et al., 8 Oct 2025).

These verification- and serving-oriented methods shift decode sparsity from a purely algorithmic selection problem to a pipeline design problem. The sparse subset must not only preserve accuracy; it must also be computable, transferable, and reusable without reintroducing the very latency that sparsification was intended to remove.

5. Systems realization and hardware co-design

The practical value of decode sparsity depends on whether the sparse pattern can be executed faster than the dense baseline on real hardware. DejaVu makes this point explicitly: LLM inference at small batch is described as memory-bandwidth or I/O bound rather than compute bound, so sparsity helps only when it reduces memory movement and is supported by efficient sparse kernels (Liu et al., 2023). Its implementation therefore fuses indexing and multiplication in Triton kernels and stores certain matrices in layouts that make selected sparse slices contiguous. The paper reports sparse MLP up to $N$9 faster than a PyTorch baseline and sparse attention up to $O(N)$0 faster, yielding end-to-end gains on OPT-175B (Liu et al., 2023).

Mustafar reaches a similar conclusion from the KV-cache side. Its bitmap-based compressed format uses $O(N)$1 tiles with 64-bit bitmaps and tile offsets, and its kernel follows a load-as-compressed, compute-as-dense strategy in which compressed tiles are loaded from global memory, decompressed into shared memory, and then multiplied densely (Joo et al., 28 May 2025). The paper reports that this reduction in memory movement is large enough to offset pruning and compression overhead, with sparse matrix–vector cost falling to 61.87% of dense cuBLAS execution time at 70% sparsity for Llama-2-7B, and throughput gains up to $O(N)$2 at equal batch size and $O(N)$3 when the reduced KV footprint enables larger effective batch (Joo et al., 28 May 2025).

“At scale” sparse decode kernels for irregular token-level context selection are reported in “Inference Time Context Sparsity: Illusion or Opportunity?”. Built on FlashInfer and evaluated on NVIDIA H100 80GB HBM3 with fp16, $O(N)$4, $O(N)$5, head dimension $O(N)$6, page size 16, and NHD layout, the backend is said to support per-query, per-head, token-level sparsity without requiring block structure (Joshi et al., 22 May 2026). The paper reports speedups over FlashInfer ranging roughly from $O(N)$7 to $O(N)$8 at $O(N)$9 sparsity depending on batch size, and up to $O(N)$0 at $O(N)$1 sparsity is highlighted in the abstract (Joshi et al., 22 May 2026).

Hardware–algorithm co-design appears again in work on compute-in-memory acceleration of sparse block-diagonal LLMs. There, structured sparsity is expressed with Monarch matrices $O(N)$2, where $O(N)$3 and $O(N)$4 are block-diagonal and only parameterized matmuls in MHA and FFN are transformed (Lima et al., 13 Oct 2025). The key issue is not merely having fewer nonzeros but mapping them efficiently to crossbar arrays. The DenseMap strategy reportedly raises average array utilization to 78.8%, requires 87% fewer CIM arrays than a linear baseline, and improves latency and energy by $O(N)$5 and $O(N)$6, respectively, relative to dense mapping (Lima et al., 13 Oct 2025). This line of work shows that decode sparsity is increasingly inseparable from storage layout, ADC-sharing regime, and scheduling policy (Lima et al., 13 Oct 2025).

An adjacent systems example outside autoregressive LLMs is test-time sparsity for action diffusion. There, the method dynamically predicts which self-attention, cross-attention, and FFN residual computations can be skipped for each denoising step, reuses cached features along block, timestep, and rollout dimensions, and overlaps pruner and decoder execution (Ji et al., 13 May 2026). The paper reports 95% sparsity, 92% FLOPs reduction, $O(N)$7 wall-clock speedup, and 47.5 Hz inference frequency without performance degradation in the best settings (Ji et al., 13 May 2026). This suggests that decode-sparsity ideas are generalizing from LLM token generation to other iterative generative loops.

6. Failure modes, theoretical perspectives, and open directions

A persistent misconception is that lower per-token decode cost necessarily reduces end-to-end inference cost. “Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage” directly contests that assumption (Hu et al., 6 Jan 2026). The paper argues that sparse attention can induce information loss severe enough to make the model generate longer, more redundant outputs, summarized by the relation

$O(N)$8

Across GSM8K, MATH-500, and AIME with DSR, DSL, and Qwe, sparse methods are reported to generate outputs up to 90% longer than full attention, even though per-token time is lower (Hu et al., 6 Jan 2026). The paper interprets this as information loss followed by attempted reconstruction, including cases in which the model produces the correct answer early but continues reasoning and later forgets it (Hu et al., 6 Jan 2026).

Lil proposes Guardian, an early-stopping algorithm that periodically compresses the running sequence with LZ77 and stops when the increase in compressed length falls below a threshold. With $O(N)$9 and $Wh$0, the paper reports up to 90% token savings with less than 2% accuracy degradation across reasoning-intensive benchmarks (Hu et al., 6 Jan 2026). This is not itself a sparsification method, but it reframes decode sparsity as a closed-loop control problem: if sparsification alters sequence dynamics, the stopping rule may need to change as well.

Another limitation is model dependence. The large 20-model study on context sparsity reports that smaller standard transformer models can degrade substantially under deterministic OracleTopK, whereas hybrid families such as Qwen3.5 and Gemma3 are markedly more robust at aggressive sparsity levels (Joshi et al., 22 May 2026). The same paper identifies agentic coding failure modes in which sparse runs enter degenerate command loops, and also notes serving-stack artifacts such as InternalServerError, Timeout, and occasional git apply rejection, indicating that sparse-vs-dense gaps may partly reflect infrastructure instability rather than pure model-quality loss (Joshi et al., 22 May 2026). FineGates, for its part, demonstrates inference-time speedups only through CPU validation wall-clock measurements rather than a dedicated GPU autoregressive decode benchmark, and explicitly notes that speedups depend on whether the runtime actually exploits the learned structured sparsity (Svirsky et al., 9 Feb 2026). Mustafar similarly reports that batch size 1 can be slower than dense because the sparse kernel underutilizes the GPU (Joo et al., 28 May 2025).

Theoretical arguments have begun to shift the discourse from heuristic efficiency to representational necessity. “Inference Time Context Sparsity: Illusion or Opportunity?” proves that if $Wh$1 and $Wh$2, then the map $Wh$3 from the attention simplex to the post-attention output is not injective, so distinct dense attention distributions can collapse to the same output vector (Joshi et al., 22 May 2026). The paper interprets this as evidence that dense attention over very long contexts is already an inherently lossy compression process, making extreme context sparsity a principled rather than ad hoc direction (Joshi et al., 22 May 2026). A plausible implication is that future training objectives and architectures may explicitly encourage sparse retrieval and sparse state usage instead of treating dense attention as the normative reference.

Inference-time decode sparsity has therefore evolved from simple pruning heuristics into a multi-layer research area spanning reasoning-aware cache retention, contextual and unstructured sparse execution, speculative verification, asynchronous serving, and hardware-aware kernel design. Its central question is no longer whether some tokens, channels, experts, or residual paths can be omitted, but under what algorithmic, systems, and architectural conditions such omission yields lower true end-to-end cost without destabilizing long-horizon generation.