Sparse-dLLM: Dynamic Cache for Diffusion LLMs

• Sparse-dLLM is a training-free inference framework that accelerates diffusion LLM decoding by dynamically pruning low-salience tokens.
• It employs delayed bidirectional sparse caching, leveraging stable attention patterns to retain only pivotal tokens during iterative denoising.
• Empirical results demonstrate up to 10× throughput gains with comparable reasoning performance and reduced memory usage on benchmark tasks.

Sparse-dLLM is a training-free inference framework for diffusion LLMs that accelerates decoding by combining dynamic cache eviction with sparse attention through delayed bidirectional sparse caching. It was introduced for masked or discrete diffusion LLMs such as LLaDA and Dream, whose bidirectional denoising procedure repeatedly reprocesses the full sequence and therefore incurs prohibitive quadratic attention cost and substantial memory overhead during inference. Its central premise is that token saliency in dLLM attention is temporally stable across decoding steps: pivotal tokens tend to remain salient, while low-relevance tokens remain unimportant, making selective retention of key–value states feasible without retraining (Song et al., 4 Aug 2025).

1. Problem setting and conceptual basis

Sparse-dLLM is defined in the setting of diffusion LLMs that generate by iterative denoising rather than left-to-right autoregression. A typical dLLM begins from a sequence of prompt tokens followed by masked positions,

$$\mathbf{x}^T = (c_0,\dots,c_{p-1},[\text{MASK}],\dots,[\text{MASK}]),$$

then repeatedly applies a transformer $f_\theta$ and a transition operator $S$: $$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$ Because attention is bidirectional at every denoising step, the model recomputes attention over the entire sequence of length $L$ instead of appending one token to a causal KV cache as in autoregressive decoding. Per-layer self-attention cost per step is therefore

$O(HL^2 d_k) \approx O(L^2 d),$

and over $T$ diffusion steps the overall complexity scales as

$O(TL^2 d).$

Earlier dLLM acceleration methods adapted KV caching to diffusion inference, but dense caches still stored full-layer states or all KV states outside the current decoding block, which reduced computation while preserving a large cache footprint. Sparse-dLLM addresses this mismatch by treating the cache itself as a sparsifiable object. The method does not modify model parameters or training objectives; instead, it prunes the set of cached tokens online according to attention-derived token importance.

The conceptual basis is an empirical analysis of dLLM attention maps. Across layers and denoising steps, the authors observed two stable structures: bright diagonals indicating local attention and bright vertical bands indicating a small set of pivotal tokens that attract attention from many positions. Most other positions remain dark and low-salience. This persistent cross-layer sparsity, and its consistency across steps, motivates dynamic cache eviction: low-salience tokens outside the current block can be removed from the cache with limited impact on quality (Song et al., 4 Aug 2025).

2. Attention saliency and dynamic cache eviction

Sparse-dLLM operates in a block-wise decoding regime. Let $b$ denote block length, $p$ prompt length, $f_\theta$0 full sequence length, and $f_\theta$1 the starting index of the current block. For a layer with key and value states $f_\theta$2, the candidate tokens for caching are all positions outside the current block: $f_\theta$3

The main text describes saliency scoring through attention between current-block queries $f_\theta$4 and these outside-block keys,

$f_\theta$5

while the appendix algorithm averages the block queries before scoring. In both descriptions, the essential objective is to estimate which prefix and suffix tokens are most informative for subsequent steps in the current block.

Sparse-dLLM then aggregates these attention scores with one-dimensional max pooling: $f_\theta$6 where $f_\theta$7 is a pooling kernel size. If the retention ratio is $f_\theta$8, the number of retained tokens is

$f_\theta$9

and the kept token indices are selected by top-$S$0: $S$1 The sparse cache for that layer is then

$S$2

Two aspects are technically distinctive. First, the eviction is bidirectional. Unlike autoregressive cache-pruning methods that only remove prefix tokens, Sparse-dLLM prunes both prefix and suffix entries because dLLMs attend bidirectionally. Second, the scoring is attention-guided rather than heuristic in the abstract sense of distance or mask position. The method therefore treats sparse caching and sparse attention as one mechanism: the cache contains only retained pivotal tokens, and subsequent attention is computed only over the current block plus that compressed cache.

This implies a different scaling law from dense caching. On cache-reuse steps, full attention from a block of size $S$3 to the entire sequence of size $S$4 becomes attention from $S$5 queries to $S$6 keys. In the large-$S$7 regime, the dominant term is reduced from $S$8 to approximately $S$9, with the compression factor governed by the retention ratio.

3. Delayed bidirectional sparse caching in the inference loop

Sparse-dLLM uses what the paper terms delayed bidirectional sparse caching. The delay addresses a specific empirical phenomenon: outside-block KV states change most strongly between the first two diffusion steps of a block, then become much more stable. To avoid caching unstable states, the method does not construct the sparse cache at the first block step.

The implementation tracks three cache states. When $$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$0, the model performs full attention and does not use a sparse cache. When $$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$1, it updates the cache using the dynamic eviction procedure described above. When $$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$2, it reuses the existing sparse cache. For the one-step delayed policy used in the main experiments, the schedule is:

• at block step $$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$3, no cache;
• at block step $$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$4, build the sparse cache;
• at later block steps, reuse the sparse cache.

The paper also gives an $$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$5-step generalization,

$$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$6

but the reported ablations identify one-step delay as the best trade-off between quality and efficiency.

During cache reuse, a layer concatenates the retained external KV states with the current block’s freshly computed KV: $$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$7 and block attention is then evaluated as

$$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$8

This procedure is repeated independently per layer, so sparsity decisions are layer-specific rather than globally shared.

A common misunderstanding is to view Sparse-dLLM as merely a memory-saving variant of dense dLLM caching. In fact, its cache management is inseparable from its computation model. Tokens not selected into $$\mathbf{z}^t = f_\theta(\mathbf{x}^t), \qquad \mathbf{x}^{t-1} = S(\hat{\mathbf{x}}^t,\mathbf{x}^t).$$9 are neither stored nor attended to during reuse steps; cache eviction therefore directly reduces attention work, not only memory occupancy.

4. Empirical performance and operating characteristics

Sparse-dLLM was evaluated on LLaDA-8B-Instruct, LLaDA-1.5, Dream-v0-7B-Base, and Dream-v0-7B-Instruct, with standard reasoning, science, and code benchmarks including MMLU, ARC-Challenge, PIQA, GPQA, GSM8K, MATH, and HumanEval. The main configuration used block length $L$0, retention ratio $L$1, and pooling kernel size $L$2, on an NVIDIA RTX 4090 with 48 GB memory (Song et al., 4 Aug 2025).

The framework’s headline claim is up to $L$3 higher throughput than vanilla dLLMs, with comparable performance and similar peak memory costs. On individual tasks, the reported throughput gains are large. For LLaDA-8B-Instruct on GSM8K, throughput rises from 4.57 tokens/s to 26.45 tokens/s; on GPQA it rises from 6.30 to 28.82 tokens/s. For Dream-v0-7B-Instruct, throughput increases from 7.27 to 34.10 tokens/s on GSM8K and from 6.29 to 32.63 tokens/s on GPQA. Averaged across tasks, the paper reports roughly $L$4 throughput gain for LLaDA-8B-Instruct and roughly $L$5 for Dream-v0-7B-Instruct.

Quality preservation is task-dependent but generally strong. The average accuracy reported for LLaDA-8B-Instruct changes from 59.24 for the vanilla model to 59.86 for Sparse-dLLM. LLaDA-1.5 changes from 60.90 to 60.99. Dream-v0-7B-Base changes from 59.18 to 59.21. Dream-v0-7B-Instruct decreases from 65.65 to 64.36, indicating that the framework is not uniformly lossless. The HumanEval results are particularly important: the paper explicitly notes that Sparse-dLLM underperforms baseline and cache-based alternatives on code generation, and attributes this to the possibility that code tasks require more complete context retention.

Memory behavior is one of the framework’s distinguishing outcomes. On LLaDA models, peak memory remains effectively at baseline, with ratios around $L$6. On Dream models, Sparse-dLLM can be slightly lower than baseline because block-wise sampling no longer needs to preserve full logits outside the current block. The paper gives a representative Dream-v0-7B-Base MMLU case of 15.64 GB for baseline versus 14.74 GB for Sparse-dLLM. By contrast, dense cache methods such as dLLM-Cache, dKV-Cache, and Fast-dLLM increase peak memory to roughly $L$7–$L$8 baseline.

Ablations further delimit the operating regime. Delay-step experiments show that one-step delay nearly matches the best accuracy while preserving higher throughput than longer delays. Retention-ratio sweeps show that performance improves steeply up to about $L$9 and then saturates, while memory grows linearly with $O(HL^2 d_k) \approx O(L^2 d),$0. Kernel-size sweeps show that $O(HL^2 d_k) \approx O(L^2 d),$1 gives the best GSM8K accuracy among tested values, whereas $O(HL^2 d_k) \approx O(L^2 d),$2 and larger kernels such as $O(HL^2 d_k) \approx O(L^2 d),$3 degrade performance.

5. Position within the dLLM acceleration literature

Sparse-dLLM became a reference point for later dLLM acceleration work, which often defines itself relative to its dynamic cache-eviction design. Subsequent papers place it within a specific family of methods: sparsification inside the standard diffusion loop, with the entire sequence still serving as the conceptual unit of computation at each step (Xiong et al., 9 Jun 2026).

Prefilling-dLLM distinguishes itself from Sparse-dLLM by introducing a prefill–decode disaggregation. In that formulation, the long prefix is partitioned into chunks, chunk KV states are computed once, and decoding attends only to a selected subset of cached chunks and tokens. The paper explicitly contrasts this with Sparse-dLLM, characterizing Sparse-dLLM as dynamic cache eviction or sparse attention inside the ordinary dLLM loop, whereas Prefilling-dLLM applies static retrieval and token pruning before iterative denoising begins (Xiong et al., 9 Jun 2026). This distinction is conceptually important: Sparse-dLLM is a within-step dynamic sparsifier, not a prefix-retrieval architecture.

Focus-dLLM makes a different critique. It argues that sparse attention in dLLMs is harder than in autoregressive models because the identities of tokens that will be unmasked next are unknown before attention is computed. In that framing, Sparse-dLLM and SparseD are described as relying on coarse block-level heuristics or reuse of prior patterns, while Focus-dLLM uses past confidence to predict future unmasked positions and combines that with sink-aware pruning (Long et al., 2 Feb 2026). Here Sparse-dLLM represents a first-generation dynamic eviction baseline: effective, training-free, but limited by coarse query-side targeting.

DyLLM moves in another direction, exploiting temporal sparsity in token representations across denoising steps. Rather than pruning cached prefix/suffix KV states, it identifies salient tokens by cosine similarity of attention contexts between adjacent steps and recomputes only those tokens’ attention and FFN paths. In that taxonomy, Sparse-dLLM is still a sparse diffusion method, but one centered on cache management and attention-guided token eviction, not on activation reuse and layer-wise temporal saliency (Lee et al., 9 Mar 2026).

These later comparisons clarify what Sparse-dLLM is and is not. It is neither a prefix-level retrieval system nor an activation-caching framework. Its defining innovation is delayed bidirectional sparse caching driven by attention stability across steps.

6. Limitations, misconceptions, and subsequent significance

Sparse-dLLM’s main limitation is that its guarantees are empirical rather than formal. The paper offers strong observational evidence for temporal consistency of token saliency and for the stabilization of out-of-block KV states after the first block step, but it does not provide theoretical error bounds on quality degradation under eviction. Its retention policy is therefore heuristic in the strict sense, even if the heuristic is tightly grounded in measured attention behavior (Song et al., 4 Aug 2025).

A second limitation is task sensitivity. The HumanEval results show that selective eviction is less reliable when tasks appear to require dense, fine-grained long-range context, and the paper explicitly notes that code tasks may need a more complete context. This rules out a simplistic reading of Sparse-dLLM as universally lossless. The method is better understood as a favorable quality–throughput–memory trade-off whose operating point depends on task class.

A third limitation is architectural dependence. Sparse-dLLM assumes access to block-wise decoding, per-layer Q/K/V states, and an implementation in which cache policies can be altered at inference time. That makes it easy to layer onto existing dLLMs such as LLaDA and Dream, but less straightforward to deploy in inference stacks that hide or fuse these components.

Its significance lies in establishing dynamic cache eviction as a viable principle for diffusion inference. Later work repeatedly treats Sparse-dLLM as the canonical example of within-loop sparse caching, against which prefix retrieval, confidence-guided attention focusing, and temporal-saliency activation reuse are compared. In that historical sense, Sparse-dLLM occupies a role analogous to early KV-cache pruning methods in autoregressive inference: it demonstrated that the full-sequence recomputation pattern of diffusion decoding is not immutable, and that a small, stable subset of tokens can often stand in for a much larger bidirectional context with substantial gains in throughput and little or no loss on many reasoning benchmarks.