Focus-dLLM: Accelerating Long-Context dLLMs

• Focus-dLLM is a training-free inference acceleration framework for long-context diffusion LLMs that leverages confidence-guided prediction to identify tokens likely to be unmasked next.
• It employs sink-aware sparse attention to prune redundant prompt-history computations while preserving key attention sinks for semantic continuity.
• Empirical results demonstrate a lossless acceleration exceeding 29x at 32K context length while matching or improving task quality benchmarks.

Focus-dLLM is a training-free inference acceleration framework for long-context diffusion LLMs (dLLMs) that reduces the cost of bidirectional full attention during iterative denoising by combining confidence-guided query prediction with sink-aware sparse attention (Long et al., 2 Feb 2026). It is designed for the long-context regime, where a dLLM repeatedly denoises a fixed-length sequence and normally performs full bidirectional attention over the entire context at every step, making inference prohibitively expensive as context length grows. The method predicts which masked tokens are likely to become unmasked at the next step, expands those predictions into local query windows, prunes prompt-history attention blocks, explicitly preserves attention sinks, and reuses sink locations across layers. Reported results show more than $29\times$ lossless speedup under $32$K context length while maintaining comparable or better task quality on the evaluated benchmarks (Long et al., 2 Feb 2026).

1. Problem formulation and computational setting

Focus-dLLM addresses a specific inefficiency in long-context dLLM inference: at each denoising step, the model must update a sequence of length $L$ under bidirectional full attention, and this full-context attention is repeated over many steps (Long et al., 2 Feb 2026). The paper formulates the per-token prediction at step $t$ as

$$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$

where $c_i^{(t)}$ is the token confidence and a subset of highest-confidence masked positions is unmasked while the rest are remasked (Long et al., 2 Feb 2026).

The bottleneck is not merely repeated state recomputation, but repeated attention over the full long context. The paper states that approximate KV cache methods reduce recomputation of token states, yet still leave attention over the full cached context expensive. This is why long-context dLLM inference remains costly even when cache-based acceleration is present (Long et al., 2 Feb 2026).

The framework is motivated by the observation that sparse attention is harder in dLLMs than in autoregressive models. In a diffusion setting, the set of tokens that will be unmasked next is unknown in advance, so standard sparse methods that rely on current query importance are poorly matched to the inference procedure. Prior approaches are described as either dependent on current query tokens that are not yet available for future unmasked positions, or reliant on coarse block-level heuristics that may miss important tokens and degrade quality (Long et al., 2 Feb 2026).

2. Empirical observations underlying the method

Focus-dLLM is grounded in two empirical findings reported by the paper. The first is temporal consistency of token confidence across adjacent denoising steps. The paper states that $c_i^{(t)}$ correlates strongly with $c_i^{(t-1)}$, and that tokens unmasked at step $t$ often overlap with positions that had the highest confidence at step $t-1$ (Long et al., 2 Feb 2026). On LLaDA-8B-Instruct, selecting the top-4 remasked tokens by confidence at step $32$0 achieved a 96.1% average recall for the tokens decoded at the next step (Long et al., 2 Feb 2026). This establishes past confidence as a predictive signal for future active queries.

The second observation concerns attention structure. The paper reports that dLLM attention is strongly local, showing a diagonal bias, but also contains attention sinks: tokens that receive unusually high attention and strongly influence semantic continuity (Long et al., 2 Feb 2026). It further reports that sink locations are cross-layer consistent within a denoising step, so sink positions identified at one layer can be reused in deeper layers (Long et al., 2 Feb 2026).

These two findings define the architecture of the method. Confidence persistence makes it possible to anticipate which masked tokens will become relevant before they are actually decoded, and sink stability makes it possible to preserve a small set of high-influence tokens without rediscovering them at every layer. A plausible implication is that Focus-dLLM substitutes prediction and structural priors for dense attention coverage, rather than attempting unstructured sparsification.

3. Confidence-guided focusing and sparse attention mechanism

The inference procedure begins by predicting candidate unmasked positions from the previous step’s confidence scores. Given the current masked set $32$1, Focus-dLLM selects the top-$32$2 positions according to $32$3: $32$4 where $32$5 is the number of tokens expected to be unmasked at step $32$6, and $32$7 is the prediction expansion factor (Long et al., 2 Feb 2026). This component is described as the past confidence-guided indicator.

The selected positions are then expanded into local windows to preserve semantic coherence: $32$8 The resulting set $32$9 defines the active queries for sparse attention (Long et al., 2 Feb 2026). The paper notes that very small windows hurt recall, whereas overly large windows introduce irrelevant tokens.

For prompt-side pruning, the prompt is partitioned into contiguous blocks, and each block $L$0 is represented by its mean key vector: $L$1 Using the predicted focus queries, the method computes block relevance scores

$L$2

then selects the top

$L$3

blocks as $L$4 (Long et al., 2 Feb 2026).

The sparse attention pattern is explicitly sink-aware. Rather than retaining only the most relevant prompt blocks, the method preserves a union of sink tokens and tokens within the selected relevant blocks: $L$5 The effective keys and values become

$L$6

and attention is computed only over this restricted set: $L$7 This formulation preserves response-side states while pruning redundant prompt-history attention (Long et al., 2 Feb 2026).

4. Sink preservation, layer reuse, and algorithmic flow

A central component of Focus-dLLM is its treatment of attention sinks. The method designates the first $L$8 layers as dense attention layers, because shallow layers are described as more sensitive to sparsification (Long et al., 2 Feb 2026). At a cut-off dense layer $L$9, the model computes sink scores using aggregated active-token queries: $t$0 and selects

$t$1

as the sink set (Long et al., 2 Feb 2026).

The paper emphasizes that sink positions are stable across layers within the same decoding step, which allows the sink set discovered at the cut-off dense layer to be reused for deeper layers rather than recomputed repeatedly (Long et al., 2 Feb 2026). This cross-layer reuse is both an accuracy-preserving and overhead-reducing device. The paper explicitly motivates it as a way to avoid repeated sink discovery in a regime where any additional long-context overhead becomes expensive.

The appendix-level inference flow is summarized as follows. The system initializes

$t$2

along with an empty KV cache and confidence scores $t$3. For each denoising step $t$4, it determines the number of tokens to unmask $t$5 from a transfer scheduler $t$6, sets $t$7, performs a full refresh at block entry, and otherwise executes confidence-guided focusing, window expansion, dense attention for layers $t$8, and sparse sink-aware attention for layers $t$9 (Long et al., 2 Feb 2026). This is an inference-time modification rather than a model reparameterization or retraining scheme.

5. Experimental setup and reported results

Focus-dLLM is evaluated on UltraLLaDA and Dream-7B-Instruct, with LongBench and Niah as the reported benchmarks, and NVIDIA H200 GPUs as the hardware platform (Long et al., 2 Feb 2026). The implementation uses a sparse sink-aware operator in Triton and FlashAttention for dense attention (Long et al., 2 Feb 2026). The main hyperparameters reported for the primary setup are:

• $$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$0
• $$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$1
• $$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$2
• $$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$3
• $$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$4
• prompt block size $$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$5

For Dream-7B-Instruct, the final four transformer layers are also kept dense because those layers are stated to be more sensitive to sparsification (Long et al., 2 Feb 2026).

The principal efficiency claim is context-length scaling. The paper reports that speedup over Vanilla grows from about $$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$6 at 8K context to $$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$7 at 32K context, and summarizes the headline result as a lossless speedup of more than $$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$8 at 32K context length (Long et al., 2 Feb 2026). It also reports up to $$\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),$$9 speedup over Fast-dLLM at 32K context (Long et al., 2 Feb 2026).

The quality results are reported in relative terms rather than as a single universal metric. On LongBench, Focus-dLLM achieves the best average score on UltraLLaDA among the compared acceleration methods, and on Dream-7B-Instruct it is reported as on par with Vanilla and better than Fast-dLLM and Sparse-dLLM, while being slightly below SparseD in some settings but substantially more efficient (Long et al., 2 Feb 2026). On Niah, it shows stronger needle-in-a-haystack retrieval than Fast-dLLM and Sparse-dLLM, and in the reported setup even surpasses the vanilla baseline at deeper layers (Long et al., 2 Feb 2026).

The ablation evidence supports both major components. The paper reports that adding the past confidence-guided indicator alone gives a small throughput gain but can slightly hurt accuracy when used without the full sparse-attention mechanism, while adding sink-aware sparse attention produces a large throughput gain and also improves accuracy; combining both yields the best trade-off (Long et al., 2 Feb 2026). It also reports that adding attention sinks improves average LongBench score by +1.35 on Dream-7B-Instruct (Long et al., 2 Feb 2026).

6. Position within the dLLM acceleration literature

Focus-dLLM belongs to a broader class of dLLM inference optimization methods, but its target and mechanism are distinct from several closely related systems.

Method	Primary bottleneck targeted	Core mechanism
Focus-dLLM	Long-context bidirectional full attention	Confidence-guided query prediction, sink-aware sparse attention, sink reuse across layers
FOCUS	Block-diffusion decoding redundancy	Dynamic focusing on decodable tokens and on-the-fly eviction of non-decodable tokens
dLLM-Cache	Repeated transformer computation across denoising steps	Long-interval prompt caching and adaptive short-interval response caching

FOCUS, despite the similar name, addresses a different inefficiency. It targets block-diffusion decoding in which only a small subset of tokens is decodable at each step, and computation on the rest is largely redundant; with $c_i^{(t)}$0, the paper reports that only about $c_i^{(t)}$1 of tokens are decoded per step, implying $c_i^{(t)}$2 redundant blockwise computation (Liang et al., 30 Jan 2026). Its mechanism is early importance estimation, dynamic budgeting, and token eviction, yielding up to $c_i^{(t)}$3 throughput improvement over LMDeploy (Liang et al., 30 Jan 2026). By contrast, Focus-dLLM is framed around long-context attention sparsification rather than compute waste within a fixed block.

dLLM-Cache addresses a different axis again: it exploits the static prompt and partially dynamic response to reuse prompt and response features across denoising steps. It combines long-interval prompt caching with adaptive short-interval response caching, and reports up to 9.1× speedup over standard inference while largely preserving quality (Liu et al., 17 May 2025). The relation between dLLM-Cache and Focus-dLLM is complementary rather than identical: one reuses intermediate computations, whereas the other reduces prompt-history attention work through query prediction and sink-aware sparsification. This suggests that long-context dLLM acceleration is becoming a systems problem with multiple orthogonal levers: caching, token-level eviction, and sparse attention.

The paper itself compares Focus-dLLM against prior sparse-attention approaches. It describes Fast-dLLM as focusing on approximate KV caching while still computing attention over the full cached context, Sparse-dLLM as using dynamic cache eviction and sparse attention with coarse token or block importance estimates, and SparseD as reusing sparse patterns that often depend on dense attention in early steps and use less dynamic, precomputed patterns (Long et al., 2 Feb 2026). Focus-dLLM is characterized as query-aware, sink-aware, layer-aware, and training-free (Long et al., 2 Feb 2026).

7. Limitations, assumptions, and common points of confusion

The paper states two explicit limitations. First, multimodal extension is not yet explored; the method is demonstrated on text-only diffusion LLMs. Second, hyperparameters are manually chosen, so the reported configuration may not be optimal across domains, and a fully adaptive parameter mechanism could improve robustness (Long et al., 2 Feb 2026).

The method also rests on several assumptions identified by the paper: that previous-step confidence is informative about which tokens will be unmasked next, that attention locality is strong enough for window expansion to be effective, and that sink patterns are stable enough across layers to justify reuse (Long et al., 2 Feb 2026). A plausible implication is that the realized speed-quality trade-off may depend on model family, remasking schedule, and task distribution more strongly than in simpler cache-only methods.

A recurring source of confusion is nomenclature. Focus-dLLM and FOCUS are not the same system. Focus-dLLM concerns long-context attention sparsification via confidence-guided context focusing and sink-aware pruning (Long et al., 2 Feb 2026), whereas FOCUS concerns compute-bound block-diffusion decoding and token eviction within the denoising block (Liang et al., 30 Jan 2026). Another potential misconception is that Focus-dLLM is a training method; the paper explicitly describes it as training-free and as an inference-time modification rather than a model reparameterization (Long et al., 2 Feb 2026).

Within the diffusion-LLM literature, Focus-dLLM is therefore best understood as a specialized long-context serving framework. Its central contribution is not merely making attention sparse, but making it sparse in a setting where future query positions are unknown by using previous-step confidence to anticipate active regions and by preserving attention sinks that would otherwise be easy to prune incorrectly (Long et al., 2 Feb 2026).