---
title: 'Focus-dLLM: Accelerating Long-Context dLLMs'
url: https://www.emergentmind.com/topics/focus-dllm
type: topic
---

# Focus-dLLM: Accelerating Long-Context dLLMs

Focus-dLLM is a training-free inference acceleration framework for long-context diffusion large language models (dLLMs) that reduces the cost of bidirectional full attention during iterative denoising by combining confidence-guided query prediction with sink-aware sparse attention [2602.02159]. It is designed for the long-context regime, where a dLLM repeatedly denoises a fixed-length sequence and normally performs full bidirectional attention over the entire context at every step, making inference prohibitively expensive as context length grows. The method predicts which masked tokens are likely to become unmasked at the next step, expands those predictions into local query windows, prunes prompt-history attention blocks, explicitly preserves attention sinks, and reuses sink locations across layers. Reported results show more than \(29\times\) lossless speedup under \(32\)K context length while maintaining comparable or better task quality on the evaluated benchmarks [2602.02159].

## 1. Problem formulation and computational setting

Focus-dLLM addresses a specific inefficiency in long-context dLLM inference: at each denoising step, the model must update a sequence of length \(L\) under bidirectional full attention, and this full-context attention is repeated over many steps [2602.02159]. The paper formulates the per-token prediction at step \(t\) as
\[
\hat{x}_i^{(t)} = \arg\max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),\qquad
c_i^{(t)} = \max_{v\in\mathcal{V}} p(x_i=v\mid \mathbf{x}^{(t)}),
\]
where \(c_i^{(t)}\) is the token confidence and a subset of highest-confidence masked positions is unmasked while the rest are remasked [2602.02159].

The bottleneck is not merely repeated state recomputation, but repeated attention over the full long context. The paper states that approximate KV cache methods reduce recomputation of token states, yet still leave attention over the full cached context expensive. This is why long-context dLLM inference remains costly even when cache-based acceleration is present [2602.02159].

The framework is motivated by the observation that sparse attention is harder in dLLMs than in autoregressive models. In a diffusion setting, the set of tokens that will be unmasked next is unknown in advance, so standard sparse methods that rely on current query importance are poorly matched to the inference procedure. Prior approaches are described as either dependent on current query tokens that are not yet available for future unmasked positions, or reliant on coarse block-level heuristics that may miss important tokens and degrade quality [2602.02159].

## 2. Empirical observations underlying the method

Focus-dLLM is grounded in two empirical findings reported by the paper. The first is temporal consistency of token confidence across adjacent denoising steps. The paper states that \(c_i^{(t)}\) correlates strongly with \(c_i^{(t-1)}\), and that tokens unmasked at step \(t\) often overlap with positions that had the highest confidence at step \(t-1\) [2602.02159]. On LLaDA-8B-Instruct, selecting the top-4 remasked tokens by confidence at step \(t-1\) achieved a **96.1% average recall** for the tokens decoded at the next step [2602.02159]. This establishes past confidence as a predictive signal for future active queries.

The second observation concerns attention structure. The paper reports that dLLM attention is strongly local, showing a diagonal bias, but also contains attention sinks: tokens that receive unusually high attention and strongly influence semantic continuity [2602.02159]. It further reports that sink locations are cross-layer consistent within a denoising step, so sink positions identified at one layer can be reused in deeper layers [2602.02159].

These two findings define the architecture of the method. Confidence persistence makes it possible to anticipate which masked tokens will become relevant before they are actually decoded, and sink stability makes it possible to preserve a small set of high-influence tokens without rediscovering them at every layer. A plausible implication is that Focus-dLLM substitutes prediction and structural priors for dense attention coverage, rather than attempting unstructured sparsification.

## 3. Confidence-guided focusing and sparse attention mechanism

The inference procedure begins by predicting candidate unmasked positions from the previous step’s confidence scores. Given the current masked set \(\mathcal{M}^{(t)}\), Focus-dLLM selects the top-\(k\) positions according to \(c_j^{(t-1)}\):
\[
\mathcal{I}_{\text{focus}} =
\left\{ i \mid c_i^{(t-1)} \in \text{top-}k\left(\{c_j^{(t-1)}\}_{j\in\mathcal{M}^{(t)}}\right) \right\},
\qquad
k = \lfloor \rho \, n^{(t)} \rceil,
\]
where \(n^{(t)}\) is the number of tokens expected to be unmasked at step \(t\), and \(\rho\) is the prediction expansion factor [2602.02159]. This component is described as the past confidence-guided indicator.

The selected positions are then expanded into local windows to preserve semantic coherence:
\[
\mathcal{I}_{\text{active}} =
\bigcup_{i \in \mathcal{I}_{\text{focus}}}
\left\{ l \mid i - \lfloor w/2 \rfloor \le l \le i + \lfloor w/2 \rfloor \right\}.
\]
The resulting set \(\mathcal{I}_{\text{active}}\) defines the active queries for sparse attention [2602.02159]. The paper notes that very small windows hurt recall, whereas overly large windows introduce irrelevant tokens.

For prompt-side pruning, the prompt is partitioned into contiguous blocks, and each block \(b\) is represented by its mean key vector:
\[
\bar{K}_b = \text{Mean}_{j\in \text{Block}_b}(K_j).
\]
Using the predicted focus queries, the method computes block relevance scores
\[
R_b = \frac{1}{H}\sum_{h=1}^{H}
\left( Q_{\mathcal{I}_{\text{focus}}}^{h} \cdot \bar{K}_b^{h} \right),
\]
then selects the top
\[
C = \lfloor \alpha \cdot N_{\text{total\_blocks}} \rfloor
\]
blocks as \(\mathcal{B}_{\text{relevant}} = \operatorname{Top}\text{-}C(R_b)\) [2602.02159].

The sparse attention pattern is explicitly sink-aware. Rather than retaining only the most relevant prompt blocks, the method preserves a union of sink tokens and tokens within the selected relevant blocks:
\[
\mathcal{I}_p =
\mathcal{I}_{\text{sink}} \cup
\bigcup_{b \in \mathcal{B}_{\text{relevant}}}
\{\, i \mid i \in \text{Block}_b \,\}.
\]
The effective keys and values become
\[
K_{\text{attn}} = \mathrm{concat}(K_{\mathcal{I}_p}, K_{\text{resp}}),\qquad
V_{\text{attn}} = \mathrm{concat}(V_{\mathcal{I}_p}, V_{\text{resp}}),
\]
and attention is computed only over this restricted set:
\[
\mathrm{Attn} = \mathrm{Softmax}\!\left(
\frac{ Q_{\mathcal{I}_{\text{active}}} K_{\text{attn}}^\top }{\sqrt{d}}
\right) V_{\text{attn}}.
\]
This formulation preserves response-side states while pruning redundant prompt-history attention [2602.02159].

## 4. Sink preservation, layer reuse, and algorithmic flow

A central component of Focus-dLLM is its treatment of attention sinks. The method designates the first \(l_{\text{dense}}\) layers as dense attention layers, because shallow layers are described as more sensitive to sparsification [2602.02159]. At a cut-off dense layer \(l_{\text{dense}}\), the model computes sink scores using aggregated active-token queries:
\[
S_j = \frac{1}{H} \sum_{h=1}^{H}
\operatorname{Softmax}_{j}
\left(
\frac{Q_{\mathcal{I}_{\text{active}}}^{h} \cdot K_j^h}{\sqrt{d}}
\right),
\]
and selects
\[
\mathcal{I}_{\text{sink}} = \operatorname{Top}\text{-}N_{\text{sink}}(S_j)
\]
as the sink set [2602.02159].

The paper emphasizes that sink positions are stable across layers within the same decoding step, which allows the sink set discovered at the cut-off dense layer to be reused for deeper layers rather than recomputed repeatedly [2602.02159]. This cross-layer reuse is both an accuracy-preserving and overhead-reducing device. The paper explicitly motivates it as a way to avoid repeated sink discovery in a regime where any additional long-context overhead becomes expensive.

The appendix-level inference flow is summarized as follows. The system initializes
\[
\mathbf{x}^{(0)} = [\mathbf{p}, \text{[MASK]}_1,\dots,\text{[MASK]}_N],
\]
along with an empty KV cache and confidence scores \(\mathbf{c}^{(0)} = 0\). For each denoising step \(t\), it determines the number of tokens to unmask \(n^{(t)}\) from a transfer scheduler \(\mathcal{S}\), sets \(k = \lfloor \rho n^{(t)} \rceil\), performs a full refresh at block entry, and otherwise executes confidence-guided focusing, window expansion, dense attention for layers \(l \le l_{\text{dense}}\), and sparse sink-aware attention for layers \(l > l_{\text{dense}}\) [2602.02159]. This is an inference-time modification rather than a model reparameterization or retraining scheme.

## 5. Experimental setup and reported results

Focus-dLLM is evaluated on **UltraLLaDA** and **Dream-7B-Instruct**, with **LongBench** and **Niah** as the reported benchmarks, and **NVIDIA H200 GPUs** as the hardware platform [2602.02159]. The implementation uses a sparse sink-aware operator in **Triton** and **FlashAttention** for dense attention [2602.02159]. The main hyperparameters reported for the primary setup are:

- \(\rho = 4\)
- \(w = 8\)
- \(l_{\text{dense}} = 6\)
- \(\alpha = 0.5\)
- \(N_{\text{sink}} = 0.01 \times M\)
- prompt block size \(= 64\)

For Dream-7B-Instruct, the final four transformer layers are also kept dense because those layers are stated to be more sensitive to sparsification [2602.02159].

The principal efficiency claim is context-length scaling. The paper reports that speedup over Vanilla grows from about **\(9.4\times\) at 8K context** to **\(29.6\times\) at 32K context**, and summarizes the headline result as a **lossless speedup of more than \(29\times\) at 32K context length** [2602.02159]. It also reports up to **\(2.05\times\)** speedup over Fast-dLLM at 32K context [2602.02159].

The quality results are reported in relative terms rather than as a single universal metric. On **LongBench**, Focus-dLLM achieves the best average score on UltraLLaDA among the compared acceleration methods, and on Dream-7B-Instruct it is reported as on par with Vanilla and better than Fast-dLLM and Sparse-dLLM, while being slightly below SparseD in some settings but substantially more efficient [2602.02159]. On **Niah**, it shows stronger needle-in-a-haystack retrieval than Fast-dLLM and Sparse-dLLM, and in the reported setup even surpasses the vanilla baseline at deeper layers [2602.02159].

The ablation evidence supports both major components. The paper reports that adding the past confidence-guided indicator alone gives a small throughput gain but can slightly hurt accuracy when used without the full sparse-attention mechanism, while adding sink-aware sparse attention produces a large throughput gain and also improves accuracy; combining both yields the best trade-off [2602.02159]. It also reports that adding attention sinks improves average LongBench score by **+1.35** on Dream-7B-Instruct [2602.02159].

## 6. Position within the dLLM acceleration literature

Focus-dLLM belongs to a broader class of dLLM inference optimization methods, but its target and mechanism are distinct from several closely related systems.

| Method | Primary bottleneck targeted | Core mechanism |
|---|---|---|
| Focus-dLLM | Long-context bidirectional full attention | Confidence-guided query prediction, sink-aware sparse attention, sink reuse across layers |
| FOCUS | Block-diffusion decoding redundancy | Dynamic focusing on decodable tokens and on-the-fly eviction of non-decodable tokens |
| dLLM-Cache | Repeated transformer computation across denoising steps | Long-interval prompt caching and adaptive short-interval response caching |

FOCUS, despite the similar name, addresses a different inefficiency. It targets block-diffusion decoding in which only a small subset of tokens is decodable at each step, and computation on the rest is largely redundant; with \(B=32\), the paper reports that only about \(\sim 10\%\) of tokens are decoded per step, implying \(\sim 90\%\) redundant blockwise computation [2601.23278]. Its mechanism is early importance estimation, dynamic budgeting, and token eviction, yielding up to **\(3.52\times\)** throughput improvement over LMDeploy [2601.23278]. By contrast, Focus-dLLM is framed around long-context attention sparsification rather than compute waste within a fixed block.

dLLM-Cache addresses a different axis again: it exploits the static prompt and partially dynamic response to reuse prompt and response features across denoising steps. It combines long-interval prompt caching with adaptive short-interval response caching, and reports up to **9.1× speedup** over standard inference while largely preserving quality [2506.06295]. The relation between dLLM-Cache and Focus-dLLM is complementary rather than identical: one reuses intermediate computations, whereas the other reduces prompt-history attention work through query prediction and sink-aware sparsification. This suggests that long-context dLLM acceleration is becoming a systems problem with multiple orthogonal levers: caching, token-level eviction, and sparse attention.

The paper itself compares Focus-dLLM against prior sparse-attention approaches. It describes **Fast-dLLM** as focusing on approximate KV caching while still computing attention over the full cached context, **Sparse-dLLM** as using dynamic cache eviction and sparse attention with coarse token or block importance estimates, and **SparseD** as reusing sparse patterns that often depend on dense attention in early steps and use less dynamic, precomputed patterns [2602.02159]. Focus-dLLM is characterized as query-aware, sink-aware, layer-aware, and training-free [2602.02159].

## 7. Limitations, assumptions, and common points of confusion

The paper states two explicit limitations. First, **multimodal extension is not yet explored**; the method is demonstrated on text-only diffusion LLMs. Second, **hyperparameters are manually chosen**, so the reported configuration may not be optimal across domains, and a fully adaptive parameter mechanism could improve robustness [2602.02159].

The method also rests on several assumptions identified by the paper: that previous-step confidence is informative about which tokens will be unmasked next, that attention locality is strong enough for window expansion to be effective, and that sink patterns are stable enough across layers to justify reuse [2602.02159]. A plausible implication is that the realized speed-quality trade-off may depend on model family, remasking schedule, and task distribution more strongly than in simpler cache-only methods.

A recurring source of confusion is nomenclature. **Focus-dLLM** and **FOCUS** are not the same system. Focus-dLLM concerns long-context attention sparsification via confidence-guided context focusing and sink-aware pruning [2602.02159], whereas FOCUS concerns compute-bound block-diffusion decoding and token eviction within the denoising block [2601.23278]. Another potential misconception is that Focus-dLLM is a training method; the paper explicitly describes it as **training-free** and as an inference-time modification rather than a model reparameterization [2602.02159].

Within the diffusion-LLM literature, Focus-dLLM is therefore best understood as a specialized long-context serving framework. Its central contribution is not merely making attention sparse, but making it sparse in a setting where future query positions are unknown by using previous-step confidence to anticipate active regions and by preserving attention sinks that would otherwise be easy to prune incorrectly [2602.02159].

Source: https://www.emergentmind.com/topics/focus-dllm