SAID: Accelerating Diffusion-Based Language Models via Scaffold-Aware Iterative Decoding

Abstract: Diffusion LLMs (DLLMs) enable non-autoregressive generation by iteratively denoising corrupted token sequences with bidirectional context. Despite their ability to update multiple positions in parallel, inference remains costly due to the many denoising steps required for high-quality generation. We propose SAID, a Scaffold-Aware Iterative Decoding framework that accelerates DLLMs by reallocating computation across tokens. SAID first spends denoising computation on scaffold tokens to establish the coarse semantic structure, and then completes predictable detail tokens with fewer steps. We further adapt SAID to block-wise diffusion decoding and introduce Confidence-Hierarchical Layered Generation (CHLG), which assigns additional steps only to low-confidence tokens. Experiments on LLaDA-8B and LLaDA 1.5 across math, coding, and knowledge benchmarks show that SAID significantly accelerates DLLM inference with a maximum speedup of 9.1x while maintaining competitive performance. Our code is publicly available: https://github.com/TH-AI-Lab-PKU/SAID.

• The paper introduces SAID, a scaffold-aware decoding method that concentrates iterative denoising on anchor tokens to reduce computational overhead.
• It employs a two-stage approach with confidence-guided detail filling, achieving up to 9.1x latency reduction on key benchmarks while maintaining accuracy.
• The method extends to block-wise inference, demonstrating robust efficiency improvements without retraining or modifying the base model architecture.

Scaffold-Aware Iterative Decoding: Accelerating Diffusion LLM Inference

Introduction and Motivation

Diffusion-based LLMs (DLLMs) have emerged as a promising non-autoregressive alternative to conventional autoregressive (AR) approaches for sequence generation, enabling iterative bidirectional denoising of corrupted token sequences. Despite their intrinsic parallelism and flexibility in handling bidirectional context, DLLMs incur significant inference latency due to the large number of denoising steps required to reach high-quality outputs. The "SAID: Accelerating Diffusion-Based LLMs via Scaffold-Aware Iterative Decoding" paper (2606.04974) addresses this inefficiency by introducing a novel scaffold-aware decoding framework that rethinks how denoising computation is allocated across tokens during generation.

The core insight is that not all tokens require equal computational attention during decoding. Once a coarse semantic "scaffold" of key tokens is established, most remaining positions can be resolved with minimal computation, as their values largely become determined by context. SAID leverages this property to accelerate DLLM inference, moving beyond uniform denoising schedules to a coarse-to-fine, confidence-guided paradigm.

Methodology

Scaffold-Aware Iterative Decoding (SAID)

SAID divides inference into two stages:

1. Semantic Modeling Stage: Iterative denoising is concentrated on a selected subset of "anchor" tokens (e.g., even-indexed positions), which establish the global semantic scaffold of the output. These anchors are unmasked in confidence-ranked order using a standard masked diffusion denoising loop, with $T$ refinement steps restricted to the anchor subset.
2. Detail Filling Stage: The remaining "detail" tokens (e.g., odd-indexed positions) are predicted in a single forward pass, conditioned on the fixed anchors, bypassing iterative refinement.

This approach reallocates the majority of denoising computation to structurally important tokens, while resolving predictable positions with minimal overhead. Because the anchor subset can be chosen with simple heuristics (parity-based partition), SAID requires no retraining or architectural modification of the base DLLM.

Confidence-Hierarchical Layered Generation (CHLG)

While most detail tokens become easy to predict once anchors are fixed, some low-confidence positions remain ambiguous and would be error-prone in a single-step filling scheme. The CHLG module enhances SAID by stratifying detail tokens into easy and hard subsets based on peak model confidence ($c_j = \max_v p_\theta(v \mid \mathbf{x})$) at each position:

• High-confidence detail tokens ($c_j \geq \lambda$) are immediately committed.
• Low-confidence tokens receive additional targeted denoising steps ($T_{\text{hard}}$), conditioned on both anchors and recently committed high-confidence tokens.

This hierarchical strategy further concentrates computation where it is most needed and preserves generation quality without introducing auxiliary models or requiring error-revocation mechanisms.

Block-Level Extension

To support the block-parallel inference paradigms increasingly used for scaling DLLMs, SAID is extended to block-wise decoding. The sequence is partitioned into consecutive blocks (e.g., $\ell=16$, $32$, $64$), with left-to-right processing:

• In each block, SAID applies the same anchor-detail two-stage decoding, with committed tokens from previous blocks serving as fixed left context.
• CHLG is used per block to assign extra denoising steps to uncertain detail tokens.

This enables compatibility with state-of-the-art block diffusion frameworks and preserves the overall semantic coherence while reducing per-block denoising budget.

Experimental Evaluation

SAID is evaluated on the LLaDA-8B and LLaDA~1.5 masked diffusion LLMs across diverse benchmarks spanning code generation (MBPP), scientific and commonsense reasoning (GPQA, ARC-C), mathematical problem solving (GSM8K, MATH), and multi-domain knowledge (MMLU-Pro).

Key Numerical Highlights

• Latency Reduction: On GPQA and ARC-C, SAID achieves up to $9.1\times$ and $5.77\times$ reduction in per-sample decoding latency, respectively, with competitive or improved accuracy. On longer sequence tasks (MBPP, GSM8K), speedup is $\sim2\times$.
• Quality Preservation: In most benchmarks, accuracy is maintained within $c_j = \max_v p_\theta(v \mid \mathbf{x})$0 of the base LLaDA decoder; ARC-C and GPQA even show marginal accuracy improvements, demonstrating the adequacy of the scaffold-guided regime for structured reasoning tasks.
• Blockwise Results: With block lengths of 16–64, block-level SAID attains $c_j = \max_v p_\theta(v \mid \mathbf{x})$1–$c_j = \max_v p_\theta(v \mid \mathbf{x})$2 speedup while matching or marginally surpassing baseline accuracy for ARC-C. Larger block sizes lead to stronger acceleration at a minor trade-off in task accuracy, reflecting the expected quality–efficiency Pareto trend.

Ablation studies confirm:

• Omission of CHLG slightly degrades accuracy on complex tasks, particularly when committing all detail tokens in one step.
• Varying the CHLG confidence threshold $c_j = \max_v p_\theta(v \mid \mathbf{x})$3 (0.6–0.9) yields robust accuracy, with mild sensitivity only on MBPP.
• The choice of anchor set (even vs. odd vs. random) can impact accuracy, particularly for reasoning-intensive domains.

Practical and Theoretical Implications

SAID offers a paradigm shift for DLLM inference, demonstrating that semantically-structured computation schedules can yield substantial acceleration without retraining. Practically, this brings DLLMs closer to latency bounds needed for user-facing and real-time applications, narrowing the gap with speculative and parallel decoding schemes used in AR models while retaining the advantages of bidirectional refinement.

Theoretically, SAID draws a clear connection between sequence structure and denoising resource allocation, suggesting future avenues for adaptive or learned scaffold selection. The coarse-to-fine approach is also relevant for compact cache management and hybrid decoding strategies, as seen in dynamic caching (Ma et al., 21 May 2025, 2509.23094) and blockwise AR-diffusion hybrids.

Limitations and Future Prospects

Limitations include reliance on fixed even/odd partitioning, which creates a ceiling on achievable speedup and might not be optimal for all generation tasks, especially those with highly variable local dependencies or morphological complexity in non-English languages. Moreover, evaluation metrics focus on accuracy, omitting fluency, calibration, and other generation qualities.

Future research could pursue:

• Adaptive Surface Selection: Dynamic, content-aware strategies to select anchors based on sequence semantics, reducing denoising overhead further.
• Multilingual and Open-ended Generation: Assessing SAID in less-structured or morphologically rich language settings.
• Integration with Advanced Caching: Synergizing scaffold-aware schedules with token- or block-level cache reuse and speculative inference for hybrid acceleration.

Conclusion

SAID provides a training-free, scaffold-aware decoding scheme that accelerates DLLM inference by distributing denoising computation according to semantic structure and confidence. The method achieves up to $c_j = \max_v p_\theta(v \mid \mathbf{x})$4 speedup with comparable or improved accuracy in diverse language modeling benchmarks, and extends naturally to block-parallel architectures. By demonstrating the value of structure-aware, confidence-guided decoding, SAID sets a trajectory for further research on efficient parallel generation in diffusion-based LLMs (2606.04974).