DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding

Abstract: As LLMs scale up, accuracy improves, but the autoregressive (AR) nature of decoding increases latency since each token requires a serial forward pass. Speculative decoding addresses this by employing a fast drafter to propose multi-token drafts, which are then verified in parallel by the target model. However, many deployments still rely on AR drafters, where sequential passes limit wall-clock gains. We revisit the drafting stage and present DiffuSpec, a training-free drop-in framework that uses a pretrained diffusion LLM (DLM) to produce multi-token drafts in a single forward pass, while remaining compatible with standard AR verifiers. Because DLM drafts are generated under bidirectional conditioning, parallel per-position candidates form a token lattice in which the locally highest-probability token at each position need not form a causal left-to-right path. Moreover, DLM drafting requires pre-specifying a draft length, inducing a speed-quality trade-off. To address these challenges, we introduce two practical components: (i) a causal-consistency path search (CPS) over this lattice that extracts a left-to-right path aligned with AR verification; and (ii) an adaptive draft-length (ADL) controller that adjusts next proposal size based on recent acceptance feedback and realized generated length. Across benchmarks, DiffuSpec yields up to 3x wall-clock speedup, establishing diffusion-based drafting as a robust alternative to autoregressive drafters for speculative decoding.

• The paper introduces DiffuSpec, a training-free framework that uses a pretrained diffusion language model as a multi-token drafter, achieving a 3.08× mean speedup and 6.99 mean accepted length on Spec-Bench.
• The method combines causal-consistency path search, which aligns diffusion proposals with left-to-right verification, and adaptive draft-length control, which adjusts proposal size to improve acceptance and latency.
• The results show diffusion-based drafting can outperform training-free baselines without added training, while open challenges include DLM inference overhead, weaker causal proxies, and untested sampling-based verification.

DiffuSpec is a training-free speculative decoding framework that replaces the conventional autoregressive (AR) drafter with a pretrained diffusion LLM (DLM), proposing multi-token drafts in a single forward pass while remaining compatible with standard AR verifiers (2510.02358). The work addresses two structural obstacles that arise when diffusion proposals meet AR verification: the loss of strict left-to-right causality in DLM outputs, and the need to pre-specify draft length. The proposed remedies—causal-consistency path search (CPS) and an adaptive draft-length (ADL) controller—yield up to $3\times$ wall-clock speedup on Spec-Bench, the strongest result among training-free baselines.

Motivation: why DLMs as drafters

Speculative decoding accelerates AR generation by having a fast drafter propose a block of tokens that the target model verifies in parallel, preserving the target distribution while reducing latency. In practice, most deployments use a small AR drafter whose sequential passes dilute verifier-side parallelism; training-based alternatives such as Medusa, Hydra, and EAGLE achieve high acceptance but require additional learned components. Large pretrained DLMs such as Dream-7B can emit an entire token block with per-position candidate distributions in essentially one refinement pass, matching the drafter desiderata of high per-step throughput and strong proposal quality—without any additional training.

Two properties of DLM drafting complicate this integration. First, because DLMs refine tokens jointly under bidirectional conditioning, probability mass spreads across positions, and the per-position argmax need not form a causal left-to-right path acceptable to the AR verifier. Second, DLM drafting requires fixing a block length $k_t$ in advance; empirically, accepted length grows with $k_t$ and then saturates at an instance-dependent point, so any fixed length either wastes compute or throttles progress.

Method

Causal-consistency path search (CPS). From the final DLM pass, DiffuSpec extracts per-position top-$M$ candidates to form a token lattice, then prunes it entropy-adaptively by retaining the smallest top-$m$ prefix whose cumulative mass exceeds a threshold $\tau$ (capped at $M_{\max}$). Expansion stops after the first EOS, since diffusion drafts tend to pad with EOS once content is complete. A left-to-right beam search over the pruned lattice scores paths with a mixture $\lambda$ of DLM confidence and a causal proxy log-score (a 3-gram KenLM fitted on each dataset's training split), selecting the path most likely to survive AR verification. Per-step complexity is $O(B\,\bar C\,m_{\max})$ for beam size $B$ and average branching factor $k_t$0.

Adaptive draft-length controller (ADL). Two signals are tracked via exponential moving averages: the EOS-aware generated length $k_t$1 (how much content the DLM is ready to produce) and the verified accepted prefix length $k_t$2. The next draft length is set by a one-line clipped policy that grows $k_t$3 only when acceptance keeps pace with generation, within guardrails $k_t$4.

Correctness. Acceptance uses a left-to-right proxy score $k_t$5, obtained by masking all future in-block positions when scoring each drafted token. Under the standard speculative-decoding acceptance rule with this proxy, the classical unbiasedness argument with respect to the target distribution applies, so output quality is preserved under greedy verification.

Results

On Spec-Bench across six task families (multi-turn conversation, translation, summarization, QA, math reasoning, RAG), with Qwen2.5-32B as target and Dream-7B as drafter, DiffuSpec achieves a mean MAT of 6.99 and mean speedup of $k_t$6, versus $k_t$7–$k_t$8 for Lookahead, PLD, Recycling, SAMD, and SPS. Task-level speedups reach $k_t$9 on math reasoning and $k_t$0 on translation. Relative to SPS—the strongest comparable baseline using a 7B AR drafter—DiffuSpec adds $k_t$1 MAT and $k_t$2 speedup. Training-based systems are reported only as context (they use Vicuna-33B targets), but EAGLE-2's mean speedup of $k_t$3 sits below DiffuSpec's figure, suggesting diffusion-based drafting approaches training-based efficiency without extra training.

Ablations attribute most gains to CPS: CPS-only raises mean MAT from 6.05 to 6.43 ($k_t$4 relative to plain) and speedup by $k_t$5, whereas ADL-only adds $k_t$6 MAT and $k_t$7. ADL's role is primarily to convert acceptance into wall-clock acceleration by right-sizing proposals; a fixed-$k_t$8 study confirms this, with speedup peaking at $k_t$9 ($M$0) and declining at $M$1 ($M$2), while ADL attains both the highest MAT (6.99) and speedup ($M$3).

Hyperparameter sweeps show CPS knobs ($M$4, $M$5, $M$6) are robust over broad ranges, with chosen values ($M$7, $M$8, $M$9) near optima. The one sharp trade-off is the number of DLM refinement steps: increasing $m$0 from 1 to 10 raises mean MAT from 6.99 to 7.33 but collapses mean speedup from $m$1 to $m$2, so the framework fixes $m$3.

Limitations and open questions

The paper concedes several constraints. The headline comparison against training-based methods is indirect, since Medusa/Hydra/EAGLE results use a different target model (Vicuna-33B) and decoding stack; EAGLE-3 was excluded entirely for lack of a compatible checkpoint. The causal proxy is a lightweight 3-gram KenLM, which may be brittle outside domains where n-gram statistics transfer well—an assumption the paper does not stress-test. The quality guarantee holds under greedy verification; behavior under sampling-based verification is not evaluated. Finally, the system-level cost of DLM drafting remains unoptimized: the authors identify KV-cache-style reuse and fused kernels for DLM inference, stronger causal proxies or verifier-aware scoring, and joint online control of draft length and search breadth as concrete open directions.

Conclusion

DiffuSpec demonstrates that pretrained DLMs can serve as effective, training-free drafters for speculative decoding when two diffusion-specific mismatches—relaxed causality and preset draft length—are handled explicitly. CPS supplies most of the acceptance gain by aligning lattice paths with AR verification, and ADL stabilizes proposal sizing near the instance-specific speed–quality sweet spot. With a $m$4 mean speedup over AR decoding at matched quality, the paper establishes diffusion-based drafting as a practical alternative to both AR drafters and training-based speculative systems.