DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference

Abstract: Speculative decoding is an effective and lossless approach for accelerating LLM inference. However, existing widely adopted model-based draft designs, such as EAGLE3, improve accuracy at the cost of multi-step autoregressive inference, resulting in high drafting latency and ultimately rendering the drafting stage itself a performance bottleneck. Inspired by diffusion-based LLMs (dLLMs), we propose DART, which leverages parallel generation to reduce drafting latency. DART predicts logits for multiple future masked positions in parallel within a single forward pass based on hidden states of the target model, thereby eliminating autoregressive rollouts in the draft model while preserving a lightweight design. Based on these parallel logit predictions, we further introduce an efficient tree pruning algorithm that constructs high-quality draft token trees with N-gram-enforced semantic continuity. DART substantially reduces draft-stage overhead while preserving high draft accuracy, leading to significantly improved end-to-end decoding speed. Experimental results demonstrate that DART achieves a 2.03x--3.44x wall-clock time speedup across multiple datasets, surpassing EAGLE3 by 30% on average and offering a practical speculative decoding framework. Code is released at https://github.com/fvliang/DART.

• The authors devise DART, a diffusion-inspired parallel prediction framework for speculative decoding that reduces drafting forward latency by 53.3× relative to standard methods and up to 3.44× over current state-of-the-art.
• By replacing the traditional autoregressive drafting process with a single-pass, masked prediction approach, DART significantly reduces total inference time on tasks like LLM drafting by eliminating a 20%-40% time overhead.
• DART combines dual log-loss likelihood and N-gram continuity assessment to refine draft candidates, enhancing logit accuracy by up to 15.4 percent (hit@1) over the baseline for early token predictions.

DART is a speculative decoding framework that replaces the autoregressive drafting stage with a single-pass, diffusion-inspired parallel prediction of multiple future token distributions. The central observation motivating the work is that in modern model-based drafters such as EAGLE3, the drafting forward pass itself has become a bottleneck: although a single customized layer keeps per-step cost low, sequential rollout forces the drafter to consume 20%–40% of total inference time, and over 75% of total time when using Qwen3-1.7B to draft for Qwen3-14B with draft length 5 (2601.19278). DART addresses this by predicting logits for all $d$ future masked positions in one forward pass of a lightweight, target-coupled drafter, then pruning the resulting combinatorial candidate space into a compact draft tree using an N-gram continuity signal.

Motivation and design constraints

The paper argues that directly repurposing diffusion-based LLMs (dLLMs) such as Dream7B as drafters is unsuitable for three reasons: their bidirectional, full-sequence denoising objective conflicts with the strictly prefix-conditioned causal requirement of speculative decoding; standalone dLLMs carry parameter counts whose per-step cost exceeds lightweight drafters by tens of times; and practical issues such as tokenizer incompatibility arise. DART instead adopts only the masked parallel prediction idea while retaining causal attention and a limited drafting horizon ($d=8$). The authors also note a positional importance bias: because verification accepts tokens sequentially from the prefix, errors at early positions terminate acceptance immediately, so early-position accuracy dominates end-to-end gains.

Drafting architecture and training

The DART drafter is a single Transformer decoder layer operating on concatenated intermediate hidden states of the target model (layers 1, $\text{num\_layers}/2 - 1$, and $\text{num\_layers}-4$, following EAGLE3's selection), projected through an FC layer and combined with shifted token embeddings. Appending $d-1$ trainable mask tokens to the prefix yields logits for all future positions in one pass. Two design choices are notable:

Shifted logits prediction: each position's logit predicts the next token rather than the token at its own position. This raises first-position accuracy on Qwen3-4B from 57.7% to 71.1% (hit@1) and from 87.1% to 93.2% (hit@10), with consistent smaller gains at later positions.

Prefix-shared masked training: multiple prefixes within one sequence are trained jointly via a sparse attention mask (prompt-causal, mask-to-prefix, block-inner causal; cross-block attention disabled), implemented efficiently with Flex-Attention. Supervision uses an annealed KL divergence against the target model's distributions, weighted by $\lambda_t = \gamma^{t-1}$ with $\gamma = 0.6$. Ablations show a clear trade-off: smaller $\gamma$ improves early-position accuracy at the expense of later positions, and $\gamma=0.6$ maximizes average acceptance length $\tau$ (3.63 on HumanEval versus 3.48 without annealing).

N-gram-based tree pruning

Parallel logits induce a factorized distribution over $d=8$0 positions—an exponentially large implicit token tree—whose naive combinations can be locally improbable. DART prunes this space with a beam-style expansion (beam width $d=8$1, top-$d=8$2 candidates per position, final tree size $d=8$3) scored by a combination of log-softmax logit likelihood and a 3-gram continuity score built from Dolma 3 Mix (~1.3 billion trie nodes). The logit weight decays as $d=8$4, deliberately shifting influence toward the N-gram signal at deeper levels where draft-logit accuracy degrades. The N-gram trie is implemented in C++, NUMA-pinned, and costs roughly 100 GB of CPU RAM but only ~6 μs per query after warmup; the full tree search adds ~2 ms per iteration. Ablations confirm the N-gram constraint is essential: it lifts $d=8$5 by 0.48–0.74 across HumanEval, Alpaca, Math500, and CodeAlpaca (e.g., CodeAlpaca rises from 3.85 to 4.59).

Empirical results

Across seven benchmarks (MT-Bench, Alpaca, CodeAlpaca, HumanEval, LiveCodeBench, Math500, MBPP) and Qwen3 targets from 1.7B to 32B plus LLaMA2-Chat-7B, DART achieves 2.03×–3.44× wall-clock speedup over vanilla autoregressive decoding, exceeding EAGLE3 by ~30% on average and up to 65% on code-centric workloads (CodeAlpaca with Qwen3-14B: 3.44× vs. 2.08×). Mean speedups range from 2.42× (Qwen3-32B, $d=8$6) to 2.87× (Qwen3-4B); results hold at temperature 1 and transfer to A100-40G hardware with similar ~30% margins over EAGLE3. Critically, these gains come despite $d=8$7 being comparable to—not consistently above—EAGLE3 (within 0.2), which supports the paper's claim that minimizing drafting latency, rather than maximizing $d=8$8 alone, is the binding constraint on end-to-end performance. On Qwen3-14B, DART reduces drafting forward latency to 1.5 ms, a 6.8× reduction versus EAGLE3 and 53.3× versus standard speculative sampling. Under larger batch sizes, gains decay as inference becomes compute-bound, but DART retains an advantage (e.g., 1.45× vs. 1.22× at batch size 64 on Qwen3-4B/HumanEval); notably, on Qwen3-8B both methods approach parity beyond batch size 24, indicating the benefit is concentrated in memory-bound regimes.

Limitations and open questions

Several caveats bear directly on the reported results. First, the N-gram component requires ~100 GB of CPU RAM and a 43.5 GB on-disk trie; the authors justify this as underutilized server memory shared across processes, but this assumption will not hold in memory-constrained deployments, and the method's sensitivity to N-gram corpus quality or domain shift is not evaluated. Second, comparisons with DiffuSpec and SpecDiff were omitted because those implementations are closed-source, so the claimed superiority over dLLM-based drafters rests partly on qualitative argumentation rather than direct measurement. Third, evaluation is restricted to batch size 1 in the main tables, with larger-batch benefits shown to shrink substantially on deeper models (Qwen3-8B reaches near-parity at high batch sizes). Fourth, the hyperparameters of the pruning score ($d=8$9, $\text{num\_layers}/2 - 1$0 decay, $\text{num\_layers}/2 - 1$1 level weighting) appear empirically tuned; no sensitivity analysis for these specific choices is provided. Open questions include whether the shifted-prediction scheme generalizes beyond the tested model families, how the fixed draft length of 8 interacts with varying task difficulty, and whether learned (rather than count-based) continuity models could further improve tree quality without added latency.

Conclusion

DART demonstrates that a causally-masked, single-pass parallel predictor coupled tightly to the target model—combined with N-gram-constrained tree pruning—can eliminate autoregressive drafting overhead while preserving acceptance lengths competitive with state-of-the-art autoregressive drafters. The result reframes drafter design around drafting latency as a first-class objective and provides strong empirical evidence that parallel distribution-level drafting is a viable alternative to sequential rollout in lossless speculative decoding.