DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees

Abstract: Speculative decoding losslessly accelerates autoregressive LLMs by verifying multiple draft tokens in parallel. Diffusion-based drafters further reduce proposal latency by predicting an entire token block in parallel, but their position-wise distributions are marginal rather than conditioned on tokens selected along each draft path. Existing recurrent correction incorporates causal information along a single draft chain, whereas diffusion-based tree construction broadens candidate coverage without carrying this correction along individual branches. We introduce DARTree, a training-free speculative decoding method that extends a pretrained AR correction head from chains to trees. DARTree first constructs a fixed-width candidate tree by expanding and scoring all nodes at each depth in a single batch, and then only applies best-first pruning to select the verification tree, decoupling AR-head inference from sequential heap operations. Across seven math, code, and chat benchmarks, DARTree achieves the highest average acceptance length and speedup in all four model--temperature configurations, accepting up to 12.97 tokens per verification round, 98.6\% more than DFlash and 27.9\% more than Domino in the same setting, and reaching up to 9.73$\times$ lossless speedup over locally measured autoregressive decoding.

• The paper introduces DARTree, a training-free method that combines depth-wise batched causal correction with deferred top-$B$ pruning to efficiently construct speculative draft trees without changing target-model verification.
• DARTree achieves up to 12.97 accepted tokens per round and 9.73× speedup on GSM8K with Qwen3-4B, outperforming DFlash, DDTree, and Domino across four model–temperature settings.
• The method preserves target-model correctness and transfers across correction heads, but its benefits are strongest at low concurrency because tree verification adds computational and memory overhead at high batch sizes.

DARTree is a training-free speculative decoding method that extends causally corrected block-parallel diffusion drafting from a single draft chain to a speculative tree (2608.13524). The paper addresses a specific efficiency bottleneck that arises when combining two complementary techniques: causal correction heads, which restore intra-block dependencies in parallel draft proposals, and best-first tree construction, which broadens candidate coverage under a fixed verification budget. Their combination forces correction-head inference to proceed one node at a time, interleaved with heap operations. DARTree resolves this by decoupling the two: it expands a fixed-width candidate supertree with depth-wise batched correction, then applies a single global top-$B$ pruning step to produce the verification tree.

Background and motivation

Speculative decoding accelerates autoregressive (AR) LLM inference by drafting multiple tokens and verifying them in parallel with the target model via rejection sampling, preserving the target distribution exactly (Chen et al., 2023). Diffusion-based block drafters such as DFlash predict an entire 16-token block in one forward pass, reducing proposal latency and enabling higher-capacity drafters (Chen et al., 5 Feb 2026). However, position-wise predictions are marginal distributions conditioned only on the verified prefix, not on tokens realized earlier within the same block — the "causal mismatch" of parallel drafting. Domino and DSpark mitigate this with lightweight autoregressive or Markov correction heads that propagate realized draft tokens along a single chain (Huang et al., 28 May 2026, Cheng et al., 6 Jul 2026).

Independently, DDTree constructs prefix trees from the marginal distributions of a block-parallel drafter, selecting the top-$B$ prefixes by probability mass to maximize a surrogate expected acceptance length using an $O(B\log B)$ max-heap (Ringel et al., 14 Apr 2026). The paper's central observation is that these two lines are complementary but incompatible at scale: once scores become path-dependent through causal correction, exact best-first construction requires that each heap pop be followed by correction-head inference on the popped node's children before the next expansion. Measured directly, this sequential variant achieves acceptance lengths comparable to DARTree but requires roughly 70 ms per round — more than twice DARTree's latency — making tree construction itself a dominant cost.

Method

DARTree modifies only candidate-tree construction; target-model verification via tree attention is inherited unchanged. The procedure has three components:

Depth-wise batched supertree construction. The block-parallel backbone runs once, producing shared representations $\{H_d\}$ for all $\gamma$ positions. At each depth $d$, DARTree batches all active branches, applies the pretrained correction head over the top-$K$ candidate tokens ($K=64$) from the base logits, ranks extensions by corrected cumulative score, retains the global top-$W$ children, and carries their correction states forward. Dependencies remain sequential across depths, but correction, scoring, selection, and state updates within each depth are tensorized. Because the lightweight AR head is evaluated in batch, increasing layer width adds little drafting latency, so DARTree deliberately builds a supertree wider than the final verification budget.

Deferred best-first pruning. Each materialized prefix $u_{1:d}$ receives score $B$0, where $B$1 is a depth bonus. The paper proves a heap–top-$B$2 equivalence lemma: since extending a prefix adds a non-positive log-probability plus a non-positive bonus, no child can outrank its parent, so the global top-$B$3 set is prefix-closed and identical to what sequential best-first heap selection would return. This allows the entire heap machinery to be replaced by one vectorized top-$B$4 operation after the supertree is fully scored. The equivalence provably fails for positive $B$5, which could let descendants outrank ancestors.

Verification. The pruned tree is flattened and scored by the target model in a single forward pass with a tree-attention mask, following standard practice.

The method is training-free in the sense that it reuses the released Domino correction head without modification; the authors note plainly that this property does not extend to naive diffusion drafters lacking a correction head.

Experimental results

Evaluations cover Qwen3-4B and Qwen3-8B at temperatures 0 and 1 across seven benchmarks spanning math (GSM8K, MATH-500, AIME25), code (HumanEval, MBPP), and chat (MT-Bench, Alpaca), on a single RTX 6000 Ada GPU with batch size one. DARTree (pruned), using supertree width $B$6, budget $B$7, and $B$8, achieves the highest overall average acceptance length $B$9 and speedup in all four model–temperature configurations. Headline results include:

Setting	DFlash	DDTree	Domino	DARTree (pruned)
GSM8K, Qwen3-4B, $O(B\log B)$0	5.24× / τ=6.53	6.62× / τ=8.47	7.37× / τ=10.14	9.73× / τ=12.97
Overall avg, Qwen3-4B, $O(B\log B)$1	4.58×	5.79×	5.17×	6.99×
Overall avg, Qwen3-8B, $O(B\log B)$2	4.41×	5.29×	5.03×	6.49×

On GSM8K with Qwen3-4B at $O(B\log B)$3, acceptance length reaches 12.97 tokens per round — 98.6% higher than DFlash and 27.9% higher than Domino. Relative to DDTree and Domino, DARTree improves $O(B\log B)$4 by up to 28.9% and 46.8%, and speedup by up to 22.7% and 40.1%. Gains persist under stochastic sampling at $O(B\log B)$5. The single exception among individual benchmarks is AIME25 with Qwen3-8B at $O(B\log B)$6, where DDTree is marginally faster (4.56× vs. 4.50×), though DARTree still attains higher acceptance.

Ablations isolate the design choices. Against the exact sequential-correction-with-heap alternative, DARTree matches acceptance quality (e.g., τ = 12.967 vs. 12.840 on GSM8K) at less than half the round time (~28 ms vs. ~63 ms). Against a cheaper baseline that corrects only a single chain before applying DDTree-style construction, DARTree improves $O(B\log B)$7 by up to 16.4%, confirming that path-specific correction of multiple branches — not merely faster construction — drives the gains. Transfer experiments with the released DSpark Markov correction head show acceptance improvements of 14.6–40.6% and speedup gains up to 34.3% across six settings, supporting the claim that DARTree is a general tree-construction method rather than a Domino-specific modification. Hyperparameter sweeps show diminishing returns beyond $O(B\log B)$8, speedup peaking near $O(B\log B)$9–128 and declining at $\{H_d\}$0, and insensitivity to $\{H_d\}$1 between 32 and 512 (full-vocabulary expansion substantially hurts latency).

Acceptance-distribution analysis shows DARTree combines DDTree's strong early-position acceptance with Domino's extended late-position reach; on GSM8K nearly half of rounds accept almost the entire draft block. Tree-shape analysis indicates DARTree allocates budget to deeper levels, whereas DDTree concentrates nodes in wide shallow layers, producing repetitive parent–child transitions in qualitative examples.

Limitations and applicability

The paper is explicit about scope constraints. DARTree requires a pretrained drafter with a causal correction head, so its training-free property does not apply to naive diffusion drafters, and evaluation is restricted to publicly available compatible models. As with all speculative decoding, total FLOPs increase rather than decrease; tree-based verification adds further computational overhead beyond non-tree methods. Concurrency experiments under SGLang show that at batch size one DARTree delivers the largest throughput gains (6.19× on GSM8K at concurrency 2), but with a fixed large tree the advantage erodes as concurrency grows, falling to roughly 1.06× at concurrency 32 — where an adaptive schedule reducing $\{H_d\}$2 and $\{H_d\}$3 maintains about 2.25×. The method is therefore recommended for low-concurrency, memory-bandwidth-bound serving rather than compute-saturated high-batch deployments. Two questions remain open: whether drafters supporting longer blocks can convert the observed concentration of rounds at the drafting boundary into further gains, and how to make budget and tree shape adaptive to runtime load and content-dependent confidence signals.

Conclusion

DARTree demonstrates that path-conditioned causal correction and broad candidate-tree coverage can be combined without sacrificing batching efficiency. By replacing node-wise best-first search with depth-wise batched expansion followed by a provably equivalent deferred top-$\{H_d\}$4 pruning, it removes a substantial source of drafting latency while preserving lossless verification. The consistent superiority across seven benchmarks, four model–temperature configurations, and two distinct correction heads establishes the approach as a general construction layer for causally corrected parallel drafters, with its principal constraint being the additional compute cost inherent to tree-based verification at high concurrency.