DominoTree: Conditional Tree-Structured Decoding

• DominoTree is a conditional tree-structured drafting procedure that integrates Domino’s GRU-based causal correction with best-first expansion.
• It replaces conventional marginal scoring with explicit path-dependent corrections, boosting accepted token rates and overall inference speed.
• Empirical results on Qwen3-4B and Qwen3-8B demonstrate significant throughput improvements over related speculative decoding methods.

Searching arXiv for DominoTree and closely related speculative decoding papers to ground the article with current citations. arxiv_search(query="DominoTree Conditional Tree-Structured Drafting with Domino for Speculative Decoding", max_results=5, sort_by="relevance") arxiv_search(query="speculative decoding DFlash DDTree CaDDTree Domino arXiv", max_results=10, sort_by="relevance") arxiv_search(query="DominoTree Conditional Tree-Structured Drafting with Domino for Speculative Decoding", max_results=10, sort_by="relevance") {"query":"DominoTree Conditional Tree-Structured Drafting with Domino for Speculative Decoding","max_results":5,"sort_by":"relevance"} DominoTree is a training-free speculative decoding method for LLM inference that combines Domino’s GRU-based causal correction with best-first tree drafting. It is designed for the setting in which a cheap drafter proposes multiple candidate continuations and an expensive target model verifies them in parallel. Relative to block-diffusion drafters such as DFlash and marginal-tree methods such as DDTree and CaDDTree, DominoTree replaces factorized marginal path scoring with conditional, non-factorized path scoring along each root-to-node path, while using a GPU-native CUDA-graph builder to keep tree construction cheap (Lin et al., 9 Jul 2026).

1. Position within speculative decoding

Speculative decoding accelerates an expensive target LLM by pairing it with a cheaper drafter. At each round, the drafter proposes a sequence or tree of candidates, the target model scores all candidates in one parallel pass, and the system accepts the longest prefix or root-to-leaf path whose tokens match what the target would sample. End-to-end speedup $\eta$ over pure autoregressive decoding satisfies

$$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$

where $\tau$ is mean accepted tokens per round. The two central levers are improving draft quality, which raises $\tau$, and reducing draft cost $T_{\text{draft}}$ (Lin et al., 9 Jul 2026).

Within this landscape, block-diffusion DFlash drafts $B$ tokens in one parallel pass, but its logits

$$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$

are marginals rather than true conditionals, so acceptance is capped. DDTree and CaDDTree build best-first trees over those marginals, increasing accepted length at the cost of verification work and either a fixed or adaptive budget. Domino augments the DFlash backbone with a cheap sequential GRU-based causal correction that recovers conditional information without extra backbone passes, but the released Domino decoder follows only a single chain. DominoTree occupies the intersection of these lines of work: it preserves the block-parallel backbone and best-first tree expansion, but scores nodes with Domino’s conditional correction rather than with per-position marginals (Lin et al., 9 Jul 2026).

A common misconception is that DominoTree is simply DDTree with a different ranking heuristic. The methodological distinction is sharper than that. DDTree and CaDDTree assume a factorized marginal path score, whereas DominoTree carries a branch-specific recurrent state and recomputes the correction on each path, so the score is explicitly path-dependent.

2. Conditional scoring via Domino’s GRU correction

Given a verified prefix $x_{\le t}$, a single parallel DFlash pass computes hidden states and base logits for positions $i=1,\dots,B-1$:

$$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$

These quantities do not condition on $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$0. Domino adds a causal encoder and a logit-space correction:

$$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$1

$$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$2

$$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$3

It then samples $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$4, embeds it, and updates $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$5 (Lin et al., 9 Jul 2026).

Formally, the position-wise draft distribution becomes

$$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$6

Here $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$7 is path-independent, while $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$8 is path-dependent via $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$9. This distinction is the basis of DominoTree’s scoring rule. For a tree node $\tau$0 corresponding to tokens $\tau$1 and GRU state $\tau$2, the root-to-node score is

$\tau$3

Each branch therefore carries its own recurrent state, and the correction is recomputed specifically for that path (Lin et al., 9 Jul 2026).

This formulation makes explicit why the released Domino decoder and marginal-tree methods are not equivalent. Domino already represents conditional structure that a factorized marginal tree cannot represent; DominoTree is the tree-search procedure that makes that structure usable.

3. Tree construction, top-$\tau$4 restriction, and best-first expansion

The main computational difficulty is that recomputing $\tau$5 for the full vocabulary at every node is expensive. DominoTree addresses this by restricting each node expansion to a candidate top-$\tau$6. For each depth $\tau$7, it extracts once the marginal top-$\tau$8 tokens under $\tau$9, packs the corresponding $\tau$0 rows of $\tau$1 and base logits, and then computes $\tau$2 only on that $\tau$3-slice. This changes the per-node projection from $\tau$4 to $\tau$5, after which the method selects the top-$\tau$6 children from those corrected logits (Lin et al., 9 Jul 2026).

Under a fixed node budget $\tau$7, the expansion is best-first. The heap is initialized with children of the root. While the tree has fewer than $\tau$8 nodes and the heap is nonempty, the algorithm pops the node with highest $\tau$9, adds it to the tree, and, if depth permits, calls the child-construction routine on the branch-specific state to obtain top-$T_{\text{draft}}$0 children together with their log-probabilities and next states. These children are then pushed back with updated cumulative score. The resulting tree is prefix-closed and is verified in one target-model pass (Lin et al., 9 Jul 2026).

This design is significant because it preserves the tree-search advantages of DDTree and CaDDTree while changing the object being optimized. The expansion is no longer over a fixed per-depth token set with sibling-invariant probabilities. Instead, sibling scores can diverge because each child inherits a different recurrent state. A plausible implication is that the tree better matches the target model when conditional dependencies omitted by marginal drafting are important.

4. GPU-native CUDA-graph implementation

DominoTree’s practicality depends not only on the scoring rule but also on how the tree is built. The implementation uses a GPU-native, CUDA-graph builder that is bit-identical to a reference Python implementation, so acceptance is unchanged (Lin et al., 9 Jul 2026).

The heavy operations inside the per-node correction are pre-captured as three small CUDA graphs, described as root, below-prefix, and above-prefix. Static input and output buffers avoid Python-to-GPU kernel-launch overhead. Best-first heap operations remain in Python, but the per-node correction routine is replayed from the captured graphs. The implementation still performs a single cuda.synchronize() per pop in order to read tokens back, yet the tens of individual kernel launches in eager PyTorch collapse into one per graph (Lin et al., 9 Jul 2026).

The paper emphasizes bit-for-bit equivalence to the Python reference: at $T_{\text{draft}}$1, every accepted token matches row-for-row. This point matters because speculative decoding speedups are only useful if they preserve the target model’s acceptance behavior. DominoTree’s systems contribution is therefore not merely acceleration of the drafter; it is acceleration under an equivalence guarantee for the acceptance path.

5. Empirical performance on Qwen3-4B and Qwen3-8B

On Qwen3-4B with block $T_{\text{draft}}$2, node budget $T_{\text{draft}}$3, and $T_{\text{draft}}$4, DominoTree attains the strongest reported results among the evaluated methods. At $T_{\text{draft}}$5, the reported figures are: DFlash, $T_{\text{draft}}$6 speedup with $T_{\text{draft}}$7; DDTree(16), $T_{\text{draft}}$8 with $T_{\text{draft}}$9; CaDDTree, $B$0 with $B$1; Domino chain with the best CUDA graph, $B$2 with $B$3; and DominoTree(16), $B$4 with $B$5. The paper describes DominoTree here as achieving the highest $B$6 and speedup (Lin et al., 9 Jul 2026).

Across eight datasets in the “Overall” aggregate on Qwen3-4B, DominoTree reports speedup $B$7 and $B$8, compared with Domino at speedup $B$9 and $$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$0. This corresponds to $$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$1 throughput versus Domino with $$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$2 confidence interval $$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$3, $$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$4 versus DDTree with confidence interval $$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$5, and $$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$6 versus CaDDTree. At $$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$7, DominoTree keeps the highest $$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$8, up to $$L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})$$9 at $x_{\le t}$0, and wins throughput versus Domino by $x_{\le t}$1–$x_{\le t}$2 and versus DDTree/CaDDTree by $x_{\le t}$3–$x_{\le t}$4, with only Code sometimes flat or slightly negative at high $x_{\le t}$5 (Lin et al., 9 Jul 2026).

On Qwen3-8B under the same block and default settings, the profile is more mixed but remains favorable overall. At $x_{\le t}$6, DominoTree reports $x_{\le t}$7 speedup with $x_{\le t}$8, compared with Domino at $x_{\le t}$9 and DDTree at $i=1,\dots,B-1$0, yielding $i=1,\dots,B-1$1 versus Domino, marked not significant, and $i=1,\dots,B-1$2 versus DDTree. At $i=1,\dots,B-1$3, it is $i=1,\dots,B-1$4 versus Domino and a tie versus DDTree. At $i=1,\dots,B-1$5, it remains $i=1,\dots,B-1$6 versus Domino but is reported at $i=1,\dots,B-1$7 versus DDTree, characterized as a small loss on Code (Lin et al., 9 Jul 2026).

These results constrain overly broad claims. DominoTree consistently has the highest accepted length at every tested temperature on Qwen3-8B, but the throughput edge over DDTree/CaDDTree narrows at higher temperature. The method’s advantage is therefore strongest when conditional scoring gains outweigh the additional tree-build cost.

6. Ablations, operating regimes, and terminological disambiguation

The ablation results make the operating trade-offs explicit. For the node budget $i=1,\dots,B-1$8, the reported outcomes are: $i=1,\dots,B-1$9 is best on chat workloads, with Alpaca $$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$0 and GSM8K flat; $$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$1 gives a balanced win on both, with GSM8K $$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$2 and Alpaca $$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$3, but at doubled build cost; and $$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$4 is too slow to win throughput. For candidate width $$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$5, the paper states that $$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$6 under-samples, $$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$7 recovers almost full gain, and $$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$8 shows $$H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).$$9 saturation. For Qwen3-4B at $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$00, the Python builder takes $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$01 ms build per round, while the GPU builder takes $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$02 ms, a reduction of $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$03 ms. With the GPU builder, DominoTree beats DDTree by $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$04 throughput; with the Python builder, it would tie or lose at high $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$05. A conditioning ablation comparing Cond@16 and Marg@16 isolates a $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$06 throughput gain from the score function, and an adaptive-budget variant, CondAdaptive, is reported as ineffective because over-confident path-prob calibration makes the rule saturate at the cap, motivating use of fixed $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$07 instead (Lin et al., 9 Jul 2026).

The summary offered in the source is correspondingly specific. DominoTree “delivers the best of both worlds—block-parallel backbone and tree-structured conditional scoring—by rerunning only a lightweight GRU+head per node.” Candidate restriction preserves most $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$08 while cutting per-node cost by approximately $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$09, and a GPU-native CUDA-graph builder removes per-node PyTorch overhead. The paper identifies fixed budget $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$10 and $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$11 as a strong default, while noting that larger $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$12 or $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$13 can raise $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$14 further at the cost of build time and sometimes net throughput. For deployment, it notes that one would integrate the GPU builder into a multi-stream, batched serving stack such as vLLM or SGLang (Lin et al., 9 Jul 2026).

The term “DominoTree” is not unique across arXiv. In graph-theoretic survey material, “DominoTree” refers to the parameter $$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$15, the least width of a tree-decomposition in which each vertex appears in at most two bags; that usage concerns spread-$$\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},$$16 tree-decompositions and is unrelated to speculative decoding (Distel et al., 7 Apr 2026). In the LLM-inference literature, by contrast, DominoTree denotes the conditional tree-structured drafting procedure described above (Lin et al., 9 Jul 2026).