---
title: 'DominoTree: Conditional Tree-Structured Decoding'
url: https://www.emergentmind.com/topics/dominotree
type: topic
---

# DominoTree: Conditional Tree-Structured Decoding

Searching arXiv for DominoTree and closely related speculative decoding papers to ground the article with current citations.
arxiv_search(query="DominoTree Conditional Tree-Structured Drafting with Domino for Speculative Decoding", max_results=5, sort_by="relevance")
arxiv_search(query="speculative decoding DFlash DDTree CaDDTree Domino arXiv", max_results=10, sort_by="relevance")
arxiv_search(query="DominoTree Conditional Tree-Structured Drafting with Domino for Speculative Decoding", max_results=10, sort_by="relevance")
{"query":"DominoTree Conditional Tree-Structured Drafting with Domino for Speculative Decoding","max_results":5,"sort_by":"relevance"}
DominoTree is a training-free speculative decoding method for large language model inference that combines Domino’s GRU-based causal correction with best-first tree drafting. It is designed for the setting in which a cheap drafter proposes multiple candidate continuations and an expensive target model verifies them in parallel. Relative to block-diffusion drafters such as DFlash and marginal-tree methods such as DDTree and CaDDTree, DominoTree replaces factorized marginal path scoring with conditional, non-factorized path scoring along each root-to-node path, while using a GPU-native CUDA-graph builder to keep tree construction cheap [2607.08642].

## 1. Position within speculative decoding

Speculative decoding accelerates an expensive target LLM by pairing it with a cheaper drafter. At each round, the drafter proposes a sequence or tree of candidates, the target model scores all candidates in one parallel pass, and the system accepts the longest prefix or root-to-leaf path whose tokens match what the target would sample. End-to-end speedup $\eta$ over pure autoregressive decoding satisfies
$$
\eta \approx \tau / (T_{\text{draft}}+T_{\text{verify}})/T_{\text{target}},
$$
where $\tau$ is mean accepted tokens per round. The two central levers are improving draft quality, which raises $\tau$, and reducing draft cost $T_{\text{draft}}$ [2607.08642].

Within this landscape, block-diffusion DFlash drafts $B$ tokens in one parallel pass, but its logits
$$
L_i^{\text{base}} \approx \log p(x_{t+i}\mid x_{\le t})
$$
are marginals rather than true conditionals, so acceptance is capped. DDTree and CaDDTree build best-first trees over those marginals, increasing accepted length at the cost of verification work and either a fixed or adaptive budget. Domino augments the DFlash backbone with a cheap sequential GRU-based causal correction that recovers conditional information without extra backbone passes, but the released Domino decoder follows only a single chain. DominoTree occupies the intersection of these lines of work: it preserves the block-parallel backbone and best-first tree expansion, but scores nodes with Domino’s conditional correction rather than with per-position marginals [2607.08642].

A common misconception is that DominoTree is simply DDTree with a different ranking heuristic. The methodological distinction is sharper than that. DDTree and CaDDTree assume a factorized marginal path score, whereas DominoTree carries a branch-specific recurrent state and recomputes the correction on each path, so the score is explicitly path-dependent.

## 2. Conditional scoring via Domino’s GRU correction

Given a verified prefix $x_{\le t}$, a single parallel DFlash pass computes hidden states and base logits for positions $i=1,\dots,B-1$:
$$
H_i \in \mathbb{R}^d,\qquad L_i^{\text{base}} = \mathrm{LMHead}(H_i).
$$
These quantities do not condition on $x_{t+1:i-1}$. Domino adds a causal encoder and a logit-space correction:
$$
S_{i-1}=\mathrm{GRU}(S_{i-2}, E_{x_{t+i-1}}),
$$
$$
\Delta L_i = W_2\,\sigma(W_1[H_i;S_{i-1}]),
$$
$$
L_i = L_i^{\text{base}} + \Delta L_i.
$$
It then samples $x_{t+i}\sim\mathrm{softmax}(L_i)$, embeds it, and updates $S_i$ [2607.08642].

Formally, the position-wise draft distribution becomes
$$
q(x_{t+i}\mid x_{\le t},x_{t+1:t+i-1})
= \mathrm{softmax}\!\left(L_i^{\text{base}} + \Delta L_i(H_i,S_{i-1}(x_{t+1:t+i-1}))\right).
$$
Here $L_i^{\text{base}}$ is path-independent, while $\Delta L_i$ is path-dependent via $S_{i-1}$. This distinction is the basis of DominoTree’s scoring rule. For a tree node $u$ corresponding to tokens $x_{t+1:t+d}$ and GRU state $S_d$, the root-to-node score is
$$
\sigma(u) = \sum_{i=1}^d \log q_{\text{Domino}}(x_{t+i}\mid x_{\le t},x_{t+1:t+i-1})
= \sum_{i=1}^d \log \mathrm{softmax}(L_i^{\text{base}} + \Delta L_i(H_i,S_{i-1}))_{x_{t+i}}.
$$
Each branch therefore carries its own recurrent state, and the correction is recomputed specifically for that path [2607.08642].

This formulation makes explicit why the released Domino decoder and marginal-tree methods are not equivalent. Domino already represents conditional structure that a factorized marginal tree cannot represent; DominoTree is the tree-search procedure that makes that structure usable.

## 3. Tree construction, top-\(M\) restriction, and best-first expansion

The main computational difficulty is that recomputing $\Delta L_i$ for the full vocabulary at every node is expensive. DominoTree addresses this by restricting each node expansion to a candidate top-$M$. For each depth $d$, it extracts once the marginal top-$M$ tokens under $L_d^{\text{base}}$, packs the corresponding $M$ rows of $W_2$ and base logits, and then computes $\Delta L_i$ only on that $M$-slice. This changes the per-node projection from $\mathbb{R}^{256}\to\mathbb{R}^{|V|}$ to $\mathbb{R}^{256}\to\mathbb{R}^{M}$, after which the method selects the top-$k$ children from those corrected logits [2607.08642].

Under a fixed node budget $n$, the expansion is best-first. The heap is initialized with children of the root. While the tree has fewer than $n$ nodes and the heap is nonempty, the algorithm pops the node with highest $\sigma(u)$, adds it to the tree, and, if depth permits, calls the child-construction routine on the branch-specific state to obtain top-$k$ children together with their log-probabilities and next states. These children are then pushed back with updated cumulative score. The resulting tree is prefix-closed and is verified in one target-model pass [2607.08642].

This design is significant because it preserves the tree-search advantages of DDTree and CaDDTree while changing the object being optimized. The expansion is no longer over a fixed per-depth token set with sibling-invariant probabilities. Instead, sibling scores can diverge because each child inherits a different recurrent state. A plausible implication is that the tree better matches the target model when conditional dependencies omitted by marginal drafting are important.

## 4. GPU-native CUDA-graph implementation

DominoTree’s practicality depends not only on the scoring rule but also on how the tree is built. The implementation uses a GPU-native, CUDA-graph builder that is bit-identical to a reference Python implementation, so acceptance is unchanged [2607.08642].

The heavy operations inside the per-node correction are pre-captured as three small CUDA graphs, described as root, below-prefix, and above-prefix. Static input and output buffers avoid Python-to-GPU kernel-launch overhead. Best-first heap operations remain in Python, but the per-node correction routine is replayed from the captured graphs. The implementation still performs a single `cuda.synchronize()` per pop in order to read tokens back, yet the tens of individual kernel launches in eager PyTorch collapse into one per graph [2607.08642].

The paper emphasizes bit-for-bit equivalence to the Python reference: at $T=0$, every accepted token matches row-for-row. This point matters because speculative decoding speedups are only useful if they preserve the target model’s acceptance behavior. DominoTree’s systems contribution is therefore not merely acceleration of the drafter; it is acceleration under an equivalence guarantee for the acceptance path.

## 5. Empirical performance on Qwen3-4B and Qwen3-8B

On Qwen3-4B with block $=16$, node budget $=16$, and $M=64$, DominoTree attains the strongest reported results among the evaluated methods. At $T=0$, the reported figures are: DFlash, $4.48\times$ speedup with $\tau=6.59$; DDTree(16), $4.96\times$ with $\tau=7.47$; CaDDTree, $4.97\times$ with $\tau=7.47$; Domino chain with the best CUDA graph, $6.41\times$ with $\tau=10.27$; and DominoTree(16), $6.63\times$ with $\tau=10.67$. The paper describes DominoTree here as achieving the highest $\tau$ and speedup [2607.08642].

Across eight datasets in the “Overall” aggregate on Qwen3-4B, DominoTree reports speedup $=4.81\times$ and $\tau=7.98$, compared with Domino at speedup $=4.43\times$ and $\tau=7.17$. This corresponds to $+9.2\%$ throughput versus Domino with $95\%$ confidence interval $[7.9,10.6]$, $+7.7\%$ versus DDTree with confidence interval $[5.6,9.8]$, and $+7.3\%$ versus CaDDTree. At $T=\{0.5,1.0\}$, DominoTree keeps the highest $\tau$, up to $9.89$ at $T=0.5$, and wins throughput versus Domino by $9$–$10\%$ and versus DDTree/CaDDTree by $2.6$–$5.2\%$, with only Code sometimes flat or slightly negative at high $T$ [2607.08642].

On Qwen3-8B under the same block and default settings, the profile is more mixed but remains favorable overall. At $T=0$, DominoTree reports $7.32\times$ speedup with $\tau=10.53$, compared with Domino at $7.39\times$ and DDTree at $5.07\times$, yielding $+4.3\%$ versus Domino, marked not significant, and $+24.0\%$ versus DDTree. At $T=0.5$, it is $+6.0\%$ versus Domino and a tie versus DDTree. At $T=1.0$, it remains $+6.0\%$ versus Domino but is reported at $-3.5\%$ versus DDTree, characterized as a small loss on Code [2607.08642].

These results constrain overly broad claims. DominoTree consistently has the highest accepted length at every tested temperature on Qwen3-8B, but the throughput edge over DDTree/CaDDTree narrows at higher temperature. The method’s advantage is therefore strongest when conditional scoring gains outweigh the additional tree-build cost.

## 6. Ablations, operating regimes, and terminological disambiguation

The ablation results make the operating trade-offs explicit. For the node budget $n$, the reported outcomes are: $n=16$ is best on chat workloads, with Alpaca $+16.6\%$ and GSM8K flat; $n=32$ gives a balanced win on both, with GSM8K $+9.9\%$ and Alpaca $+10.9\%$, but at doubled build cost; and $n=64$ is too slow to win throughput. For candidate width $M$, the paper states that $M=32$ under-samples, $M=64$ recovers almost full gain, and $M\ge 128$ shows $\tau$ saturation. For Qwen3-4B at $T=0$, the Python builder takes $3.67$ ms build per round, while the GPU builder takes $2.31$ ms, a reduction of $1.36$ ms. With the GPU builder, DominoTree beats DDTree by $+7.7\%$ throughput; with the Python builder, it would tie or lose at high $T$. A conditioning ablation comparing Cond@16 and Marg@16 isolates a $+9.2\%$ throughput gain from the score function, and an adaptive-budget variant, CondAdaptive, is reported as ineffective because over-confident path-prob calibration makes the rule saturate at the cap, motivating use of fixed $n$ instead [2607.08642].

The summary offered in the source is correspondingly specific. DominoTree “delivers the best of both worlds—block-parallel backbone and tree-structured conditional scoring—by rerunning only a lightweight GRU+head per node.” Candidate restriction preserves most $\tau$ while cutting per-node cost by approximately $|V|/M$, and a GPU-native CUDA-graph builder removes per-node PyTorch overhead. The paper identifies fixed budget $n=16$ and $M=64$ as a strong default, while noting that larger $n$ or $M$ can raise $\tau$ further at the cost of build time and sometimes net throughput. For deployment, it notes that one would integrate the GPU builder into a multi-stream, batched serving stack such as vLLM or SGLang [2607.08642].

The term “DominoTree” is not unique across arXiv. In graph-theoretic survey material, “DominoTree” refers to the parameter $\dtw(G)$, the least width of a tree-decomposition in which each vertex appears in at most two bags; that usage concerns spread-$2$ tree-decompositions and is unrelated to speculative decoding [2604.05690]. In the LLM-inference literature, by contrast, DominoTree denotes the conditional tree-structured drafting procedure described above [2607.08642].

Source: https://www.emergentmind.com/topics/dominotree