Speculative Token Trees in AI Decoding

• Speculative token trees are rooted, usually prefix-closed structures that organize candidate continuations for parallel verification in autoregressive models.
• They employ fixed and adaptive construction strategies, including DP-optimized and diffusion-derived methods, to maximize accepted tokens within a node budget.
• Verification techniques range from top-down token-level checks to sequence-aware traversals, balancing token acceptance with throughput and hardware cost considerations.

A speculative token tree is a rooted, usually prefix-closed structure used in speculative decoding to organize multiple draft continuations of an autoregressive model so that a larger target model can verify many candidate tokens in parallel in a single forward pass. In this formulation, the root represents the current accepted context or last confirmed token, each non-root node represents a proposed token or partial continuation, and each root-to-leaf path corresponds to one draft sequence. The central quantity is the acceptance length: the number of speculative tokens that can be committed after target-side verification. Across the literature, speculative token trees appear as fixed heuristic trees, dynamic or adaptive trees, diffusion-derived trees, and hardware-aware or cost-aware trees; they also appear with distinct verification rules, including top-down token-level verification, sampling-without-replacement verification, and leaf-to-root traversal verification (Miao et al., 2023, Chen et al., 2024, Wang et al., 2024, Xiong et al., 2024, Weng et al., 18 May 2025, Ringel et al., 14 Apr 2026, Zhang et al., 1 Jun 2026).

1. Formal structure and objective

The common formalization is a rooted directed tree whose nodes encode candidate continuations under a lightweight draft model. SpecInfer defines a token tree as a rooted tree whose nodes each represent a hypothesized continuation, with each node labeled by a token and each root-to-node path defining a candidate sequence (Miao et al., 2023). Sequoia, OPT-Tree, DDTree, and CaDDTree use closely related views: a tree is characterized by total node budget, maximum depth, and branching pattern, while every root-to-leaf path denotes one speculative continuation (Chen et al., 2024, Wang et al., 2024, Ringel et al., 14 Apr 2026, Zhang et al., 1 Jun 2026).

A recurring surrogate objective is the expected accepted-token count or expected acceptance length. In OPT-Tree, the expectation is written as

$E[A] = \sum_{v\in T}\hat p(v),$

where $\hat p(v)$ is the product of draft-model probabilities along the path to node $v$ (Wang et al., 2024). Sequoia uses an equivalent additive form,

$F(T)=\sum_{v\in T} f(v),$

with $f(v)$ defined as the product of positional acceptance probabilities along the path (Chen et al., 2024). For diffusion draft trees, DDTree and CaDDTree define a surrogate expected acceptance length

$\sum_{u\in T} Q(u),$

or equivalently $\Phi(T)=\sum_{s\in T}\pi_s$, again exploiting additive decomposition over nodes (Ringel et al., 14 Apr 2026, Zhang et al., 1 Jun 2026). This convergence across formulations suggests that speculative token trees are most naturally understood as budgeted prefix sets whose value is the cumulative mass of their retained prefixes.

The tree abstraction generalizes single-chain speculative decoding. A chain wastes work when an early rejection invalidates all descendants, whereas a tree retains multiple alternatives and allows verification to continue along other branches. C2T states this contrast directly: in vanilla speculative decoding the draft tokens are arranged in a single chain, while in tree-structured speculation only a rejected node’s subtree is pruned rather than the entire chain (Huo et al., 19 Feb 2025).

2. Construction strategies

Early tree construction strategies were predominantly fixed. SpecInfer builds trees by expanding each leaf with top-$k_d$ children at depth $d$, optionally merging trees from multiple small speculative models (Miao et al., 2023). Sequoia retains the fixed-budget setting but replaces hand-crafted shapes with a dynamic programming optimizer over tree size and depth; under its positional-acceptance assumption, the optimal tree solves

$$c(n)=\max_{a_1+\cdots+a_{n-1}=n-1}\Bigl\{1+\sum_{i=1}^{n-1} p_i\,c(a_i)\Bigr\},$$

with a depth-bounded variant $\hat p(v)$0 for bounded trees (Chen et al., 2024).

Subsequent work shifted toward runtime adaptation. OPT-Tree searches, at each decoding step, for the tree structure that maximizes the mathematical expectation of the acceptance length under a node budget $\hat p(v)$1, and reports that if the draft model is powerful enough and the node budget is sufficient, it can generate more than ten tokens in a single step (Wang et al., 2024). DySpec uses a greedy max-heap over expandable nodes, scoring each candidate by estimated marginal contribution derived from draft probabilities, and proves greedy optimality under mild assumptions; it reports $\hat p(v)$2 tree construction for a tree of size $\hat p(v)$3 (Xiong et al., 2024). TALON adopts a budget-driven, confidence-aware expansion rule: after a robust top-$\hat p(v)$4 root initialization, it retains nodes satisfying

$\hat p(v)$5

which yields “deep-and-narrow” trees in deterministic contexts and “shallow-and-wide” trees in uncertain contexts (Liu et al., 12 Jan 2026).

Other methods change the scoring signal rather than the search procedure. C2T augments joint probability with entropy and depth, using

$\hat p(v)$6

and a lightweight classifier to prune candidate nodes; empirically it reduces the total number of candidate tokens by approximately $\hat p(v)$7 while maintaining or improving acceptance length (Huo et al., 19 Feb 2025). SMART reframes expansion as a hardware-aware marginal analysis problem and expands a node only when its marginal benefit-cost ratio exceeds the current tree-level speedup, rather than merely when its likelihood is high (Wang et al., 9 Apr 2026).

A separate branch of work derives trees from non-autoregressive or relaxed drafters. DDTree constructs a prefix-closed tree directly from per-position marginals of a block diffusion drafter by selecting the top-$\hat p(v)$8 prefixes according to a surrogate factorized probability, using a best-first max-heap in $\hat p(v)$9 time (Ringel et al., 14 Apr 2026). CaDDTree keeps the same diffusion-tree basis but treats budget as a runtime variable rather than a fixed hyperparameter (Zhang et al., 1 Jun 2026). In visual autoregressive models, LANTERN++ uses static tree drafting with a fixed branching schedule and a multiplicative relaxed acceptance condition, motivated by token-selection ambiguity in vision generation (Park et al., 10 Feb 2025).

Family	Representative methods	Main criterion
Fixed or DP-optimized trees	SpecInfer, Sequoia, LANTERN++	Predefined or DP-selected topology under a budget
Dynamic construction	OPT-Tree, DySpec, TALON, C2T, SMART	Runtime search using probability, confidence, classifier, or marginal utility
Diffusion-derived trees	DDTree, CaDDTree	Best-first selection from block marginals

3. Verification algorithms and exactness

The standard verification pattern is a single target-model forward pass over the entire tree using a tree-specific attention mask. EVICT describes this as feeding the pruned draft tree into the target model in one forward pass by using a block-sparse “tree” attention mask, after which verification proceeds layer by layer while exactly preserving the target model’s original autoregressive sampling distribution (Pan et al., 1 May 2026). DDTree uses an ancestor-only attention mask: each node attends to the past context and only to its own ancestors in the tree (Ringel et al., 14 Apr 2026). SpecInfer calls the same general mechanism tree-based parallel decoding and proves exact distribution preservation for its multi-step speculative sampling verifier (Miao et al., 2023).

A substantial line of work modifies the verifier rather than the tree. Sequoia replaces i.i.d. draft sampling with sampling without replacement and updates residual target and draft distributions after each rejection; its theorem states both an optimal-transport property for $v$0 and a cover property when the draft support covers the target support within $v$1 draws (Chen et al., 2024). Recursive Speculative Decoding likewise builds draft trees with Gumbel-Top-$v$2 or stochastic beam search without replacement and applies recursive rejection sampling, with Theorem 3.1 guaranteeing exact recovery of the target distribution (Jeon et al., 2024).

Traversal Verification departs more sharply from token-level top-down checking. It verifies a tree by visiting deepest leaves first, tracking sequence-level acceptance variables and preserving potentially valid subsequences that top-down schemes would discard. It proves that the resulting distribution is identical to that of the target model and that, on a single chain, Traversal Verification attains the maximum possible expected acceptance length among all valid chain verifiers (Weng et al., 18 May 2025). Dynamic Delayed Tree Expansion later reports that, under matched settings, Traversal Verification dominates consistently over OT-based methods, and motivates delayed branching so that i.i.d. expansion occurs deeper in the tree, where draft and target distributions diverge more (Thomas et al., 19 Feb 2026).

4. Cost models, utility criteria, and adaptive verification

A major development in the literature is the shift from acceptance-length maximization to throughput maximization. SMART names the core failure mode the “efficiency paradox”: draft and verification costs can grow super-linearly, so larger trees may increase accepted tokens yet reduce wall-clock speedup, especially when GPUs move from memory-bound to compute-bound regimes (Wang et al., 9 Apr 2026). Its runtime controller accepts a node only if

$v$3

where $v$4 is the current global speedup estimate (Wang et al., 9 Apr 2026).

Sequoia already included a hardware-aware speedup model,

$v$5

and chooses tree size and depth by combining its dynamic program with an empirically fitted verification-time model (Chen et al., 2024). CaDDTree pushes this further for diffusion drafters by optimizing

$v$6

and proving unimodality of the throughput curve under convex verification cost, which enables a greedy stopping rule that finds the optimal budget per round without offline budget search (Zhang et al., 1 Jun 2026).

EVICT introduces an expert-aware criterion specialized to sparse Mixture-of-Experts targets. In MoE verification, each tree node can activate a different subset of experts, so the union of activated experts grows rapidly with tree size, increasing memory-bandwidth demand and latency. EVICT estimates benefit by

$v$7

profiles verification latency offline as $v$8, and chooses

$v$9

It is training-free, hyperparameter-free, and lossless, and reports up to $F(T)=\sum_{v\in T} f(v),$0 speedup over autoregressive decoding, an average $F(T)=\sum_{v\in T} f(v),$1 speedup over EAGLE-3, $F(T)=\sum_{v\in T} f(v),$2 fewer unique experts per iteration, and a $F(T)=\sum_{v\in T} f(v),$3 reduction in target verification latency on average (Pan et al., 1 May 2026).

Training objectives have also become tree-aware. GTO argues that draft models are commonly trained for a single greedy path while inference uses a tree policy. It defines a Draft Tree Reward as a log-sum-exp aggregation over branchwise expected acceptance lengths and reports that optimizing this reward increases acceptance length by $F(T)=\sum_{v\in T} f(v),$4 and yields an additional $F(T)=\sum_{v\in T} f(v),$5 speedup over EAGLE-3 on average (Hu et al., 26 Sep 2025).

5. Systems, hardware backends, and architectural variants

Speculative token trees are tightly coupled to systems design. SpecInfer’s tree-based parallel decoding uses a topology-aware causal mask and a fused pass over the entire tree; it reports $F(T)=\sum_{v\in T} f(v),$6-$F(T)=\sum_{v\in T} f(v),$7 speedup for distributed LLM inference and $F(T)=\sum_{v\in T} f(v),$8-$F(T)=\sum_{v\in T} f(v),$9 for offloading-based inference while preserving the same generative performance (Miao et al., 2023). Sequoia explicitly fits performance models to target hardware and reports up to $f(v)$0 on Llama2-7B, $f(v)$1 on Llama2-13B, $f(v)$2 on Vicuna-33B, and $f(v)$3 in an offloading setting for exact Llama2-70B inference latency on L40 (Chen et al., 2024).

Backend portability is nontrivial. EAGLE-Pangu ports EAGLE-3-style tree speculative decoding to a Pangu teacher backend on Ascend NPUs using an explicit branch/commit cache manager, accelerator-safe tree tensorization with nonnegative indices, and a fused-kernel-compatible verification path with an eager fallback. On 240 turns from MT-Bench and HumanEval-style prompts, it reports $f(v)$4 average throughput improvement and up to $f(v)$5 at p99 over teacher-only greedy decoding (Han et al., 9 Mar 2026).

Tree verification is no longer confined to Transformer-only targets. STree introduces the first scalable tree-based speculative decoding algorithm for state-space models and hybrid SSM/Transformer architectures. It defines a binary tree mask $f(v)$6 with $f(v)$7 exactly when node $f(v)$8 lies on the unique path from the root to node $f(v)$9, and uses a fused TreeScan kernel to avoid materializing per-branch states. It reports end-to-end generation speedups of about $\sum_{u\in T} Q(u),$0 on MT-Bench and about $\sum_{u\in T} Q(u),$1-$\sum_{u\in T} Q(u),$2 on HumanEval and GSM-8K relative to autoregressive decoding, with memory usage reduced by $\sum_{u\in T} Q(u),$3-$\sum_{u\in T} Q(u),$4 against unrolled baselines in some settings (Wu et al., 20 May 2025).

The abstraction also extends beyond token-level decoding. SPEX defines a speculative token tree over partial reasoning states in Tree-of-Thought search, implements it in SGLang, and reports $\sum_{u\in T} Q(u),$5 speedup for different ToT reasoning algorithms; when combined with token-level speculative decoding, it reports cumulative speedups of up to $\sum_{u\in T} Q(u),$6 (Zhong et al., 11 May 2026).

6. Empirical behavior, trade-offs, and recurring misconceptions

One persistent misconception is that larger trees are always better. Multiple papers reject this. SMART shows that large trees can produce negative wall-clock speedup when verification overhead dominates (Wang et al., 9 Apr 2026). CaDDTree states directly that expected acceptance length is non-decreasing in budget and therefore always favors larger trees regardless of verification cost, which is precisely why a throughput objective is needed (Zhang et al., 1 Jun 2026). EVICT reaches the same conclusion in MoE models, where expert-union growth can make verification disproportionately expensive (Pan et al., 1 May 2026).

A second misconception is that depth is the dominant predictor of acceptance. An empirical study across code generation, mathematical reasoning, logical reasoning, and open-ended chat finds that task type is a stronger predictor of acceptance than tree depth. Using TinyLlama-1.1B as draft and Llama-2-7B-Chat-GPTQ as target over 99,768 speculative nodes from 200 prompts, it reports that only the chat domain consistently yields an expected accepted length exceeding $\sum_{u\in T} Q(u),$7 token per step under a $\sum_{u\in T} Q(u),$8, $\sum_{u\in T} Q(u),$9 budget, while the entropy-acceptance correlation is consistently negative but weak, with $\Phi(T)=\sum_{s\in T}\pi_s$0 in $\Phi(T)=\sum_{s\in T}\pi_s$1 (Mahmoud, 16 Apr 2026). This suggests that speculation budgets, draft-model choice, and even whether to speculate at all can be domain dependent.

A third misconception is that one verification rule dominates in all senses. Traversal Verification reports acceptance-length improvements of roughly $\Phi(T)=\sum_{s\in T}\pi_s$2-$\Phi(T)=\sum_{s\in T}\pi_s$3 and throughput gains of about $\Phi(T)=\sum_{s\in T}\pi_s$4-$\Phi(T)=\sum_{s\in T}\pi_s$5 over existing methods in its experiments (Weng et al., 18 May 2025), while Dynamic Delayed Tree Expansion argues that delayed branching plus a dynamic neural selector can allow OT-based methods such as SpecInfer to outperform Traversal Verification for the first time, achieving $\Phi(T)=\sum_{s\in T}\pi_s$6 higher average throughput across a wide range of models, datasets, and sampling settings (Thomas et al., 19 Feb 2026). The literature therefore presents a genuine methodological tension: token-level top-down verification, sequence-aware traversal, without-replacement schemes, and OT-based solvers optimize different parts of the draft–verify trade-off.

Empirically, the gains from speculative token trees are substantial but heterogeneous. DySpec reports throughput improvement up to $\Phi(T)=\sum_{s\in T}\pi_s$7 and latency reduction up to $\Phi(T)=\sum_{s\in T}\pi_s$8 on Llama2-70B under low temperature, and throughput up to $\Phi(T)=\sum_{s\in T}\pi_s$9 under high temperature (Xiong et al., 2024). TALON reports up to $k_d$0 end-to-end speedup over autoregressive decoding (Liu et al., 12 Jan 2026). LANTERN++ reports up to $k_d$1 latency speedup for visual autoregressive models while maintaining high image quality (Park et al., 10 Feb 2025). These results indicate that speculative token trees are not a single algorithm but a design space spanning tree construction, verification, system realization, and cost modeling. The unifying principle is stable: organize speculative continuations as a tree, verify them in parallel, and make each verified token count.