Draft Tree Reward in LLM Reasoning

• Draft Tree Reward is a method that values partial solution nodes to provide dense, stepwise guidance during tree search.
• It encompasses techniques like sibling-relative averaging, subtree correctness estimation, and branch-wise reward shaping.
• Applications include on-policy RL and speculative decoding, where it improves efficiency and reasoning accuracy.

Searching arXiv for the cited paper and closely related work on tree-structured rewards and reward-guided tree search. Draft Tree Reward denotes a family of reward constructions that assign values to draft nodes—partial solutions, reasoning prefixes, or draft branches—within a tree-structured search or training procedure, rather than waiting for a single terminal reward at the end of a full trajectory. In contemporary LLM reasoning work, this idea appears as step-wise expected final reward in TreeRPO, per-node process reward in TreeRL, inferred process reward in ReST-MCTS*, and branch-level expected acceptance-length objectives in speculative decoding. The common motivation is explicit in TreeRPO: rewards defined at the full trajectory level provide insufficient guidance for optimizing the intermediate steps of a reasoning process, whereas tree sampling can produce fine-grained and dense reward signals for partial hypotheses (Yang et al., 5 Jun 2025). The term itself is not yet fully standardized: the survey on tree search and reward design states that “Draft Tree Reward” does not appear verbatim there, but can be understood as a special case of the Reward Formulation component, specifically a per-step, tree-structured reward used to guide search during Test-Time Scaling (Wei et al., 11 Oct 2025).

1. Conceptual definition and scope

Within LLM reasoning, a draft tree is a tree whose nodes represent partial continuations of a solution or decoding process. A Draft Tree Reward is the value attached to such nonterminal nodes or branches. In TreeRPO, the tree consists of partial solutions grown up to depth $D$; leaf rewards are obtained from a verifiable reward function $\phi$, and internal-node rewards are the mathematical expectations of downstream rewards estimated by backward propagation over the sampled tree (Yang et al., 5 Jun 2025). In TreeRL, the corresponding object is a per-node “process” reward defined from the correctness frequency of leaves in the node’s subtree, together with global and local advantage terms (Hou et al., 13 Jun 2025). In ReST-MCTS*, the reward is described as a dense, per-step reward signal “drafted” on the fly by repeated tree-search rollouts under an oracle or final-answer verifier (Zhang et al., 2024).

This family of constructions is broader than outcome supervision. Outcome-supervised methods attach reward only to complete trajectories; draft-tree methods estimate the value of continuing from here. TreeRPO states this explicitly as estimating “what happens if I continue from here” via explicit tree sampling (Yang et al., 5 Jun 2025). A plausible implication is that the defining feature of the paradigm is not the particular optimizer—PPO, MCTS, or beam-style search—but the relocation of supervision from terminal completions to intermediate tree states.

The same structural idea appears outside mathematical reasoning. In Group Tree Optimization for speculative decoding, the reward is not correctness but expected acceptance length of a draft tree under the target model; the reward is still attached to draft branches before final verification, and is designed to align training with the decoding-time tree policy (Hu et al., 26 Sep 2025). RADAR likewise formulates draft tree generation as an MDP and assigns reward to the decision to continue or stop tree growth, trading acceptance length against drafting cost (Ma et al., 16 Dec 2025).

2. Tree construction and reward propagation

A canonical implementation begins by expanding an $N$-ary tree of partial solutions. TreeRPO gives an implementation-ready specification: given input question $q$, at each step $t$ each active node spawns $N$ children sampled under the current policy $\pi_{\theta_{\rm old}}$; tree growth stops at depth $D$; each sampled branch produces at most $L_{\rm step}$ new tokens; and branches terminate early if they emit a stop token or exceed the token limit (Yang et al., 5 Jun 2025). Once leaves are collected, each complete path $P(v_{\rm leaf})$ is scored by a final-answer verifier,

$\phi$0

For any nonleaf node $\phi$1, reward is propagated bottom-up as the average of children’s rewards,

$\phi$2

Numbering nodes at depth $\phi$3 as $\phi$4, TreeRPO defines

$\phi$5

so $\phi$6 is the expected final reward if one continues from node $\phi$7 (Yang et al., 5 Jun 2025).

TreeRL uses a closely related subtree estimator. For a node $\phi$8, let $\phi$9 be the leaves in its subtree; then

$N$0

which is the Monte-Carlo probability that the subtree under $N$1 eventually yields a correct leaf (Hou et al., 13 Jun 2025). ReST-MCTS* presents two estimators for per-step correctness probability from an $N$2-rollout tree: a hard estimate that checks whether any reachable leaf equals the oracle answer, and a soft estimate

$N$3

with $N$4 or $N$5 used as the step’s correctness probability (Zhang et al., 2024).

The computational pattern is therefore stable across methods: expand a reasoning tree, evaluate terminal continuations by a verifier or target model, and back up statistics to internal draft nodes. This suggests that Draft Tree Reward is best understood as a credit-assignment mechanism over prefixes rather than as a single reward formula.

3. Relative, weighted, and branchwise reward formulations

Although the basic tree-backup template is shared, the reward attached to a draft node can be normalized or transformed in several distinct ways.

TreeRPO inherits the group-relative training mechanism of GRPO. At each depth $N$6, siblings of the same parent form a group $N$7. For siblings with rewards $N$8, TreeRPO computes

$N$9

It also prunes low-variance groups by discarding any group with reward spread $q$0, where $q$1 is defined from the max–min reward difference within the sibling set (Yang et al., 5 Jun 2025). The stated purpose is to avoid weak learning signals.

TreeRL decomposes node quality into global and local advantages relative to the virtual root and the parent: $q$2 and defines the process reward

$q$3

To avoid over-crediting non-leaf nodes that appear in multiple trajectories, TreeRL reweights: $q$4 The paper further reports that removing either the local or the global advantage term drops accuracy by 1–2 points on math benchmarks, and that discarding the $q$5 reweight similarly degrades performance (Hou et al., 13 Jun 2025).

ReST-MCTS* uses a weighted reward and cumulative quality value. Let $q$6 be the quality value up to step $q$7, $q$8 the estimated remaining steps, and $q$9 the PRM score. Then

$t$0

The paper states that $t$1 and $t$2 (Zhang et al., 2024).

In speculative decoding, Group Tree Optimization defines a branch-level expected acceptance length. For branch $t$3 of length $t$4,

$t$5

and aggregates branch rewards using

$t$6

The paper characterizes this as a sampling-free objective equal to the expected acceptance length of the draft tree under the target model (Hu et al., 26 Sep 2025).

Setting	Draft-tree reward object	Construction
TreeRPO	$t$7 at node $t$8	Average child reward propagated from verifier-scored leaves
TreeRL	$t$9, $N$0	Subtree correctness value plus global/local advantages and reweighting
ReST-MCTS*	$N$1, $N$2, $N$3	Rollout-based step correctness probability and weighted backup
GTO	$N$4	Log-sum-exp over branch expected acceptance lengths

These differences are consequential. TreeRPO emphasizes sibling-relative discrimination, TreeRL emphasizes decomposed process advantage, ReST-MCTS* emphasizes value shaping for self-training, and GTO emphasizes direct alignment with decoding-time acceptance behavior. A plausible implication is that “Draft Tree Reward” names a design space of tree-conditioned reward estimators rather than a unique algorithmic object.

4. Optimization objectives and learning regimes

Draft Tree Reward can be used in at least three learning regimes: on-policy RL, search-guided self-training, and inference-time-only search.

TreeRPO explicitly treats each sampled token decision as an on-policy RL transition. For each kept sibling node $N$5, it records state $N$6, action $N$7, instantaneous reward $N$8, and advantage $N$9. The policy objective is a PPO-style clipped surrogate plus KL penalty,

$\pi_{\theta_{\rm old}}$0

with likelihood ratio

$\pi_{\theta_{\rm old}}$1

Empirically, TreeRPO reports that the average Pass@1 accuracy of Qwen-2.5-Math on test benchmarks increases from $\pi_{\theta_{\rm old}}$2 to $\pi_{\theta_{\rm old}}$3, and that it outperforms GRPO by $\pi_{\theta_{\rm old}}$4 while reducing average response length by $\pi_{\theta_{\rm old}}$5 (Yang et al., 5 Jun 2025).

TreeRL also uses on-policy RL, but with process rewards computed directly from the tree’s own leaves, eliminating the need for a separate reward model training. Its objective is written either as a policy-gradient objective

$\pi_{\theta_{\rm old}}$6

or as a PPO-style surrogate (Hou et al., 13 Jun 2025). The paper states that existing approaches typically train a separate process reward model, which can suffer from distribution mismatch and reward hacking; TreeRL presents its tree-derived process reward as an alternative.

ReST-MCTS* uses inferred draft rewards in a mutual self-training loop rather than centering PPO updates. The inferred per-step rewards serve dual purposes: they act as value targets for refining the process reward model $\pi_{\theta_{\rm old}}$7, trained with

$\pi_{\theta_{\rm old}}$8

and they facilitate selection of high-quality traces for supervised self-training of the policy model $\pi_{\theta_{\rm old}}$9 (Zhang et al., 2024).

A distinct regime appears in STILL-1 and DARWIN, where tree rewards guide search without necessarily becoming direct RL targets. STILL-1 integrates a policy model, reward model, and MCTS-style procedure in which reward estimates from full-chain rollouts guide expansion choices and subsequent policy fine-tuning by direct preference loss (Jiang et al., 2024). DARWIN uses a learned preference reward $D$0 to score partial beams every $D$1 tokens and prune low-reward beams in favor of high-reward ones during decode-time alignment (Hung et al., 2024).

5. Draft-tree rewards in speculative decoding and non-reasoning settings

The term “draft” has a second, distinct usage in speculative decoding: the draft model proposes tokens or branches that a target model verifies in parallel. Here Draft Tree Reward measures decoding utility rather than reasoning correctness.

RADAR models draft tree generation as a finite MDP $D$2, where the state is the vector of top-$D$3 confidence scores from the draft model, actions are “continue” or “stop,” and reward trades off accepted tokens against generation cost. At nonterminal steps the reward is $D$4; at stopping time $D$5 the terminal reward is

$D$6

where $D$7 is acceptance length and $D$8 is estimated generate-time cost (Ma et al., 16 Dec 2025). The paper reports a speedup of $D$9–$L_{\rm step}$0 over auto-regressive decoding, a $L_{\rm step}$1–$L_{\rm step}$2 improvement over EAGLE-3, almost unchanged average acceptance length within $L_{\rm step}$3 of EAGLE-3, and a reduction in average draft calls of $L_{\rm step}$4–$L_{\rm step}$5 (Ma et al., 16 Dec 2025).

Group Tree Optimization addresses what it calls draft policy misalignment: existing training objectives optimize only a single greedy draft path, while decoding follows a tree policy that reranks and verifies multiple branches. Its Draft Tree Reward is explicitly defined as a sampling-free objective equal to the expected acceptance length of the draft tree under the target model, and the paper states that increasing this reward provably improves acceptance length and speedup (Hu et al., 26 Sep 2025). Empirically, GTO increases acceptance length by $L_{\rm step}$6 and yields an additional $L_{\rm step}$7 speedup over EAGLE-3, with average acceptance length $L_{\rm step}$8 rising from $L_{\rm step}$9 to $P(v_{\rm leaf})$0 and average speedup ratio from $P(v_{\rm leaf})$1 to $P(v_{\rm leaf})$2 in the reported table (Hu et al., 26 Sep 2025).

These works show that Draft Tree Reward is not restricted to mathematical verification or process supervision. The unifying property is tree-level evaluation of partial candidate structure before final commitment. In reasoning RL, the quantity approximates downstream correctness; in speculative decoding, it approximates expected accepted-prefix utility under the target model.

6. Relation to interpretable reward trees, misconceptions, and open problems

Draft Tree Reward should be distinguished from earlier work on tree-structured reward functions in preference-based RL. In “Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions,” the reward is represented by a binary reward-tree over state-action space with axis-aligned tests and leaf rewards $P(v_{\rm leaf})$3, so that

$P(v_{\rm leaf})$4

The tree is a transparent model of the reward function itself, learned from pairwise preferences (Bewley et al., 2021). “Reward Learning with Trees: Methods and Evaluation” similarly treats reward trees as axis-aligned decision trees over transition features, optimized to fit preference labels (Bewley et al., 2022). These are not draft-tree rewards in the LLM reasoning sense; they are interpretable reward parameterizations. The overlap is structural rather than procedural.

A common misconception is therefore to treat every tree-structured reward as a Draft Tree Reward. The recent survey offers a more precise classification: Reward Formulation consists of a single-step reward model $P(v_{\rm leaf})$5 and a partial-solution value model $P(v_{\rm leaf})$6; Draft Tree Reward falls squarely in the Process Reward Model category, where per-step intermediate rewards guide search on partial nodes (Wei et al., 11 Oct 2025). In that taxonomy, the essential criterion is not whether the reward function is represented by a tree, but whether reward is assigned to draft states within a search tree.

Another misconception is that dense draft rewards necessarily require a separate step reward model. TreeRPO states the opposite: unlike prior methods that rely on a separate step reward model, it directly estimates these rewards through tree sampling (Yang et al., 5 Jun 2025). TreeRL makes a related claim, arguing that direct on-policy tree-derived process rewards eliminate the need for separate reward model training and may reduce distribution mismatch and reward hacking (Hou et al., 13 Jun 2025). By contrast, STILL-1 and ReST-MCTS* do rely on learned reward or value models, showing that the presence of a Draft Tree Reward does not uniquely determine the architecture of supervision (Jiang et al., 2024).

Open questions are already articulated in the survey literature. The stated unresolved challenges include scalability of reward models beyond math and code, robustness and overoptimization when reward models are imperfect, applicability in irreversible environments where tree backtracking is impossible, computational efficiency given the slowdowns of current MCTS methods, and integration into self-improvement loops that harvest draft rewards during search and reuse them as training signals later (Wei et al., 11 Oct 2025). This suggests that the central research problem is not merely how to define a better node value, but how to balance fidelity, variance, cost, and train–test alignment when reward is moved from terminal outputs to draft structures.

In present usage, Draft Tree Reward thus names a technical shift in credit assignment for LLM systems: from sparse full-trajectory outcome supervision to tree-conditioned valuation of partial continuations. Whether instantiated as subtree correctness, sibling-relative expected return, weighted process value, or expected acceptance length, the method seeks to make intermediate branches legible to optimization and search, thereby coupling exploration structure with finer-grained reward signals (Yang et al., 5 Jun 2025).