---
title: Draft Tree Reward in LLM Reasoning
url: https://www.emergentmind.com/topics/draft-tree-reward
type: topic
---

# Draft Tree Reward in LLM Reasoning

Searching arXiv for the cited paper and closely related work on tree-structured rewards and reward-guided tree search.
Draft Tree Reward denotes a family of reward constructions that assign values to *draft* nodes—partial solutions, reasoning prefixes, or draft branches—within a tree-structured search or training procedure, rather than waiting for a single terminal reward at the end of a full trajectory. In contemporary LLM reasoning work, this idea appears as step-wise expected final reward in TreeRPO, per-node process reward in TreeRL, inferred process reward in ReST-MCTS*, and branch-level expected acceptance-length objectives in speculative decoding. The common motivation is explicit in TreeRPO: rewards defined at the full trajectory level provide insufficient guidance for optimizing the intermediate steps of a reasoning process, whereas tree sampling can produce fine-grained and dense reward signals for partial hypotheses [2506.05183]. The term itself is not yet fully standardized: the survey on tree search and reward design states that “Draft Tree Reward” does not appear verbatim there, but can be understood as a special case of the Reward Formulation component, specifically a per-step, tree-structured reward used to guide search during Test-Time Scaling [2510.09988].

## 1. Conceptual definition and scope

Within LLM reasoning, a draft tree is a tree whose nodes represent partial continuations of a solution or decoding process. A Draft Tree Reward is the value attached to such nonterminal nodes or branches. In TreeRPO, the tree consists of partial solutions grown up to depth \(D\); leaf rewards are obtained from a verifiable reward function \(\phi\), and internal-node rewards are the mathematical expectations of downstream rewards estimated by backward propagation over the sampled tree [2506.05183]. In TreeRL, the corresponding object is a per-node “process” reward defined from the correctness frequency of leaves in the node’s subtree, together with global and local advantage terms [2506.11902]. In ReST-MCTS*, the reward is described as a dense, per-step reward signal “drafted” on the fly by repeated tree-search rollouts under an oracle or final-answer verifier [2406.03816].

This family of constructions is broader than outcome supervision. Outcome-supervised methods attach reward only to complete trajectories; draft-tree methods estimate the value of *continuing from here*. TreeRPO states this explicitly as estimating “what happens if I continue from here” via explicit tree sampling [2506.05183]. A plausible implication is that the defining feature of the paradigm is not the particular optimizer—PPO, MCTS, or beam-style search—but the relocation of supervision from terminal completions to intermediate tree states.

The same structural idea appears outside mathematical reasoning. In Group Tree Optimization for speculative decoding, the reward is not correctness but expected acceptance length of a draft tree under the target model; the reward is still attached to draft branches before final verification, and is designed to align training with the decoding-time tree policy [2509.22134]. RADAR likewise formulates draft tree generation as an MDP and assigns reward to the decision to continue or stop tree growth, trading acceptance length against drafting cost [2512.14069].

## 2. Tree construction and reward propagation

A canonical implementation begins by expanding an \(N\)-ary tree of partial solutions. TreeRPO gives an implementation-ready specification: given input question \(q\), at each step \(t\) each active node spawns \(N\) children sampled under the current policy \(\pi_{\theta_{\rm old}}\); tree growth stops at depth \(D\); each sampled branch produces at most \(L_{\rm step}\) new tokens; and branches terminate early if they emit a stop token or exceed the token limit [2506.05183]. Once leaves are collected, each complete path \(P(v_{\rm leaf})\) is scored by a final-answer verifier,
\[
r(v_{\rm leaf})=\phi(P(v_{\rm leaf})).
\]
For any nonleaf node \(v\), reward is propagated bottom-up as the average of children’s rewards,
\[
r(v)=\mathbb{E}_{c\in\mathrm{Children}(v)}[r(c)]
=\frac{1}{|\mathrm{Children}(v)|}\sum_{c\in\mathrm{Children}(v)}r(c).
\]
Numbering nodes at depth \(t\) as \(v_{t,1},\dots,v_{t,G_t}\), TreeRPO defines
\[
R_{i,t}=r(v_{t,i}),
\]
so \(R_{i,t}\) is the expected final reward if one continues from node \(v_{t,i}\) [2506.05183].

TreeRL uses a closely related subtree estimator. For a node \(s_n\), let \(L(s_n)\) be the leaves in its subtree; then
\[
V(s_n)=\frac{1}{|L(s_n)|}\sum_{l\in L(s_n)}\mathbf{1}(l\text{ correct}),
\]
which is the Monte-Carlo probability that the subtree under \(s_n\) eventually yields a correct leaf [2506.11902]. ReST-MCTS* presents two estimators for per-step correctness probability from an \(N\)-rollout tree: a hard estimate that checks whether any reachable leaf equals the oracle answer, and a soft estimate
\[
A_{s_k}^{SE}=(1/N)\sum_{j=1}^N \mathbf{1}(A_j=a^*),
\]
with \(r_{s_k}=A_{s_k}^{SE}\) or \(1-A_{s_k}^{HE}\) used as the step’s correctness probability [2406.03816].

The computational pattern is therefore stable across methods: expand a reasoning tree, evaluate terminal continuations by a verifier or target model, and back up statistics to internal draft nodes. This suggests that Draft Tree Reward is best understood as a *credit-assignment mechanism over prefixes* rather than as a single reward formula.

## 3. Relative, weighted, and branchwise reward formulations

Although the basic tree-backup template is shared, the reward attached to a draft node can be normalized or transformed in several distinct ways.

TreeRPO inherits the group-relative training mechanism of GRPO. At each depth \(t\), siblings of the same parent form a group \(\mathcal{G}\). For siblings with rewards \(\{R_i\}_{i=1}^N\), TreeRPO computes
\[
\mu=\frac1N\sum_{i=1}^N R_i,\qquad
\sigma=\mu(1-\mu),\qquad
A_i=\frac{R_i-\mu}{\sigma}.
\]
It also prunes low-variance groups by discarding any group with reward spread \(\Delta R_{\mathcal{G}}\le \tau\), where \(\Delta R_{\mathcal{G}}\) is defined from the max–min reward difference within the sibling set [2506.05183]. The stated purpose is to avoid weak learning signals.

TreeRL decomposes node quality into global and local advantages relative to the virtual root and the parent:
\[
G_A(s_n)=V(s_n)-V(s_0),\qquad
L_A(s_n)=V(s_n)-V(p(s_n)),
\]
and defines the process reward
\[
R(s_n)=G_A(s_n)+L_A(s_n).
\]
To avoid over-crediting non-leaf nodes that appear in multiple trajectories, TreeRL reweights:
\[
r(s_n)=\frac{R(s_n)}{\sqrt{|L(s_n)|}}
\quad\text{for non-leaf } s_n,
\qquad
r(s_n)=R(s_n)
\quad\text{for leaf } s_n.
\]
The paper further reports that removing either the local or the global advantage term drops accuracy by 1–2 points on math benchmarks, and that discarding the \(1/\sqrt{|L(s_n)|}\) reweight similarly degrades performance [2506.11902].

ReST-MCTS* uses a weighted reward and cumulative quality value. Let \(v_{k-1}\) be the quality value up to step \(k-1\), \(m_k\) the estimated remaining steps, and \(r_{s_k}\) the PRM score. Then
\[
w_{s_k}=\frac{(1-v_{k-1})}{m_k+1}\cdot (1-2r_{s_k}),
\qquad
v_k=\max(v_{k-1}+w_{s_k},0),\quad v_0=0.
\]
The paper states that \(v_k\in[0,1]\) and \(w_{s_k}\le 1-v_{k-1}\) [2406.03816].

In speculative decoding, Group Tree Optimization defines a branch-level expected acceptance length. For branch \(\mathbf{S}_{t,i}\) of length \(l_i\),
\[
\mathbf{L}_{t,i}
=\sum_{j=1}^{l_i}\prod_{k=1}^{j}
\mathcal{T}(\bar x_{t+k,i}\mid \mathbf{x}_{1:t},\bar x_{t+1:t+k-1,i}),
\]
and aggregates branch rewards using
\[
\mathbf{r}_t
=\mathcal{R}(\mathbf{T}_t;\eta)
=\frac{1}{\eta}\log\Bigl(\sum_{i=1}^N \exp(\eta\,\mathbf{L}_{t,i})\Bigr).
\]
The paper characterizes this as a sampling-free objective equal to the expected acceptance length of the draft tree under the target model [2509.22134].

| Setting | Draft-tree reward object | Construction |
|---|---|---|
| TreeRPO | \(R_{i,t}\) at node \(v_{t,i}\) | Average child reward propagated from verifier-scored leaves |
| TreeRL | \(R(s_n)\), \(r(s_n)\) | Subtree correctness value plus global/local advantages and reweighting |
| ReST-MCTS* | \(r_{s_k}\), \(w_{s_k}\), \(v_k\) | Rollout-based step correctness probability and weighted backup |
| GTO | \(\mathbf{r}_t\) | Log-sum-exp over branch expected acceptance lengths |

These differences are consequential. TreeRPO emphasizes sibling-relative discrimination, TreeRL emphasizes decomposed process advantage, ReST-MCTS* emphasizes value shaping for self-training, and GTO emphasizes direct alignment with decoding-time acceptance behavior. A plausible implication is that “Draft Tree Reward” names a *design space* of tree-conditioned reward estimators rather than a unique algorithmic object.

## 4. Optimization objectives and learning regimes

Draft Tree Reward can be used in at least three learning regimes: on-policy RL, search-guided self-training, and inference-time-only search.

TreeRPO explicitly treats each sampled token decision as an on-policy RL transition. For each kept sibling node \(v_{t,i}\), it records state \(s_{t,i}=(q,\text{history up to }t-1)\), action \(a_{t,i}\), instantaneous reward \(r_{t,i}=R_{i,t}\), and advantage \(\hat A_{i,t}=A_i\). The policy objective is a PPO-style clipped surrogate plus KL penalty,
\[
\mathcal{J}_{\rm TreeRPO}(\theta)
=
\mathbb{E}\Biggl[
\frac1G\sum_{i=1}^G\frac1{|o_i|}\sum_{t=1}^{|o_i|}
\min\bigl(r_{i,t}(\theta)\hat A_{i,t},
\mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\hat A_{i,t}\bigr)
\Biggr]
-\beta D_{\rm KL}(\pi_\theta\|\pi_{\rm ref}),
\]
with likelihood ratio
\[
r_{i,t}(\theta)
=
\frac{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}
{\pi_{\theta_{\rm old}}(o_{i,t}\mid q,o_{i,<t})}.
\]
Empirically, TreeRPO reports that the average Pass@1 accuracy of Qwen-2.5-Math on test benchmarks increases from \(19.0\%\) to \(35.5\%\), and that it outperforms GRPO by \(2.9\%\) while reducing average response length by \(18.1\%\) [2506.05183].

TreeRL also uses on-policy RL, but with process rewards computed directly from the tree’s own leaves, eliminating the need for a separate reward model training. Its objective is written either as a policy-gradient objective
\[
J(\theta)
=
\mathbb{E}_{\tau\sim\pi_\theta}
\Bigl[\sum_{i=1}^{T}r(s_i)\log\pi_\theta(a_i\mid s_i)\Bigr]
-\beta\,\mathrm{KL}[\pi_\theta\|\pi_{\rm ref}],
\]
or as a PPO-style surrogate [2506.11902]. The paper states that existing approaches typically train a separate process reward model, which can suffer from distribution mismatch and reward hacking; TreeRL presents its tree-derived process reward as an alternative.

ReST-MCTS* uses inferred draft rewards in a mutual self-training loop rather than centering PPO updates. The inferred per-step rewards serve dual purposes: they act as value targets for refining the process reward model \(V_\theta\), trained with
\[
L_{MSE}=\mathbb{E}_{(q,p,v)\sim D_V}[|V_\theta(p|q)-v|^2],
\]
and they facilitate selection of high-quality traces for supervised self-training of the policy model \(\pi_S\) [2406.03816].

A distinct regime appears in STILL-1 and DARWIN, where tree rewards guide search without necessarily becoming direct RL targets. STILL-1 integrates a policy model, reward model, and MCTS-style procedure in which reward estimates from full-chain rollouts guide expansion choices and subsequent policy fine-tuning by direct preference loss [2411.11694]. DARWIN uses a learned preference reward \(R_\theta(s,I)\) to score partial beams every \(m\) tokens and prune low-reward beams in favor of high-reward ones during decode-time alignment [2406.15193].

## 5. Draft-tree rewards in speculative decoding and non-reasoning settings

The term “draft” has a second, distinct usage in speculative decoding: the draft model proposes tokens or branches that a target model verifies in parallel. Here Draft Tree Reward measures decoding utility rather than reasoning correctness.

RADAR models draft tree generation as a finite MDP \((S,A,P,r,\gamma)\), where the state is the vector of top-\(k\) confidence scores from the draft model, actions are “continue” or “stop,” and reward trades off accepted tokens against generation cost. At nonterminal steps the reward is \(-\alpha\); at stopping time \(T\) the terminal reward is
\[
\frac{\ell_{\rm acc}}{T_{\rm gen}(T)},
\]
where \(\ell_{\rm acc}\) is acceptance length and \(T_{\rm gen}(t)\) is estimated generate-time cost [2512.14069]. The paper reports a speedup of \(3.17\times\)–\(4.82\times\) over auto-regressive decoding, a \(3\%\)–\(29\%\) improvement over EAGLE-3, almost unchanged average acceptance length within \(1.2\%\) of EAGLE-3, and a reduction in average draft calls of \(9.3\%\)–\(34.3\%\) [2512.14069].

Group Tree Optimization addresses what it calls draft policy misalignment: existing training objectives optimize only a single greedy draft path, while decoding follows a tree policy that reranks and verifies multiple branches. Its Draft Tree Reward is explicitly defined as a sampling-free objective equal to the expected acceptance length of the draft tree under the target model, and the paper states that increasing this reward provably improves acceptance length and speedup [2509.22134]. Empirically, GTO increases acceptance length by \(7.4\%\) and yields an additional \(7.7\%\) speedup over EAGLE-3, with average acceptance length \(\tau\) rising from \(6.07\) to \(6.52\) and average speedup ratio from \(3.46\times\) to \(3.73\times\) in the reported table [2509.22134].

These works show that Draft Tree Reward is not restricted to mathematical verification or process supervision. The unifying property is tree-level evaluation of *partial candidate structure* before final commitment. In reasoning RL, the quantity approximates downstream correctness; in speculative decoding, it approximates expected accepted-prefix utility under the target model.

## 6. Relation to interpretable reward trees, misconceptions, and open problems

Draft Tree Reward should be distinguished from earlier work on tree-structured reward *functions* in preference-based RL. In “Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions,” the reward is represented by a binary reward-tree over state-action space with axis-aligned tests and leaf rewards \(r_x\), so that
\[
R_T(s,a)=r^T\phi_T(s,a).
\]
The tree is a transparent model of the reward function itself, learned from pairwise preferences [2112.11230]. “Reward Learning with Trees: Methods and Evaluation” similarly treats reward trees as axis-aligned decision trees over transition features, optimized to fit preference labels [2210.01007]. These are not draft-tree rewards in the LLM reasoning sense; they are interpretable reward parameterizations. The overlap is structural rather than procedural.

A common misconception is therefore to treat every tree-structured reward as a Draft Tree Reward. The recent survey offers a more precise classification: Reward Formulation consists of a single-step reward model \(R_\theta(s_i)=r_{s_i}\) and a partial-solution value model \(V_\theta(p_i)=v_i\); Draft Tree Reward falls squarely in the Process Reward Model category, where per-step intermediate rewards guide search on partial nodes [2510.09988]. In that taxonomy, the essential criterion is not whether the reward function is represented by a tree, but whether reward is assigned to draft states within a search tree.

Another misconception is that dense draft rewards necessarily require a separate step reward model. TreeRPO states the opposite: unlike prior methods that rely on a separate step reward model, it directly estimates these rewards through tree sampling [2506.05183]. TreeRL makes a related claim, arguing that direct on-policy tree-derived process rewards eliminate the need for separate reward model training and may reduce distribution mismatch and reward hacking [2506.11902]. By contrast, STILL-1 and ReST-MCTS* do rely on learned reward or value models, showing that the presence of a Draft Tree Reward does not uniquely determine the architecture of supervision [2411.11694].

Open questions are already articulated in the survey literature. The stated unresolved challenges include scalability of reward models beyond math and code, robustness and overoptimization when reward models are imperfect, applicability in irreversible environments where tree backtracking is impossible, computational efficiency given the slowdowns of current MCTS methods, and integration into self-improvement loops that harvest draft rewards during search and reuse them as training signals later [2510.09988]. This suggests that the central research problem is not merely how to define a better node value, but how to balance fidelity, variance, cost, and train–test alignment when reward is moved from terminal outputs to draft structures.

In present usage, Draft Tree Reward thus names a technical shift in credit assignment for LLM systems: from sparse full-trajectory outcome supervision to tree-conditioned valuation of partial continuations. Whether instantiated as subtree correctness, sibling-relative expected return, weighted process value, or expected acceptance length, the method seeks to make intermediate branches legible to optimization and search, thereby coupling exploration structure with finer-grained reward signals [2506.05183].

Source: https://www.emergentmind.com/topics/draft-tree-reward