Value-Guided Efficient Decoding

• Value-Guided Efficient Decoding is a design pattern that integrates a learned value function into autoregressive generation to guide search toward high-reward continuations.
• VED employs techniques like Monte Carlo Tree Search, value-guided top-k sampling, and blockwise beam search to overcome the limitations of purely likelihood-based decoding.
• By aligning inference-time decisions with long-horizon reward objectives, VED improves generation quality and diversity, as evidenced by experiments in sentiment steering and recommendation.

Value-Guided Efficient Decoding (VED) denotes a family of decoding-time control methods in which a learned value function scores partial generations and steers search or sampling toward higher-reward continuations, rather than relying only on next-token likelihood or retraining the full policy with RLHF. Across recent work, the shared premise is that autoregressive generation is a sequential decision problem with long-horizon, reward-based objectives, while standard decoding remains locally likelihood-driven; VED addresses this discrepancy by injecting prefix-level value estimates into inference-time decision rules, including Monte-Carlo Tree Search, value-guided top-$k$ sampling, blockwise beam search, and budgeted tree expansion (Liu et al., 2023, Liu et al., 4 Mar 2025, Jiang et al., 11 Feb 2026).

1. Conceptual scope and motivation

VED arises from a recurring mismatch between how language or recommendation models are trained and how they are decoded. In PPO-based controlled text generation, the model is trained against a terminal reward and a value function over partial states, yet standard deployment commonly keeps only the policy and decodes with greedy decoding or top-$p$ sampling. The criticism is that such decoding is purely local, whereas the task objective is long-horizon and reward-based. In generative recommendation, the corresponding failure mode is described as a probability-reward mismatch: likelihood-dominated decoding expands the most probable prefixes rather than the most valuable ones, which can prune high-reward but low-probability branches early and produce homogeneous candidate sets (Liu et al., 2023, Jiang et al., 11 Feb 2026).

Within this broad framing, VED is not a single algorithm but a design pattern. One strand uses the value model already produced by PPO training to guide Monte-Carlo Tree Search during decoding. Another treats value-guided decoding as an efficient alternative to RLHF and focuses on improving the value estimator itself through Monte Carlo supervision and iterative on-policy refinement. A third develops a budget-aware tree decoder for generative recommendation, where value and uncertainty determine where extra search is worth spending (Liu et al., 2023, Liu et al., 4 Mar 2025, Jiang et al., 11 Feb 2026).

A common misconception is that search becomes unnecessary once a model has been aligned with PPO or related objectives. The cited work argues the opposite: inference-time search can still improve output quality, and the under-explored asset is often the value model rather than the policy alone (Liu et al., 2023).

2. Formal foundations

The underlying formalization is sequential. In controlled text generation, given a prompt $w$, the state is

$s_t = (w, x_{<t}),$

the action is the next token

$a_t = x_t,$

and the policy model is $p_\theta(a_t \mid s_t)$. PPO optimizes a terminal reward $r(s_{T+1})$ with step rewards that include a KL penalty to a reference policy $p_{\theta_0}$. It jointly trains a policy $p_\theta(a_t \mid s_t)$ and a value model $V_\phi(s_t)$, where the value objective fits $p$0 to the return

$p$1

The central claim is that $p$2 is already trained to evaluate partial outputs under the associated policy, so discarding it at inference discards the model component most aligned with prefix scoring (Liu et al., 2023).

A more general VED formulation is given through a KL-regularized RL objective. Let $p$3 be a pretrained or SFT model and $p$4 a reward model over complete responses. The objective is

$p$5

Under the deterministic token-level MDP used in that work, the optimal policy takes the exponential-tilting form

$p$6

with

$p$7

In this formulation, decoding-time control reduces to estimating a prefix value function accurately enough that it can reweight or rank candidate continuations (Liu et al., 4 Mar 2025).

In generative recommendation, the same principle is expressed on a prefix tree of Semantic IDs. The model likelihood factorizes autoregressively,

$p$8

but reward depends on final item quality $p$9. The paper explicitly states that there can exist sequences $w$0 such that

$w$1

VED is then introduced as a way to direct limited decoding budget toward prefixes with higher expected downstream return rather than higher local likelihood (Jiang et al., 11 Feb 2026).

3. PPO-MCTS and value-guided tree decoding

"Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding" presents PPO-MCTS, a decoding algorithm that integrates the PPO value network and policy network during inference-time generation. For each token, the method builds a tree rooted at the current partial sequence $w$2, runs $w$3 simulations, and selects the next token from the root children’s visit counts. Nodes correspond to states, edges to actions, and the tree stores $w$4, $w$5, and $w$6 (Liu et al., 2023).

The simulation has four stages. In selection, a PUCT-style rule is used: $w$7 where $w$8 is the child state after action $w$9. The paper explicitly uses $s_t = (w, x_{<t}),$0 rather than $s_t = (w, x_{<t}),$1 in the selection rule because PPO has step-level KL penalties and discounting, so edge-level $s_t = (w, x_{<t}),$2 better reflects the actual return structure. In expansion, the algorithm computes $s_t = (w, x_{<t}),$3 at the first unexplored node and expands the top-$s_t = (w, x_{<t}),$4 actions. In evaluation, the node is scored by

$s_t = (w, x_{<t}),$5

or by the terminal reward if $s_t = (w, x_{<t}),$6 is terminal. The method does not use Monte-Carlo rollouts for efficiency. In backup, the appendix gives

$s_t = (w, x_{<t}),$7

$s_t = (w, x_{<t}),$8

$s_t = (w, x_{<t}),$9

After $a_t = x_t,$0 simulations, decoding uses

$a_t = x_t,$1

If the search reaches $a_t = x_t,$2, it jumps directly to backup rather than expanding further (Liu et al., 2023).

One of the paper’s key modifications is “initializing $a_t = x_t,$3 with $a_t = x_t,$4”: $a_t = x_t,$5 The reported reason is practical rather than merely aesthetic: without this initialization, exploration is severely suppressed because PPO value scales can be large, the search degenerates toward greedy decoding, and diversity suffers. Raising $a_t = x_t,$6 can partially compensate, but goal satisfaction then drops (Liu et al., 2023).

The empirical evaluation covers sentiment steering on OpenWebText, toxicity reduction on RealToxicityPrompts, knowledge introspection on several commonsense QA datasets, and helpful and harmless chatbots on HH-RLHF. Reported results include $a_t = x_t,$7 desired sentiment for positive steering versus $a_t = x_t,$8 for PPO, $a_t = x_t,$9 desired sentiment for negative steering versus $p_\theta(a_t \mid s_t)$0 for PPO, average max toxicity dropping from $p_\theta(a_t \mid s_t)$1 to $p_\theta(a_t \mid s_t)$2, QA accuracy improving from $p_\theta(a_t \mid s_t)$3 to $p_\theta(a_t \mid s_t)$4, usefulness improving from $p_\theta(a_t \mid s_t)$5 to $p_\theta(a_t \mid s_t)$6, and a $p_\theta(a_t \mid s_t)$7 absolute higher human win rate for helpful/harmless chatbots. The paper also reports that PPO-MCTS outperforms longer PPO training and best-of-$p_\theta(a_t \mid s_t)$8 decoding (Liu et al., 2023).

These results are presented as evidence that search itself matters. The stepwise-value baseline, which evaluates top-$p_\theta(a_t \mid s_t)$9 tokens independently without tree search, has much lower goal satisfaction and much worse fluency, indicating that simply possessing a value model is not equivalent to conducting value-guided search (Liu et al., 2023).

4. Iterative value-function optimization for decoding-time control

"Iterative Value Function Optimization for Guided Decoding" treats value-guided decoding as a cheaper alternative to RLHF because it keeps model weights frozen and modifies only inference-time search. Its central claim is that the bottleneck is value accuracy: decoding-time control is only as good as the value estimates plugged into it. The proposed Iterative Value Function Optimization (IVO) framework therefore improves the value model rather than the policy network (Liu et al., 4 Mar 2025).

IVO has two components. The first is Monte Carlo Value Estimation. For each prompt $r(s_{T+1})$0, the method samples multiple trajectories

$r(s_{T+1})$1

scores each full trajectory by the reward model,

$r(s_{T+1})$2

and assigns the observed terminal reward as a Monte Carlo target to every prefix on the sampled trajectory: $r(s_{T+1})$3 The value model $r(s_{T+1})$4 is then trained with squared regression,

$r(s_{T+1})$5

The stated motivation is that multiple sampled trajectories reduce variance and provide better coverage of the state space than one-trajectory estimation (Liu et al., 4 Mar 2025).

The second component is Iterative On-Policy Optimization. Starting from a current value function $r(s_{T+1})$6, the method defines a value-guided policy

$r(s_{T+1})$7

samples trajectories from this policy, scores them with the reward model, retrains $r(s_{T+1})$8, and repeats. The paper describes the loop as better value model $r(s_{T+1})$9 better guided policy $p_{\theta_0}$0 better trajectories $p_{\theta_0}$1 better value model. This is meant to reduce the distribution shift that arises when value learning uses only base-policy trajectories (Liu et al., 4 Mar 2025).

The decoding layer contains two practical approximations. In value-guided top-$p_{\theta_0}$2 sampling, value guidance is applied only to top-$p_{\theta_0}$3 next-token candidates under the base model. In value-guided blockwise beam search, the decoder maintains $p_{\theta_0}$4 candidate sequences, samples $p_{\theta_0}$5 continuation blocks of length $p_{\theta_0}$6, scores the resulting $p_{\theta_0}$7 candidates with $p_{\theta_0}$8, keeps the top $p_{\theta_0}$9, and repeats until completion. An appendix introduces a more efficient version that applies value scoring only every $p_\theta(a_t \mid s_t)$0 tokens, which reduces inference cost while keeping most of the benefit (Liu et al., 4 Mar 2025).

The experimental evaluation spans summarization on TL;DR, multi-turn dialogue on Anthropic HH, and instruction following on UltraFeedback. In summarization, IVO achieves the best reward/KL tradeoff, and in blockwise beam search it reaches the highest reward, around $p_\theta(a_t \mid s_t)$1. In multi-turn dialogue, GPT-4 win rate against the base policy is reported as $p_\theta(a_t \mid s_t)$2 for FUDGE, $p_\theta(a_t \mid s_t)$3 for VAS, $p_\theta(a_t \mid s_t)$4 for DPO, $p_\theta(a_t \mid s_t)$5 for IPO, and $p_\theta(a_t \mid s_t)$6 for IVO. In instruction following, IVO achieves $p_\theta(a_t \mid s_t)$7 win rate on AlpacaEval 2. The paper also states that main experiments use $p_\theta(a_t \mid s_t)$8 sampled trajectories per prompt, that a second optimization iteration usually helps a lot, and that more than $p_\theta(a_t \mid s_t)$9 iterations yields diminishing returns (Liu et al., 4 Mar 2025).

A plausible implication is that VED can be decomposed into two partially independent problems: estimating a prefix value function and designing a decoder that can exploit it. IVO concentrates almost entirely on the first problem, whereas PPO-MCTS concentrates on the second.

5. Budget-aware VED in generative recommendation

"Spend Search Where It Pays: Value-Guided Structured Sampling and Optimization for Generative Recommendation" introduces VED as the candidate-construction and decoding component of V-STAR. Here the aim is not exhaustive tree search but budgeted search on a SID prefix tree under strict decoding budget. The motivating failures are insufficient exploration and advantage compression. The paper defines the reward range of a candidate set $V_\phi(s_t)$0 as

$V_\phi(s_t)$1

and the GRPO-style group advantage as

$V_\phi(s_t)$2

It then bounds the magnitude of $V_\phi(s_t)$3 by $V_\phi(s_t)$4, and via Popoviciu’s inequality bounds $V_\phi(s_t)$5 by $V_\phi(s_t)$6. The intended conclusion is that when beam search produces a collapsed candidate set with small $V_\phi(s_t)$7, the advantages become nearly indistinguishable; this is called advantage compression (Jiang et al., 11 Feb 2026).

The value model estimates downstream return from a prefix state $V_\phi(s_t)$8: $V_\phi(s_t)$9 The implementation uses a lightweight value head on top of the policy backbone: a shallow Transformer block plus an MLP regressor. Because exact-match terminal reward is sparse, the paper defines dense semantic step rewards using frozen text-encoder item embeddings and trains the value function by TD learning: $p$00 This gives the decoder a reward-aligned lookahead signal over prefixes (Jiang et al., 11 Feb 2026).

VED ranks prefixes with a joint acquisition score

$p$01

where

$p$02

The interpretation given in the paper is that $p$03 measures whether the prefix is promising, while $p$04 measures whether the next step is still uncertain enough that extra search might help (Jiang et al., 11 Feb 2026).

The algorithm has four stages. It begins with a low-cost probability-guided beam search to build a shallow tree. Every node in the initial tree is evaluated once to populate $p$05. Selection uses a UCB-style score

$p$06

and repeatedly chooses the child with highest $p$07 until a leaf or terminal node is reached. Expansion is gated by depth-wise decisiveness: if $p$08 denotes all nodes at depth $p$09, then

$p$10

and a visited node $p$11 is expanded if and only if

$p$12

Expansion adds one new child sampled from yet-unexpanded valid children according to normalized policy probabilities. Search continues until

$p$13

where cost is measured as the number of backbone forward tokens consumed during decoding (Jiang et al., 11 Feb 2026).

The training configuration relevant to VED includes Qwen2.5-1.5B as backbone, $p$14 candidates per query, SID length $p$15, hierarchical weights $p$16, discount factor $p$17, acquisition exploration coefficient $p$18, and beam width $p$19 for VED initialization. The paper notes that standard beam search is used at inference by default for serving efficiency, so VED is primarily a training-time mechanism, though it can also be used at inference when extra compute is available (Jiang et al., 11 Feb 2026).

On Amazon Review subsets Industrial and Office Products, the decoding ablation reports the following values.

Decoder	Industrial	Office
Beam Search	NDCG@10 0.1194, HR@10 0.1606	NDCG@10 0.1299, HR@10 0.1684
Top-K	NDCG@10 0.1090, HR@10 0.1538	NDCG@10 0.1226, HR@10 0.1644
VED	NDCG@10 0.1217, HR@10 0.1641	NDCG@10 0.1340, HR@10 0.1746

The paper also reports that the combined acquisition score outperforms value-only and entropy-only variants, that on candidate pools of size $p$20 VED achieves the best or tied-best diversity and the highest best-in-set reward, and that in a 5-day A/B test on WeChat Channels, V-STAR improved GMV by $p$21 and GMV-Normal by $p$22 over BeamSearch+GRPO (Jiang et al., 11 Feb 2026).

6. Comparative properties, efficiency trade-offs, and limitations

Across these works, the most stable pattern is that VED uses a prefix value estimator to correct the myopia of likelihood-based decoding, but the exact role of search differs.

Method	Decoder form	Reported emphasis
PPO-MCTS	MCTS with PPO policy prior and PPO value model	Reduce train/test mismatch in partial-sequence scoring
IVO	Value-guided top-$p$23 sampling and blockwise beam search	Improve value accuracy for decoding-time control
VED in V-STAR	Budgeted tree search with value-plus-entropy acquisition	Spend search on decisive prefixes under strict budget

PPO-MCTS emphasizes policy-specific prefix evaluation. The value model is preferable to a reward model for MCTS guidance because it is trained on partial sequences and tailored to the associated policy. The ablation replacing the value model with the reward model performs worse, which is presented as evidence that full-sequence reward prediction is not an adequate substitute for partial-state value estimation in this setting (Liu et al., 2023).

IVO emphasizes estimation quality and distribution shift. It explicitly argues that if trajectories come only from $p$24, the learned value function mostly covers the base-policy region of the output space rather than the better regions that guided decoding seeks. Its iterative on-policy data collection is designed to move supervision toward those better regions (Liu et al., 4 Mar 2025).

VED in V-STAR emphasizes budget allocation. It does not attempt exhaustive search and instead combines value with policy entropy so that extra search is concentrated on high-value, high-uncertainty prefixes. The paper states that uncertainty is most useful as an expansion gate rather than as a standalone objective, since the combined score performs best in the acquisition ablation (Jiang et al., 11 Feb 2026).

The compute trade-off is explicit in all three formulations. PPO-MCTS is about $p$25 times slower than direct decoding if policy and value networks are the same size, although KV caching still applies, the subtree under a decoded token can be reused for the next token, and at least $p$26 nodes need not be recomputed. IVO is still more expensive than plain decoding because value evaluation adds inference overhead, especially for tokenwise guidance, but blockwise sampling optimization reduces inference cost while keeping most of the benefit. VED in V-STAR is designed for strict latency constraints and therefore uses a shallow initialization, selective deepening, and a cost budget defined by backbone forward tokens (Liu et al., 2023, Liu et al., 4 Mar 2025, Jiang et al., 11 Feb 2026).

The limitations are likewise method-specific but structurally related. PPO-MCTS requires access to the PPO value model, and some PPO implementations make this harder through reward whitening, adaptive KL, or other details that force approximations; the paper also notes that if the value model were manipulated or poorly aligned, search could potentially steer the system toward harmful behavior. IVO depends on reward model quality, requires multiple rounds of data collection, and remains constrained by base-policy support because guidance is often applied over top-$p$27 tokens and base-policy sampling. VED in V-STAR still depends on the quality of the learned value model, cannot recover every low-probability branch under a fixed budget, and is not the default production decoder in the strictest serving setting (Liu et al., 2023, Liu et al., 4 Mar 2025, Jiang et al., 11 Feb 2026).

A final misconception addressed by the literature is that value guidance is merely a heuristic add-on to standard decoding. The technical arguments are more specific. In PPO-MCTS, the value model is the component explicitly trained to evaluate partial sequences under the current policy. In IVO, the optimal KL-regularized decoder is written directly in terms of exponential reweighting by the value function. In V-STAR, the acquisition rule, gating condition, and budgeted tree traversal all use value as the principal signal, with entropy providing an auxiliary uncertainty estimate. This suggests that VED is best understood not as ad hoc rescoring, but as a family of inference-time control procedures whose quality is determined jointly by prefix value estimation and the structure of the search policy (Liu et al., 2023, Liu et al., 4 Mar 2025, Jiang et al., 11 Feb 2026).