Reward-Guided Controlled Decoding

• Reward-guided controlled decoding is a set of inference-time methods that adjust token probabilities using auxiliary reward signals to steer fixed generative models.
• The paradigm encompasses diverse strategies such as tokenwise reweighting, search-based control, and latent-space interventions to balance model plausibility with task-specific rewards.
• Empirical studies highlight improvements in alignment, efficiency, and robustness across tasks like text generation, multimodal captioning, and vision-language-action control.

Reward-guided controlled decoding denotes a family of inference-time methods that steer a frozen generative model toward high-utility outputs by combining the model’s native decoding distribution with an auxiliary reward, value, or preference signal. Rather than updating the base model’s parameters, these methods modify token probabilities, branch selection, beam pruning, proposal acceptance, or latent-path aggregation during generation. The paradigm appears in controlled language generation, multimodal captioning, vision-language-action control, lexical query rewriting, and sequential decoding, with representative formulations in “Controlled Decoding from LLMs” (Mudgal et al., 2023), “Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model” (Deng et al., 2023), “The Missing Piece in Pre-trained Model Evaluation: Reward-Guided Decoding Unlocks Task-Oriented Behavior Without Parameter Updates” (Wang et al., 27 May 2026), and “MPCoT: Reward-Guided Multi-Path Latent Reasoning for Test-Time Scalable Vision-Language-Action” (Zhang et al., 4 Jun 2026).

1. Foundational formulation

A common formalization treats decoding as a control problem over a fixed base distribution. In Controlled Decoding, the frozen base LLM is denoted by $p(z \mid [x,y^t])$, and the tokenwise KL-regularized RL objective is

$$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$

with unique maximizer

$$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$

This establishes a value-guided decoding rule in which the reward signal appears as an exponential tilt of the base model (Mudgal et al., 2023).

Several later methods instantiate the same idea with different levels of approximation. Reward-Augmented Decoding rescales next-token logits with a unidirectional reward model,

$\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$

equivalently

$$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$

thereby steering local sampling toward high-reward continuations (Deng et al., 2023). ARGS uses the related scoring rule

$$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$

over a top-$k$ candidate set, again replacing pure likelihood decoding with reward-weighted search (Khanov et al., 2024).

A more global variant appears in Energy-Based Decoding, which optimizes over full-sequence densities

$$\max_q \; \mathbb{E}_{y \sim q(\cdot \mid x)}[S_x(y)] - \tfrac{1}{\beta}\mathrm{KL}(q(\cdot \mid x)\|p_\theta(\cdot \mid x)),$$

yielding the reward-tilted posterior

$$\pi^*_\beta(y \mid x) = \frac{1}{Z_\beta(x)} p_\theta(y \mid x)\exp(\beta S_x(y)).$$

Here controlled decoding is no longer only a local logit perturbation; it becomes approximate sampling from a sequence-level posterior defined by both prior plausibility and reward (Wang et al., 27 May 2026).

These formulations share a precise structural feature: the base model remains fixed, while control is exerted through a separate reward, value, or scorer module. This separates inference-time alignment from post-training and makes the reward source modular.

2. Control granularity and algorithmic families

The literature spans tokenwise steering, blockwise or search-based control, targeted intervention, and latent-space control. The following table organizes representative systems by the locus of intervention.

Family	Representative methods	Mechanism
Tokenwise reweighting	RAD, ARGS, PARGS, FaRMA, OPAD, CARD	Add reward/value corrections to token logits
Blockwise or search-based control	CD blockwise, MRGD, STORM, EBD, SMC	Sample branches or blocks, score them, retain high-reward continuations
Targeted or structured intervention	GGRO, Collab, RSD, MPCoT	Intervene only at selected positions, agents, thresholds, or latent paths

In tokenwise methods, the reward enters directly into the next-token distribution. PARGS defines

$$\pi(y^i \mid x, y^{1(i-1)}) \propto \pi_{\rm ref}(y^i \mid x, y^{1(i-1)}) \exp[\beta \cdot r_\phi(y^{1i}\mid x)],$$

and explicitly trains the reward model on prefixes rather than only on complete sequences (Rashid et al., 2024). OPAD instead compares a principle-prompted surrogate policy $$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$0 to the unconstrained base model $$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$1, then constructs the aligned policy

$$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$2

making principle adherence itself the reward signal (Zhu et al., 20 Feb 2025). CARD applies personalization only at decoding time through user preference vectors $$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$3 and low-rank logit corrections $$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$4, added to cluster-specific logits (Song et al., 9 Jan 2026).

Search-based methods widen the control horizon. MRGD samples $$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$5 continuations from a multimodal policy and selects the one maximizing

$$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$6

thereby trading object precision against recall at test time (Mañas et al., 15 Aug 2025). STORM repeatedly evaluates partial query rewrites against a BM25 index, converts the partial retrieval score into incremental token rewards

$$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$7

and prunes beams with low reward (Satouf et al., 9 Jun 2026). Sequential Monte Carlo methods generalize this further by defining a reward-augmented target over complete sequences and maintaining weighted particles rather than a single trajectory (Markovic-Voronov et al., 7 Apr 2026).

A distinct class intervenes selectively rather than at every token. GGRO monitors token-level entropy, flags high-uncertainty positions, and injects “nudging tokens” derived from reward-model gradients (Lin et al., 8 Jun 2026). Collab performs token-level policy switching among multiple aligned agents by estimating a long-term utility $$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$8 for each agent-token pair and choosing the maximizer (Chakraborty et al., 27 Mar 2025). Reward-Guided Speculative Decoding accepts the draft token if a Process Reward Model score exceeds a threshold $$J_\lambda([x,y^t];\pi) := \lambda \cdot A([x,y^t];\pi) - D([x,y^t];\pi),$$9, and otherwise falls back to the target model (Liao et al., 31 Jan 2025).

MPCoT moves the entire control process into latent space. At each control step it initializes $$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$0 latent hypotheses, refines each for $$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$1 weight-tied steps, scores them, and softly aggregates them into a final latent state before action decoding. It preserves the original 8-step action interface and generates zero reasoning tokens, so the control mechanism is internal to the policy backbone rather than an explicit textual search (Zhang et al., 4 Jun 2026).

3. Reward, value, and scorer construction

The effectiveness of reward-guided controlled decoding depends on how reward information is represented. One major line uses scalar prefix scorers. Controlled Decoding introduces a learned prefix scorer $$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$2 with two training recipes: CD-FUDGE, which regresses every prefix to the final sequence reward, and CD-Q, which uses Bellman-backup targets. Architecturally, $$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$3 is a small Transformer or MLP that shares token embedding and positional encoding with the base model but has its own parameters (Mudgal et al., 2023).

Another line constructs tokenwise reward heads. FaRMA maps a prefix to a vector in $$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$4 so that all next-token reward scores are produced in a single forward pass. It alternates a Bradley–Terry loss on full-sequence preferences with a Bellman-style constraint,

$$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$5

to ensure that greedy decoding follows prefixes that admit optimal continuations (Rashid et al., 6 Feb 2025). Low-rank RAD replaces a high-rank reward head with the factorization

$$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$6

allowing reward scoring with only a single reward-model call per generated token (Troshin et al., 2024).

A third line internalizes reward modeling inside the policy. Reward Transformer adds a reward channel $$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$7 alongside the policy channel $$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$8 and outputs both $$\pi^*_\lambda(z \mid [x,y^t]) \propto p(z \mid [x,y^t]) \cdot \exp(\lambda \cdot V^*([x,y^t,z])).$$9 and a self-reward estimate $\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$0. Token-level self-reward scores are then trained from pairwise human preferences through a Bradley–Terry loss over averaged prefix rewards (Zhang et al., 24 Feb 2025). MPCoT uses a different internalization strategy: during training only, each final latent branch is decoded into a candidate action chunk and evaluated by expert-action consistency, world-model/VLM progress, and success feedback. These are combined into

$\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$1

and used in an advantage-weighted log-probability loss over latent path choices. At inference, the reward machinery is discarded and only the learned path scorer remains (Zhang et al., 4 Jun 2026).

Reward-guided multimodal control requires specialized reward models. MRGD builds two small multimodal reward models: an object-hallucination reward using a PaliGemma backbone trained on pairwise preferences, and an object-recall reward based on OWLv2, POS tagging, and Sentence-BERT semantic matching (Mañas et al., 15 Aug 2025). Robust multi-objective decoding requires multiple value predictors. RMOD assumes $\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$2 value functions $\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$3 and solves for worst-case simplex weights $\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$4 before decoding under a KL-regularized best-response policy (Son et al., 11 Mar 2025).

The literature also contains a critical correction to early practice. “A Critical Look At Tokenwise Reward-Guided Text Generation” proves a partial-sequence pathology: a Bradley–Terry reward model trained only on full sequences can satisfy the usual loss while assigning arbitrary values to partial-sequence rewards. The proposed remedy is explicit Bradley–Terry training on all prefixes (Rashid et al., 2024). This result sharply distinguishes principled prefix scorers from heuristically reused full-sequence reward models.

4. Empirical behavior and efficiency trade-offs

Empirical results show that reward-guided controlled decoding can improve task reward, long-horizon success, or alignment quality without base-model fine-tuning, but the gains depend strongly on control granularity and reward quality.

In text alignment, ARGS improves the average reward of LLaMA-7B-SFT on HH-RLHF from 5.732 under greedy decoding to 6.872 under ARGS-greedy, a reported $\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$5 gain, and attains a GPT-4 win-or-tie score of 64.33% against greedy decoding (Khanov et al., 2024). Controlled Decoding reports that blockwise CD-Q matches or exceeds the best-of-$\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$6 reward–KL tradeoff while using far smaller $\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$7, and that blockwise CD-Q strongly outperforms PPO, DPO, and IPO in the low-KL regime (Mudgal et al., 2023). FaRMA reduces total model calls per response on TLDR from 656.7 for ARGS, 642.3 for PARGS, and 662.9 for CD to 106.6, while improving reward to $\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$8 and reducing inference time from roughly 29–32 minutes to 5 minutes (Rashid et al., 6 Feb 2025).

Efficiency-oriented designs show comparable trends. Low-rank ARM performs on par with higher-rank RAD on detoxification and sentiment control while requiring only a single reward-model call per generated token, with reported 20×–300× speedups in the reward step depending on $\widetilde z_{t,i} = z_{t,i} + \beta \rho_{t,i},$9 (Troshin et al., 2024). Reward-Guided Speculative Decoding delivers up to $$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$0 fewer FLOPs than decoding with the target model only and achieves up to +3.5 accuracy points over the parallel decoding method on average (Liao et al., 31 Jan 2025). Energy-Based Decoding improves Qwen3-8B-Base on AlpacaEval2.0 from 8.8 to 44.5 and reduces Mistral-7B Math500 latency by 18.9x relative to prior decoding work (Wang et al., 27 May 2026).

Search-heavy methods often deliver larger quality gains at a higher, but structured, compute cost. Streaming Looking Ahead with Token-level Self-reward reports an overall win rate of 79.7% against greedy decoding with a frozen policy model, and 89.4% when combined with DPO (Zhang et al., 24 Feb 2025). The Sequential Monte Carlo framework reaches 87.8% on HumanEval and 78.4% on MATH500 with Qwen2.5-7B, consistently outperforming GRPO (Markovic-Voronov et al., 7 Apr 2026). GGRO outperforms sampling-based baselines on several tasks, for example improving ARC-Challenge accuracy from 84.0% for Vanilla and 92.8% for BoN to 94.3%, while keeping peak GPU usage lower than BoN(64) (Lin et al., 8 Jun 2026).

Multimodal and embodied settings exhibit the same compute–quality trade-off. In MRGD on LLaVA-1.5 for COCO, greedy decoding has $$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$1 and $$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$2, whereas MRGD with $$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$3 reduces $$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$4 to $$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$5 with $$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$6; varying $$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$7 moves smoothly along a precision–recall frontier (Mañas et al., 15 Aug 2025). MPCoT improves LIBERO all-suite single-policy performance from Avg SR = 96.8% and Long SR = 95.3% for OpenVLA-OFT to Avg SR = 98.9% and Long SR = 98.9% for MPCoT $$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$8, and raises CALVIN 5-step success from 72.9% to 89.4%. Even $$p_{\rm RAD}(x_t \mid X) \propto p_{\rm LM}(x_t \mid X)\exp(\beta R(x_{\le t})),$$9 increases per-step latency only from 24 ms to 38 ms, whereas a standard textual CoT baseline rises to 110–160 ms (Zhang et al., 4 Jun 2026).

These results establish a recurrent empirical pattern: test-time reward guidance is most effective when it preserves a strong prior from the base model while using reward information to concentrate exploration, scoring, or latent deliberation.

5. Limitations, misconceptions, and robustness concerns

A central misconception is that any reward model trained on full sequences can be safely queried on prefixes. The partial-sequence pathology result shows otherwise: full-sequence Bradley–Terry training leaves prefix rewards underdetermined, so tokenwise steering can become theoretically ungrounded (Rashid et al., 2024). This objection does not invalidate controlled decoding as a whole, but it does constrain which reward-model constructions are principled.

A second limitation is reward-model error. Streaming-looking-ahead methods note that Token-level Self-reward Modeling is only an approximation of the true reward model, and that too large a search width $$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$0 can inject noise and hurt accuracy (Zhang et al., 24 Feb 2025). GGRO identifies a related failure mode in inference-time alignment: sampling-intensive methods can over-optimize imperfect reward models, producing reward hacking. Its MMLU-Pro experiments show that Best-of-$$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$1 can continue increasing reward while accuracy drops, whereas GGRO’s accuracy increases monotonically with compute (Lin et al., 8 Jun 2026). REFORM turns this vulnerability into a training signal by using reward-guided decoding to generate false negatives and false positives for the reward model itself, then retraining on the discovered failures; average robustness drops under perturbations fall from approximately 12% to approximately 5% on Anthropic HH and PKU according to its summary (Pathmanathan et al., 8 Jul 2025).

A third issue is objective imbalance. Standard multi-objective decoding often relies on fixed weights or averages, which can sacrifice one objective for another. RMOD addresses this by solving a maximin game over reward weights and the decoding policy, reducing the inner problem to a convex optimization over the simplex and targeting worst-case reward improvement; on HH it reports up to 70% worst-case win-rate versus 50% for Uniform (Son et al., 11 Mar 2025). This indicates that “high reward” is not a uniquely defined target once multiple reward heads are active.

Compute cost remains a practical constraint. Early tokenwise methods require one reward-model call per candidate token, and search-based methods can scale with the number of branches, blocks, or particles. Much of the subsequent literature can be read as an engineering response to this bottleneck: unidirectional reward-model caching in RAD, low-rank reward heads in ARM, all-vocabulary reward vectors in FaRMA, internal reward channels in Reward Transformer, thresholded fallback in RSD, selective entropy-triggered intervention in GGRO, and latent-space reasoning in MPCoT (Deng et al., 2023, Troshin et al., 2024, Rashid et al., 6 Feb 2025, Zhang et al., 24 Feb 2025, Liao et al., 31 Jan 2025, Lin et al., 8 Jun 2026, Zhang et al., 4 Jun 2026).

6. Scope, generalization, and research directions

Reward-guided controlled decoding is no longer confined to text-only alignment. Multimodal captioning uses separate reward models for hallucination avoidance and object recall, with user-set control over both relative weighting and search breadth (Mañas et al., 15 Aug 2025). Vision-language-action control uses latent path scoring to improve long-horizon execution while preserving the original action interface (Zhang et al., 4 Jun 2026). Query rewriting uses retrieval metrics as stepwise rewards and remains compatible with a standard inverted index (Satouf et al., 9 Jun 2026). Personalized generation injects per-user preference vectors only at decoding time (Song et al., 9 Jan 2026). LDPC decoding formulates cluster scheduling as an MDP and learns a reward-guided policy by Q-learning, with AM-RELDEC extending this to changing channel conditions through meta-reinforcement learning (Habib et al., 2021).

Across these domains, the recurring design pattern is modular: define a frozen generator or decoder, attach a lightweight reward or value mechanism, and expose explicit inference controls such as $$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$2, $$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$3, $$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$4, block length, beam width, depth, width, or threshold. MPCoT explicitly formulates this as “width” $$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$5 and “depth” $$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$6 in latent reasoning (Zhang et al., 4 Jun 2026); MRGD exposes the precision–recall weight $$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$7, sample count $$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$8, and evaluation frequency $$s(v,c_{<t}) = LM(v \mid c_{<t}) + w \cdot r([c_{<t},v]),$$9 (Mañas et al., 15 Aug 2025); RSD uses a reward threshold $k$0 to allocate expensive target-model calls (Liao et al., 31 Jan 2025). This suggests that controllable compute allocation is not an incidental feature but a defining property of the paradigm.

The research frontier is correspondingly broad. The surveyed works propose adaptive width/depth scheduling based on uncertainty or confidence, hybrid search with value or Q-function estimates, learned switching networks for mixture-of-agents decoding, reward-model extensions for additional multimodal attributes, and broader use of sequence-level samplers such as MCMC and Sequential Monte Carlo (Zhang et al., 24 Feb 2025, Chakraborty et al., 27 Mar 2025, Mañas et al., 15 Aug 2025, Wang et al., 27 May 2026, Markovic-Voronov et al., 7 Apr 2026). A plausible implication is that reward-guided controlled decoding is evolving from a narrow alignment heuristic into a general inference-time control layer for generative systems.

In its most general form, reward-guided controlled decoding is the study of how a fixed generative prior can be reshaped at test time by explicit utility signals. The literature shows that this reshaping can be local or global, exact or approximate, tokenwise or latent, single-objective or robustly multi-objective, and training-free or training-light. Its central challenge is not whether reward can influence decoding, but how to do so with correct credit assignment, bounded overhead, and resistance to reward-model error.