Reward-Guided Test-Time Compute (RTTC)

• RTTC is a framework where reward signals guide extra compute allocation during inference, adapting model strategies dynamically.
• It employs methods like candidate reranking, trajectory pruning, and parameter adaptation based on local and global reward estimates.
• RTTC enables efficiency gains in diverse domains such as PDEs, language, and image generation by focusing compute on challenging inputs.

Reward-Guided Test-Time Compute (RTTC) denotes a class of inference-time and test-time adaptation methods in which additional computation is allocated under the control of a reward, verifier, or value signal rather than spent uniformly. The term is used explicitly for query-adaptive selection among test-time strategies in “RTTC: Reward-Guided Collaborative Test-Time Compute” (Muñoz et al., 7 Aug 2025), while closely related work describes the same underlying pattern as reward-model-driven inference-time scaling for PDE foundation models, verifier-guided adaptive test-time inference, dual-phase reward-guided search, reward-guided decoding, and test-time reinforcement learning (Mansingh et al., 2 Sep 2025). Across these formulations, RTTC couples a generator with a reward source and uses the resulting scores to choose among candidate outputs, intermediate trajectories, search actions, or adaptation updates.

1. Definition and conceptual scope

RTTC emerged from the broader test-time scaling literature as a response to the inefficiency of uniform compute allocation. In the surveyed work, the baseline pattern is usually one of three forms: sample more complete outputs and rerank them, search longer trajectories, or adapt model parameters during inference. RTTC modifies this pattern by making the extra computation conditional on reward estimates. In the query-adaptive collaborative framework, the system decides per query whether to do no adaptation, use retrieval-augmented generation, use test-time training, or run both and select the better answer with a pretrained reward model (Muñoz et al., 7 Aug 2025). In verifier-guided adaptive inference for mathematical reasoning, the controller chooses whether to plan, which tools to invoke, which search strategy to use, and what exploration parameter to allocate, again using a process reward model as the control signal (Bilal et al., 1 Feb 2026).

This scope is broader than reranking-only inference. Some RTTC methods use rewards only after full candidate generation, but others apply rewards during generation, during local trajectory repair, or during online parameter updates. Adaptive Rectification Sampling, for example, checks reasoning one step at a time and inserts a trigger sentence only when the current step receives a low process reward score, thereby directing extra generation to a specific failure point instead of lengthening the whole chain (Tan et al., 2 Apr 2025). Test-Time Reinforcement Learning and its descendants go further: they convert extra sampling into pseudo-rewards and then update the policy on unlabeled test inputs, so test-time compute becomes both a search budget and a learning signal (Zuo et al., 22 Apr 2025).

A common misconception is that RTTC is synonymous with Best-of-$N$. Best-of-$N$ is one possible substrate, but several papers define RTTC precisely by moving beyond fixed-width repeated sampling. The adaptive inference framework of (Bilal et al., 1 Feb 2026) distinguishes itself from plain Best-of-$N$, reranking-only verifiers, fixed tree search, and uniform compute scaling. The dual-phase DREAM framework similarly argues that reward signals should govern not only which completed trajectory to keep, but also where, when, and how much sampling budget to spend within a plan–execution decomposition (Cui et al., 29 Sep 2025).

2. Reward signals and objective functions

The reward in RTTC is not a single object. The literature uses analytical physics constraints, learned process reward models, outcome reward models, pseudo-rewards from self-consensus, tool-verified pseudo-labels, tail-based value estimates, and implicit rewards derived from density ratios. In the PDE setting, Analytical Reward Models (ARMs) score adjacent predicted states by conservation error for mass, momentum, or energy, while learned Process Reward Models (PRMs) are trained on stochastic one-step predictions ranked by actual MSE to ground truth (Mansingh et al., 2 Sep 2025). In mathematical reasoning, PRMs typically score local correctness of intermediate steps and aggregate those step scores into a trajectory-level reward by the mean (Bilal et al., 1 Feb 2026). In reward-guided text generation, the target policy is explicitly reward-tilted,

$$\pi_{\beta,B}(y\mid x)\propto \pi_B(y\mid x)\exp(\beta\,r(x,y)),$$

so test-time alignment is formalized as KL-regularized reward maximization under a frozen base model (Geuter et al., 4 Jun 2025).

Process-level and outcome-level rewards play different roles. Process rewards support localized control because they score prefixes, steps, or plan–execution pairs. Outcome rewards support selection among complete outputs or states with estimated long-run payoff. The tail-guided search literature makes this distinction explicit by defining a state value

$$V_N(s):=\mathbb{E}\Big[\max_{1\le i\le N}R_s^{(i)}\Big],$$

which measures the expected best reward obtainable from state $s$ under a continuation budget $N$ (Li et al., 1 Feb 2026). This value is not a step correctness score; it is a prediction of future scaling potential.

Several papers broaden the notion of reward still further. Test-Time Reinforcement Learning uses majority voting across self-generated answers to construct a binary pseudo-reward on unlabeled test inputs (Zuo et al., 22 Apr 2025). Tool Verification for Test-Time Reinforcement Learning replaces plain majority vote with verification-aware voting in which rollouts supported by code-execution evidence receive higher weight, thereby producing a more reliable pseudo-label for RL updates (Liao et al., 2 Mar 2026). In diffusion LLMs, “reward-free guidance” parameterizes the process reward by the log-likelihood ratio between an enhanced model and a reference model, yielding a stepwise guidance rule without training an explicit reward model (Chen et al., 29 Sep 2025). For diffusion image generators, reward-tilted target distributions in output or noise space are written as

$$p^\star(x)\propto p^{\text{base}}(x)\exp(r(x)/\alpha),$$

or equivalently as a tilted noise distribution pushed forward through a frozen generator (Eyring et al., 13 Aug 2025).

This diversity of reward constructions suggests that RTTC is best understood operationally rather than by one specific reward formalism. What matters is that reward estimates alter the allocation of inference-time or adaptation-time computation.

3. Algorithmic patterns of reward-guided allocation

The methods in the literature differ mainly in what object receives additional compute and at what granularity reward intervenes.

Control target	Typical mechanism	Representative papers
Candidate outputs	Best-of-$B$, soft best-of-$n$, verifier reranking	(Mansingh et al., 2 Sep 2025, Geuter et al., 4 Jun 2025, Rashid et al., 6 Feb 2025)
Prefixes or trajectories	PRM-guided pruning, lookahead, local rectification	(Bilal et al., 1 Feb 2026, Cui et al., 29 Sep 2025, Tan et al., 2 Apr 2025)
Query-level strategy	Route between direct inference, RAG, and TTT	(Muñoz et al., 7 Aug 2025)
Parameters at test time	RL or TTT on unlabeled inputs using pseudo- or critic rewards	(Zuo et al., 22 Apr 2025, Liao et al., 2 Mar 2026, Zhang et al., 21 Apr 2026)
Latents or prompts	Noise optimization, trajectory refinement, prompt rewriting	(Uehara et al., 20 Feb 2025, Dai et al., 21 May 2026, Kim et al., 20 Jun 2025)

At the simplest end, RTTC is a sequential search-and-selection procedure. For PDE foundation models, the stochastic autoregressive predictor samples a branching set of next states

$N$0

the reward model scores each candidate pair $N$1, and the rollout keeps

$N$2

The branching factor $N$3 is the explicit test-time compute knob (Mansingh et al., 2 Sep 2025).

More elaborate methods place reward inside the search loop. The verifier-guided adaptive framework of (Bilal et al., 1 Feb 2026) uses a PRM to score steps, aggregates those scores into trajectory rewards, and uses them both within iterations and across iterations. Beam Search prunes prefixes according to running PRM averages, and Lookahead Search commits the next step from the highest-scoring rollout while discarding alternatives. DREAM makes the control problem phase-aware: planning and execution are searched separately, each with its own reward model, and a two-threshold policy allows early stopping when enough high-reward candidates exist or extra budget allocation when all rewards remain poor (Cui et al., 29 Sep 2025). AR-Sampling pushes reward guidance to the step level by checking whether

$N$4

falls below a threshold $N$5, and only then inserting a trigger that tells the model to rethink from that step (Tan et al., 2 Apr 2025).

RTTC also appears as selective use of heavier adaptation mechanisms. In the collaborative RTTC system, the reward model first scores the base answer; if the score exceeds a threshold $N$6, the system returns it immediately. Otherwise it retrieves examples, tries RAG, compares reward again, and escalates to LoRA-based test-time training only when needed (Muñoz et al., 7 Aug 2025). This is reward-guided conditional computation at the query level.

4. Domain-specific instantiations

Although the literature is often discussed through LLM reasoning, RTTC is now explicitly cross-domain. In mathematical reasoning, dynamic PRM-guided inference improves Qwen-2.5-7B on MATH-500 from 71.2% to 81.4% under the “Dynamic + PRM Selection” setting and yields gains on AIME24 and AMO-Bench as well (Bilal et al., 1 Feb 2026). DREAM(+) reports about a 2% accuracy gain over DREAM at comparable token budgets on GSM8K and shows that around 80% of reasoning steps trigger early stopping on GSM8K, compared with around 5% on MATH, indicating that adaptive budget release is especially useful when step difficulty is heterogeneous (Cui et al., 29 Sep 2025). Test-time training variants extend RTTC from inference-only control to online adaptation: TTRL improves Qwen2.5-Math-7B on AIME 2024 from 16.7 to 43.3 using only unlabeled test data (Zuo et al., 22 Apr 2025), while TEMPO pushes OLMO3-7B on AIME 2024 from 33.0% to 51.1% and Qwen3-14B from 42.3% to 65.8% by alternating policy refinement on unlabeled questions with critic recalibration on labeled data (Zhang et al., 21 Apr 2026).

The PDE literature introduces RTTC into operator learning. “Towards Reasoning for PDE Foundation Models” instantiates the first test-time computing strategy for PDEs as greedy best-of-$N$7 branching over stochastic autoregressive predictions, scored either by analytical conservation laws or by learned local reward models (Mansingh et al., 2 Sep 2025). The work reports that PRM-guided TTC is consistently stronger than ARM guidance, that downstream accuracy can be reached after finetuning on only $N$8 of the training data required by an equivalent baseline FM without TTC, and that the strongest model is about 5M parameters while cited baseline PDE models range from 21M to 0.7B (Mansingh et al., 2 Sep 2025). This suggests that in PDE settings, test-time compute can partially substitute for training data and model scale.

Diffusion and image-generation work expands RTTC into latent-space and prompt-space optimization. HyperNoise amortizes reward-guided test-time noise optimization by training a Noise Hypernetwork that perturbs the initial latent in one extra forward pass; on SD-Turbo, GenEval improves from 0.49 to 0.57 at 0.3 s, compared with ReNO at 0.63 and 20.0 s (Eyring et al., 13 Aug 2025). Reward-guided trajectory scaling for diffusion models uses reward-guided noise optimization plus sparse key-step refinement, improving FLUX-1.dev from 0.6516 to 0.7535 on GenEval and from 0.9159 to 1.4703 on ImageReward (Dai et al., 21 May 2026). In text-to-image prompt search, RATTPO iteratively rewrites prompts with an optimizer LLM and a reward-aware hint generator; on ImageReward it reaches 1.132 on Lexica, compared with 0.880 for Paraphrase, and it surpasses other test-time search baselines using up to 3.5 times less inference budget (Kim et al., 20 Jun 2025).

Reward-guided diffusion refinement also appears in scientific design. RERD alternates partial noising and reward-guided denoising for protein and DNA design, and on protein cRMSD it reports median performance of 1.68 against 8.2 for SVDD and 6.3 for a genetic algorithm baseline (Uehara et al., 20 Feb 2025). This extends RTTC beyond language and image synthesis to structured scientific objectives with black-box rewards.

5. Efficiency, scaling laws, and theoretical accounts

A notable feature of the RTTC literature is that it is accompanied by several distinct theoretical programs. One line studies reward-aligned sampling as a tilted distribution. Guided Speculative Inference targets the soft best-of-$N$9 policy

$N$0

uses a smaller proposal model plus a likelihood-ratio correction, and proves a KL bound between its corrected speculative distribution and the optimal tilted policy under a coverage condition (Geuter et al., 4 Jun 2025). Cost-effective reward-guided text generation makes a complementary point at token level: the reward model architecture itself can determine whether inference-time alignment is practical, and a vocabulary-wide next-token reward model reduces reward-model calls from order-$N$1 per decoding step to one per step (Rashid et al., 6 Feb 2025).

A second line studies compute allocation directly. Tail-guided search argues that best-of-$N$2 performance is controlled by the upper tail of the reward distribution and defines the state value $N$3 accordingly. Its Scaling-Law Guided Search predicts future payoff from pilot rollouts, proves vanishing regret compared to perfect-information oracles, and shows expected rewards that would otherwise require a polynomially larger compute budget under BoN (Li et al., 1 Feb 2026). Reward Model Generalization for Compute-Aware Test-Time Reasoning links the generalization error of a PRM to test-time compute optimality. Its answer-accuracy bound shows that lower reward-model error and larger reward margins reduce the number of samples required to find correct answers, and its CATS controller uses reward distributions plus a sparsity proxy for PRM generalization to allocate search budget dynamically (Song et al., 23 May 2025).

A third line focuses on reward reliability under prolonged test-time adaptation. TTRL demonstrates that majority-vote pseudo-rewards can drive RL on unlabeled data (Zuo et al., 22 Apr 2025), but later work identifies failure modes. T$N$4RL analyzes false-popular mode collapse caused by unverified consensus and adds verification-aware voting backed by tool execution, showing that spending compute on verification can be more effective than spending the same budget on more unverified samples (Liao et al., 2 Mar 2026). TEMPO formalizes alternating critic recalibration and policy refinement through an EM lens, arguing that prior test-time training methods are incomplete because they omit the recalibration step and therefore suffer reward drift (Zhang et al., 21 Apr 2026). Together, these papers imply that RTTC scaling is governed not only by how much compute is added, but by whether the reward signal remains informative as the search or policy changes.

6. Limitations, misconceptions, and open directions

The central limitation across RTTC methods is reward quality. Poorly aligned or weakly calibrated rewards can misallocate compute, and extra inference budget can then amplify error rather than correct it. In PDE RTTC, ARM guidance based on conservation laws can degrade MSE because the training data itself may violate exact conservation, so a pure conservation reward misranks candidates (Mansingh et al., 2 Sep 2025). In tool-verified TTRL, a weak 0.5B verifier can make T$N$5RL worse than TTRL, illustrating that a bad verifier may be more harmful than none (Liao et al., 2 Mar 2026). In test-time training for reasoning models, self-generated rewards can drift as the policy evolves, producing performance plateaus and diversity collapse unless the critic is recalibrated (Zhang et al., 21 Apr 2026).

A second misconception is that any increase in compute is beneficial if enough samples are drawn. Several papers argue the opposite. Adaptive inference for math explicitly contrasts its method with uniform test-time scaling and reranking-only verification (Bilal et al., 1 Feb 2026). DREAM shows that dynamic budget allocation helps more when paired with phase-aware search than when added to ordinary beam search (Cui et al., 29 Sep 2025). Sparse trajectory scaling for diffusion reports that full scaling across all denoising steps performs worse than sparse scaling at selected steps (Dai et al., 21 May 2026). These results suggest that RTTC is fundamentally about allocation policy, not raw budget expansion.

A third constraint is systems cost. The PDE paper reports that generating predictions over 500 trajectories with branching factor 1000 took about 2 hours on 8 H100 GPUs after PRM training (Mansingh et al., 2 Sep 2025). RTS raises FLUX runtime from 9.8 s and 50 NFEs to 217.7 s and 1000 NFEs (Dai et al., 21 May 2026). RATTPO reports that one full optimization of 20 iterations for one initial prompt takes about 10 minutes on an A6000 + API setup (Kim et al., 20 Jun 2025). RTTC therefore remains a deployment tradeoff rather than a universally cheap substitute for larger models or retraining.

Open directions in the surveyed literature are comparatively consistent. PDE RTTC is framed as a foundational first step toward richer adaptive reasoning and reinforcement-learning-based PDE algorithms (Mansingh et al., 2 Sep 2025). Adaptive verifier-guided inference still uses a prompt-engineered controller with fixed outer-loop iteration count rather than a learned budget controller (Bilal et al., 1 Feb 2026). Tail-guided search identifies recursive search, broader agentic workflows, and reward-model misspecification as open problems (Li et al., 1 Feb 2026). Diffusion work points toward amortization and implicit reward surrogates: HyperNoise learns a reusable approximation to reward-guided noise optimization (Eyring et al., 13 Aug 2025), while reward-free guidance in diffusion LLMs replaces explicit process rewards with log-likelihood-ratio guidance (Chen et al., 29 Sep 2025). This suggests that the next phase of RTTC research will likely center on better reward estimation, learned or budget-aware controllers, and mechanisms that preserve the benefits of reward-guided search while reducing its runtime overhead.