Reward-Refine Methods

Updated 7 July 2026

Reward-Refine is a framework that employs iterative, reward-guided refinement to optimize outputs, reward functions, and training data.
It integrates proposal generation, evaluation, and local revision, forming a closed loop used in text generation, robotics, and privacy-preserving synthesis.
These methods drive performance gains by feedback-driven revisions, achieving measurable improvements in metrics such as ICAT scores and success rates.

Searching arXiv for papers on Reward-Refine and closely related frameworks. Reward-Refine denotes a family of methods in which a reward signal, judge, process critic, or diagnostic is used not only to score outputs but to iteratively improve them, the underlying reward specification, or the training data itself. Across retrieval-augmented generation, long-form reasoning, reinforcement learning, privacy-preserving synthesis, and robotics, the recurring design is a closed loop in which candidate plans, answers, trajectories, or reward functions are generated, evaluated, revised, and then either selected, distilled, or optimized further (Salemi et al., 10 Apr 2025). The broader reward-modeling literature places such systems within a pipeline of data collection, feedback acquisition, reward modeling, and policy optimization, while formal work on the reward hypothesis specifies the exact preference conditions under which scalar reward maximization is an adequate representation of goals (Yu et al., 18 Jun 2025, Yang et al., 2022).

1. Historical emergence and scope

Early precursors framed reward refinement as automated reward shaping rather than answer reranking. “Self-Supervised Online Reward Shaping in Sparse-Reward Environments” alternates classification-based reward inference and policy update steps, using the original sparse reward to rank observed trajectories and infer a denser reward for subsequent RL updates (Memarian et al., 2021). “Auto MC-Reward” then made reward code itself the object of iterative refinement through a Reward Designer, Reward Critic, and Trajectory Analyzer, so that dense reward functions for Minecraft were repeatedly proposed, checked, and revised from trajectory evidence (Li et al., 2023). In reasoning, “ReST-MCTS*” introduced an explicit process-reward-guided tree search and self-training loop, in which inferred step values guide search and the resulting traces are reused to refine both policy and value models (Zhang et al., 2024).

From 2025 onward, the pattern diversified. Some systems use reward as an inference-time selector; some use it to curate training data; some refine reward code; some co-evolve policy and reward estimation inside one model. The survey “Reward Models in Deep Reinforcement Learning: A Survey” groups such methods by reward source, mechanism, and learning paradigm, which provides a useful background taxonomy for Reward-Refine systems even when the individual papers target different domains (Yu et al., 18 Jun 2025).

System	Domain	Reward-Refine mechanism
Plan-and-Refine	Retrieval-augmented generation	multiple plans, iterative editing, reward-based answer selection
Refine-n-Judge	Preference-data curation	iterative refinement gated by an LLM judge
RewardDS	Privacy-preserving fine-tuning	reward-guided filtering and self-optimizing refinement
RDA	Robotics RL	VLM trajectory analysis and reward-code revision
SPARK	LLM/LVLM post-training	verifiable rewards recycled into pointwise, pairwise, and reflection training

2. Canonical architecture

Taken together, the literature suggests a common architecture with four stages: proposal generation, reward or diagnostic evaluation, local or structural revision, and final selection or optimization. In “Plan-and-Refine,” this appears as global exploration over plans $P=\{p_i\}_{i=1}^N$ , local exploitation through iterative edits $o_t=M_E(x,p,o_{t-1})$ , and reward-based final choice

$o_f=\arg\max_{o\in O_F} M_R(x,o),$

where the reward model scores factuality and coverage jointly (Salemi et al., 10 Apr 2025). In “Refine-n-Judge,” the same pattern is instantiated without a separate learned reward network: the LLM refines an answer $Ans_t$ , judges whether $Ans_{t+1}$ is preferred to $Ans_t$ , and stops when no further improvement is preferred, producing monotone preference chains $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ (Cayir et al., 3 Aug 2025).

A second recurring distinction is between reranking-style and optimization-style Reward-Refine. “Plan-and-Refine” and “Review, Remask, Refine” are inference-time systems: they refine candidate outputs and then select among them, without using the reward model as a reinforcement signal for the generator (Salemi et al., 10 Apr 2025, Mounier et al., 7 Jul 2025). By contrast, “SPARK” and “Reason, Reward, Refine” feed the reward signal back into policy updates. SPARK builds on RLVR, recycles verifiable rewards from rollouts, and simultaneously trains the model itself as a generative reward model through pointwise reward score, pairwise comparison, and reflection-conditioned evaluation, creating what the paper calls a positive co-evolving feedback loop (Liu et al., 26 Sep 2025). “Reason, Reward, Refine” identifies the first reasoning error, generates structured feedback, and applies policy gradient with KL regularization so that the model learns to revise its own physics solutions without using ground-truth solutions as generation targets (Jaiswal et al., 6 Jul 2026).

A third distinction concerns what is being refined. Some methods refine answers; some refine datasets; some refine reward functions; some refine labels used to train process reward models. This suggests that Reward-Refine is better understood as a systems pattern than as a single algorithmic family.

3. Retrieval, text generation, and LLM post-training

In retrieval-augmented generation, “Plan-and-Refine: Diverse and Comprehensive Retrieval-Augmented Generation” is a direct and explicit instantiation. A planner $M_P$ samples diverse plans $p=\{(a_i,q_i,r_i)\}_{i=1}^{|p|}$ , where $a_i$ is an aspect title, $o_t=M_E(x,p,o_{t-1})$ 0 a retrieval query, and $o_t=M_E(x,p,o_{t-1})$ 1 a justification. Retrieved evidence $o_t=M_E(x,p,o_{t-1})$ 2 is conditioned into an initial response $o_t=M_E(x,p,o_{t-1})$ 3, after which an editor $o_t=M_E(x,p,o_{t-1})$ 4 iteratively improves coverage, factual alignment, and coherence; a ModernBERT-base reward model then chooses the candidate with highest predicted ICAT quality (Salemi et al., 10 Apr 2025). On ANTIQUE, the full system improves $o_t=M_E(x,p,o_{t-1})$ 5 from $o_t=M_E(x,p,o_{t-1})$ 6 to $o_t=M_E(x,p,o_{t-1})$ 7, a relative gain of about $o_t=M_E(x,p,o_{t-1})$ 8, and on TREC the reported relative gain is $o_t=M_E(x,p,o_{t-1})$ 9 or $o_f=\arg\max_{o\in O_F} M_R(x,o),$ 0, depending on the comparison stated in the paper (Salemi et al., 10 Apr 2025).

For iterative text generation with masked diffusion, “Review, Remask, Refine (R3)” uses a Process Reward Model to review intermediate blocks, converts low PRM scores into higher remasking probabilities, and then forces the diffusion model to regenerate the targeted spans (Mounier et al., 7 Jul 2025). On 127 MATH problems, simple diffusion attains $o_f=\arg\max_{o\in O_F} M_R(x,o),$ 1 accuracy, R3 reaches $o_f=\arg\max_{o\in O_F} M_R(x,o),$ 2 with $o_f=\arg\max_{o\in O_F} M_R(x,o),$ 3, and block-wise Best-of- $o_f=\arg\max_{o\in O_F} M_R(x,o),$ 4 reaches $o_f=\arg\max_{o\in O_F} M_R(x,o),$ 5, while R3 uses far fewer PRM calls than block-wise BoN (Mounier et al., 7 Jul 2025). The mechanism is still Reward-Refine, but the “refine” operator is remasking rather than editing.

For automatic data curation, “Refine-n-Judge” turns the judge itself into the reward mechanism. The same LLM acts as refiner and judge, iteratively improving answers on criteria including accuracy, completeness, clarity, conciseness, and relevance, and stopping once the refinement is no longer preferred (Cayir et al., 3 Aug 2025). Applied to approximately $o_f=\arg\max_{o\in O_F} M_R(x,o),$ 6k TULU queries, the resulting datasets improve downstream SFT: for Llama 3.1-8B, AlpacaEval rises from $o_f=\arg\max_{o\in O_F} M_R(x,o),$ 7 to $o_f=\arg\max_{o\in O_F} M_R(x,o),$ 8, AlpacaEval 2.0 from $o_f=\arg\max_{o\in O_F} M_R(x,o),$ 9 to $Ans_t$ 0, and MT-Bench from $Ans_t$ 1 to $Ans_t$ 2; cross-judge win-rates against TULU-only models range from $Ans_t$ 3 to $Ans_t$ 4 for the 8B model and $Ans_t$ 5 to $Ans_t$ 6 for the 70B model (Cayir et al., 3 Aug 2025).

For open-ended reasoning, “Direct Reasoning Optimization” introduces the Reasoning Reflection Reward (R3), computed internally by the same model over a reference outcome, and combines it with a length reward in

$Ans_t$ 7

so that RL can be applied even when no generic verifiable reward exists for long-form outputs (Xu et al., 16 Jun 2025). The paper positions this as a fully self-contained setup for open-ended reasoning tasks such as ParaRev, where reasoning must be rewarded through consistency with the reference outcome rather than exact answer matching (Xu et al., 16 Jun 2025).

In RL, Reward-Refine frequently targets the reward specification itself. “RewardDS” is a privacy-preserving example: the client trains a generation proxy model and a reward proxy model with DP-SGD, each with privacy budget $Ans_t$ 8, for a total privacy cost of $Ans_t$ 9, and the server then uses reward-guided filtering and self-optimizing refinement to improve synthetic data before fine-tuning the target model (Wang et al., 23 Feb 2025). The reward model is trained with a Bradley-Terry-style logistic loss, and the refinement loop repeatedly uses reward differences and feedback to regenerate higher-quality synthetic responses. On medical QA, RewardDS reaches ROUGE-1 $Ans_{t+1}$ 0, ROUGE-L $Ans_{t+1}$ 1, and PPL $Ans_{t+1}$ 2; on financial QA it reaches ROUGE-1 $Ans_{t+1}$ 3, ROUGE-L $Ans_{t+1}$ 4, and PPL $Ans_{t+1}$ 5; on code generation it reaches Pass@1 $Ans_{t+1}$ 6 and Pass@10 $Ans_{t+1}$ 7 (Wang et al., 23 Feb 2025).

In robotics, “RDA: Reward Design Agent for Reinforcement Learning” refines reward code from trajectory semantics rather than scalar training curves alone. A VLM decomposes tasks into subtasks, visually scores trajectories at the subtask level, summarizes failure modes, and revises reward code accordingly (Lee et al., 1 Jun 2026). On 12 ManiSkill tasks, RDA reports average alignment $Ans_{t+1}$ 8 and success $Ans_{t+1}$ 9, versus Eureka’s $Ans_t$ 0 and $Ans_t$ 1; on 4 HumanoidBench tasks, RDA reports alignment $Ans_t$ 2 and success $Ans_t$ 3, compared with Eureka’s alignment $Ans_t$ 4 and success of about $Ans_t$ 5, highlighting the paper’s emphasis on instruction alignment rather than raw success alone (Lee et al., 1 Jun 2026).

“Auto MC-Reward” does something similar in Minecraft through a Reward Designer, Reward Critic, and Trajectory Analyzer (Li et al., 2023). The reward function is emitted as executable Python with dense and sparse components, the critic checks syntax and semantic consistency, and trajectory analysis feeds back failure causes such as lava avoidance or oscillatory movement. On diamond-ore exploration, the full system reaches success $Ans_t$ 6 versus $Ans_t$ 7 for MineCLIP and $Ans_t$ 8 for sparse-reward RL; on approach-tree it reaches $Ans_t$ 9 success with $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ 0 average steps, versus $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ 1 and $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ 2 for MineCLIP (Li et al., 2023).

The debugging interpretation is made explicit in “When LLM Reward Design Fails.” The paper identifies two dominant one-shot failure modes—reward flooding and semantic/API misunderstanding—and a rarer weak-shaping case, then uses probe-training diagnostics and a taxonomy prompt to revise reward code (Wang et al., 27 May 2026). On DoorKey-8×8, refinement raises success from $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ 3 to $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ 4; on KeyCorridor, from $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ 5 to $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ 6; metrics-only re-prompting drops these results sharply, while a static-vocabulary control recovers much of the gain, which the paper interprets as evidence that taxonomy-guided debugging is a major mechanism (Wang et al., 27 May 2026).

5. Process reward modeling and step-level reasoning supervision

A major branch of Reward-Refine focuses on process rewards for reasoning. “ReST-MCTS*” addresses the standard problem that final-answer correctness is too sparse and admits false-positive traces with flawed intermediate reasoning (Zhang et al., 2024). It infers step-wise rewards from search trees and final-answer supervision, defines quality values $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ 7 over partial solutions, and uses these values to guide a modified MCTS* search. The resulting traces are then used to improve both the value model and the policy model over multiple self-training iterations. On GSM8K and MATH500, the paper reports that its value model used as a verifier improves over ORM and MATH-SHEPHERD, reaching $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ 8 on GSM8K and $Ans_0 \prec Ans_1 \prec \dots \prec Ans_n$ 9 on MATH500 in the reported setting (Zhang et al., 2024).

“Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning” identifies a different bottleneck: reasoning traces are often segmented too finely, so redundant steps and non-incremental steps receive supervision as if they were meaningful progress (Hu et al., 23 Jan 2025). CFPRM merges adjacent steps into coarse windows, assigns the label of the last step to the merged block, then gradually reduces the window size to $M_P$ 0, yielding a coarse-to-fine curriculum over process data. The method is loss-agnostic across BCE, MSE, and Q-ranking, and in the reported experiments often improves BoN performance over ShepHerd, RestMCTS*, and PQM, although one Q-ranking setting on MATH500 shows a small degradation (Hu et al., 23 Jan 2025).

“Towards Robust Process Reward Modeling via Noise-aware Learning” argues that MCE labels are intrinsically policy-dependent and therefore noisy, generating both false positives and false negatives (Xie et al., 19 Jan 2026). Its first stage applies reflection-aware label correction with an LLM judge to suppress overestimated rewards when a trajectory succeeds only because it later corrects the current step. Its second stage, Noise-Aware Iterative Training, progressively replaces labels that disagree strongly with PRM confidence, producing soft refined labels over several stages. The paper reports that average F1 on ProcessBench rises from about $M_P$ 1 for a raw MC-based 7B PRM to $M_P$ 2 for the NAIT variant, with GSM8K F1 reaching $M_P$ 3 and MATH F1 $M_P$ 4 (Xie et al., 19 Jan 2026).

For domain-specific step-level RL, “Reason, Reward, Refine: Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small LLMs” localizes the first error in a seven-tag XML reasoning trace, classifies it as Problem Miscomprehension, Conceptual Error, or Computational Error, and computes the step-level reward

$M_P$ 5

with $M_P$ 6 when no error is found (Jaiswal et al., 6 Jul 2026). The framework then routes the case to problem restatement, retrieval-augmented concept feedback, or code-based calculation correction, and updates the policy with REINFORCE-style policy gradient plus KL regularization. Across five physics benchmarks, it reports $M_P$ 7– $M_P$ 8 gains over CoT prompting and $M_P$ 9– $p=\{(a_i,q_i,r_i)\}_{i=1}^{|p|}$ 0 over the strongest baseline, while reducing calculation errors from $p=\{(a_i,q_i,r_i)\}_{i=1}^{|p|}$ 1 to $p=\{(a_i,q_i,r_i)\}_{i=1}^{|p|}$ 2 and miscomprehension errors from $p=\{(a_i,q_i,r_i)\}_{i=1}^{|p|}$ 3 to $p=\{(a_i,q_i,r_i)\}_{i=1}^{|p|}$ 4 in the best observed cases (Jaiswal et al., 6 Jul 2026).

6. Evaluation, limitations, and theoretical boundaries

The literature also converges on recurring limitations. Reward approximation and misalignment remain central in systems that learn or proxy reward rather than using exact verifiers. “Plan-and-Refine” explicitly notes that its reward model approximates ICAT and is used only for reranking, not for direct RL of the generator, which limits optimization but also reduces reward-hacking risk (Salemi et al., 10 Apr 2025). “RewardDS” similarly relies on a DP-trained reward proxy that is only an indirect measure of task usefulness, while “When LLM Reward Design Fails” shows that diagnostic heuristics can misfire, especially in dense locomotion where success-based rules falsely signal reward hacking (Wang et al., 23 Feb 2025, Wang et al., 27 May 2026).

Judge reliability is another recurrent issue. “Refine-n-Judge” reports that judge agreement is near $p=\{(a_i,q_i,r_i)\}_{i=1}^{|p|}$ 5 early in refinement chains but drops toward $p=\{(a_i,q_i,r_i)\}_{i=1}^{|p|}$ 6 near termination, which the paper treats as uncertainty about whether further refinements are improvements (Cayir et al., 3 Aug 2025). “Reason, Reward, Refine” depends on GPT-4o to localize the first physics error and to classify its type, so any misclassification can route the sample into the wrong feedback channel (Jaiswal et al., 6 Jul 2026). “Towards Robust Process Reward Modeling via Noise-aware Learning” shows the same general problem in process supervision: labels derived from continuation success are contaminated by self-correction dynamics, which is why reflection-aware denoising is needed (Xie et al., 19 Jan 2026).

Theoretical work sharpens the boundary of what reward refinement can represent. “Settling the Reward Hypothesis” proves that goals and purposes are representable as maximization of expected cumulative scalar Markov reward if and only if the induced preference relation satisfies the von Neumann–Morgenstern axioms and Temporal $p=\{(a_i,q_i,r_i)\}_{i=1}^{|p|}$ 7-Indifference (Yang et al., 2022). This result implies that Reward-Refine can be exact only when the target preferences admit such a scalar representation; constrained, lexicographic, or otherwise non-compensatory objectives may require augmented state, vector-valued reward, or objectives beyond scalar expected return (Yang et al., 2022). The survey on reward models reaches a compatible systems-level conclusion: reward models can be learned from demonstrations, goals, preferences, AI feedback, or intrinsic signals, but they must ultimately be evaluated not only by downstream policy performance but also by reward distances, interpretability, and resistance to reward hacking (Yu et al., 18 Jun 2025).

A plausible synthesis is that Reward-Refine is most reliable when three conditions hold simultaneously: the interface between reward and task is semantically faithful, the reward or judge signal is calibrated to the distribution being optimized, and the refinement operator is strong enough to exploit that signal without collapsing into reward hacking or verbosity drift. The cited literature demonstrates that these conditions can be met in several distinct ways, but it also shows that the engineering burden shifts rather than disappears: one must refine the reward, the judge, the process labels, the reward code, or the diagnostic vocabulary until the optimization loop and the intended objective are once again aligned.