Unified Reward & Policy Optimization (URPO)

• Unified Reward & Policy Optimization (URPO) is a framework that couples reward learning and policy improvement into one optimization loop, eliminating separate training of reward models.
• It recasts diverse alignment data into a single generative format and employs Group-Relative Policy Optimization to jointly train generation and evaluation.
• URPO boosts performance by integrating verifiable reasoning, preference supervision, and self-generated scores, yielding improved instruction following and reasoning results.

Unified Reward & Policy Optimization (URPO) designates a class of alignment methods in which reward learning and policy improvement are not staged as separate post-training components but are coupled inside a common optimization framework. In the canonical large-language-model formulation, introduced in "URPO: A Unified Reward & Policy Optimization Framework for LLMs," a single model serves simultaneously as instruction-following "player" and reward-modeling "referee"; preference pairs, verifiable reasoning data, and open-ended instructions are recast into one generative format; and a single Group-Relative Policy Optimization (GRPO) loop trains generation and evaluation together (Lu et al., 23 Jul 2025). Closely related work uses the same idea more broadly for multimodal critics, adaptive trust-region RL, and unified text-image optimization, indicating that URPO is both a specific framework and a broader design pattern for coupling reward signals and policy updates (Wang et al., 7 Mar 2025).

1. Conceptual scope and research lineage

The explicit URPO framework for LLMs appears in 2025 as a response to the standard RLHF pipeline in which a policy model is first tuned for instruction following, then a separate reward model is trained on preference data, and finally the policy is optimized against that frozen reward signal (Lu et al., 23 Jul 2025). The central claim is that this separation creates pipeline complexity, a static-reward performance ceiling, and a competence mismatch between an improving policy and a fixed evaluator.

Several adjacent papers articulate closely related unification principles. UnifiedReward proposes a single multimodal critic for both understanding and generation across images and videos, and explicitly states that its pipeline naturally suggests a “Unified Reward & Policy Optimization (URPO)” loop in which a unified reward model supplies signals for multiple multimodal policies and a common preference-optimization engine consumes the resulting pairs (Wang et al., 7 Mar 2025). PPO-BR describes itself as “the first unified dual-signal trust region adaptation framework” for PPO, replacing static clipping with a bounded adaptive trust region driven jointly by policy entropy and reward progression; the paper explicitly presents this as conceptually close to URPO because reward-side and policy-side signals are fused inside one update mechanism (Rahman, 23 May 2025). UniGRPO extends the unification idea to reasoning-driven visual generation by formulating text reasoning and image synthesis as one MDP with a shared terminal reward and a joint objective over discrete and continuous actions (Liu et al., 24 Mar 2026).

Other neighboring proposals clarify the design space from the policy-optimization side. RPO removes parametric value learning entirely in scalar-feedback RLHF by estimating a per-prompt partition function directly from data and regressing policy log-ratios to centered rewards (Faye et al., 16 Jun 2025). GOPO argues that when reward models are trained on pairwise preferences, policy optimization should also use only ordinal information rather than reward magnitudes, replacing z-scored rewards with fixed rank-based advantages (Choi et al., 1 Feb 2026). Taken together, these works suggest that URPO is best understood as a family of methods organized around one question: how tightly the reward mechanism and the policy update should be unified, and in what representation that unification should occur.

2. Canonical URPO for LLMs

In the LLM setting, URPO replaces the conventional two-model architecture with a single model $\pi_\theta$ that plays two roles. As a player, it generates solutions, answers, and open-ended completions. As a referee, it reads a prompt together with multiple candidate responses and generates textual scores or rankings for those candidates (Lu et al., 23 Jul 2025). There is no separate reward model $r_\phi$; evaluation is performed by the same network under a different prompt format.

This reorganization also changes the training pipeline. Instead of a sequence such as SFT $\rightarrow$ reward modeling $\rightarrow$ PPO/GRPO, URPO uses one training phase and one optimizer. All post-training data are converted into a single generative interface so that preference supervision, verifiable reasoning, and open-ended instruction-following all enter the same GRPO loop. Preference examples train the model in referee mode, verifiable reasoning trains it in player mode with external correctness checks, and open-ended instruction following uses self-generated scores from the same model as rewards (Lu et al., 23 Jul 2025).

The framework is therefore not merely “joint training” in a loose sense. Its defining property is functional unification: the same parameters are used for answer generation, answer evaluation, and policy optimization. This differs both from standard RM-based RLHF and from pipelines such as UnifiedReward, where the reward model is unified across tasks but is still trained once and then frozen before downstream DPO (Wang et al., 7 Mar 2025).

3. Unified data representation and reward sources

URPO’s unification is operationalized by recasting heterogeneous alignment data into a shared generative format. Preference data are turned into evaluation prompts in which the model is asked to score multiple responses to the same question and emit an output of the form Scores: \boxed{s_1, s_2, ..., s_N}. The resulting score vector induces a ranking, and the reward for that evaluation rollout is Kendall’s $\tau$ with the ground-truth ranking (Lu et al., 23 Jul 2025):

$$R(q,\{a_i\}_{i=1}^N) = \tau_K(\mathbf r,\mathbf g) = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} \operatorname{sign}\big[(s_i-s_j)(g_i-g_j)\big].$$

Verifiable reasoning data are handled differently. For math or code, the model simply generates solutions as the player, and a rule-based verifier assigns reward, typically based on whether the final answer is correct. Open-ended instruction data have no external ground truth; here URPO first samples a group of candidate answers, then feeds those candidates back into the same model in referee mode, parses the generated scores, and uses those self-assigned scores as rewards for GRPO updates (Lu et al., 23 Jul 2025).

This yields three reward pathways inside one training system:

Task category	Primary role	Reward source
Verifiable reasoning	Player	Rule-based verifier
Open-ended generation	Player	Self-generated scores
Preference alignment	Referee	Kendall’s $\tau$ vs ground-truth rank

The same idea appears in other domains with different implementations. UnifiedReward constructs synthetic preference data by combining pairwise ranking with pointwise sifting: a unified multimodal critic first selects better candidates within pairs, then assigns pointwise scores, and the best chosen and worst rejected samples become DPO pairs (Wang et al., 7 Mar 2025). UniGRPO likewise uses one terminal reward $R(x_0,c)$ for the entire multimodal trajectory $$\text{Prompt} \rightarrow \text{Reasoning} \rightarrow \text{Image}$$, so that both reasoning tokens and image-denoising steps share a common advantage signal (Liu et al., 24 Mar 2026). The common structural theme is that reward construction is no longer an external preprocessing step; it is embedded in the same system that drives policy learning.

4. Optimization mechanics

URPO uses GRPO rather than vanilla PPO. In canonical GRPO, for a prompt $q$, one samples a group of outputs $r_\phi$0, computes a scalar reward for each output, forms a group-relative advantage $r_\phi$1, and applies a clipped policy-gradient objective over all tokens in each trajectory (Lu et al., 23 Jul 2025). URPO retains this value-free groupwise structure across all data types.

The paper then introduces specific modifications, following DAPO for stability. It removes per-response length normalization $r_\phi$2 to reduce length bias, uses an asymmetric clipping rule with importance ratio constrained to $r_\phi$3, and sets the KL penalty to zero, i.e. $r_\phi$4, to encourage exploration. In simplified form, the update becomes

$r_\phi$5

Training uses a two-stage curriculum. The first 100 steps are a warm-up phase containing only verifiable reasoning and preference data, so that the referee acquires grounded evaluative competence before self-rewarding is introduced. The unified phase then adds open-ended instruction data, mixing reasoning, preference, and instruction samples in ratios such as $r_\phi$6, $r_\phi$7, or related ablations (Lu et al., 23 Jul 2025).

The reported implementation uses global batch size 256 prompts, group size $r_\phi$8, sampling temperature 1.0, maximum generation length 4096 tokens, AdamW with learning rate $r_\phi$9 and cosine decay, and four policy-update epochs per collected batch (Lu et al., 23 Jul 2025). The primary model is Qwen2.5-7B-Base, with additional experiments on Qwen3-8B-Base and Llama3.1-8B-derived checkpoints. In the broader URPO-related literature, different optimizers instantiate the same coupling principle differently: UnifiedReward uses offline DPO or Diffusion-DPO on preference pairs produced by a frozen unified critic (Wang et al., 7 Mar 2025), PPO-BR modifies PPO’s clipping threshold by the dual-signal rule

$\rightarrow$0

and UniGRPO uses a joint objective $\rightarrow$1 with one shared group-relative advantage for text and image trajectories (Rahman, 23 May 2025, Liu et al., 24 Mar 2026).

5. Empirical performance and emergence of an internal evaluator

On Qwen2.5-7B-Base, URPO improves the instruction-following score on AlpacaEval from 42.24 for a strong baseline using a separate generative reward model to 44.84, and improves the composite reasoning average from 32.66 to 35.66 (Lu et al., 23 Jul 2025). On individual reasoning benchmarks, it reaches 91.96 on GSM8K, 77.40 on MATH-500, 17.08 on AIME 2024, and 15.83 on AIME 2025, all above the corresponding RM-gen baseline numbers 90.75, 74.80, 13.33, and 10.83.

The same pattern persists when the starting point is a Qwen2.5-7B SFT checkpoint. In that setting, URPO raises AlpacaEval to 42.36 and the reasoning average to 31.83, exceeding GRPO with an RM-score baseline at 28.32 and 30.34 and GRPO with an RM-gen baseline at 39.38 and 31.64 (Lu et al., 23 Jul 2025). On Qwen3-8B-Base, URPO achieves AlpacaEval 39.25 and reasoning average 39.57, compared with 36.77 and 39.75 for RM-gen. On Llama3.1-8B-derived OctoThinker, URPO reaches AlpacaEval 41.49 and reasoning average 27.43, above RM-gen at 36.15 and 26.42.

A central empirical claim is that the unified model also becomes a stronger evaluator. On RewardBench, a dedicated generative reward model scores 83.55 mean, whereas URPO’s internal evaluator reaches 85.15 under the 1:1:0 mixture of preference and reasoning data, and 85.31 under 2:1:0 (Lu et al., 23 Jul 2025). This is not automatic: when preference data are removed and the mixture is 0:1:1, the RewardBench mean collapses to 62.39. Preference supervision is therefore essential for evaluator grounding, while reasoning data materially improve evaluator quality.

Data-mixture ablations show that balanced training best supports joint capability. The 1:1:1 mixture yields AlpacaEval 44.84 and reasoning average 35.66, which is the best overall trade-off reported. By contrast, 1:1:0 produces high reasoning average 35.03 but low AlpacaEval 31.43, while 1:0:1 produces AlpacaEval 43.11 but lower reasoning average 28.58 (Lu et al., 23 Jul 2025). The results therefore support the paper’s core thesis: co-training generation and evaluation is most effective when preference, reasoning, and instruction data are all present, but they contribute different functional roles.

6. Relations to adjacent methods, misconceptions, and open directions

A common misconception is that URPO is simply “self-rewarding.” The empirical evidence does not support that reading. In the LLM paper, self-generated rewards are used only after a warm-up stage on preference and verifiable reasoning data, and evaluator quality degrades sharply without preference supervision (Lu et al., 23 Jul 2025). URPO is therefore not reward-free; rather, it internalizes reward modeling into the same model that is being optimized.

Another misconception is that all URPO-like methods share one optimization recipe. The literature instead reveals several distinct variants. UnifiedReward uses a unified multimodal VLM critic trained by autoregressive cross-entropy on evaluation text, then freezes that critic and applies DPO or Diffusion-DPO to downstream policies; its key technical device is pair ranking plus pointwise sifting (Wang et al., 7 Mar 2025). PPO-BR does not unify model roles at all; it unifies signals at the trust-region level by letting entropy and reward progression jointly determine the clipping threshold while preserving bounded adaptation and a monotonic-improvement guarantee (Rahman, 23 May 2025). UniGRPO unifies text reasoning and image synthesis inside one MDP and one shared terminal reward, but remains specific to reasoning-driven visual generation and introduces CFG-free linear rollouts plus an MSE penalty on velocity fields to mitigate reward hacking (Liu et al., 24 Mar 2026). RPO argues for a value-free scalar-feedback objective based on empirical reward partitioning rather than joint reward-policy learning (Faye et al., 16 Jun 2025). GOPO argues that when reward models are trained on ordinal preferences, policy optimization should discard reward magnitudes and use only ranks, yielding consistently higher training and validation reward trajectories and improved LLM-as-judge evaluations than GRPO in non-verifiable settings (Choi et al., 1 Feb 2026).

These divergences define the main theoretical fault lines around URPO. One concerns where unification occurs: inside one model, inside one trust-region rule, or only through a shared reward-construction pipeline. A second concerns what kind of reward signal the policy should consume: self-generated scores, scalar rewards, relative ranks, or verifier outputs. A third concerns how to prevent reward overfitting: external anchoring through preference data in URPO (Lu et al., 23 Jul 2025), periodic reward refresh or online DPO as future work in UnifiedReward (Wang et al., 7 Mar 2025), velocity-space regularization in UniGRPO (Liu et al., 24 Mar 2026), bounded clipping in PPO-BR (Rahman, 23 May 2025), and value-free analytic normalization in RPO (Faye et al., 16 Jun 2025).

Open directions follow directly from these papers. UnifiedReward points to online DPO and periodic reward-model refresh as a natural extension beyond its current offline pipeline (Wang et al., 7 Mar 2025). UniGRPO identifies multimodal process reward models as a way to move from sparse terminal rewards toward dense feedback on intermediate reasoning steps (Liu et al., 24 Mar 2026). PPO-BR raises the possibility of state-dependent or action-dependent adaptive signals beyond scalar entropy and reward deltas (Rahman, 23 May 2025). The LLM URPO paper itself suggests scaling to larger and multimodal models, incorporating safety-specific preference data, and exploring alternative GRPO stabilization strategies (Lu et al., 23 Jul 2025). Across all of these directions, the durable idea is the same: reward formation and policy improvement are not independent modules but mutually constraining parts of one alignment system.