Reward-Aware Preference Optimization

• Reward-aware Preference Optimization is a unified framework that aligns a language model’s implicit reward with a target explicit reward using a chosen distance metric.
• It systematically recovers methods like DPO, IPO, and SimPO by varying optimization geometry, response count, and reward scaling to match preference margins.
• The framework emphasizes design axes such as reward quality, online versus offline training, and reference policy roles to optimize model alignment performance.

Reward-aware Preference Optimization (RPO) is a mathematical framework for LLM alignment that casts preference optimization as matching a policy’s implicit reward to a target explicit reward under a chosen distance metric, thereby unifying methods that otherwise appear structurally different, including DPO, IPO, SimPO, and REINFORCE Leave-One-Out (RLOO) (Sun et al., 31 Jan 2025). In this formulation, the central design variables are the optimization geometry, the number of responses per prompt, the role of the reference policy, and the distinction between implicit and explicit reward models. The same term is also used more broadly for reward-informed preference learning systems outside the original framework, but the named framework in the literature is the one introduced in "Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment" (Sun et al., 31 Jan 2025).

1. Scope, terminology, and disambiguation

Within the exact terminology of the literature, Reward-aware Preference Optimization refers to the unifying framework introduced in (Sun et al., 31 Jan 2025). Its purpose is not primarily to propose a single replacement loss, but to expose a common structure behind a fragmented alignment landscape in which offline pairwise methods, listwise methods, and online RL-style methods differ mainly by a small number of explicit choices.

The acronym RPO is overloaded across adjacent research areas. "Fine-Tuning Diffusion Generative Models via Rich Preference Optimization" uses RPO to mean Rich Preference Optimization, a critique-and-revision pipeline for synthetic preference construction in text-to-image diffusion tuning (Zhao et al., 13 Mar 2025). "Value-Free Policy Optimization via Reward Partitioning" uses RPO for Reward Partitioning Optimization, a scalar-feedback method for $(x,y,r)$ triplets rather than pairwise preferences (Faye et al., 16 Jun 2025). "Robust Preference Optimization: Aligning LLMs with Noisy Preference Feedback" uses RPO for Robust Preference Optimization, an EM-based noise-robust meta-framework (Cao et al., 29 Sep 2025). "Auto-Rubric as Reward" uses RPO for Rubric Policy Optimization in multimodal online RL (Tian et al., 8 May 2026). This naming collision matters because many papers are relevant to reward-aware preference learning while not instantiating the specific framework of (Sun et al., 31 Jan 2025).

More broadly, the phrase “reward-aware preference optimization” is also used informally for methods in which reward signals shape pair construction, negative sampling, online advantages, or fairness constraints. This broader usage is useful descriptively, but it should not be conflated with the precise framework defined in (Sun et al., 31 Jan 2025).

2. Core mathematical formulation

The framework begins from the standard KL-regularized RLHF objective

$$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$

From the closed-form optimizer of this objective, any policy $\pi$ induces an implicit reward

$$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$

where $\log Z(x)$ is prompt-dependent and independent of $y$ (Sun et al., 31 Jan 2025). This identity is the foundation of the framework: preference optimization can be interpreted as aligning the policy’s implicit reward to some target reward signal.

For pairwise data $(x,y^1,y^2)$, the RPO loss is defined by comparing the policy’s implicit reward margin to a target explicit reward margin: $$\mathbb{D} \left[ r_{\pi_{\theta}}(x, y^1) - r_{\pi_\theta}(x, y^2) \,\|\, \eta r^\star(x, y^1) - \eta r^\star(x, y^2) \right].$$ Here $\mathbb D$ is a distance or divergence, $r^\star$ is the target explicit reward model, $$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$0 is the reference policy, $$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$1 is the implicit-reward scale inherited from KL regularization, and $$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$2 rescales the target reward (Sun et al., 31 Jan 2025). This formulation makes reward-awareness explicit: the policy is trained not only to order responses correctly, but to reproduce a reward-structured target.

Two distance choices are central. The first is squared distance,

$$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$3

which directly regresses the implicit reward margin onto the target margin. The second is backward Bernoulli KL,

$$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$4

where $$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$5. This interprets scalar margins as Bernoulli logits and matches target and predicted preference probabilities (Sun et al., 31 Jan 2025).

The framework also extends to $$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$6 responses per prompt. Because $$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$7 contains the additive prompt-only term $$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$8, the multivariate distance must cancel constants. The computable listwise form is

$$\max_{\pi_{\theta}} \mathbb E_{x, y\sim \pi_{\theta}} \left[r_{\phi}(x, y) - \beta \, \mathrm{KL}\!\left(\pi_{\theta}(y|x)\,\|\,\pi_{ref}(y|x)\right)\right].$$9

For squared-distance objectives, (Sun et al., 31 Jan 2025) introduces a leave-one-out centered variant $\pi$0; for probabilistic objectives, it introduces backward categorical KL based on softmax-normalized reward vectors.

3. Recovery of existing methods

A central feature of the framework is that many seemingly distinct methods are recovered by specific choices of $\pi$1, the target reward, and the response structure (Sun et al., 31 Jan 2025).

Method	RPO instantiation	Brief characterization
DPO	$\pi$2, $\pi$3	Hard-target Bernoulli probability matching
cDPO	$\pi$4, $\pi$5	Finite target preference probability
IPO	$\pi$6, fixed target margin $\pi$7	Squared regression on reward margin
Distill-DPO	$\pi$8	Regress policy margin onto teacher reward margin
RLOO	Online $\pi$9	REINFORCE-style leave-one-out scaling
Online RPO-bwd	Online backward categorical KL	Softmax-normalized policy-vs-target reward matching

The DPO recovery is especially important. Under backward Bernoulli KL, if the target reward says one response is infinitely better than the other, the target Bernoulli becomes deterministic and the objective reduces to the usual DPO logistic loss. This gives a precise interpretation of DPO as a limiting hard-target instance of reward-aware probability matching rather than as a fundamentally separate object.

IPO appears as the squared-distance counterpart: it fixes a target reward margin and regresses the policy’s implicit reward gap toward it. Distill-DPO is the most direct explicit-reward regression instance, because the target is itself a teacher reward difference. SimPO is also placed in the framework formally, although the corresponding $$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$0 is noted to be not a mathematically valid probability distribution in that recovery, so the equivalence is structural rather than literal (Sun et al., 31 Jan 2025).

The online side is equally important. Differentiating the listwise objective produces a REINFORCE-form gradient

$$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$1

where $$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$2 depends on the chosen distance. For $$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$3, the resulting scale is exactly leave-one-out centered and recovers RLOO. For backward categorical KL, the scale becomes the difference between the softmax over policy-induced implicit rewards and the softmax over target rewards, yielding the online RPO-bwd optimizer (Sun et al., 31 Jan 2025).

4. Design axes and empirical behavior

The framework is meant to disentangle a small set of design axes: the optimization objective, the number of responses per prompt, the use of implicit versus explicit reward models, the reference policy, offline versus online response generation, and single-iteration versus iterative alignment (Sun et al., 31 Jan 2025). To study these axes cleanly, the paper adopts a controlled evaluation pipeline built around a Ground-Truth Judge, Nemotron-4-340B-RM, so that algorithmic differences are not conflated with reward-model mismatch.

The empirical picture is specific. In offline training with $$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$4, DPO, SimPO, and RPO-bwd perform similarly and best, while KTO and RPO-sqloo are weaker. On LMSYS test for the 8B model, the average reward is $$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$5 for DPO, $$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$6 for SimPO, $$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$7 for RPO-bwd, $$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$8 for KTO, and $$r_{\pi}(x, y) = \beta \log \frac{\pi (y| x)}{\pi_{ref} (y|x)} + \beta \log Z(x),$$9 for RPO-sqloo (Sun et al., 31 Jan 2025). Thus, in the controlled offline setting, the principal benefit of the framework is interpretive unification more than large raw gains from changing the loss alone.

Increasing the number of responses per prompt from $\log Z(x)$0 to $\log Z(x)$1 yields little benefit in the reported setup. For the 70B model with offline RPO-bwd, LMSYS test average reward moves only from $\log Z(x)$2 to $\log Z(x)$3, while win rate changes from $\log Z(x)$4 to $\log Z(x)$5 (Sun et al., 31 Jan 2025). This suggests that additional listwise structure is not automatically useful unless other parts of the pipeline, especially reward quality and response diversity, also improve.

Online training is markedly stronger when a strong reward model is available. For the 70B model, offline $\log Z(x)$6 RPO-bwd reaches LMSYS test reward $\log Z(x)$7 and AlpacaEval win rate $\log Z(x)$8, whereas online $\log Z(x)$9 RPO-bwd reaches $y$0 and $y$1 (Sun et al., 31 Jan 2025). Within online methods, RPO-bwd is substantially better and more stable than RPO-sqloo/RLOO; the reported example gives reward $y$2 and win rate $y$3 for online RPO-bwd versus reward $y$4 and win rate $y$5 for online RPO-sqloo/RLOO. The interpretation offered in the paper is that the backward-KL geometry provides a normalized probability-matching signal that stabilizes optimization.

The strongest caveat is that online performance depends critically on reward-model quality. When the same online procedure is driven by a learned reward model instead of the ground-truth reward model, performance collapses close to SFT level; the paper reports an online RPO-bwd win rate of only $y$6 under the learned reward model condition (Sun et al., 31 Jan 2025). Reward-aware preference optimization is therefore only as strong as the reward signal it aligns to.

Iterative alignment helps consistently. For 70B online RPO-bwd, the paper reports AlpacaEval reward and win rate progressing from Iter-1 $y$7, $y$8, to Iter-2 $y$9, $(x,y^1,y^2)$0, and Iter-3 $(x,y^1,y^2)$1, $(x,y^1,y^2)$2 (Sun et al., 31 Jan 2025). This result reinforces the framework’s emphasis on data generation, reference updating, and reward quality as core design axes.

5. Broader reward-aware formulations and extensions

The exact framework of (Sun et al., 31 Jan 2025) has been extended, reinterpreted, or paralleled in several directions. "Preference Optimization via Contrastive Divergence" reframes preference optimization as approximate maximum likelihood in an energy-based model, where dispreferred samples are importance samples for estimating the partition function; DPO is recovered as the $(x,y^1,y^2)$3 case of RNCE, and the proposed MC-PO samples reward-aware hard negatives via contrastive divergence rather than heuristic pair construction (Chen et al., 6 Feb 2025). This makes reward-awareness stronger at the level of negative-sample selection.

"MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge" injects prior reward knowledge directly into pairwise preference optimization through the reward gap

$(x,y^1,y^2)$4

yielding a loser term scaled by $(x,y^1,y^2)$5. In that formulation, DPO is recovered when $(x,y^1,y^2)$6, and SFT-like behavior appears when $(x,y^1,y^2)$7. This is a direct reward-gap-aware extension of DPO-style alignment and is plugin-compatible with SimPO, IPO, and CPO (Lan et al., 27 Jul 2025).

"Fairness Aware Reward Optimization" moves reward-awareness to the reward-learning stage itself. Faro imposes demographic parity, equalized odds, or counterfactual fairness constraints on reward models and proves reward-to-policy fairness transfer under KL-regularized optimization, including extensions to DPO-like settings with implicit rewards (Choi et al., 8 Feb 2026). This broadens RPO from utility matching to constrained reward design.

In multimodal and diffusion settings, reward-awareness often enters through richer supervision interfaces rather than through the exact objective of (Sun et al., 31 Jan 2025). "Fine-Tuning Diffusion Generative Models via Rich Preference Optimization" argues that scalar reward labels are too opaque and uses critique-and-revision to build more informative preference pairs before standard Diffusion-DPO training (Zhao et al., 13 Mar 2025). "Visual Preference Optimization with Rubric Rewards" uses instance-specific rubrics to score multimodal responses and mine higher-quality DPO pairs (Yu et al., 14 Apr 2026). "Auto-Rubric as Reward" goes further: ARR externalizes implicit multimodal preference knowledge into prompt-specific rubrics, and its Rubric Policy Optimization uses rubric-conditioned pairwise judgments as online PPO-style binary rewards (Tian et al., 8 May 2026). For flow models, "G²RPO: Granular GRPO for Precise Reward in Flow Models" localizes stochasticity to a single denoising step and aggregates advantages across multiple denoising granularities to improve reward attribution (Zhou et al., 2 Oct 2025). Collectively, these results suggest that “reward-aware preference optimization” functions not only as one named framework, but also as a broader design space spanning target-reward matching, reward-gap priors, structured rubric rewards, and precise online credit assignment.

6. Limitations, controversies, and open questions

The named RPO framework is a framework, not a single universally superior optimizer. Its main strength is conceptual: it organizes preference optimization methods through a common reward-matching lens. Its practical conclusions are conditional. In the reported experiments, offline DPO, SimPO, and RPO-bwd are close; simply increasing the number of responses per prompt does little; and the largest gains come from online training with a strong reward model rather than from the formalism alone (Sun et al., 31 Jan 2025).

Several limitations are explicit. The framework is sequence-level and “does not subsume PPO” because it does not provide token-level credit assignment. Reward-model training is out of scope even though online performance is shown to depend critically on reward-model quality. Preference data generation is not optimized in the study; prompts, response diversity, on-policy/off-policy mixing, and judging strategy remain open design variables (Sun et al., 31 Jan 2025). Some recoveries are also formal rather than literal, with the SimPO mapping relying on a $(x,y^1,y^2)$8 that is not a valid probability distribution.

A broader controversy concerns what “reward-aware” should mean. The exact framework of (Sun et al., 31 Jan 2025) interprets preference optimization as reward matching between implicit and explicit reward functions. Other work instead treats reward-awareness as hard-negative sampling in an energy-based model, reward-gap priors, fairness-constrained reward estimation, rubric-mediated judgment, or improved online advantage assignment. This suggests that the phrase denotes a family resemblance rather than a unique algorithmic recipe.

Finally, the overloaded acronym creates a bibliographic problem. RPO may denote Reward-aware Preference Optimization, Rich Preference Optimization, Reward Partitioning Optimization, Robust Preference Optimization, or Rubric Policy Optimization, depending on the paper. For technical precision, the expanded title and arXiv identifier are therefore essential whenever “RPO” appears in current alignment literature.