RewardDPO: Reward-Aware DPO Methods

• RewardDPO is a family of methods that inject explicit reward signals into the DPO framework to guide preference learning beyond simple binary labels.
• It employs diverse mechanisms such as sample reweighting, adaptive margins, and token-level modulation to incorporate quantitative reward differences.
• RewardDPO approaches show empirical improvements in structured and multimodal settings, though they face challenges in reward quality and standardization.

Searching arXiv for papers on RewardDPO and closely related reward-aware DPO variants. RewardDPO denotes a reward-aware strand of Direct Preference Optimization in which explicit reward information influences preference learning more richly than a bare chosen/rejected label. The term is not standardized across the literature. In the narrow sense, “RewardDPO” names the training-time preference-optimization component of SMART-Editor, where reward-aligned layout pairs are used to distill reward-guided editing preferences into a one-pass editor (Mondal et al., 30 Jul 2025). In the broader sense, the label covers DPO variants and adjacent formulations that inject reward differences, adaptive margins, token-level rewards, reward-conditioned signals, or rubric-derived judgments into DPO losses or DPO data construction (Sun et al., 31 Jan 2025).

1. Terminological scope and conceptual boundaries

The literature does not present RewardDPO as a single canonical algorithm. Instead, several lines of work occupy the same design space while differing in where reward enters the pipeline. Some methods alter the optimization objective directly by weighting or reshaping the DPO loss with reward-derived quantities. Others preserve a standard DPO loss and use reward models to construct or filter preference pairs. Still others condition the policy on structured reward signals, or replace sequence-level preference structure with token-level reward guidance.

A precise boundary follows from the distinction between pairwise preference supervision and scalar-reward supervision. Classical DPO consumes $(x,y_w,y_l)$ triples. By contrast, Reward Partitioning Optimization uses $(x,y,r)$ triplets and replaces value learning with an empirical partition estimate, so it is best understood as a scalar-reward direct optimization method adjacent to RewardDPO rather than a literal DPO variant (Faye et al., 16 Jun 2025). A second boundary concerns whether the optimized object is a policy or a reward model: GFRIEND uses a DPO-like multi-level objective to train a generative reward judge in few-shot regimes, which is RewardDPO-like in mechanism but reward-model-centric in purpose (Zhao et al., 10 Jun 2025). This suggests that “RewardDPO” functions more as a family resemblance term than as a uniquely fixed formalism.

2. Mathematical basis: from implicit reward in DPO to explicit reward-aware preference optimization

The common mathematical starting point is the DPO identity that a policy relative to a reference policy induces an implicit reward,

$$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$

so the DPO classification logit is an implicit reward difference between a chosen and a rejected response (Su et al., 5 Feb 2025). Under the Bradley–Terry model, standard DPO therefore trains the policy so that its reference-relative log-probability ratio matches observed pairwise preferences, without explicitly fitting a separate reward model.

Reward-aware Preference Optimization generalizes this picture by introducing an explicit reward target $r^\star(x,y)$ and matching the policy’s implicit reward difference to an explicit reward difference under a chosen distance $\mathbb D$: $$\mathbb D\!\left[r_{\pi_\theta}(x,y^1)-r_{\pi_\theta}(x,y^2)\,\|\,\eta r^\star(x,y^1)-\eta r^\star(x,y^2)\right].$$ Within this framework, standard DPO is recovered as a qualitative special case in which the target reward gap is effectively infinite, whereas reward-aware variants use finite, structured, or continuous reward differences (Sun et al., 31 Jan 2025). This is the cleanest formal answer to what RewardDPO changes: it replaces or augments binary pairwise supervision with quantitative reward structure.

A nearby but distinct scalar-feedback formulation appears in Reward Partitioning Optimization, which starts from the KL-regularized optimum

$$\pi^*(y|x)=\frac{\pi_{\mathrm{ref}}(y|x)\exp(r(x,y)/\tau)}{Z(x)}$$

and then fits policy log-ratios to partition-normalized scalar rewards without learning a value model. This is conceptually close to a “reward analog of DPO,” but its regression target and single-trajectory supervision distinguish it from pairwise RewardDPO proper (Faye et al., 16 Jun 2025).

3. Principal mechanisms for injecting reward into DPO

Reward-aware DPO systems can be organized by the locus at which reward intervenes: objective weighting, adaptive margins, token-level credit assignment, reward conditioning, or upstream pair construction.

Mechanism	Representative formulations	Key operation
Sample reweighting	RDO (Wang et al., 2024), MADPO (Rho, 6 Oct 2025)	Multiply each pair loss by a reward-derived coefficient
Adaptive reward margin	$\alpha$-DPO (Wu et al., 2024), MADPO (Rho, 6 Oct 2025)	Make the effective target margin pair-dependent
Token-level modulation	TPO (Gu et al., 2024), TGDPO (Zhu et al., 17 Jun 2025), $\mathrm{D}^2\mathrm{PO}$ (Shao et al., 20 Feb 2025)	Reweight token contributions inside the preference logit
Reward-conditioned preference learning	MCDPO (Jang et al., 11 Dec 2025)	Condition the policy on structured reward outcomes
Reward-guided pair construction	PDS-DPO (Wijaya et al., 2024), rDPO (Yu et al., 14 Apr 2026), RewardDPO in SMART-Editor (Mondal et al., 30 Jul 2025)	Use rewards to build better chosen/rejected pairs before standard DPO

In Reward Difference Optimization, the DPO loss is multiplied by a coefficient derived from the estimated reward gap

$\mathcal R=r_\phi(x,y_w)-r_\phi(x,y_l),$

or from a dedicated difference model, yielding a reward-strength-sensitive version of offline RLHF. The important point is that the DPO mechanics remain intact; what changes is how much each pair contributes to the gradient (Wang et al., 2024).

MADPO also modifies contribution strength, but does so through a two-stage pipeline in which a reward model first estimates a margin $(x,y,r)$0, and a bounded continuous function of that margin defines the per-example weight $(x,y,r)$1. The resulting loss

$(x,y,r)$2

preserves data, avoids batchwise compromise temperatures, and is analyzed as inducing an adaptive effective target margin that is amplified for hard pairs and dampened for easy pairs (Rho, 6 Oct 2025).

$(x,y,r)$3-DPO targets the same broad problem—heterogeneous margin structure—but does not require an external reward model. Instead it introduces an adaptive implicit reference

$(x,y,r)$4

which yields a pair-specific margin term $(x,y,r)$5 inside the sigmoid objective. The result is a reward-margin-based bridge between SimPO-style fixed-margin optimization and DPO-style reference-relative reward differences (Wu et al., 2024).

4. Token-level, structured, and conditioned rewards

A major development within RewardDPO is the shift from sequence-level preference labels to structured reward signals over tokens or reward dimensions. The central motivation is that sequence-level DPO treats all tokens equally, even though alignment-relevant evidence is often sparse, localized, or dimension-specific.

Token Preference Optimization addresses this in LVLM hallucination mitigation by defining a self-calibrated token reward from the difference between the generated token’s raw logit under the clean image and under a corrupted image: $(x,y,r)$6 This score is mapped to a token reward $(x,y,r)$7, flipped between winner and loser branches, and multiplied across the sequence as a visual-aware factor. The operational meaning is that visually anchored tokens receive greater emphasis inside the DPO-style policy ratio, without any fine-grained human token annotations (Gu et al., 2024).

TGDPO develops a more general token-level reward-guided DPO. Starting from a decomposition of sequence-level PPO into token-level subproblems, it derives an optimal token-level policy under reward-guided scaling and then converts the result back into a Bradley–Terry loss. Its final DPO-like objective weights each token’s reference-relative log-ratio by a reward-dependent factor $(x,y,r)$8, so different tokens can deviate from the reference policy to different degrees. In the practical instantiation, the token reward is often the induced DPO reward from a pretrained DPO model, and the shaping functions are

$(x,y,r)$9

This is reward injection at the token-credit level rather than at the example level (Zhu et al., 17 Jun 2025).

$$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$0 is a simpler token-credit variant that does not require an external reward model. It discounts per-token implicit reward terms by a temporal decay factor $$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$1, so earlier tokens contribute more strongly than later ones. The method is motivated by DPO’s length bias and can be interpreted as temporal reward shaping within the implicit reward decomposition itself (Shao et al., 20 Feb 2025).

A different structured extension appears in Multi Reward Conditional DPO for diffusion alignment. Standard scalar DPO aggregates multiple reward dimensions into a single latent scalar, which the paper identifies as a source of reward conflict. MCDPO instead conditions the policy on a discrete per-dimension outcome vector $$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$2, uses a disentangled Bradley–Terry objective, and applies dimensional reward dropout to prevent domination by easy axes. RewardDPO here becomes reward-conditioned DPO: not merely a scalar reward margin, but a structured family of optimization directions inside one conditioned model (Jang et al., 11 Dec 2025).

5. Reward-guided pair construction and application-specific instantiations

Not all RewardDPO systems modify the loss. A substantial branch uses reward models upstream to construct higher-quality preference pairs and then trains with otherwise standard DPO. This design is especially common in multimodal and structured-generation settings, where pair quality rather than objective form is often the bottleneck.

In multimodal synthetic alignment, pretrained reward models are used twice: ImageReward selects the best image among several Stable Diffusion candidates, and Llama-3-8B-ArmoRM selects the best and worst responses among multiple MLLM outputs. The resulting chosen/rejected pairs are then fed into ordinary DPO on LLaVA-1.5-7B. The reward model is thus an offline oracle for data curation rather than an online optimization signal (Wijaya et al., 2024).

Visual Preference Optimization with Rubric Rewards pushes this idea further by constructing a reusable instruction-rubric pool. For each image-instruction pair, it drafts instance-specific essential and additional criteria, scores on-policy candidate responses at criterion level with credits in $$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$3, aggregates them into a scalar reward

$$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$4

and mines chosen/rejected pairs only if they satisfy essential-quality constraints and a reward-gap threshold. The final optimization remains standard DPO, but the pair labels become rubric-grounded and criterion-aware (Yu et al., 14 Apr 2026).

SMART-Editor’s RewardDPO is the clearest literal use of the name. It operates on layout editing rather than free-form language, learns from reward-aligned pairs $$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$5, and focuses mainly on layout rather than content. Preference data are formed by choosing a high-reward layout and generating negatives via targeted structural corruptions such as overlap, misalignment, or broken semantic order. The paper’s written objective is a simplified pairwise softmax over layout probabilities rather than canonical reference-ratio DPO, but functionally it is a reward-aligned preference learner distilled into a one-pass editor (Mondal et al., 30 Jul 2025).

Reward-oriented DPO can also target reward artifacts themselves rather than end-user responses. DPO-PRO fine-tunes an LLM to generate reward functions for downstream public-health RMABs from natural-language objectives, using pairwise preferences over candidate reward programs and robustifying the conditional preference probability with a distributionally robust formulation. This suggests a broader interpretation in which RewardDPO can align reward-function generators, not only response policies (Kim et al., 2 Sep 2025).

6. Empirical behavior, limitations, and unresolved questions

Empirically, RewardDPO-style methods are strongest when the injected reward signal is both informative and appropriately localized. TGDPO reports gains of up to $$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$6 points on MT-Bench, $$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$7 points on AlpacaEval 2, and $$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$8 points on Arena-Hard over DPO baselines, while also avoiding the severe degradation that standard DPO and SimPO can exhibit at apparent loss convergence (Zhu et al., 17 Jun 2025). In multimodal hallucination mitigation, TPO on LLaVA-1.5-7B improves AMBER F1 from $$r_\pi(x,y)=\beta \log \frac{\pi(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta \log Z(x),$$9 to $r^\star(x,y)$0, raises MMHal score from $r^\star(x,y)$1 to $r^\star(x,y)$2, and reduces MMHal hallucination rate from $r^\star(x,y)$3 to $r^\star(x,y)$4, which is strong evidence that token-level reward structure matters when only a subset of tokens is visually grounded (Gu et al., 2024).

Reward-aware methods also show clear benefits in structured or rubric-rich domains. SMART-Editor reports up to $r^\star(x,y)$5 gains in structured settings for RewardDPO, with especially strong human preferences on posters and websites, while Reward-Refine remains more competitive on natural images (Mondal et al., 30 Jul 2025). In rubric-grounded visual preference optimization, rubric-based filtering raises the macro average to $r^\star(x,y)$6, whereas outcome-based filtering drops it to $r^\star(x,y)$7 from an $r^\star(x,y)$8 base model, indicating that criterion-level rewards can be more valuable than coarse outcome rewards even when the final objective is still vanilla DPO (Yu et al., 14 Apr 2026). MADPO, in a sentiment-generation setting, reports gains of up to $r^\star(x,y)$9 on High Quality data and $\mathbb D$0 on Low Quality data over the next-best method, supporting the claim that reward-margin-aware reweighting can stabilize heterogeneous preference optimization (Rho, 6 Oct 2025).

The strongest caveat is reward quality. Online reward-aware optimization improves markedly when a strong reward model or verifier is available, but weak learned reward models can induce reward hacking and erase the benefit of reward-aware training (Sun et al., 31 Jan 2025). The same issue recurs in synthetic-pair pipelines, where biases of ImageReward, ArmoRM, or proxy reward models can propagate directly into chosen/rejected pairs (Wijaya et al., 2024). Multi-dimensional conditioning methods inherit the quality ceiling of their proxy reward axes, and structured multimodal methods often pay extra preprocessing and judging cost to obtain usable criterion-level feedback (Jang et al., 11 Dec 2025).

A second unresolved issue is standardization. Some methods modify the DPO objective itself; others only alter data construction; others use DPO-like losses outside policy alignment, such as reward inference or reward-function generation. This suggests that RewardDPO is best treated as a research program rather than a single algorithmic object. Within that program, the central unresolved question is where reward should enter: as an explicit scalar target, as a token-level modulator, as a structured condition, or as a judge for pair construction. The literature so far indicates that no single answer dominates across domains; instead, the best intervention point appears to depend on whether the task’s reward-relevant structure is sequence-level, token-level, dimension-level, or latent in the quality of the preference pairs themselves.