Preference PPO: RLHF Optimization Methods

• The paper introduces Preference PPO as a family of methods combining pairwise preference feedback with PPO-style proximal updates and KL regularization.
• It extends traditional RLHF by incorporating both sequence- and token-level comparisons, addressing gradient imbalances and refining credit assignment.
• Empirical results indicate that variants like P3O and PbPO improve performance in reasoning, coding, and safety tasks through robust, data-driven optimization.

Preference-based Proximal Policy Optimization, often shortened informally to Preference PPO, denotes a family of RLHF and preference-optimization procedures that retain PPO’s proximal or trust-region policy-update structure while using pairwise preference feedback as the supervisory signal. In the canonical pipeline used for LLM alignment, a reward model is first trained from preference pairs $(x,y_w,y_l)$, and PPO then optimizes a KL-regularized policy objective under that reward model; later work extends the same basic idea to trajectory-wise pairwise policy gradients, robust min–max optimization against uncertain reward models, dynamic clipping, and trust-region preference learning for reasoning (Ma et al., 28 Feb 2025, Wu et al., 2023, Jia, 17 Nov 2025, Zhang et al., 2023, Su et al., 6 Apr 2025).

1. Scope and terminology

Across recent work, the label is applied to several related but non-identical constructions. The common thread is not a single loss, but the use of preferences as the primary source of optimization signal together with a PPO-like proximal update, a KL trust region, or an explicit policy-improvement step relative to a reference model. This suggests that “Preference PPO” is best understood as a methodological family rather than a uniquely fixed algorithm (Ma et al., 28 Feb 2025, Wu et al., 2023, Jia, 17 Nov 2025).

Formulation	Core mechanism	Representative source
Canonical PPO-style RLHF	Preference pairs $\rightarrow$ reward model $\rightarrow$ KL-regularized PPO	(Ma et al., 28 Feb 2025)
Pairwise Proximal Policy Optimization	Trajectory-wise pairwise policy gradient with PPO-style clipping	(Wu et al., 2023)
Preference-based Policy Optimization (PbPO)	Min–max policy optimization against a confidence-set-constrained reward model	(Jia, 17 Nov 2025)
Preference based PPO (Pb-PPO)	Multi-armed-bandit selection of PPO clipping bounds	(Zhang et al., 2023)
Trust Region Preference Approximation	Trust-region preference optimization with rule-derived preference levels	(Su et al., 6 Apr 2025)
Pessimistic Reward-based Policy Optimization	PPO-style RLHF with pessimism over preference uncertainty	(Gupta et al., 10 Mar 2025)

One terminological complication is acronym reuse. “P3O” names two distinct proposals in the supplied literature: “Pairwise Proximal Policy Optimization” for trajectory-wise RLHF (Wu et al., 2023), and a separate pessimistic preference-optimization method paired with PRPO in robust RLHF (Gupta et al., 10 Mar 2025). The former is a direct pairwise PPO variant; the latter is part of a pessimistic min–max framework.

2. Canonical preference-conditioned PPO pipeline

In RLHF pipelines such as those used for InstructGPT and subsequent LLMs, PPO is applied on top of a reward model learned from human preferences. Operationally, a reward model $r_\phi(x,y)$ is trained from pairwise preferences $(x,y_w,y_l)$ via a Bradley–Terry-style loss,

$$\mathcal{L}_R(r_\phi) = \mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \big[\log \sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big)\big],$$

and the gradients with respect to $r_\phi(x,y_w)$ and $r_\phi(x,y_l)$ are symmetric in norm, so reward learning is balanced between preferred and dispreferred responses (Ma et al., 28 Feb 2025).

The policy is then updated with a KL-regularized PPO objective of the form

$$\max_{\pi_\theta} \mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot\mid x)} \Big[ r_\phi(x,y) -\beta \log \frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)} \Big].$$

This is the standard preference-to-reward-to-policy decomposition that underlies reward-model RLHF (Zhu et al., 17 Jun 2025). At a high level, it is the same KL-constrained objective often written as

$$\max_{\pi}\; \mathbb{E}_{x\sim\mathcal{P},\,y\sim\pi(\cdot\mid x)}[r(x,y)] -\beta\,\mathbb{D}_{\mathrm{KL}}\big[\pi(\cdot\mid x)\,\|\,\pi_{\mathrm{SFT}}(\cdot\mid x)\big],$$

with PPO used as the optimizer in the policy stage (Liu et al., 2023).

A central observation in later analysis is that this two-stage pipeline induces a specific gradient structure. Reward-model training is balanced at the pair level, but policy optimization is positively imbalanced: with an optimal reward model $\rightarrow$0, PPO tends to focus more on winning responses, because high-reward outputs receive larger positive gradients while low-reward outputs are unchanged or softly suppressed by the KL term (Ma et al., 28 Feb 2025). In token-level implementations, this is usually realized by attaching the sequence-level reward to the final token and adding per-token KL penalties relative to the reference policy, followed by value estimation and a clipped PPO update (Ivison et al., 2024).

3. Relation to direct preference optimization

Direct Preference Optimization was proposed as a reward-free alternative to PPO-based RLHF. Instead of fitting $\rightarrow$1 and then running policy-gradient RL, DPO directly optimizes a logistic comparison between winner and loser log-likelihood ratios: $\rightarrow$2 This collapses reward modeling and PPO into a single supervised-style objective on preference pairs (Ma et al., 28 Feb 2025).

The main analytical argument for why PPO-style RLHF often remained stronger is gradient geometry. For DPO, the winner/loser probability-gradient ratio satisfies

$\rightarrow$3

Once training has made winners more probable than losers, the loser receives a larger gradient magnitude, so DPO is negatively imbalanced. By contrast, reward-model PPO is balanced in reward learning and positively imbalanced in policy learning. The same analysis links DPO’s negative imbalance to sensitivity under offline distribution shift, noisier updates under label noise, and brittle behavior in complex tasks; Balanced-DPO was proposed as a corrective reweighting scheme to move the winner/loser gradient ratio closer to $\rightarrow$4 and, over training, closer to PPO-like winner emphasis (Ma et al., 28 Feb 2025).

Several later works reinterpret this contrast more structurally. TGDPO decomposes sequence-level PPO with token-level rewards into a sequence of token-level PPO subproblems, derives a closed-form optimal token-level policy, and then uses the induced token-level reward inside a Bradley–Terry model to obtain a DPO-style loss with token-level reward guidance. In that perspective, ordinary DPO is a sequence-level special case of token-level “preference-based / bandit PPO,” while token-guided variants restore some of the per-token control that PPO obtains naturally from dense rewards (Zhu et al., 17 Jun 2025).

Iterative preference-optimization work makes the same connection from a proximal viewpoint. AIPO explicitly treats DPO- and IPO-style updates as trust-region-like preference optimization relative to a reference model, with agreement-aware weighting and an NLL regularizer acting as a stabilization term analogous to PPO’s KL or SFT regularization (Shen et al., 2024). APO goes further and shows that iterative preference optimization can be viewed as a proximal point method, then adds Nesterov momentum to accelerate the alignment dynamics; in that account, iterative DPO and related methods are already PPO-like in policy space, though not reward-model-based in the standard RLHF sense (He et al., 2024).

4. Representative algorithmic variants

The most direct attempt to redesign PPO around comparative feedback is Pairwise Proximal Policy Optimization. P3O formulates language generation as a contextual bandit over full trajectories, replaces scalar advantages with pairwise reward differences, and applies PPO-style clipping to trajectory-level ratios or log-ratios. Its defining theoretical claim is invariance to equivalent reward functions $\rightarrow$5, a property the paper proves for P3O and DPO but not for standard PPO. Because it operates trajectory-wise, P3O removes the value function and GAE machinery that standard token-level PPO requires for sparse sequence rewards (Wu et al., 2023).

PbPO introduces a different extension: a preference-based policy optimization framework formulated as a min–max game between the main policy and a reward model constrained to a likelihood-based confidence set. Its core objective is

$\rightarrow$6

Operationally, PbPO is fully online: it alternates reference-policy rollouts, uncertainty-seeking exploration with an enhancer policy, confidence-set reward-model updates, and clipped policy-gradient ascent. It provides regret bounds for both sequence-level and token-level reward models, and the implementation is explicitly described as PPO-like: on-policy sampling, clipped updates, and a trust-region flavor (Jia, 17 Nov 2025).

Pb-PPO uses the same name in a different sense. Here the “preference” is not pairwise human feedback over responses, but the RL task’s preference for higher return. The method treats the PPO clipping bound $\rightarrow$7 as a decision variable in a bi-level optimization problem and uses a multi-armed bandit to select the clipping bound that maximizes current return. The outer loop updates UCB-style statistics over candidate clipping bounds, while the inner loop is standard PPO with the selected $\rightarrow$8 (Zhang et al., 2023).

TRPA adapts trust-region preference optimization to reasoning tasks. It constructs four rule-based preference levels—correct format plus correct answer, correct format plus wrong answer, correct format plus incomplete answer, and incorrect format—forms preference pairs from rollouts of the current policy, and optimizes a DPO-like preference loss together with a KL trust-region penalty against the previous policy. The method is presented as a preference-based algorithm with a theoretical monotonic improvement guarantee, intended to recover the stability of PPO-style trust regions without explicit scalar reward optimization and its associated reward-hacking failure modes (Su et al., 6 Apr 2025).

Robust RLHF work adds pessimism to this picture. The pessimistic framework develops a restricted pessimistic Nash objective over policies and uncertain preference functions, then derives P3O and PRPO as practical algorithms. PRPO is the reward-model instantiation: it keeps the KL-regularized RLHF structure but adversarially updates the BTL reward model inside a pessimistic objective so that policy optimization becomes robust to overoptimization outside the support of the fixed preference dataset. The same work argues that KL regularization alone is a blunt way to suppress hacking, whereas pessimism directly addresses uncertainty in the learned preference or reward model (Gupta et al., 10 Mar 2025).

5. Empirical picture across tasks and domains

A controlled comparison of preference-learning pipelines finds that PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains, with the strongest gains in reasoning, coding, and safety. The same study concludes that all four components—preference data, learning algorithm, reward model, and policy-training prompts—matter, but data quality matters most, followed by algorithm choice, then reward-model quality, and finally unlabeled prompt selection (Ivison et al., 2024).

Pairwise P3O reports a systematically better KL–Reward trade-off than PPO and online-DPO. On TL;DR summarization, P3O-V2 reaches almost the same highest reward as DPO with 25% lower KL divergence, and on Anthropic HH the method is reported to achieve a 57.0% GPT-4 win rate against PPO while also outperforming SFT by 69.3% in GPT-4 comparisons (Wu et al., 2023). These results support the original claim that pairwise, trajectory-level optimization better matches preference-trained reward models than token-wise PPO with a critic.

PbPO extends the empirical case for online, robust Preference PPO. On five benchmarks, PbPO with a sequence-level reward model reaches an average score of 57.1 on LLaMA2-7B, and PbPO with a token-level reward model reaches 57.6; the corresponding baselines are 49.2 for PPO with an offline reward model, 49.9 for online PPO, 49.7 for DPO, and 50.8 for Online DPO. Similar trends are reported on Qwen2-7B, where PbPO averages roughly 71–72 versus online PPO at about 67.6 (Jia, 17 Nov 2025).

TRPA shows that preference-based trust-region optimization can be competitive even in reasoning-heavy settings usually dominated by reward-based RL. On K×K logic puzzles, the TRPA-trained Qwen2.5-7B-Instruct-1M reaches 0.938 average accuracy, above Logic RL at 0.889 and slightly above o3-mini-high at 0.935. On the math suite built from the DeepScaleR-Preview dataset, TRPA improves DeepSeek-R1-Distill-Qwen-7B from 0.57 average accuracy to 0.63, and on AIME 2024 it improves the base model from 0.43 to 0.57 (Su et al., 6 Apr 2025).

The preference-conditioned PPO idea also transfers outside text generation. In lyric-to-song generation, a PER-based reward model is trained from hallucination-focused preference data, and PPO then optimizes the audio LLM with sequence reward plus KL regularization. In that setting, PPO and GRPO yield PER reductions of 4.9% and 4.7%, respectively, while DPO yields a 7.4% PER reduction; all three suppress hallucinations while preserving musical quality (Zhang et al., 7 Aug 2025).

6. Design principles and unresolved issues

Several recurrent design principles emerge. First, objective shape matters, not only optimum sets. The gradient-imbalance analysis shows that two preference objectives can implicitly target similar rewards yet produce very different optimization trajectories; inspecting the winner/loser gradient ratio is therefore a substantive design criterion for any direct preference objective intended to replace PPO (Ma et al., 28 Feb 2025). Second, dense or token-level reward information remains valuable. Token-level reward guidance in TGDPO is motivated precisely by the observation that sequence-level preference optimization is bandit-like and coarse, whereas PPO benefits from finer-grained credit assignment (Zhu et al., 17 Jun 2025).

Third, coverage and sampling are central. Preference Optimization via Contrastive Divergence reinterprets preference learning as NLL estimation for an energy-based model,

$\rightarrow$9

with dispreferred completions acting as Monte Carlo samples for the partition function. In that view, hard-negative sampling is not an implementation detail but part of the statistical approximation, and online sampling from the current policy can make the estimator of the log-normalizer gradient unbiased (Chen et al., 6 Feb 2025). Statistical Rejection Sampling Optimization makes a related point from a different angle: the ideal MLE for the target optimal policy requires preference pairs sampled from that policy, whereas DPO and SLiC typically train on pairs from $\rightarrow$0 or other off-policy sources. Rejection sampling toward the estimated optimal policy $\rightarrow$1 is proposed as an offline approximation to on-policy preference collection (Liu et al., 2023).

Fourth, overoptimization cannot be reduced to KL tuning alone. The pessimistic framework for P3O and PRPO argues that fixed preference datasets induce unavoidable uncertainty outside support, and that explicit pessimism over preference or reward models is a principled response to reward or preference hacking. This shifts the focus from merely constraining policy movement to modeling epistemic uncertainty in the preference signal itself (Gupta et al., 10 Mar 2025).

Finally, the literature does not support a single universally best instantiation. Canonical PPO-style RLHF remains strong when high-quality preference data and sufficiently capable reward models are available (Ivison et al., 2024). Pairwise and trust-region preference methods reduce some of PPO’s calibration burdens and often simplify optimization (Wu et al., 2023, Su et al., 6 Apr 2025). Min–max and pessimistic variants strengthen robustness under distribution shift (Jia, 17 Nov 2025, Gupta et al., 10 Mar 2025). Dynamic-clipping and token-guided approaches show that even within PPO-style preference learning, exploration control and credit assignment remain open algorithmic degrees of freedom (Zhang et al., 2023, Zhu et al., 17 Jun 2025). Taken together, these works suggest that Preference PPO is less a single algorithm than an active design space at the intersection of RLHF, trust-region optimization, pairwise comparison modeling, and robust policy improvement.