PGPO: Preference-Guided Policy Optimization

Updated 13 July 2026

PGPO is a family of policy optimization methods that uses pairwise preference signals instead of scalar rewards to guide learning.
PGPO methods span diverse applications including offline reinforcement learning, combinatorial optimization, molecular lead optimization, and LLM planning.
PGPO enhances sample efficiency by extracting multiple learning signals per batch and preserving the comparative structure of trajectory pairs.

Searching arXiv for papers using the term “Preference-Guided Policy Optimization” and related variants. Preference-Guided Policy Optimization (PGPO) denotes a family of policy-optimization methods in which pairwise or comparative preference information directly shapes policy learning, rather than serving only as supervision for a separately trained scalar reward model. In recent arXiv usage, the term covers several non-identical formulations: offline preference-based reinforcement learning that bypasses explicit reward fitting, entropy-regularized policy updates in combinatorial optimization that reparameterize latent reward differences by policy likelihoods, multi-turn reinforcement learning with intra-trajectory preference losses, planning-oriented direct preference optimization for LLM agents, and two-stage reference-guided DPO variants. This body of work collectively treats preferences as first-class optimization signals and not merely as a precursor to conventional reward maximization (Kang et al., 2023, Pan et al., 13 May 2025, Wang et al., 26 Sep 2025, Cao et al., 2 Jun 2025, Pan et al., 22 Apr 2025).

1. Scope, terminology, and conceptual boundaries

The literature does not present a single canonical PGPO algorithm. Instead, closely related papers use the label for distinct update rules, model classes, and supervision pipelines. "Beyond Reward: Offline Preference-guided Policy Optimization" introduces OPPO in offline preference-based reinforcement learning, where offline trajectories and human preferences are modeled in one step through a contextual policy and a learned hindsight-information representation $z$ , explicitly avoiding a separately learned scalar reward (Kang et al., 2023). "Preference Optimization for Combinatorial Optimization Problems" presents a formulation summarized as PGPO that converts quantitative rewards into pairwise preference signals and optimizes an entropy-regularized objective through a policy-based reparameterization of latent reward differences (Pan et al., 13 May 2025). "POLO: Preference-guided multi-turn Optimization for Lead Optimization" uses PGPO for a dual-level update in which PPO over full optimization trajectories is combined with turn-level preference learning over intermediate molecules (Wang et al., 26 Sep 2025). "PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization" uses the acronym for a DPO-style framework driven by plan quality and plan-following comparisons in LLM agents (Cao et al., 2 Jun 2025). "Pre-DPO" describes a preference-guided policy optimization framework in which an already optimized model becomes a guiding reference for a second DPO stage (Pan et al., 22 Apr 2025).

This suggests that PGPO is better understood as a design pattern than as a single algorithm. Across formulations, the recurring idea is to preserve comparative structure—preferred versus dispreferred trajectories, responses, plans, molecules, or solutions—during optimization, instead of collapsing that structure immediately into one scalar target.

2. Preference signal construction

A central difference among PGPO variants is how the preference signal is produced. Some methods rely on human pairwise labels, whereas others derive preferences automatically from oracle scores, local search improvements, or model-based comparisons.

Variant	Preference source	Update mechanism
OPPO	Human labels $y\in\{0,0.5,1\}$ on offline trajectory pairs $(\tau^i,\tau^j,y)$	Hindsight-information matching plus preference modeling over $z^*$ (Kang et al., 2023)
Combinatorial PGPO	Reward comparisons $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$; LS( $\tau$ ) vs. $\tau$ pairs in fine-tuning	Bradley–Terry or related comparison model with policy-guided preference probability (Pan et al., 13 May 2025)
POLO PGPO	Intermediate molecules within a trajectory, with a pair $(i,j)$ whenever $r_j>r_i$	PPO trajectory objective plus DPO-style list-wise ranking loss (Wang et al., 26 Sep 2025)
Planning PGPO	Contrastive datasets $D_p$ from outcome reward and $y\in\{0,0.5,1\}$ 0 from plan-following reward	Two DPO losses plus supervised cloning on winning expert trajectories (Cao et al., 2 Jun 2025)
Pre-DPO	Offline preference triples $y\in\{0,0.5,1\}$ 1, reweighted by a guiding reference model	Two-stage DPO in which the guide induces per-sample weights $y\in\{0,0.5,1\}$ 2 (Pan et al., 22 Apr 2025)

A common misconception is that preference-guided optimization necessarily means direct human labeling at every step. The surveyed formulations do not support that interpretation. In the combinatorial and molecular settings, preference pairs are induced from objective improvements or oracle evaluations; in the planning setting, preferences arise from reward comparisons between generated and expert trajectories; and in the DPO-based setting, the guide modulates the strength of each offline preference update. Preference guidance is therefore about the structure of the supervision signal, not about a single data-collection protocol.

3. Objective families and update rules

In offline preference-based reinforcement learning, OPPO introduces two coupled objectives. The offline hindsight information matching objective

$y\in\{0,0.5,1\}$ 3

trains a transformer-based contextual policy $y\in\{0,0.5,1\}$ 4 so that conditioning on the encoded hindsight information $y\in\{0,0.5,1\}$ 5 reproduces trajectories whose decoded information matches the original trajectory. The preference modeling objective

$y\in\{0,0.5,1\}$ 6

searches for the optimal context $y\in\{0,0.5,1\}$ 7 by making it closer to embeddings of preferred trajectories than to embeddings of dispreferred ones. Training alternates updates on $y\in\{0,0.5,1\}$ 8 and $y\in\{0,0.5,1\}$ 9, and deployment uses $(\tau^i,\tau^j,y)$ 0 (Kang et al., 2023).

In combinatorial optimization, PGPO begins from the maximum-entropy objective

$(\tau^i,\tau^j,y)$ 1

and exploits the analytical optimum

$(\tau^i,\tau^j,y)$ 2

From this, the latent reward differences can be reparameterized by log-policy differences: $(\tau^i,\tau^j,y)$ 3 Substituting this into a Bradley–Terry comparison model yields a preference probability directly parameterized by the policy, and the resulting gradient estimate operates on pairwise trajectory comparisons instead of raw reward regression (Pan et al., 13 May 2025).

POLO’s PGPO explicitly combines two optimization levels. The trajectory objective is PPO over full multi-turn lead-optimization trajectories, while the turn-level preference objective ranks intermediate molecules within each trajectory. The combined objective is

$(\tau^i,\tau^j,y)$ 4

with $(\tau^i,\tau^j,y)$ 5 defined through a DPO-style log-ratio $(\tau^i,\tau^j,y)$ 6 and a list-wise ranking loss over selected preference pairs $(\tau^i,\tau^j,y)$ 7. The method further subsamples the top $(\tau^i,\tau^j,y)$ 8 most informative pairs per trajectory to reduce compute (Wang et al., 26 Sep 2025).

The planning-oriented PGPO for LLM agents also follows a direct preference optimization paradigm, but with two distinct contrastive datasets. $(\tau^i,\tau^j,y)$ 9 compares plan-and-rollout tuples under the plan-driven reward $z^*$ 0, while $z^*$ 1 compares partial executions under the Monte Carlo plan-following reward $z^*$ 2. The optimization minimizes $z^*$ 3, where $z^*$ 4 is a supervised cloning term on winning expert trajectories added specifically to prevent collapse of the preference objective (Cao et al., 2 Jun 2025).

Pre-DPO reframes preference-guided optimization as reference-guided data weighting. Starting from an SFT model, it first runs a preference-optimization method $z^*$ 5 to obtain $z^*$ 6, then sets $z^*$ 7, and finally reruns DPO from the original SFT initialization using $z^*$ 8 as the reference. In the gradient view of DPO, the induced

$z^*$ 9

acts as a learned per-sample weight, concentrating updates on pairs the guiding model regards as more learnable or more aligned (Pan et al., 22 Apr 2025).

4. Representative instantiations across domains

The offline RL instantiation centers on static trajectory data. OPPO assumes a dataset $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$0 of offline trajectories and a preference set $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$1, with $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$2 serving as a hindsight-information extractor and $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$3 implemented as a transformer-based contextual policy. The design premise is that dynamics and task information are orthogonal, so a direct one-step model of trajectories and preferences can avoid the information bottleneck of fitting a scalar reward before running an off-the-shelf offline RL algorithm (Kang et al., 2023).

In neural combinatorial optimization, PGPO is used for TSP, CVRP, and FFSP. The method samples multiple trajectories per instance, computes pairwise labels among them, and in the fine-tuning stage occasionally refines trajectories with a light local search procedure. The examples given are $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$4-Opt for TSP and Swap* for CVRP. Because $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$5, the pair $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$6 is inserted into the same preference objective, turning local-search improvements into expert demonstrations without importance sampling (Pan et al., 13 May 2025).

In molecular lead optimization, PGPO operates inside an LLM-mediated MDP whose state contains the full in-context conversation, including the lead molecule, optimization history, and past oracle scores. Actions are molecular proposals in the format > …<answer>SMILES</answer>, episodes are approximately $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$7 turns, and the reward combines oracle improvements with a similarity constraint relative to the original lead compound. The training framework first samples rollouts, then performs two-stage trajectory filtering—keeping top $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$8-variance groups and then top $y_{1,2}=\mathbbm{1}(r(x,\tau_1)>r(x,\tau_2))$9-score trajectories—before applying one gradient step on the combined trajectory-plus-preference loss (Wang et al., 26 Sep 2025).

For interactive LLM agents, PGPO is tied to pseudocode-style plans rather than free-form natural-language plans. A P-code Plan is a structured sequence of planning steps with identifiers, high-level function names, parameters, return values, and optional control flow. The agent emits the plan at the start of the episode and then conditions subsequent thoughts and actions on $\tau$ 0. Full-parameter fine-tuning is performed on open-source LLMs including LLaMA-2 7B/13B, LLaMA-3 8B, Mistral-7B, and Qwen2.5, and the Monte Carlo estimator for $\tau$ 1 uses $\tau$ 2 samples (Cao et al., 2 Jun 2025).

The reference-guided DPO instantiation is architecturally simpler but methodologically important. It does not alter the DPO loss family; instead, it changes how the reference model is chosen. The core claim is that the reference model acts as a data weight adjuster, so using a guiding reference that has already been preference-optimized once can improve data utilization relative to the standard choice $\tau$ 3 (Pan et al., 22 Apr 2025).

5. Empirical profile

Reported empirical outcomes are heterogeneous because each PGPO variant targets a different regime, but several patterns recur. In offline PbRL, OPPO is evaluated on D4RL Gym-Mujoco tasks—Hopper, Walker2d, and HalfCheetah under Medium, Medium-Replay, and Medium-Expert datasets—and on Robosuite tasks "Lift" and "Can." On the Gym sum of 9 tasks, OPPO reports $\tau$ 4, compared with $\tau$ 5 for DT+ $\tau$ 6, $\tau$ 7 for DT+ $\tau$ 8, $\tau$ 9 for CQL+ $\tau$ 0, $\tau$ 1 for IQL+ $\tau$ 2, and $\tau$ 3 for BC. On Robosuite, OPPO sums to $\tau$ 4 versus $\tau$ 5 for IQL+ $\tau$ 6 and $\tau$ 7 for IQL+PT. The ablation labeled "OPPO-a" shows that omitting the cross-gradient influence produces a drop of approximately $\tau$ 8 points in Gym, and the Hopper label-efficiency study reports a total sum of approximately $\tau$ 9 with $(i,j)$ 0k labels versus $(i,j)$ 1 for $(i,j)$ 2k labels (Kang et al., 2023).

In combinatorial optimization, PGPO is reported to improve both convergence efficiency and final solution quality. For POMO on TSP/CVRP with $(i,j)$ 3, the RF gap $(i,j)$ 4 is reduced to $(i,j)$ 5 in the same time, and PGPO plus fine-tuning reaches a $(i,j)$ 6 gap. On FFSP with MatNet, the makespan gap changes from $(i,j)$ 7 under RF to $(i,j)$ 8 under PO, and test-time augmentation with PO+Aug reaches $(i,j)$ 9. The paper further states that PGPO typically needs $r_j>r_i$ 0– $r_j>r_i$ 1 of the epochs to match RF performance, and that zero-shot library benchmarks show ELG+RF versus ELG+PGPO gaps of $r_j>r_i$ 2 on TSPLib and $r_j>r_i$ 3 on CVRPLib (Pan et al., 13 May 2025).

In molecular lead optimization, POLO reports an average success rate of $r_j>r_i$ 4 on single-property tasks versus $r_j>r_i$ 5 for the best baseline, and $r_j>r_i$ 6 on multi-property tasks versus $r_j>r_i$ 7 for the best baseline, using only $r_j>r_i$ 8 oracle evaluations per lead across $r_j>r_i$ 9 test leads. The hardest three-objective task reaches $D_p$ 0 success versus $D_p$ 1– $D_p$ 2 for other methods. The paper attributes part of the sample-efficiency gain to the fact that PGPO extracts $D_p$ 3 learning signals per batch, compared with $D_p$ 4 for standard RL (Wang et al., 26 Sep 2025).

For LLM agents, the planning-oriented PGPO is evaluated on ALFWorld, WebShop, TextCraft, and ScienceWorld. On the LLaMA-2-7B backbone, the reported average reward is $D_p$ 5 for PGPO versus $D_p$ 6 for IPR, $D_p$ 7 for ETO, and $D_p$ 8 for SFT across ALFWorld seen and unseen, WebShop, and TextCraft. On ALFWorld, invalid action rate decreases from $D_p$ 9 for ETO and $y\in\{0,0.5,1\}$ 00 for IPR to $y\in\{0,0.5,1\}$ 01 for PGPO. On WebShop, success rate at final reward $y\in\{0,0.5,1\}$ 02 is reported as $y\in\{0,0.5,1\}$ 03 for PGPO, compared with $y\in\{0,0.5,1\}$ 04 for IPR and $y\in\{0,0.5,1\}$ 05 for ETO (Cao et al., 2 Jun 2025).

In instruction tuning, Pre-DPO reports improvements on AlpacaEval 2.0 and Arena-Hard v0.1. For Llama3.2-3B Instruct, vanilla DPO with SFT reference reaches AlpacaEval 2 length-controlled win rate $y\in\{0,0.5,1\}$ 06, raw win rate $y\in\{0,0.5,1\}$ 07, and Arena-Hard win rate $y\in\{0,0.5,1\}$ 08, while Pre-DPO with a DPO $y\in\{0,0.5,1\}$ 09 guide reports $y\in\{0,0.5,1\}$ 10, $y\in\{0,0.5,1\}$ 11, and $y\in\{0,0.5,1\}$ 12, respectively. The paper summarizes the average AlpacaEval 2 LC gain as approximately $y\in\{0,0.5,1\}$ 13 over vanilla DPO in Base/Instruct settings and Arena-Hard gains as approximately $y\in\{0,0.5,1\}$ 14– $y\in\{0,0.5,1\}$ 15 points (Pan et al., 22 Apr 2025).

Several adjacent methods clarify what is specific to PGPO and what belongs to a broader preference-optimization landscape. PbPO for LLM bootstrapping formulates policy learning as a min–max game between the policy and a reward model constrained to a confidence set $y\in\{0,0.5,1\}$ 16, and provides high-probability regret bounds of $y\in\{0,0.5,1\}$ 17 for sequence-level reward models and $y\in\{0,0.5,1\}$ 18 for token-level reward models (Jia, 17 Nov 2025). GOPO replaces GRPO’s z-scored scalar rewards with purely ordinal within-group ranks, assigning advantages in $y\in\{0,0.5,1\}$ 19 and proving both a gradient-norm inflation factor $y\in\{0,0.5,1\}$ 20 and a uniform worst-case bound $y\in\{0,0.5,1\}$ 21 (Choi et al., 1 Feb 2026). P-GRPO replaces within-batch normalization by preference-group-specific historical baselines $y\in\{0,0.5,1\}$ 22, targeting heterogeneous user preferences (Wang et al., 17 Feb 2026). TPMM-DPO addresses iterative DPO error accumulation by merging the trajectory of past policy checkpoints into a learned reference model rather than using only the immediately previous checkpoint (Fu et al., 22 May 2026).

A second misconception is that preference-guided optimization is uniformly reward-model-free. The surveyed papers do not support that claim. OPPO explicitly removes separate scalar reward learning, but PbPO centers the reward model as the adversarial follower in a min–max game, and combinatorial PGPO starts from an underlying reward $y\in\{0,0.5,1\}$ 23 before converting it into qualitative preference labels. Conversely, some methods that use the acronym PGPO are not preference-guided in the narrow sense at all: "V-tableR1" uses "Process-Guided Direct Alignment Policy Optimization," where a critic VLM supplies dense process rewards, format scores, and answer accuracy for multimodal table reasoning, leading to reported averages of $y\in\{0,0.5,1\}$ 24 on table fact verification and $y\in\{0,0.5,1\}$ 25 on table question answering for the $y\in\{0,0.5,1\}$ 26B model (Jiang et al., 22 Apr 2026). This suggests that the acronym has become overloaded across adjacent alignment literatures.

Open problems remain prominent. OPPO explicitly states that it does not provide formal convergence proofs or sample-complexity bounds, and names theoretical analysis, active preference querying, extending $y\in\{0,0.5,1\}$ 27 to a distribution over contexts, and multi-task or lifelong sharing of $y\in\{0,0.5,1\}$ 28 as future work (Kang et al., 2023). Pre-DPO points to dynamic reset strategies, combining reward modeling with DPO under guiding references, and meta-learning $y\in\{0,0.5,1\}$ 29 as natural extensions (Pan et al., 22 Apr 2025). TPMM-DPO proposes sample-adaptive fusion, uncertainty-aware merging, and integration with reward-model training as broader directions for preference-guided optimization trajectories (Fu et al., 22 May 2026). The aggregate literature therefore supports a restrained conclusion: PGPO is not a settled algorithmic primitive, but an active research area organized around the preservation of comparative supervision during policy optimization.