Cross-Pair Preference Optimization

• Cross-Pair Preference Optimization is a set of techniques that derive supervision from structured contrasts, using different sources like teacher versus student responses or multiple rollout comparisons.
• Key methods such as ORPO-Distill, GraphDPO, MPPO, BOPO, and CAPO apply distinct strategies to construct pairwise comparisons that inform model learning in tasks like LLM distillation and combinatorial optimization.
• Empirical results demonstrate that carefully designed contrast mechanisms—through varied loss formulations and handling of ties—significantly enhance performance and robustness across diverse applications.

Cross-pair preference optimization denotes, in current arXiv usage, a family of preference-learning procedures that derive supervision from structured contrasts rather than from a single isolated winner–loser pair. The phrase is used in several non-identical senses: ORPO-Distill constructs a teacher-positive versus student-negative pair for cross-architecture distillation; GraphDPO replaces independent pairs with a directed acyclic preference graph over multiple rollouts; MPPO extracts multiple pairwise comparisons from a multi-response prompt; BOPO uses best-anchored comparisons within a solution pool; and CAPO contrasts a target annotator’s response against other valid human annotations for the same item (Singh et al., 29 Sep 2025, Liu et al., 8 May 2026, Xie et al., 2024, Liao et al., 10 Mar 2025, Chen et al., 27 May 2026). Taken together, these papers suggest a common theme: the informational value of preference optimization depends heavily on how the contrasted samples are constructed.

1. Conceptual scope and defining characteristics

In ORPO-Distill, the chosen and rejected responses come from different policies and different model roles: the teacher supplies the preferred reasoning trace and the student supplies the rejected reasoning trace. The paper identifies this as the key “cross-pair” aspect, because the comparison is explicitly between a teacher positive reasoning trace and a student negative reasoning trace rather than between two teacher traces (Singh et al., 29 Sep 2025).

GraphDPO uses the term in a broader structural sense. Instead of treating supervision as a set of isolated pairs, it builds a preference DAG for each prompt and applies supervision jointly across the rollout set. In that formulation, “cross-pair” does not mean a single preferred response contrasted against a single rejected response; it means that one loss term can simultaneously cover many pairwise relations through a dominated neighborhood (Liu et al., 8 May 2026).

MPPO uses “multi pair-wise” optimization for the setting where one prompt has multiple candidate replies. Its central claim is that training should not be restricted to one annotated winner–loser tuple when multiple responses are available, because top-vs-rest or combinational pairing can extract more signal from the same prompt (Xie et al., 2024).

BOPO adopts a narrower construction. It does not perform arbitrary all-pairs comparison or cross-instance comparison. Instead, it generates multiple solutions for one instance, filters them, and anchors every pair to the best sampled solution. The paper explicitly characterizes this as best-anchored, objective-aware pairwise preference optimization rather than general all-pairs ranking (Liao et al., 10 Mar 2025).

CAPO shifts the meaning again. Its rejected response is not inferior or incorrect in the usual alignment sense, but another valid human annotation for the same input. The preference relation is target-specific imitation: the model should prefer annotator $a$’s own label-explanation output over another annotator’s valid output when conditioned on annotator $a$ (Chen et al., 27 May 2026).

2. Construction of preference relations

A central design variable across these methods is the mechanism by which preferred and rejected items are assembled.

ORPO-Distill forms triplets

$\langle \text{Prompt, Chosen, Rejected} \rangle$

where Chosen is a teacher chain-of-thought trace that leads to the correct answer and Rejected is a student chain-of-thought trace that leads to an incorrect answer. The method samples $K$ diverse positive traces from the teacher and $K$ diverse negative traces from the student, with $K=8$ chosen after testing $K \in \{4,8,12\}$. It also discards traces with ROUGE-L overlap above $0.80$, uses the same “Reason-then-Answer” format for teacher and student, and does not inject the wrong answer when generating student negatives (Singh et al., 29 Sep 2025).

The negative-trace policy in ORPO-Distill is itself part of the pair-construction procedure. The paper defines three regimes: Off-policy, where negatives are fixed from the initial student model; On-policy, where negatives are regenerated after every epoch using the latest checkpoint; and Mixed-policy, where negatives are generated by randomly mixing the base student model and the most recent checkpoint. The mixed-policy regime uses a policy fraction $\phi = 0.5$ (Singh et al., 29 Sep 2025).

GraphDPO begins from $K$ sampled responses $a$0 for a prompt $a$1 and a preference relation $a$2. It constructs a DAG

$a$3

with an edge

$a$4

For discrete or sparse signals, responses are grouped into equivalence classes

$a$5

and edges are allowed only across classes, not within a class. The paper emphasizes that “intra-class comparisons contribute zero loss,” which prevents arbitrary ordering among tied responses (Liu et al., 8 May 2026).

MPPO constructs multiple pairs from a multi-response prompt rather than from a fixed triple $a$6. It describes two broad strategies. In top-vs-rest pairing, the highest-scoring response is treated as positive and the others as negatives. In combinational pairing, two responses from the same prompt are randomly sampled, their scores are compared, and the higher-scoring response becomes the positive. In the UltraFeedback setup used in the paper, each prompt has four responses rated by GPT-4 from $a$7 to $a$8, and the scores are normalized to $a$9 by dividing by $\langle \text{Prompt, Chosen, Rejected} \rangle$0 (Xie et al., 2024).

BOPO constructs preference pairs from a solution pool for a single combinatorial optimization instance. It first performs hybrid rollout, generating $\langle \text{Prompt, Chosen, Rejected} \rangle$1 solutions from sampling and $\langle \text{Prompt, Chosen, Rejected} \rangle$2 greedy solution. It then applies uniform filtering by sorting the solutions and selecting $\langle \text{Prompt, Chosen, Rejected} \rangle$3 evenly across the quality spectrum. Finally, it performs best-anchored pairing: $\langle \text{Prompt, Chosen, Rejected} \rangle$4 so that only $\langle \text{Prompt, Chosen, Rejected} \rangle$5 preference pairs are created and every suboptimal solution is compared only against the best filtered solution (Liao et al., 10 Mar 2025).

CAPO constructs a chosen–rejected pair from human annotations on the same item. For target annotator $\langle \text{Prompt, Chosen, Rejected} \rangle$6, the chosen response is $\langle \text{Prompt, Chosen, Rejected} \rangle$7, and the rejected response is $\langle \text{Prompt, Chosen, Rejected} \rangle$8 with $\langle \text{Prompt, Chosen, Rejected} \rangle$9. The rejected response remains a valid human annotation; it is simply less target-specific. To reduce confounding by obvious label disagreement, the main pair policies are conservative: on VariErr, the main setting requires the same NLI label, and on R2 the main setting requires $K$0 (Chen et al., 27 May 2026).

3. Objective formulations

Although these methods share a preference-learning orientation, their objectives differ substantially.

ORPO-Distill uses the ORPO objective from Hong et al., combining a standard SFT term with an odds-ratio preference term. For prompt $K$1, favored positive trace $K$2, disfavored negative trace $K$3, and student distribution $K$4,

$K$5

$K$6

with

$K$7

The pipeline sets $K$8, and the paper states that this is appropriate for strongly adapting the student to prefer correct reasoning paths and “clip incorrect generation paths” (Singh et al., 29 Sep 2025).

GraphDPO defines a dominated neighborhood

$K$9

and centered scores

$K$0

Its local loss is

$K$1

which is equivalent to

$K$2

The full graph loss averages over nodes with non-empty dominated neighborhoods. The paper explicitly shows that when $K$3 with one edge, the objective reduces to standard DPO up to centering, making pairwise DPO a special case of the graph formulation (Liu et al., 8 May 2026).

MPPO replaces DPO’s reference-model-based reward reparameterization with a reward proxy defined by the geometric mean of token probabilities: $K$4 Its pairwise variants then operate on the resulting scalar scores $K$5 and $K$6. In the one-positive/one-negative case,

$K$7

For the top-vs-rest setting with $K$8 negatives, the strongest reported variant is Pair-MNM: $K$9 The paper also studies point-wise and list-wise alternatives, but concludes that the pair-wise implementation performs best (Xie et al., 2024).

BOPO defines explicit preference for a minimization COP as

$K=8$0

so that lower objective value implies higher preference. The model’s implicit preference is its average log-likelihood,

$K=8$1

and the Bradley–Terry preference probability is scaled by an adaptive factor

$K=8$2

The resulting loss is

$K=8$3

The paper interprets the ratio $K=8$4 as a preference-based adaptive scaling factor or “natural curriculum” (Liao et al., 10 Mar 2025).

CAPO uses a DPO-style reward difference relative to a frozen SFT reference model. For target annotator $K=8$5,

$K=8$6

where $K=8$7 is the frozen SFT policy for annotator $K=8$8. Its loss is

$K=8$9

with $K \in \{4,8,12\}$0 in the reported setup (Chen et al., 27 May 2026).

4. Representative frameworks and application domains

The current literature uses cross-pair constructions in several distinct domains.

Method	Cross-pair unit	Domain
ORPO-Distill	teacher positive vs student negative trace	cross-architecture LLM distillation
GraphDPO	node vs dominated neighborhood in a preference DAG	reasoning and program synthesis
MPPO	multiple response pairs from one prompt	LLM alignment with human feedback
BOPO	best solution vs suboptimal solutions from one instance	neural combinatorial optimization
CAPO	target annotator output vs other annotators’ outputs	annotator-specific explanation imitation

ORPO-Distill is motivated by cross-architecture settings in which white-box KD is not applicable because teacher and student do not share vocabulary or logits. The reported teacher is InternLM 2.5 7B-Chat; the student models are InternLM 2.5 1.8B-Chat and TinyLlama 1.1B-Instruct; and training uses full-parameter tuning for five epochs with temperature $K \in \{4,8,12\}$1 (Singh et al., 29 Sep 2025).

GraphDPO is evaluated on GSM8K, MATH-500, and APPS. It adds two elements absent from standard pairwise DPO: equivalence-class masking for ties and optional ground-truth anchoring. A verified solution can be inserted as a dominant node, and the anchoring term is weighted by an annealed schedule that is high early in training and then reduced later (Liu et al., 8 May 2026).

MPPO is trained from a Llama3-8B SFT model, using UltraChat-200k for SFT and UltraFeedback for preference optimization. Its benchmark suite is MT-Bench and Arena-Hard, and the paper emphasizes that MPPO does not require a reference model and only requires tuning the learning rate in its reported experiments (Xie et al., 2024).

BOPO is architecture agnostic and is demonstrated with MGL for JSP and FJSP and a POMO backbone for TSP. The paper presents it as a training paradigm rather than a model architecture, explicitly intended for COPs where objective values are cheap to compute (Liao et al., 10 Mar 2025).

CAPO is evaluated on two sentence-pair tasks with four annotators per item: VariErr NLI and R2 paraphrase judgment. Its base models are Qwen3-4B-Instruct-2507 and Llama-3.2-3B-Instruct. The method first trains one independent LoRA adapter per annotator by SFT and then continues from that adapter with CAPO (Chen et al., 27 May 2026).

5. Empirical findings

ORPO-Distill reports a consistent progression across five QA benchmarks: Single CoT fine-tuning improves over zero-shot; Diverse CoT fine-tuning improves further; ORPO with student negatives improves much more; and Mixed-policy ORPO is best overall. For TinyLlama 1.1B-Instruct, the reported averages are 37.58 for Diverse CoT FT, 41.36 for Off-policy ORPO, 38.97 for On-policy ORPO, and 43.17 for Mixed-policy ORPO. For InternLM 2.5 1.8B-Chat, the corresponding averages are 48.67, 53.93, 50.46, and 55.84. The paper also reports that $K \in \{4,8,12\}$2 yields higher accuracy than $K \in \{4,8,12\}$3, supporting the use of student-generated negatives (Singh et al., 29 Sep 2025).

GraphDPO reports superior performance over the compared baselines on all three reported tasks. On GSM8K, PRO scores 88.70, LiPO 89.23, GraphDPO without GT 92.42, and GraphDPO with GT 92.75. On MATH-500, the corresponding scores are 83.60, 85.60, 87.40, and 88.87. On APPS, they are 68.58, 69.32, 72.93, and 73.76. The paper further states that larger $K \in \{4,8,12\}$4 generally helps because the graph becomes richer, and that anchoring gives the biggest boost when $K \in \{4,8,12\}$5 is small (Liu et al., 8 May 2026).

MPPO finds that the pair-wise implementation is stronger than the point-wise and list-wise implementations. Pair-MNM achieves the best reported MT-Bench score, 6.16. On Arena-Hard, Pair-MNM achieves a 21.6 win rate, exceeding DPO at 15.9, KTO at 12.8, and ORPO at 10.7, while remaining below SimPO at 23.4. The paper states that Point-CE and Point-MSE perform poorly, and that List-MLE is unstable, especially on Arena-Hard (Xie et al., 2024).

BOPO reports that removing hybrid rollout, uniform filtering, or best-anchored pairing hurts performance. In its loss-function ablation, $K \in \{4,8,12\}$6 outperforms $K \in \{4,8,12\}$7, $K \in \{4,8,12\}$8, and $K \in \{4,8,12\}$9, indicating that the objective-based adaptive scaling improves performance. The paper states that POCO is evaluated on JSP, TSP, and FJSP and consistently outperforms or matches strong baselines (Liao et al., 10 Mar 2025).

CAPO reports that prompting is weakest, SFT substantially improves annotator imitation, and CAPO further improves imitation-oriented metrics and judge-based attribution while keeping decision accuracy competitive. On VariErr with Qwen3, SFT obtains label accuracy 0.638, ImiScore 0.859, and Judge Acc 0.300, while CAPO obtains label accuracy 0.627, ImiScore 0.888, and Judge Acc 0.328. On R2 with Llama3.2, SFT obtains score accuracy 0.450, MAE 1.535, ImiScore 0.972, and Judge Acc 0.512, while CAPO obtains score accuracy 0.460, MAE 1.470, ImiScore 0.995, and Judge Acc 0.530. Human validation ranks the systems as CAPO > SFT > prompting, with overall agreement 82.8% and Cohen’s $0.80$0 (Chen et al., 27 May 2026).

6. Methodological distinctions, limitations, and interpretive issues

A recurrent misconception is that cross-pair preference optimization always means “more pairwise comparisons.” The papers collectively show that this is not the only interpretation. GraphDPO argues that flattening a multi-rollout prompt into independent pairs breaks transitivity, creates redundant supervision, and can create conflicting or noisy gradients when ties are present. Its response is to aggregate supervision over graph neighborhoods instead of enumerating independent pairs (Liu et al., 8 May 2026). By contrast, MPPO reports that, among its own point-wise, pair-wise, and list-wise implementations, pair-wise is strongest. This suggests that the empirical value of richer structure depends on how preference information is represented and on the task regime, rather than on a universal ordering between pairwise and non-pairwise objectives (Xie et al., 2024).

A second misconception is that the rejected item must be objectively wrong. ORPO-Distill does use incorrect student traces as negatives, but CAPO explicitly does not: its rejected outputs are other valid human annotations, and the optimization target is target-specific imitation rather than generic quality ranking (Singh et al., 29 Sep 2025, Chen et al., 27 May 2026).

A third distinction concerns the scope of “cross.” BOPO is explicit that its construction is not general all-pairs ranking or cross-instance comparison; it is anchor-based comparison within a single instance’s solution pool (Liao et al., 10 Mar 2025). ORPO-Distill, by contrast, is genuinely cross-policy and cross-architecture: preferred and rejected traces come from different models with different roles (Singh et al., 29 Sep 2025).

The methods also identify different failure modes. ORPO-Distill reports that pure on-policy updates degrade performance relative to off-policy and mixed-policy because the negative trace distribution narrows too much (Singh et al., 29 Sep 2025). GraphDPO addresses tie-heavy data by excluding intra-class edges so that “intra-class comparisons contribute zero loss,” thereby avoiding spurious gradients (Liu et al., 8 May 2026). CAPO notes that its effectiveness depends on a small, repeated-annotator regime; larger crowdsourcing settings introduce harder negative-sampling problems, potential dilution from easy negatives, and cold-start issues for annotators with limited history (Chen et al., 27 May 2026).

Ethical and interpretive cautions also vary by domain. CAPO stresses that high imitation scores indicate recognizability rather than “better” behavior, and that annotator IDs should remain anonymized and not be used to infer sensitive attributes (Chen et al., 27 May 2026). In combinatorial optimization, BOPO’s objective-guided scaling assumes that objective values are cheaply available and reliable as preference signals, which is appropriate for the reported COP settings but not automatically transferable beyond them (Liao et al., 10 Mar 2025).

Taken together, these works indicate that cross-pair preference optimization is better understood as a design space than as a single algorithm. The central choices are the source of the contrasted samples, the structural unit of comparison, the handling of ties or near-equivalences, and the relationship between the preference signal and the task-specific error distribution.