---
title: Cross-Pair Preference Optimization
url: https://www.emergentmind.com/topics/cross-pair-preference-optimization
type: topic
---

# Cross-Pair Preference Optimization

Cross-pair preference optimization denotes, in current arXiv usage, a family of preference-learning procedures that derive supervision from structured contrasts rather than from a single isolated winner–loser pair. The phrase is used in several non-identical senses: ORPO-Distill constructs a teacher-positive versus student-negative pair for cross-architecture distillation; GraphDPO replaces independent pairs with a directed acyclic preference graph over multiple rollouts; MPPO extracts multiple pairwise comparisons from a multi-response prompt; BOPO uses best-anchored comparisons within a solution pool; and CAPO contrasts a target annotator’s response against other valid human annotations for the same item [2509.25100] [2605.08037] [2412.15244] [2503.07580] [2605.28802]. Taken together, these papers suggest a common theme: the informational value of preference optimization depends heavily on how the contrasted samples are constructed.

## 1. Conceptual scope and defining characteristics

In ORPO-Distill, the chosen and rejected responses come from different policies and different model roles: the teacher supplies the preferred reasoning trace and the student supplies the rejected reasoning trace. The paper identifies this as the key “cross-pair” aspect, because the comparison is explicitly between a teacher positive reasoning trace and a student negative reasoning trace rather than between two teacher traces [2509.25100].

GraphDPO uses the term in a broader structural sense. Instead of treating supervision as a set of isolated pairs, it builds a preference DAG for each prompt and applies supervision jointly across the rollout set. In that formulation, “cross-pair” does not mean a single preferred response contrasted against a single rejected response; it means that one loss term can simultaneously cover many pairwise relations through a dominated neighborhood [2605.08037].

MPPO uses “multi pair-wise” optimization for the setting where one prompt has multiple candidate replies. Its central claim is that training should not be restricted to one annotated winner–loser tuple when multiple responses are available, because top-vs-rest or combinational pairing can extract more signal from the same prompt [2412.15244].

BOPO adopts a narrower construction. It does not perform arbitrary all-pairs comparison or cross-instance comparison. Instead, it generates multiple solutions for one instance, filters them, and anchors every pair to the best sampled solution. The paper explicitly characterizes this as best-anchored, objective-aware pairwise preference optimization rather than general all-pairs ranking [2503.07580].

CAPO shifts the meaning again. Its rejected response is not inferior or incorrect in the usual alignment sense, but another valid human annotation for the same input. The preference relation is target-specific imitation: the model should prefer annotator \(a\)’s own label-explanation output over another annotator’s valid output when conditioned on annotator \(a\) [2605.28802].

## 2. Construction of preference relations

A central design variable across these methods is the mechanism by which preferred and rejected items are assembled.

ORPO-Distill forms triplets
\[
\langle \text{Prompt, Chosen, Rejected} \rangle
\]
where **Chosen** is a teacher chain-of-thought trace that leads to the correct answer and **Rejected** is a student chain-of-thought trace that leads to an incorrect answer. The method samples \(K\) diverse positive traces from the teacher and \(K\) diverse negative traces from the student, with \(K=8\) chosen after testing \(K \in \{4,8,12\}\). It also discards traces with ROUGE-L overlap above \(0.80\), uses the same “Reason-then-Answer” format for teacher and student, and does not inject the wrong answer when generating student negatives [2509.25100].

The negative-trace policy in ORPO-Distill is itself part of the pair-construction procedure. The paper defines three regimes: **Off-policy**, where negatives are fixed from the initial student model; **On-policy**, where negatives are regenerated after every epoch using the latest checkpoint; and **Mixed-policy**, where negatives are generated by randomly mixing the base student model and the most recent checkpoint. The mixed-policy regime uses a policy fraction \(\phi = 0.5\) [2509.25100].

GraphDPO begins from \(K\) sampled responses \(\{y_1,\ldots,y_K\}\) for a prompt \(x\) and a preference relation \(\succ_x\). It constructs a DAG
\[
\mathcal{G}_x=(V,E), \qquad V=\{1,\ldots,K\},
\]
with an edge
\[
(i \rightarrow j)\in E \quad \text{iff} \quad y_i \succ_x y_j.
\]
For discrete or sparse signals, responses are grouped into equivalence classes
\[
\mathcal{C}_1 \succ_x \cdots \succ_x \mathcal{C}_G,
\]
and edges are allowed only across classes, not within a class. The paper emphasizes that “intra-class comparisons contribute zero loss,” which prevents arbitrary ordering among tied responses [2605.08037].

MPPO constructs multiple pairs from a multi-response prompt rather than from a fixed triple \((x,y_w,y_l)\). It describes two broad strategies. In **top-vs-rest pairing**, the highest-scoring response is treated as positive and the others as negatives. In **combinational pairing**, two responses from the same prompt are randomly sampled, their scores are compared, and the higher-scoring response becomes the positive. In the UltraFeedback setup used in the paper, each prompt has four responses rated by GPT-4 from \(1\) to \(10\), and the scores are normalized to \([0.1,1]\) by dividing by \(10\) [2412.15244].

BOPO constructs preference pairs from a solution pool for a single combinatorial optimization instance. It first performs **hybrid rollout**, generating \(B-1\) solutions from sampling and \(1\) greedy solution. It then applies **uniform filtering** by sorting the solutions and selecting \(K\) evenly across the quality spectrum. Finally, it performs **best-anchored pairing**:
\[
\mathcal P = \{(\mathbf x, \mathbf y_1, \mathbf y_k) \mid k\in\{2,\cdots,K\}\},
\]
so that only \(K-1\) preference pairs are created and every suboptimal solution is compared only against the best filtered solution [2503.07580].

CAPO constructs a chosen–rejected pair from human annotations on the same item. For target annotator \(a\), the chosen response is \(z^+=z_{i,a}\), and the rejected response is \(z^-=z_{i,b}\) with \(b\neq a\). The rejected response remains a valid human annotation; it is simply less target-specific. To reduce confounding by obvious label disagreement, the main pair policies are conservative: on VariErr, the main setting requires the same NLI label, and on R2 the main setting requires \(|s_{i,a}-s_{i,b}| \le 1\) [2605.28802].

## 3. Objective formulations

Although these methods share a preference-learning orientation, their objectives differ substantially.

ORPO-Distill uses the ORPO objective from Hong et al., combining a standard SFT term with an odds-ratio preference term. For prompt \(x\), favored positive trace \(y_P\), disfavored negative trace \(y_N\), and student distribution \(q_\theta\),
\[
L_{SFT} = - \log q_\theta(y_P \mid x),
\]
\[
L_{OR} = - \log \sigma \Bigg( \log \frac{ \text{odds}\,q_\theta (y_P|x) }{ \text{odds}\,q_\theta (y_N|x) } \Bigg),
\]
with
\[
\text{odds}\,q_\theta (y|x) = \frac{ q_\theta(y|x) }{ 1 - q_\theta(y|x) }, \qquad
L_{ORPO} = L_{SFT} + \lambda L_{OR}.
\]
The pipeline sets \(\lambda = 1\), and the paper states that this is appropriate for strongly adapting the student to prefer correct reasoning paths and “clip incorrect generation paths” [2509.25100].

GraphDPO defines a dominated neighborhood
\[
\mathcal{N}^-(i)=\{j \in V : A_{ij}=1\}
\]
and centered scores
\[
\tilde{s}_i = s_\theta(x,y_i) - \frac{1}{K}\sum_{j=1}^{K}s_\theta(x,y_j).
\]
Its local loss is
\[
\ell_i = - \tilde{s}_i + \log \sum_{k \in \{i\} \cup \mathcal{N}^-(i)} \exp(\tilde{s}_k),
\]
which is equivalent to
\[
\ell_i = -\log P(i \mid \{i\} \cup \mathcal{N}^-(i)).
\]
The full graph loss averages over nodes with non-empty dominated neighborhoods. The paper explicitly shows that when \(K=2\) with one edge, the objective reduces to standard DPO up to centering, making pairwise DPO a special case of the graph formulation [2605.08037].

MPPO replaces DPO’s reference-model-based reward reparameterization with a reward proxy defined by the geometric mean of token probabilities:
\[
r_{\text{MPPO}}(x,y)=\prod_{i=1}^{|y|} \pi_{\theta}(y_i \mid x, y_{<i})^{\frac{1}{|y|}}.
\]
Its pairwise variants then operate on the resulting scalar scores \(p_w\) and \(p_l\). In the one-positive/one-negative case,
\[
\mathcal{L}_{\text{Pair-Single}}(\pi_{\theta}) = -\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma(p_w - p_l) \right].
\]
For the top-vs-rest setting with \(N\) negatives, the strongest reported variant is Pair-MNM:
\[
\mathcal{L}_{\text{Pair-MNM}}(\pi_{\theta}) =
-\mathbb{E}_{(x,y_w,y_{l_i})\sim \mathcal{D}}
\left[
\log \sigma\left( N \cdot p_w - \sum_{i=1}^{N} p_{l_i} \right)
\right].
\]
The paper also studies point-wise and list-wise alternatives, but concludes that the pair-wise implementation performs best [2412.15244].

BOPO defines explicit preference for a minimization COP as
\[
f^*(\mathbf y,\mathbf x) = -g(\mathbf y),
\]
so that lower objective value implies higher preference. The model’s implicit preference is its average log-likelihood,
\[
f_{\bm\theta}(\bm y,\bm x) = \frac{1}{|\bm y|}\log \pi_{\bm\theta}(\bm y|\bm x),
\]
and the Bradley–Terry preference probability is scaled by an adaptive factor
\[
\beta(\mathbf x,\mathbf y_w,\mathbf y_l)=\frac{g(\mathbf y_l)}{g(\mathbf y_w)}.
\]
The resulting loss is
\[
\mathcal L_{POCO}(\pi_{\bm\theta},\mathbf x,\mathbf y_w,\mathbf y_l)
= -\log \sigma\left(
\frac{g(\mathbf y_l)}{g(\mathbf y_w)}
\left(
\frac{\log \pi_\theta(\mathbf y_w|\mathbf x)}{|\mathbf y_w|}
-
\frac{\log \pi_\theta(\mathbf y_l|\mathbf x)}{|\mathbf y_l|}
\right)
\right).
\]
The paper interprets the ratio \(g(\mathbf y_l)/g(\mathbf y_w)\) as a preference-based adaptive scaling factor or “natural curriculum” [2503.07580].

CAPO uses a DPO-style reward difference relative to a frozen SFT reference model. For target annotator \(a\),
\[
r_{\theta_a}(z;x_i,a) =
\log \pi_{\theta_a}(z\mid x_i,a)
-
\log \pi_{\mathrm{ref},a}(z\mid x_i,a),
\]
where \(\pi_{\mathrm{ref},a}\) is the frozen SFT policy for annotator \(a\). Its loss is
\[
\mathcal{L}_{\mathrm{CAPO}}
=
-\log \sigma\!\Big(
\beta\big[
r_{\theta_a}(z^+;x_i,a)
-
r_{\theta_a}(z^-;x_i,a)
\big]
\Big),
\]
with \(\beta=0.1\) in the reported setup [2605.28802].

## 4. Representative frameworks and application domains

The current literature uses cross-pair constructions in several distinct domains.

| Method | Cross-pair unit | Domain |
|---|---|---|
| ORPO-Distill | teacher positive vs student negative trace | cross-architecture LLM distillation |
| GraphDPO | node vs dominated neighborhood in a preference DAG | reasoning and program synthesis |
| MPPO | multiple response pairs from one prompt | LLM alignment with human feedback |
| BOPO | best solution vs suboptimal solutions from one instance | neural combinatorial optimization |
| CAPO | target annotator output vs other annotators’ outputs | annotator-specific explanation imitation |

ORPO-Distill is motivated by cross-architecture settings in which white-box KD is not applicable because teacher and student do not share vocabulary or logits. The reported teacher is InternLM 2.5 7B-Chat; the student models are InternLM 2.5 1.8B-Chat and TinyLlama 1.1B-Instruct; and training uses full-parameter tuning for five epochs with temperature \(\tau=0.8\) [2509.25100].

GraphDPO is evaluated on GSM8K, MATH-500, and APPS. It adds two elements absent from standard pairwise DPO: equivalence-class masking for ties and optional ground-truth anchoring. A verified solution can be inserted as a dominant node, and the anchoring term is weighted by an annealed schedule that is high early in training and then reduced later [2605.08037].

MPPO is trained from a Llama3-8B SFT model, using UltraChat-200k for SFT and UltraFeedback for preference optimization. Its benchmark suite is MT-Bench and Arena-Hard, and the paper emphasizes that MPPO does not require a reference model and only requires tuning the learning rate in its reported experiments [2412.15244].

BOPO is architecture agnostic and is demonstrated with MGL for JSP and FJSP and a POMO backbone for TSP. The paper presents it as a training paradigm rather than a model architecture, explicitly intended for COPs where objective values are cheap to compute [2503.07580].

CAPO is evaluated on two sentence-pair tasks with four annotators per item: VariErr NLI and R2 paraphrase judgment. Its base models are Qwen3-4B-Instruct-2507 and Llama-3.2-3B-Instruct. The method first trains one independent LoRA adapter per annotator by SFT and then continues from that adapter with CAPO [2605.28802].

## 5. Empirical findings

ORPO-Distill reports a consistent progression across five QA benchmarks: Single CoT fine-tuning improves over zero-shot; Diverse CoT fine-tuning improves further; ORPO with student negatives improves much more; and Mixed-policy ORPO is best overall. For TinyLlama 1.1B-Instruct, the reported averages are 37.58 for Diverse CoT FT, 41.36 for Off-policy ORPO, 38.97 for On-policy ORPO, and 43.17 for Mixed-policy ORPO. For InternLM 2.5 1.8B-Chat, the corresponding averages are 48.67, 53.93, 50.46, and 55.84. The paper also reports that \((p\text{-}CoT_{Teacher}, n\text{-}CoT_{Student})\) yields higher accuracy than \((p\text{-}CoT_{Teacher}, n\text{-}CoT_{Teacher})\), supporting the use of student-generated negatives [2509.25100].

GraphDPO reports superior performance over the compared baselines on all three reported tasks. On GSM8K, PRO scores 88.70, LiPO 89.23, GraphDPO without GT 92.42, and GraphDPO with GT 92.75. On MATH-500, the corresponding scores are 83.60, 85.60, 87.40, and 88.87. On APPS, they are 68.58, 69.32, 72.93, and 73.76. The paper further states that larger \(K\) generally helps because the graph becomes richer, and that anchoring gives the biggest boost when \(K\) is small [2605.08037].

MPPO finds that the pair-wise implementation is stronger than the point-wise and list-wise implementations. Pair-MNM achieves the best reported MT-Bench score, 6.16. On Arena-Hard, Pair-MNM achieves a 21.6 win rate, exceeding DPO at 15.9, KTO at 12.8, and ORPO at 10.7, while remaining below SimPO at 23.4. The paper states that Point-CE and Point-MSE perform poorly, and that List-MLE is unstable, especially on Arena-Hard [2412.15244].

BOPO reports that removing hybrid rollout, uniform filtering, or best-anchored pairing hurts performance. In its loss-function ablation, \(\mathcal L_{POCO}\) outperforms \(\mathcal L_{SimPO}\), \(\mathcal L_{DPO}\), and \(\mathcal L_{POCO^-}\), indicating that the objective-based adaptive scaling improves performance. The paper states that POCO is evaluated on JSP, TSP, and FJSP and consistently outperforms or matches strong baselines [2503.07580].

CAPO reports that prompting is weakest, SFT substantially improves annotator imitation, and CAPO further improves imitation-oriented metrics and judge-based attribution while keeping decision accuracy competitive. On VariErr with Qwen3, SFT obtains label accuracy 0.638, ImiScore 0.859, and Judge Acc 0.300, while CAPO obtains label accuracy 0.627, ImiScore 0.888, and Judge Acc 0.328. On R2 with Llama3.2, SFT obtains score accuracy 0.450, MAE 1.535, ImiScore 0.972, and Judge Acc 0.512, while CAPO obtains score accuracy 0.460, MAE 1.470, ImiScore 0.995, and Judge Acc 0.530. Human validation ranks the systems as **CAPO > SFT > prompting**, with overall agreement 82.8% and Cohen’s \(\kappa = 0.650\) [2605.28802].

## 6. Methodological distinctions, limitations, and interpretive issues

A recurrent misconception is that cross-pair preference optimization always means “more pairwise comparisons.” The papers collectively show that this is not the only interpretation. GraphDPO argues that flattening a multi-rollout prompt into independent pairs breaks transitivity, creates redundant supervision, and can create conflicting or noisy gradients when ties are present. Its response is to aggregate supervision over graph neighborhoods instead of enumerating independent pairs [2605.08037]. By contrast, MPPO reports that, among its own point-wise, pair-wise, and list-wise implementations, pair-wise is strongest. This suggests that the empirical value of richer structure depends on how preference information is represented and on the task regime, rather than on a universal ordering between pairwise and non-pairwise objectives [2412.15244].

A second misconception is that the rejected item must be objectively wrong. ORPO-Distill does use incorrect student traces as negatives, but CAPO explicitly does not: its rejected outputs are other valid human annotations, and the optimization target is target-specific imitation rather than generic quality ranking [2509.25100] [2605.28802].

A third distinction concerns the scope of “cross.” BOPO is explicit that its construction is not general all-pairs ranking or cross-instance comparison; it is anchor-based comparison within a single instance’s solution pool [2503.07580]. ORPO-Distill, by contrast, is genuinely cross-policy and cross-architecture: preferred and rejected traces come from different models with different roles [2509.25100].

The methods also identify different failure modes. ORPO-Distill reports that pure on-policy updates degrade performance relative to off-policy and mixed-policy because the negative trace distribution narrows too much [2509.25100]. GraphDPO addresses tie-heavy data by excluding intra-class edges so that “intra-class comparisons contribute zero loss,” thereby avoiding spurious gradients [2605.08037]. CAPO notes that its effectiveness depends on a small, repeated-annotator regime; larger crowdsourcing settings introduce harder negative-sampling problems, potential dilution from easy negatives, and cold-start issues for annotators with limited history [2605.28802].

Ethical and interpretive cautions also vary by domain. CAPO stresses that high imitation scores indicate recognizability rather than “better” behavior, and that annotator IDs should remain anonymized and not be used to infer sensitive attributes [2605.28802]. In combinatorial optimization, BOPO’s objective-guided scaling assumes that objective values are cheaply available and reliable as preference signals, which is appropriate for the reported COP settings but not automatically transferable beyond them [2503.07580].

Taken together, these works indicate that cross-pair preference optimization is better understood as a design space than as a single algorithm. The central choices are the source of the contrasted samples, the structural unit of comparison, the handling of ties or near-equivalences, and the relationship between the preference signal and the task-specific error distribution.

Source: https://www.emergentmind.com/topics/cross-pair-preference-optimization