Group Contrastive Preference Optimization

• Group Contrastive Preference Optimization (GCPO) is a set-level approach that contrasts groups of candidate responses to achieve reward-based alignment.
• It leverages deviation-based weighting and group softmax objectives to prioritize high-quality responses while mitigating biases inherent in pairwise methods.
• Active variants like AMPO employ informed subset selection to handle large candidate pools, balancing computational cost with improved alignment results.

Searching arXiv for papers on group contrastive preference optimization and closely related formulations. Group Contrastive Preference Optimization (GCPO) denotes a family of preference-based post-training methods that replace pairwise chosen-versus-rejected supervision with groupwise or set-level contrasts over multiple candidate responses associated with the same prompt. In this usage, GCPO is not a single universally fixed algorithm. Rather, it refers to a common design pattern: collect or generate several responses for a query, partition them into preferred and non-preferred subsets, and optimize a contrastive objective over the entire set, often with reward-derived weighting or active selection. This family is exemplified by "Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts" (Gupta et al., 2024), extended to active self-play selection in "AMPO: Active Multi-Preference Optimization for Self-play Preference Selection" (Gupta et al., 25 Feb 2025), and instantiated in task-specific reinforcement-learning variants such as "GeometryZero: Improving Geometry Solving for LLM with Group Contrastive Policy Optimization" (Wang et al., 8 Jun 2025). The acronym GCPO is also used by later works for distinct methods, including Group Contrastive Policy Optimization with gold answers for reasoning (Wu et al., 9 Oct 2025), Group Causal Policy Optimization (Gu et al., 7 Aug 2025), and Guidance Contrastive Policy Optimization for token credit assignment (Li et al., 28 May 2026), so the term is context-dependent rather than uniquely standardized.

1. Emergence from pairwise preference optimization

Direct Preference Optimization (DPO) is the immediate antecedent for GCPO-style methods. In the pairwise setting, training data for a prompt $x$ consists of one preferred response $y^+$ and one dispreferred response $y^-$, and optimization is driven by a Bradley–Terry style contrast over their relative policy scores. The formulation synthesized for the groupwise literature defines

$$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$

with a DPO-style objective

$$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$

where $\sigma$ is the logistic sigmoid and $\beta$ controls preference strength (Gupta et al., 2024).

GCPO-style methods generalize this pairwise construction to the practical regime in which on-policy generation yields multiple responses per prompt, each of which may be scored by a reward model or by a strong evaluator. In "Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts" (Gupta et al., 2024), the central change is from a single comparison $y^+ \succ y^-$ to a set-level comparison between a preferred subset and the full candidate set. The paper explicitly frames this as optimizing over entire sets of responses by extending the Bradley–Terry model to groupwise comparisons between chosen and rejected sets (Gupta et al., 2024).

This shift is also reflected in "Sequence-level LLM Training with Contrastive Preference Optimization" (Feng et al., 23 Feb 2025), which—although not presented as GCPO—already formulates a $K$-way contrastive preference objective over a group of candidate continuations. There, one ground-truth sequence is contrasted against multiple synthetic negatives through a softmax over the whole candidate set, providing a sequence-level analogue of pairwise DPO (Feng et al., 23 Feb 2025). A plausible implication is that GCPO can be viewed as the natural listwise extension of the DPO family across both alignment and sequence-level language modeling.

2. Canonical set-level formulation

The clearest formalization of GCPO-style alignment in the provided literature is the groupwise objective synthesized from (Gupta et al., 2024). For a prompt $x$, let there be $y^+$0 candidate responses $y^+$1 with scalar rewards $y^+$2. The method first computes the per-query mean reward

$y^+$3

then the reward deviation

$y^+$4

Responses are partitioned into

$y^+$5

The model score relative to a reference policy is

$y^+$6

Using deviation-based logit adjustment,

$y^+$7

the set-level loss becomes

$y^+$8

This objective maximizes the aggregate softmax mass assigned to the preferred subset relative to all responses for that prompt (Gupta et al., 2024).

The same paper interprets this as a groupwise Bradley–Terry or set-level contrastive model. A useful derived quantity is a group score

$y^+$9

where $y^-$0 is a deviation-derived weight. Under this view, GCPO models the event that the chosen group $y^-$1 “wins” against $y^-$2 by aggregating scores over sets rather than over individual pairs (Gupta et al., 2024).

AMPO retains the same essential structure but adopts a reference-free form in its main equations. For a selected subset $y^-$3, it uses

$y^-$4

with

$y^-$5

where $y^-$6 is the reward and $y^-$7 is the selected-subset mean reward (Gupta et al., 25 Feb 2025). When $y^-$8, this reduces to a logistic contrastive loss over one preferred and one rejected response, making explicit that the groupwise objective subsumes pairwise preference optimization (Gupta et al., 25 Feb 2025).

3. Deviation-based weighting, curriculum effects, and bias reduction

A defining characteristic of the GCPO formulation in (Gupta et al., 2024) is deviation-based weighting. The synthesized presentation gives two weighting families. In exponential form,

$y^-$9

while a power-form alternative is

$$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$0

The case $$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$1 corresponds to unweighted group contrast, while $$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$2 and $$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$3 increasingly emphasize high-deviation responses (Gupta et al., 2024).

The paper interprets this as inducing a self-paced curriculum. Responses far above the mean reward receive larger positive weights, responses far below the mean receive larger negative weights, and near-mean responses contribute smaller gradients. This suggests that GCPO prioritizes high-signal outliers first and uses less decisive examples as weaker supervision (Gupta et al., 2024). The gradient analysis reported in the synthesized details reinforces this interpretation: minimizing the loss increases probabilities of responses in $$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$4 and decreases probabilities of responses outside $$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$5, with larger weights amplifying the update magnitude (Gupta et al., 2024).

The same work provides the main theoretical argument for why groupwise preference supervision can be preferable to pairwise supervision. It defines an attribute $$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$6, the expected attribute over acceptable responses $$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$7, and the model’s expected attribute after training on $$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$8 positive and $$s_\theta(y\mid x)=\log\frac{P_\theta(y\mid x)}{P_{\mathrm{ref}}(y\mid x)},$$9 negative samples per query, $$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$0. The alignment bias is

$$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$1

Under finite variance, independent sampling, sufficient model capacity, and a uniform variance bound, the paper states

$$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$2

with corollary

$$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$3

The stated interpretation is that using multiple responses per prompt reduces the risk that training overfits idiosyncratic properties of a single chosen and rejected example, such as spurious length or style biases (Gupta et al., 2024).

A related but broader theoretical perspective appears in "Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator" (Chen et al., 6 Feb 2025). That work treats preference optimization as negative log-likelihood estimation of an energy-based model

$$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$4

and shows that pairwise DPO corresponds to a special case of a groupwise RNCE-style estimator with one negative. The multi-negative loss

$$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$5

is thus a listwise cross-entropy over a candidate group, with the preferred element $$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$6 required to win against all other group members (Chen et al., 6 Feb 2025). This suggests a probabilistic interpretation of GCPO as Monte Carlo estimation of a normalized energy model rather than as a purely heuristic listwise ranking rule.

4. Active and adaptive variants

"AMPO: Active Multi-Preference Optimization for Self-play Preference Selection" (Gupta et al., 25 Feb 2025) extends GCPO from static grouped supervision to an on-policy alignment framework with active subset selection. For each query $$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$7, the current policy generates $$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$8 responses, a reward model scores them, semantic embeddings are computed, and a subset $$\mathcal{L}_{\mathrm{DPO}}(\theta)\propto -\log \sigma\left(\beta\left[s_\theta(y^+\mid x)-s_\theta(y^-\mid x)\right]\right),$$9 of size $\sigma$0 is selected to maximize informativeness before groupwise optimization (Gupta et al., 25 Feb 2025).

Three subset-selection strategies are specified. AMPO-BottomK chooses the $\sigma$1 lowest-reward negatives. AMPO-Coreset clusters the embeddings via $\sigma$2-means and selects the lowest-reward response in each cluster to cover distinct semantic regions. AMPO-Opt-Select defines a weighted coverage cost

$\sigma$3

with distances $\sigma$4 and reward-based weights

$\sigma$5

then chooses the negative set that minimizes this cost, approximately via weighted $\sigma$6-medoids local search (Gupta et al., 25 Feb 2025).

The theoretical claims in AMPO are specific. Under an $\sigma$7-Lipschitz constraint in embedding space, one free positive, and finite candidate support, the paper states that the size-$\sigma$8 set minimizing the weighted coverage cost also maximizes expected reward among Lipschitz-compliant policies that penalize $\sigma$9 negatives and keep one positive unconstrained (Gupta et al., 25 Feb 2025). It further states that 1-swap local search yields a solution within a factor of 5 of the optimal weighted $\beta$0-medoids cost, and gives a distribution-dependent coreset guarantee for the clustering-based variant under bounded cluster diameter assumptions (Gupta et al., 25 Feb 2025).

Empirically, AMPO evaluates on AlpacaEval 2, Arena-Hard v0.1, and MT-Bench using Llama‑3‑8B and reports that AMPO-Coreset achieves 52.4 AlpacaEval2 LC, 52.1 AlpacaEval2 WR, 39.4 Arena-Hard WR, and 8.12 MT-Bench GPT‑4 score, outperforming a best-vs-worst SimPO baseline at 47.6 LC and 44.7 WR (Gupta et al., 25 Feb 2025). A plausible implication is that active selection addresses a key systems-level obstacle for GCPO: if candidate pools become large, training on all responses is unnecessary so long as the retained subset spans both reward extremes and diverse failure modes.

A distinct adaptive direction appears in "GeometryZero: Improving Geometry Solving for LLM with Group Contrastive Policy Optimization" (Wang et al., 8 Jun 2025). There, GCPO is not a setwise DPO generalization but a reinforcement-learning framework over grouped rollouts. For each question, the policy generates a main rollout group $\beta$1, a group $\beta$2 where auxiliary construction is required, and a group $\beta$3 where it is prohibited. The method defines a masking rule over the auxiliary-construction reward: $\beta$4 The final reward is

$\beta$5

with a length reward

$\beta$6

This GCPO variant is group-contrastive in the sense that it contrasts groups of rollouts with and without tool use to decide whether auxiliary construction should be rewarded or penalized (Wang et al., 8 Jun 2025). It is therefore conceptually related to groupwise contrast, but methodologically distinct from set-level DPO generalizations.

5. Empirical performance and benchmark evidence

The strongest direct evidence for GCPO-style set-level alignment comes from (Gupta et al., 2024). On UltraFeedback, each instruction has 4 responses scored on a 0–10 scale. The paper fine-tunes Mistral‑7B‑Instruct‑v0.1 by splitting responses above the per-instruction mean reward into the chosen set and the rest into the rejected set (Gupta et al., 2024). On AlpacaEval2, the reported table excerpt includes the following results for Mistral‑7B‑Instruct:

Method	WR	LC-WR
DPO x (k-1)	8.41	13.36
Swepo $\beta$7	11.14	16.06
Swepo $\beta$8	11.70	16.30
Swepo $\beta$9	11.94	16.64

The accompanying text states that these correspond to approximately 24% LC-WR improvement and 42% WR improvement over the best preference baseline DPOx$y^+ \succ y^-$0 on AlpacaEval2 (Gupta et al., 2024). Additional ablations state that using all 4 responses rather than only 2 improves performance, that multi-positive versus multi-negative training is better than 1-vs-all, and that deviation-based weighting yields up to approximately 3.6% LC-WR and 7.2% WR improvements over unweighted group contrast (Gupta et al., 2024).

AMPO reports stronger absolute results under a different setup. With Llama‑3‑8B, AMPO-Coreset reaches 52.4 AlpacaEval2 LC and 52.1 AlpacaEval2 WR, compared to 47.6 and 44.7 for a best-vs-worst SimPO baseline (Gupta et al., 25 Feb 2025). All AMPO variants surpass the base model, and the diversity-aware selection methods outperform pure BottomK selection, especially on Arena-Hard (Gupta et al., 25 Feb 2025). This suggests that once candidate generation becomes on-policy and abundant, subset selection quality becomes a major determinant of GCPO effectiveness.

GeometryZero reports smaller but consistent gains in geometry reasoning. Averaged over four benchmarks, GCPO-based GeometryZero improves over GRPO by 4.23 points at 1.5B, 1.50 points at 3B, and 1.84 points at 7B; it also surpasses ToRL, whose unconditional auxiliary reward is reported as yielding marginal or negative change depending on model size (Wang et al., 8 Jun 2025). Ablations show that unconditional auxiliary reward alone can hurt performance, while the combination of group contrastive masking and length reward performs best (Wang et al., 8 Jun 2025).

Evidence from adjacent listwise or sequence-level formulations supports the same general pattern. CPO for sequence-level training contrasts one ground-truth continuation against multiple negatives via a group softmax and reports win-rate improvements over next-token prediction on instruction-following and open-ended generation (Feng et al., 23 Feb 2025). MC-PO and OnMC-PO show that multi-negative contrastive preference learning with contrastive-divergence sampling can outperform standard pairwise baselines on alignment benchmarks (Chen et al., 6 Feb 2025). A plausible implication is that the performance advantage of GCPO is not tied to one objective variant, but to the more general principle of leveraging multiple informative alternatives per prompt rather than collapsing them into one pair.

6. Terminological divergence and relation to adjacent methods

The name GCPO is not unique. In the set-level alignment literature, the phrase most naturally refers to groupwise preference optimization in the lineage of MPO/Swepo/AMPO (Gupta et al., 2024, Gupta et al., 25 Feb 2025). But later works use the same acronym for substantially different procedures.

"GCPO: When Contrast Fails, Go Gold" (Wu et al., 9 Oct 2025) defines Group Contrastive Policy Optimization as a reasoning-oriented RL algorithm that augments GRPO with gold standard answers. When all model rollouts are incorrect, it replaces one rollout with an external golden answer, assigns that trajectory reward 1, recomputes group-normalized advantages, switches from token-level to sequence-level importance sampling, and removes KL regularization (Wu et al., 9 Oct 2025). Here, “group contrastive” refers to contrast within a rollout group under verifiable rewards, not to chosen-versus-rejected set-level preference optimization.

"Group Causal Policy Optimization for Post-Training LLMs" (Gu et al., 7 Aug 2025) uses GCPO to mean Group Causal Policy Optimization. It starts from GRPO, introduces a structural causal model over candidate responses and a final integrated output, defines a causal projection $y^+ \succ y^-$1, scales advantages by a causal similarity factor $y^+ \succ y^-$2, and adds a KL term toward a causally projected reference distribution $y^+ \succ y^-$3 (Gu et al., 7 Aug 2025). The shared acronym is accidental rather than conceptual.

"Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization" (Li et al., 28 May 2026) uses GCPO for Guidance Contrastive Policy Optimization. There, the method addresses uniform token credit assignment in GRPO/DAPO-style RL by contrasting the same sequence under a positive prompt $y^+ \succ y^-$4 and a negative prompt $y^+ \succ y^-$5. It defines a per-token KL-based guidance score

$y^+ \succ y^-$6

normalizes it within the sequence, and scales the sample-level advantage by this token weight (Li et al., 28 May 2026). Again, the shared acronym does not denote a set-level chosen-versus-rejected objective.

Because of these divergent usages, precise disambiguation is necessary. In alignment contexts centered on DPO generalization, GCPO denotes groupwise contrastive preference optimization over multiple responses per prompt (Gupta et al., 2024, Gupta et al., 25 Feb 2025). In reasoning-RL contexts, the same acronym may instead denote group-relative policy optimization variants involving gold answers (Wu et al., 9 Oct 2025), causal projections (Gu et al., 7 Aug 2025), token credit assignment (Li et al., 28 May 2026), or masked tool rewards (Wang et al., 8 Jun 2025). A common misconception is therefore to assume that every GCPO paper addresses the same optimization problem; the shared acronym obscures substantial methodological differences.

7. Limitations, design choices, and open directions

Several limitations recur across the GCPO-style literature. The first is dependence on scalar rewards. The set-level formulations in (Gupta et al., 2024) and (Gupta et al., 25 Feb 2025) assume each response has a scalar reward, typically from a reward model or a strong evaluator. If only pairwise preferences are available, a separate procedure is needed to infer or approximate scalar scores before mean-threshold grouping or deviation weighting can be applied (Gupta et al., 2024). This suggests that GCPO is especially natural in pipelines already built around reward models or evaluator scores.

The second is sensitivity to group construction. The baseline grouping rule in (Gupta et al., 2024) is mean-threshold partitioning, but the same synthesized discussion notes that alternative GCPO variants could use top-$y^+ \succ y^-$7 versus bottom-$y^+ \succ y^-$8, reward thresholds, or percentile cutoffs. The ablations cited there show that multiple positives versus multiple negatives outperform 1-vs-all, implying that poor grouping can distort the learning signal by misclassifying high-quality responses as negatives (Gupta et al., 2024).

The third is computational cost. GCPO requires processing all responses in a set together, because the loss depends on the group softmax denominator. (Gupta et al., 2024) explicitly notes an approximate factor-$y^+ \succ y^-$9 increase in compute and memory per query relative to single-response training. AMPO is largely motivated by the observation that self-play can produce so many candidates that including them all in the objective becomes infeasible, motivating active subset selection (Gupta et al., 25 Feb 2025). GeometryZero similarly incurs overhead because it requires three rollout groups per question rather than one (Wang et al., 8 Jun 2025).

The fourth is robustness to correlated or noisy candidates. The bias-reduction theorem in (Gupta et al., 2024) assumes independent sampling, yet on-policy generations from a LLM can be highly correlated. The synthesized remarks explicitly identify the effect of correlation on the theoretical guarantees as an open question (Gupta et al., 2024). Relatedly, (Chen et al., 6 Feb 2025) argues that groupwise training benefits from hard-negative sampling rather than arbitrary negatives, because poor approximation of the normalization term underweights informative dispreferred modes.

Several research directions emerge from these limitations. One is richer group structure: multi-positive and multi-negative GCPO is empirically better than 1-vs-all (Gupta et al., 2024), but the literature has not fully characterized optimal grouping schemes or weighting schedules. Another is tighter integration with active sampling or online generation, as in AMPO (Gupta et al., 25 Feb 2025) and OnMC-PO (Chen et al., 6 Feb 2025). A third is extending scalar-attribute bias guarantees to richer distributional properties, since the available theory focuses on scalar $K$0 rather than multidimensional alignment behavior (Gupta et al., 2024). Finally, the terminological proliferation around GCPO itself suggests that future work may benefit from sharper naming distinctions between set-level preference optimization, group-relative RL with external positives, and prompt-contrastive token credit assignment.

In the narrow technical sense established by the multi-preference alignment literature, GCPO is best understood as the set-level generalization of DPO: a family of objectives that operate over groups of responses, maximize preferred-group probability mass within a softmax over the whole set, and often exploit reward-derived weighting or informed subset selection to improve alignment efficiency and reduce bias (Gupta et al., 2024, Gupta et al., 25 Feb 2025).