Group Direct Preference Optimization

• Group Direct Preference Optimization (GDPO) is a family of methods that extend direct preference optimization by evaluating structured groups rather than individual pairs.
• GDPO methods leverage specialized aggregation techniques to compare positive and negative subsets, yielding efficiency gains and improved model alignment.
• The approach addresses varied definitions of 'group' across applications such as text-video retrieval, diffusion models, and protein modeling, while balancing richer supervision with training costs.

Group Direct Preference Optimization (GDPO) denotes a family of groupwise preference-optimization constructions derived from, or adjacent to, Direct Preference Optimization (DPO), but it is not a single universally standardized method name. In the direct-preference sense, GDPO-style methods replace the isolated chosen–rejected pair of ordinary DPO with structured groups of candidates, regions, instances, or belief-conditioned responses, and optimize relative preference information over those groups rather than over a single pair. At the same time, the acronym is polysemous: some papers use nearby names such as “GroupDPO,” “DGPO,” “dual-group DPO,” or “Region-Grouped DPO,” while other papers use GDPO for methods that are not DPO at all but GRPO-style multi-reward policy optimization (Lee et al., 20 Sep 2025, Leng et al., 17 Apr 2026, Luo et al., 9 Oct 2025, Yi et al., 16 Mar 2026, Yao et al., 2024, Liu et al., 8 Jan 2026).

1. Terminological scope and disambiguation

In the direct-preference literature, “group” does not have a single fixed meaning. It can refer to multiple responses for one prompt, multiple captions for one video, positive and negative subsets inside a rollout group, region-level subsets inside an image, or even pluralistic preference distributions inside a human group. The result is a technically related but terminologically fragmented landscape.

Paper	Official term	Meaning of “group”
"Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimization" (Lee et al., 20 Sep 2025)	DG-DPO	Within-video and cross-video caption groups
"GroupDPO: Memory efficient Group-wise Direct Preference Optimization" (Leng et al., 17 Apr 2026)	GroupDPO	Multiple positive and negative responses per prompt
"Reinforcing Diffusion Models by Direct Group Preference Optimization" (Luo et al., 9 Oct 2025)	DGPO	Positive and negative subsets inside a sample group
"GDPO-SR: Group Direct Preference Optimization for One-Step Generative Image Super-Resolution" (Yi et al., 16 Mar 2026)	GDPO	Online sample groups for one-step ISR
"No Preference Left Behind: Group Distributional Preference Optimization" (Yao et al., 2024)	GDPO	Belief distributions within a human group
"GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization" (Liu et al., 8 Jan 2026)	GDPO	Reward-dimension groups, not DPO

A recurring source of confusion is that several papers use GDPO for methods that are not direct preference optimization. "GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization" (Liu et al., 8 Jan 2026), "GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization" (Liu et al., 15 Jun 2026), and "GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimization" (Jin et al., 26 Mar 2026) all use GDPO in a GRPO-style RL sense rather than in the DPO sense. By contrast, the direct-preference branch includes GroupDPO, DG-DPO, DGPO for diffusion, GDPO-SR, Region-Grouped DPO, and Group Distributional Preference Optimization.

2. Pairwise DPO as the baseline and the recurrent GDPO design pattern

Most direct-preference variants start from the standard DPO objective recalled in the retrieval paper: $\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$ with

$$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$

Here $x$ is the input, $y_w$ the preferred output, $y_l$ the dispreferred output, $\pi_\theta$ the trainable policy, $\pi_\text{ref}$ the frozen reference model, and $\beta$ the temperature or regularization hyperparameter (Lee et al., 20 Sep 2025).

GDPO-style methods modify this baseline along three recurring axes. First, they replace the single $(y_w,y_l)$ comparison with a structured set of candidates. Second, they redefine what makes one candidate “preferred”: some methods use retrieval relevance, some use evaluator scores, some use directional consistency, and some use belief-conditioned labels. Third, they change aggregation. Group-wise training may average over all positive–negative pairs, form positive and negative subsets, use log-sum-exp group scores, or derive per-sample coefficients from detached group statistics. There is therefore no single universal GDPO loss. GroupDPO states this abstractly as

$\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$0

where $\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$1 and $\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$2 are prompt-specific positive and negative sets and $\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$3 may instantiate Margin, MPO, Softmax, or All-Pairs objectives (Leng et al., 17 Apr 2026).

Most direct-preference variants retain a frozen reference model and compare policy scores against that reference, as in DG-DPO, GroupDPO, DGPO for diffusion, GPO for diffusion, GDPO-SR, Region-Grouped DPO, and g-DPO (Lee et al., 20 Sep 2025, Leng et al., 17 Apr 2026, Luo et al., 9 Oct 2025, Chen et al., 16 May 2025, Yi et al., 16 Mar 2026, Shuai et al., 16 Mar 2026, Ferragu et al., 22 Oct 2025). A notable exception is "DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization" (Deng et al., 11 May 2026), which uses length-normalized policy log-likelihoods, a learned Beta posterior for directional consistency, and group log-sum-exp aggregation without a reference-model ratio.

3. Groupwise direct preference optimization in language, retrieval, and protein modeling

In text-video retrieval, "Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimization" (Lee et al., 20 Sep 2025) introduces DG-DPO as a retrieval-oriented extension of DPO for training a video caption generator. Its defining move is to combine local / single-group preferences among captions generated for the same video with global / cross-group preferences among captions belonging to different videos. Preferences are induced by a retrieval model rather than by BLEU or human annotation, using a masked-video relevance score

$\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$4

DG-DPO includes both $\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$5 and $\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$6 comparisons, with a weighting factor $\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$7 balancing local and cross-group pairs. The paper reports average R@1 gains of $\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$8 for $\mathcal{L}_\text{DPO}(\pi_\theta ; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}_{\text{DPO}} \Bigl[ \log \sigma\bigl( \hat r_\theta(x, y_w)-\hat r_\theta(x, y_l) \bigr) \Bigr],$9, $$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$0 for $$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$1, and $$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$2 for $$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$3 across DiDeMo, ActivityNet, and MSRVTT, supporting the value of global retrieval-rank preferences. It also reports that DG-DPO reuses already computed log probabilities for cross-video pairings, so the additional comparisons do not require proportional extra generation cost.

In large-language-model alignment, "GroupDPO: Memory efficient Group-wise Direct Preference Optimization" (Leng et al., 17 Apr 2026) studies the case where preference data naturally contains multiple responses per prompt. Rather than collapsing each prompt to one chosen–rejected pair, it partitions responses into a positive set $$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$4 and a negative set $$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$5, and evaluates group losses such as Margin, MPO, Softmax, and All-Pairs. Its main systems contribution is a first-order-exact surrogate: it computes detached scores $$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$6, forms per-sample coefficients

$$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$7

and then optimizes

$$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$8

which preserves gradients while decoupling samples during backpropagation. Across offline and online alignment settings, the paper reports that leveraging multiple responses consistently outperforms single-pair training, and it identifies a positive-response NLL term as critical for both performance gains and training stability.

In protein language modeling, "g-DPO: Scalable Preference Optimization for Protein LLMs" (Ferragu et al., 22 Oct 2025) adapts group-DPO to experimentally labeled mutant datasets. The method first clusters sequences in mutation space by a union-mask criterion, pruning redundant and distant comparisons, and then amortizes likelihood computations by scoring all members of a sampled group under a shared union mask. DPO itself remains pairwise inside the group, but pair construction and sequence scoring are group-based. Across three protein engineering tasks, g-DPO maintains in-silico and in-vitro performance that is statistically indistinguishable from standard DPO while converging $$\hat r_\theta(x, y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_\text{ref}(y\mid x)}.$$9 to $x$0 times faster.

Reasoning alignment supplies a different groupwise semantics. "DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization" (Deng et al., 11 May 2026) organizes forward and reverse question-answer instances into preferred and dispreferred groups: $x$1 and symmetrically for $x$2. It learns a Beta posterior $x$3 for directional consistency, builds pre-activation scores $x$4 and $x$5, and aggregates them with temperature-scaled log-sum-exp into $x$6 and $x$7. The resulting margin objective yields average accuracy improvements of up to $x$8, and the paper explicitly states that if each preferred and dispreferred group contains one response, the formulation collapses to a pairwise contrastive objective.

4. Diffusion and visual-generation variants

Diffusion-model post-training has produced several of the clearest explicit group-preference objectives. "Reinforcing Diffusion Models by Direct Group Preference Optimization" (Luo et al., 9 Oct 2025) introduces DGPO for diffusion models as a direct, group-aware alternative to GRPO-style policy gradients. For each condition $x$9, a group $y_w$0 is sampled from an online model, rewards are normalized into

$y_w$1

and the group is partitioned into

$y_w$2

With weights $y_w$3, the positive and negative total weights match, which cancels the intractable partition term and yields a trainable logistic objective over weighted positive and negative sets. The method permits deterministic ODE rollouts and is reported to train around $y_w$4 times faster than Flow-GRPO while improving in-domain and out-of-domain reward metrics.

A closely related self-improvement line appears in "Towards Self-Improvement of Diffusion Models via Group Preference Optimization" (Chen et al., 16 May 2025). That paper first defines Group DPO as the sum over all ordered pairs inside a ranked group, then derives an efficient linear form with coefficients $y_w$5, and finally replaces rank coefficients with standardized evaluator scores

$y_w$6

The resulting GPO objective

$y_w$7

is used for self-improvement without explicit external preference data. On Stable Diffusion 3.5 Medium, the paper reports improvements of about 20 percentage points in accurate counting and text rendering when combined with evaluators such as YOLO and OCR.

"GDPO-SR: Group Direct Preference Optimization for One-Step Generative Image Super-Resolution" (Yi et al., 16 Mar 2026) adapts the idea to one-step ISR. It first introduces a noise-aware one-step diffusion model with an unequal-timestep strategy, then computes an attribute-aware reward

$y_w$8

standardizes it into a group-relative advantage

$y_w$9

and inserts those coefficients into a diffusion-DPO-style sigmoid objective. On RealSR, the paper reports improvements such as PSNR $y_l$0, MANIQA $y_l$1, and MUSIQ $y_l$2, while noting that training overhead grows because multiple outputs must be generated per input during training.

Spatial localization yields another branch of GDPO-style design. "GlyphPrinter: Region-Grouped Direct Preference Optimization for Glyph-Accurate Visual Text Rendering" (Shuai et al., 16 Mar 2026) argues that whole-image DPO is insufficient because glyph errors are localized. It therefore constructs region masks $y_l$3, $y_l$4, and $y_l$5 and defines both inter-sample and intra-sample terms inside a region-grouped objective

$y_l$6

The Stage-2 setting uses group size $y_l$7, $y_l$8, and $y_l$9. In ablations, full GlyphPrinter outperforms Stage 1, SFT, Mask-SFT, inter-only, and intra-only variants on multilingual and complex benchmarks, supporting the claim that region-level and group-level preference signals are complementary.

5. Distributional and group-conditioned preference alignment

Not all direct-preference uses of “group” concern multiple generated samples. In "No Preference Left Behind: Group Distributional Preference Optimization" (Yao et al., 2024), GDPO refers to alignment with the distribution of preferences within a group of people. The model factorizes response generation through discrete beliefs: $\pi_\theta$0 and optimizes a two-part objective

$\pi_\theta$1

where the first term calibrates the predicted belief distribution to the target $\pi_\theta$2 and the second is a belief-conditioned DPO-style preference loss. On controllable opinion generation, the paper reports that DPO moves belief distributions away from the target while GDPO reduces JSD during training; for example, on the US split with GPT-2 Large, JSD changes from $\pi_\theta$3 under DPO to $\pi_\theta$4 under GDPO, while BPC changes from $\pi_\theta$5 to $\pi_\theta$6.

A related but distinct adjacent framework is "Group Preference Optimization: Few-Shot Alignment of LLMs" (Zhao et al., 2023). That paper’s GPO is not direct preference optimization. It trains an independent transformer module to predict group-specific preference scores from few-shot support examples, using the conditional predictive objective

$\pi_\theta$7

Its function is modular rather than end-to-end policy optimization: the learned module can be used for reranking, Best-of-$\pi_\theta$8, or as a reward for PPO. This distinction is important because the title similarity between GPO, GDPO, and GroupDPO can obscure the fact that only some of these methods are DPO-family objectives.

6. Scalability, stabilization, and unresolved methodological questions

Several papers around DPO, while not themselves GDPO methods, directly inform how GDPO-style systems can be made data-efficient or stable. "Active Learning for Direct Preference Optimization" (Kveton et al., 3 Mar 2025) linearizes the DPO policy at the last layer, defines pairwise difference features $\pi_\theta$9, and uses the DPO Hessian

$\pi_\text{ref}$0

as the Fisher-information object for D-optimal design. Its greedy acquisition score

$\pi_\text{ref}$1

selects preference pairs that maximally increase information about DPO logits, and the paper proves that maximum logit-estimation error diminishes with more feedback. For GDPO-style methods, this establishes that preference-set construction can itself be optimized rather than treated as fixed.

Optimization stability is addressed from another angle in "Gradient-Gated DPO: Stabilizing Preference Optimization in LLMs" (Mouiche, 4 May 2026). Gate-DPO introduces a per-example gate

$\pi_\text{ref}$2

that attenuates rejected gradients in low-probability valleys while preserving standard optimization behavior elsewhere. The gated logit

$\pi_\text{ref}$3

improves chosen-response likelihood and reduces squeezing. This suggests that future GDPO variants may benefit from combining groupwise supervision with explicit control of rejected-gradient dynamics, especially when group objectives still decompose into positive and negative subsets.

The relation between large-group RL and pairwise preference optimization is sharpened by "It Takes Two: Your GRPO Is Secretly DPO" (Wu et al., 1 Oct 2025). In the binary-reward RLVR setting, the paper shows that GRPO can be rewritten as a contrastive positive-minus-negative objective, and that the two-rollout case, 2-GRPO, performs on par with 16-GRPO while using only $\pi_\text{ref}$4 of the rollouts and reducing training time by over $\pi_\text{ref}$5. This complicates a common intuition around GDPO-like methods: larger groups can enrich supervision, but they are not automatically necessary for stable learning, and the marginal benefit of more group structure depends on what additional information the group actually carries.

Taken together, the literature suggests that GDPO is best understood not as a single loss but as a design space. The main unresolved questions are how to define groups, how to convert group structure into preference weights or margins, how much reference-model structure to retain, and how to balance richer supervision against memory, sampling, and optimization costs. The surveyed papers answer these questions differently, which is precisely why the term remains productive and unsettled rather than canonically fixed.