Direct Multi-Preference Optimization (DMPO)

• DMPO is a framework that generalizes Direct Preference Optimization by comparing multiple responses simultaneously to capture richer, multi-dimensional preference signals.
• It employs methodologies such as multi-negative, multi-turn, listwise, and fine-grained aspect comparisons to optimize policy log-probabilities directly and improve recommendation performance.
• Empirical evaluations report significant AUC gains in diverse recommendation datasets, demonstrating DMPO’s robustness in handling multi-objective and trajectory-level challenges.

Direct Multi-Preference Optimization (DMPO) denotes a family of direct alignment formulations that extend Direct Preference Optimization (DPO) beyond a single static winner–loser comparison. In the literature, the acronym has been used in several closely related senses: as a multi-negative recommendation objective that directly contrasts one preferred completion against multiple dispreferred ones (Bai et al., 2024); as a multi-turn trajectory-level preference loss for language agents under a state-action occupancy-measure constraint (Shi et al., 2024); and, more broadly, as a conceptual umbrella for DPO-style methods that incorporate multiple preference dimensions, listwise supervision, groupwise multi-sample comparisons, or fine-grained aspect-specific preference signals (Sun et al., 24 Jun 2025, Wang et al., 2024, Zhang et al., 11 Aug 2025). Across these usages, the common principle is to preserve direct optimization of policy log-probabilities while expanding the preference signal from a single pair to richer multi-response or multi-objective structures.

1. Scope and nomenclature

The term “DMPO” is not used uniformly. In “Finetuning LLM for Personalized Ranking,” DMPO is the name of a specific objective for recommendation that “simultaneously maximiz[es] the probability of positive samples and minimiz[es] the probability of multiple negative samples” (Bai et al., 2024). In “Direct Multi-Turn Preference Optimization for Language Agents,” the same acronym refers to “Direct Multi-Turn Preference Optimization,” a trajectory-level extension of DPO for agentic interaction (Shi et al., 2024). Later papers also use “Direct Multi-Preference Optimization” as a conceptual description for methods that move from one static binary preference to multiple dimensions, multiple candidates, or multiple samples (Sun et al., 24 Jun 2025, Wang et al., 2024).

Usage in the literature	Core extension	Representative paper
Personalized ranking DMPO	One preferred completion vs multiple negatives	(Bai et al., 2024)
Multi-turn DMPO	Whole trajectories instead of single actions	(Shi et al., 2024)
Multi-preference/listwise DMPO	Multiple preference dimensions and listwise supervision	(Sun et al., 24 Jun 2025)
Fine-grained aspect DMPO	Multi-aspect DPO with a Preference Divergence term	(Zhang et al., 11 Aug 2025)
Multi-sample DMPO	Groupwise comparison over sets of outputs	(Wang et al., 2024)

A common misconception is that DMPO is a single universally fixed loss. The literature instead uses the label for several DPO-style generalizations. What they share is a departure from the original assumption that one prompt is paired with exactly one preferred response and exactly one rejected response under one static notion of goodness.

2. Core optimization pattern

Standard DPO is formulated on pairwise preference data $(x,y_w,y_l)$ with a reference policy $\pi_{\text{ref}}$ and a trainable policy $\pi_\theta$. One canonical form is

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( \beta\left[ \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right] \right) \right],$$

which is the direct policy-space reparameterization of a Bradley–Terry preference model under KL-regularized RL (Badrinath et al., 2024, Sun et al., 24 Jun 2025).

DMPO-style formulations preserve this direct policy optimization viewpoint but modify the structure of the comparison. In the recommendation formulation, the single loser is replaced by an average over $k$ dispreferred completions: $$\mathcal{L}_\text{DMPO}(\pi_{\theta}; \pi_{ref}) = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}} \bigg[ \log \sigma \Big( \beta \log \frac{\pi_{\theta}(y_w\mid x)}{\pi_{ref}(y_w\mid x)} - \frac{1}{k}\sum_{i=1}^{k} \beta \log \frac{\pi_{\theta}(y_{l,i} \mid x)}{\pi_{ref}(y_{l,i}\mid x)} \Big) \bigg]$$ (Bai et al., 2024). In the listwise multi-preference formulation, the binary pair is replaced by a candidate set $\{y_1,\dots,y_N\}$ and a target listwise distribution $p^\lambda$ induced by a simplex weight vector $\lambda$: $$\mathcal{L}_{\lambda\text{-DPO}}(\theta) = -\mathbb{E}_{(x, \{y_i\}), \lambda} \left[ \sum_{i=1}^{N} p^\lambda(y_i \mid x)\,\log P_\theta(y_i \mid x) \right]$$ (Sun et al., 24 Jun 2025). In the fine-grained multi-aspect setting, the generalization appears as an additive Preference Divergence term: $\pi_{\text{ref}}$0 where $\pi_{\text{ref}}$1 is the usual DPO margin and $\pi_{\text{ref}}$2 quantifies inter-aspect preference conflicts (Zhang et al., 11 Aug 2025).

These variants differ in structure, but the recurring design pattern is stable: the optimization stays in policy space, usually remains offline, and replaces the single pairwise margin by a richer comparison object.

3. DMPO as multi-negative preference optimization for recommendation

The most explicit use of the name “Direct Multi-Preference Optimization” appears in personalized ranking for LLM-based recommenders (Bai et al., 2024). The setting is binary-preference recommendation derived from ratings. Users have chronological interaction histories; items with rating $\pi_{\text{ref}}$3 are positive and items with rating $\pi_{\text{ref}}$4 are negative. After filtering, each user has at least 5 positive and 5 negative items, and the history list is truncated to length at most 40 (Bai et al., 2024).

The task is presented to the LLM as an instruction. The prompt contains a role description, the user’s historical watched list, and one or more candidate item pairs. The correct completion places a positive item before a negative item. The key limitation of SFT in this setting is that it “increases the likelihood of the correct completion but does not explicitly push down the likelihood of incorrect completions” (Bai et al., 2024). DMPO addresses this by comparing one preferred completion against multiple dispreferred completions from the same user’s negative pool.

This formulation is sequence-level and reference-relative. The implicit reward is

$\pi_{\text{ref}}$5

and training performs a direct push–pull on sequence probabilities: increase $\pi_{\text{ref}}$6 while decreasing the average $\pi_{\text{ref}}$7 (Bai et al., 2024). The method is implemented after SFT, with the SFT model used as the frozen reference policy and LoRA adapters used for preference tuning (Bai et al., 2024).

Empirically, the paper reports that DMPO “significantly improves the recommendation capabilities of LLMs across three real-world public datasets in few-shot scenarios” and “exhibits superior generalization ability in cross-domain recommendations” (Bai et al., 2024). On MovieLens-1M, Amazon Movies & TV, and Amazon Video Games, the reported AUC values for DMPO are $\pi_{\text{ref}}$8, $\pi_{\text{ref}}$9, and $\pi_\theta$0, respectively, and each is marked as significantly better than all baselines at $\pi_\theta$1 (Bai et al., 2024). The same study also reports that using $\pi_\theta$2 negatives yields stable gains over the single-negative case, with gains saturating around $\pi_\theta$3 (Bai et al., 2024). A further feature of this formulation is explainability through token-level probabilities: the model tends to assign higher conditional probabilities to tokens corresponding to the user-preferred item, and the reported case study interprets the probability gap between positive and negative candidates as an explanation signal (Bai et al., 2024).

4. Multi-turn and trajectory-level extensions

In the language-agent literature, DMPO refers to “Direct Multi-Turn Preference Optimization” (Shi et al., 2024). The motivating problem is that single-turn DPO is derived for a bandit-like setting and cannot directly handle trajectory preferences because the partition function cannot be canceled across states in multi-step interaction (Shi et al., 2024). The proposed solution replaces the policy-level KL constraint with a state-action occupancy measure constraint: $\pi_\theta$4 which yields a global partition function over occupancy measures rather than a state-dependent partition function over actions (Shi et al., 2024).

Trajectory preferences are modeled with a length-normalized Bradley–Terry construction, and the resulting loss uses discounted, length-adjusted sums over preferred and dispreferred trajectories: $\pi_\theta$5 with

$\pi_\theta$6

The paper states that the length normalization is needed both to cancel the partition function and to address length disparities between preferred and dispreferred trajectories (Shi et al., 2024).

This usage of DMPO is best understood as a trajectory-level generalization of DPO rather than a multi-aspect objective. The reported experiments on WebShop, ScienceWorld, and ALFWorld show that the DMPO loss outperforms DPO in noisy settings and outperforms or matches stronger baselines such as PPO and ETO in cleaner settings (Shi et al., 2024). Later work on segment-level preference optimization for social agents explicitly treated DMPO as a session-level baseline and argued that session-level methods are “overly coarse-grained,” introduce training noise, and require stronger theoretical support when the whole dialogue is used as the comparison unit (Kong et al., 3 Jan 2025). This criticism does not negate the original DMPO formulation; it identifies a later refinement path in which the comparison object is narrowed from full trajectories to selected segments.

5. Multi-aspect, listwise, and multi-sample DMPO

A broader strand of work uses “DMPO” as a generic label for DPO-style alignment under richer preference structure. One major axis is multiple preference dimensions. “Multi-Preference Lambda-weighted Listwise DPO for Dynamic Preference Alignment” defines $\pi_\theta$7 dimension-specific listwise distributions $\pi_\theta$8, introduces a simplex weight vector $\pi_\theta$9, and forms the aggregate target

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( \beta\left[ \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right] \right) \right],$$0

The training objective is a listwise cross-entropy between $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( \beta\left[ \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right] \right) \right],$$1 and the policy-induced listwise distribution $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( \beta\left[ \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right] \right) \right],$$2 (Sun et al., 24 Jun 2025). The paper positions this explicitly as a direct multi-preference optimization framework that “handles multiple preference dimensions jointly,” uses “listwise supervision,” and enables “dynamic interpolation” among preference dimensions via $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( \beta\left[ \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right] \right) \right],$$3 (Sun et al., 24 Jun 2025).

A second axis is fine-grained aspect-specific preferences. “Beyond Single” derives a DMPO objective from a multi-aspect PPO formulation and identifies the Preference Divergence term $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( \beta\left[ \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right] \right) \right],$$4, which “quantifies inter-aspect preference conflicts” (Zhang et al., 11 Aug 2025). Rather than directly optimizing the estimated DMPO loss, that paper uses the PD term as a data-selection principle: select the subset of data with the most negative PD values and then train with ordinary DPO on that subset (Zhang et al., 11 Aug 2025). The reported result is “over 10% relative improvement” against both a standard holistic-preference baseline and a stronger oracle on UltraFeedback with three conflict levels (Zhang et al., 11 Aug 2025). This is a data-centric interpretation of DMPO: the multi-preference formulation exposes which samples are high-consensus and therefore most useful for downstream direct preference training.

A third axis is groupwise or multi-sample comparison. “Preference Optimization with Multi-Sample Comparisons” introduces multi-sample DPO (mDPO) and multi-sample IPO (mIPO), where the comparison unit is a group $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( \beta\left[ \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right] \right) \right],$$5 rather than a single response (Wang et al., 2024). The group-level implicit reward is the average log-ratio over the group, and the resulting mDPO loss applies a logistic preference link to the difference between group-level averages (Wang et al., 2024). The paper presents mDPO and mIPO as concrete instantiations of a general DMPO design for distributional properties such as diversity, bias, and calibration (Wang et al., 2024). The reported empirical findings include improved diversity and bias mitigation for diffusion models and improved robustness under label noise in synthetic preference data (Wang et al., 2024).

Taken together, these formulations show that “multi-preference” has at least three operational meanings: multiple preference dimensions, multiple candidate responses per prompt, and multiple samples judged jointly.

6. Related formulations, limitations, and open directions

Several neighboring methods sharpen the design space around DMPO. “Unified Preference Optimization” proposes a hybrid objective of the form

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( \beta\left[ \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right] \right) \right],$$6

combining a DPO/KTO-style preference term with an offline RL-style auxiliary objective over designer rewards (Badrinath et al., 2024). The paper presents this as a way to optimize “user and designer preferences without any additional specialized or preference data, computational cost, stability tweaks, or training instability” (Badrinath et al., 2024). This is not named DMPO in the paper, but it provides a template for direct preference optimization with multiple auxiliary objectives.

Later work extends the same area in several directions. “Mix- and MoE-DPO” introduces latent expert assignments and a mixture-of-Bradley–Terry model so that different experts specialize to distinct preference modes or tasks (Bohne et al., 9 Oct 2025). “MASS-DPO” addresses the inefficiency of large negative pools by selecting compact, informative negative subsets under a Plackett–Luce-specific Fisher-information criterion (Surana et al., 11 May 2026). “Mult-DPO” replaces pairwise preferences with a tractable multinomial surrogate likelihood for set-wise recommendation events and extends this to multiple preference levels (Zhu et al., 8 Jun 2026). “Omni-DPO” keeps the DPO form but dynamically weights each pair by both external data quality and current model difficulty (Peng et al., 11 Jun 2025). “Self-Improvement Towards Pareto Optimality” argues that DPO-based multi-objective alignment suffers from preference conflicts and proposes self-generated Pareto-optimal responses to turn conflicting data into consistent DPO pairs (Li et al., 20 Feb 2025). “Direct Preference Optimization with Penalization” adds a gated penalty on reference-greedy responses and suggests a route toward integrating additional auxiliary negatives into multi-response preference training (Sun, 10 Jun 2026).

These developments also clarify the main limitations of DMPO-style methods. The first is semantic ambiguity: “multi” may refer to multiple negatives, multiple turns, multiple preference dimensions, multiple samples, or multiple preference levels. The second is data quality. Several papers emphasize that richer supervision introduces severe noise or conflicts, whether across preference aspects (Zhang et al., 11 Aug 2025), across objectives on the Pareto front (Li et al., 20 Feb 2025), or across multiple negatives with redundant gradients (Surana et al., 11 May 2026). The third is controllability. In the dynamic listwise setting, changing $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( \beta\left[ \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right] \right) \right],$$7 at inference does not necessarily mean explicit conditioning on $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( \beta\left[ \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right] \right) \right],$$8 inside the model; some formulations instead learn a single policy that is robust across a distribution of trade-offs (Sun et al., 24 Jun 2025). A plausible implication is that future work will continue to separate two questions that the acronym currently compresses into one: how to represent richer preference structure, and how to turn that structure into a stable direct optimization objective.