Divergence Minimization Preference Optimization (DMPO)

• DMPO is a principled framework that aligns generative models like LLMs and diffusion models with human preferences by directly minimizing an f-divergence between the learned and optimal policies.
• It generalizes methods such as DPO, EXO, and SimPO, offering a unified mathematical foundation to balance sample quality, diversity, and exploration-exploitation trade-offs.
• Supported by both extensive theoretical analysis and empirical evaluations across multiple domains, DMPO enables precise control over model behaviors through customizable divergence functions.

Divergence Minimization Preference Optimization (DMPO) is a principled framework for aligning generative models—including LLMs and diffusion models—with human preferences by directly minimizing a chosen $f$-divergence between the optimized policy and a reference (or optimal) policy. DMPO subsumes and generalizes existing preference optimization algorithms such as Direct Preference Optimization (DPO), Exploration Preference Optimization (EXO), and SimPO, providing a unified mathematical and algorithmic foundation for interpolation between diverse alignment objectives. By selecting different divergence functions (e.g., reverse KL, forward KL, Jensen–Shannon, $\alpha$-divergence), DMPO offers precise control over the trade-off between sample quality, diversity, and mode-seeking/mode-covering behaviors in conditional generation tasks. It is supported by extensive theoretical and empirical analysis across LLMs, text-to-image diffusion models, and combinatorial RL environments (Han et al., 2024, Wang et al., 2023, Sun et al., 2024, Li et al., 10 Jul 2025, Li et al., 19 May 2026).

1. Mathematical Formulation and Theoretical Foundation

At its core, DMPO treats offline preference alignment as a distribution matching problem. The objective is to minimize the $f$-divergence $D_f(\hat\pi_\theta \| \hat\pi^*)$ between a "twisted" version of the learned policy $\hat\pi_\theta$ and the optimal policy $\hat\pi^*$:

• The reference/fine-tuned policy: $\pi_\mathrm{ref}(y|x)$
• The optimized policy: $\pi_\theta(y|x)$
• The optimal Boltzmann policy:

$$\pi^{*}(y|x) \propto \pi_\mathrm{ref}(y|x) \exp\!\left(\frac{1}{\beta} r(x, y)\right)$$

where $r(x,y)$ is the reward or preference score and $\alpha$0 is the KL penalty parameter.

For robust interpolation, a "twisted" policy is introduced: $\alpha$1

The general DMPO loss is: $\alpha$2 with the generator function $\alpha$3 characterizing the divergence (as in Table 1 of (Han et al., 2024)).

In the case of pairwise preference data, DMPO admits a simple form: $\alpha$4 where $\alpha$5.

As shown in (Wang et al., 2023) and (Han et al., 2024), for varied $\alpha$6-divergences, the optimal policy is derived via Karush–Kuhn–Tucker (KKT) conditions, yielding tractable, closed-form mappings between reward functions and policy ratios.

2. Connections to Existing Methods and Divergence Choices

DMPO explicitly generalizes key preference alignment approaches:

• Direct Preference Optimization (DPO): $\alpha$7 (reverse KL); mode-seeking, concentrating probability mass on high-reward samples (Han et al., 2024, Wang et al., 2023).
• EXO: $\alpha$8 (forward KL); mode-covering, ensuring diversity by covering all reward modes.
• Intermediate/Novel Variants: Jensen–Shannon and $\alpha$9-divergences interpolate between these behaviors, supporting a flexible alignment-performance/diversity trade-off (Han et al., 2024, Sun et al., 2024).

Table: Generator Functions and Gradient Weights for Common $f$0-divergences

Divergence	$f$1	Gradient weight $f$2
Reverse KL	$f$3	$f$4
Forward KL	$f$5	$f$6
Jensen–Shannon	$f$7	$f$8
$f$9-divergence	$D_f(\hat\pi_\theta \| \hat\pi^*)$0	$D_f(\hat\pi_\theta \| \hat\pi^*)$1

Distinct gradient weighting directly determines the boost/suppression ratio for winners/losers in pairwise updates (Sun et al., 2024).

3. Algorithmic Recipe and Implementation in Practice

DMPO is implemented as a simple offline fine-tuning procedure. For LLMs, the high-level training loop is:

$\hat\pi^*$7

• The reference policy, divergence function $D_f(\hat\pi_\theta \| \hat\pi^*)$2, and other hyperparameters (e.g., label smoothing $D_f(\hat\pi_\theta \| \hat\pi^*)$3) are all user-selectable.
• For scaling and efficiency, approximations such as length-normalized or SimPO-style surrogates for $D_f(\hat\pi_\theta \| \hat\pi^*)$4 are used (Han et al., 2024).
• DMPO autonomously provides reverse-KL regularization when $D_f(\hat\pi_\theta \| \hat\pi^*)$5 is chosen accordingly, and can be efficiently implemented with only minor modifications over standard DPO.
• For diffusion models, the DMPO objective is ported to stepwise denoiser preference updates, with per-timestep logistic loss and no explicit reward model required (Li et al., 10 Jul 2025, Sun et al., 2024).

4. Theoretical Properties and Special Features

Key DMPO properties include:

• Global Optimality: For strictly convex $D_f(\hat\pi_\theta \| \hat\pi^*)$6, minimizing the $D_f(\hat\pi_\theta \| \hat\pi^*)$7-divergence yields $D_f(\hat\pi_\theta \| \hat\pi^*)$8, ensuring recovery of the optimal policy at convergence (Han et al., 2024, Wang et al., 2023).
• Unbiased Monte Carlo Estimators: Empirical multi-sample losses converge to the true $D_f(\hat\pi_\theta \| \hat\pi^*)$9-divergence in expectation as $\hat\pi_\theta$0.
• Mode-Seeking vs. Mode-Covering: Choice of $\hat\pi_\theta$1 governs exploration-exploitation tradeoff; reverse KL promotes conservative, safe solutions, while forward KL promotes spread/diversity. Jensen–Shannon interpolates the tradeoffs—empirically superior in balancing human-value and diversity in image generation (Sun et al., 2024).
• No Need for Explicit Reward Normalization: The normalization constant cancels in the Bradley–Terry model, so only preference differences matter (Wang et al., 2023).
• Robustness to Conflicting/Multi-Aspect Preferences: In fine-grained or multi-aspect preference datasets, DMPO yields a natural "preference divergence" used for rigorous data-selection or conflict resolution (Zhang et al., 11 Aug 2025).

5. Empirical Evaluation Across Domains

DMPO demonstrates empirical superiority and robustness across modalities:

• LLM Alignment: On Pythia-2.8B, Mistral-7B, and Llama-3-8B, $\hat\pi_\theta$2-PO ($\hat\pi_\theta$3 between 0.05 and 0.2) outperforms DPO/EXO by 5–10pp on GPT-4 judged win-rate across AlpacaEval 2, Arena-Hard, MT-Bench, and Open LLM Leaderboard v2 (Han et al., 2024).
• Diffusion Model Alignment: DMPO-based fine-tuning yields at least 64.6% PickScore win-rate advantages over all existing diffusion alignment baselines on benchmarks such as Pick-a-Pic V2 and HPS V2 (SD1.5/SDXL) (Li et al., 10 Jul 2025).
• Image-Text Alignment: Jensen–Shannon divergence yields best trade-off between human-value alignment and generation diversity for text-to-image models (Sun et al., 2024).
• Diversity-Preserving RL: Group-level forward KL surrogates via DMPO robustly prevent mode collapse in combinatorial reasoning, improving Quality Ratios by 9–12% over vanilla reverse-KL RL methods (Li et al., 19 May 2026).

Additionally, DMPO reduces expected calibration error degradation compared to PPO, and is more "divergence efficient," achieving higher reward for a given divergence constraint (Wang et al., 2023).

6. Practical Recommendations and Data-Centric Enhancements

• When only pairwise preferences are available and mode-seeking alignment is desired, reverse KL ($\hat\pi_\theta$4) or $\hat\pi_\theta$5 close to $\hat\pi_\theta$6 is recommended.
• For high-quality reward models (driving diversity coverage), forward KL or small $\hat\pi_\theta$7 is preferable.
• Empirically, $\hat\pi_\theta$8-PO with $\hat\pi_\theta$9 in $\hat\pi^*$0 consistently yields best trade-offs.
• Fine-grained datasets: In high-noise, multi-aspect preference settings, select samples with most negative "preference divergence" (inter-aspect consensus) to maximize efficient downstream alignment (Zhang et al., 11 Aug 2025).
• Hyperparameter tuning: $\hat\pi^*$1 in $\hat\pi^*$2 stabilizes the two-term loss; moderate $\hat\pi^*$3 optimizes regularization for diffusion models (Han et al., 2024, Li et al., 10 Jul 2025).

7. Extensions and Outlook

DMPO provides a unifying foundation for preference optimization across multiple generative domains—including LLMs, diffusion models, and RL agents:

• Algorithmic extensibility: Specialized variants (e.g., MC-PO with contrastive divergence negative mining) offer state-of-the-art performance in benchmarked preference ranking tasks (Chen et al., 6 Feb 2025).
• Theoretical avenues: Open research includes generalizing beyond $\hat\pi^*$4-divergences (e.g., $\hat\pi^*$5, GFlowNet), non-i.i.d. preference structures, and enhanced calibration guarantees (Han et al., 2024, Wang et al., 2023).
• Applications: DMPO is being actively integrated into standard toolkits for scalable instruction tuning, image, and multimodal alignment, leveraging its differentiable flexibility and divergence-regularized supervision.

In summary, DMPO ($\hat\pi^*$6-PO) defines a transparent, tunable, and theoretically-grounded family of offline preference-optimization algorithms, directly interpolating between known methods and enabling discovery of new, empirically superior variants for diverse generative model alignment tasks (Han et al., 2024, Li et al., 10 Jul 2025, Wang et al., 2023, Sun et al., 2024, Li et al., 19 May 2026, Zhang et al., 11 Aug 2025, Chen et al., 6 Feb 2025).