Holistic DPO: A Multi-Axis Perspective

• Holistic DPO is a comprehensive framework that broadens Direct Preference Optimization from a single-model, binary-label approach to encompassing entire compound AI systems and fine-grained preferences.
• It integrates system-level dynamics by aligning complete computational graphs and evaluating performance across multiple axes such as factuality, safety, proactivity, conciseness, and diversity.
• It employs conflict-aware data selection and fine-grained preference signals to mitigate optimization pathologies like response suppression and dispersion, enhancing overall alignment quality.

Searching arXiv for papers on Holistic DPO, system-level DPO, and DPO evaluation. Holistic DPO is best understood as an umbrella reading of Direct Preference Optimization in which preference alignment is no longer treated as a narrowly defined single-model, single-label, single-metric problem. In current literature, that reading spans at least three distinct but related developments: extending DPO from a single policy to a compound AI system via System-level DPO (SysDPO) (Wang et al., 24 Feb 2025); evaluating DPO against PPO, ORPO, and KTO across factuality, safety, conciseness, proactivity, and diversity under both in-distribution and out-of-distribution conditions (Janiak et al., 16 Sep 2025); and moving beyond one holistic “better-than” label toward fine-grained aspect-specific preferences, then using conflict-aware data selection before standard DPO training (Zhang et al., 11 Aug 2025). A complementary theoretical line argues that such broader formulations are motivated by intrinsic weaknesses of vanilla DPO, summarized as the 3D-properties: drastic drop in rejected response likelihood, degradation into response suppression, and dispersion effect on unseen responses (Yan et al., 2024). This suggests that “holistic” refers less to a single canonical loss than to a broader alignment stance that jointly considers optimization dynamics, data structure, system boundaries, and evaluation criteria.

1. Baseline DPO and the motivation for holistic formulations

Direct Preference Optimization is presented as a simplification of RLHF in which the reward modeling phase is eliminated and the policy is trained directly from preference pairs. In the single-model setting, the standard DPO objective is written as

$$\ell_{\text{DPO}}(\theta) = -\sum_{(x,a^+,a^-)} \log \sigma\left[ \beta \log \frac{\pi_\theta(a^+|x)}{\pi_0(a^+|x)} - \beta \log \frac{\pi_\theta(a^-|x)}{\pi_0(a^-|x)} \right].$$

Within the alignment evaluation literature, DPO is correspondingly described as maximizing the margin between preferred and non-preferred completions and as learning “an implicit reward function directly from the collected preference data” without a separate reward model (Yan et al., 2024).

The need for a holistic treatment follows from two observations. First, standard DPO assumes a single policy $\pi_\theta(y\mid x)$ and a single preference label per pair, whereas many deployment settings involve multiple interacting components, multiple preference aspects, or both. Second, DPO’s apparent simplicity does not imply benign optimization dynamics. A broader treatment therefore becomes necessary when alignment objectives are system-level, when preference data are multi-aspect and conflict-prone, or when evaluation must account for multiple behavior axes rather than one scalar preference score.

A practical implication is that “holistic” DPO is not merely DPO applied to more data. It typically involves one of three enlargements of scope: enlarging the optimized object from a model to a compound system, enlarging the supervision signal from a single binary preference to structured fine-grained preferences, or enlarging the evaluation target from one criterion to a multidimensional profile.

2. Optimization pathologies and the limits of vanilla DPO

A major argument for holistic reformulation comes from the analysis of DPO’s training dynamics. The 3D-properties paper shows that vanilla DPO produces an asymmetric interaction between the gradients on chosen and rejected responses. Using $\pi^+ = \pi_\theta(a^+\mid x)$ and $\pi^- = \pi_\theta(a^-\mid x)$, the key ratio is

$$\frac{ \frac{\partial \ell^{\text{DPO}}}{\partial \pi^-} }{ \frac{\partial \ell^{\text{DPO}}}{\partial \pi^+} } = \frac{\pi^+}{\pi^-}.$$

As $\pi^-$ becomes small, optimization increasingly concentrates on suppressing the rejected response. In the regime $\pi^- \to 0$ with $\beta < 1$, the paper further gives

$$\frac{\partial \ell^{\text{DPO}}}{\partial \pi^+} \propto (\pi^-)^\beta \to 0, \qquad \frac{\partial \ell^{\text{DPO}}}{\partial \pi^-} \propto (\pi^-)^{\beta-1} \to \infty.$$

The stated interpretation is that DPO degrades into “unlearning” the rejected sample, while the positive sample loses gradient support (Yan et al., 2024).

The three resulting failure modes are presented as coupled rather than independent. The drastic drop in rejected response likelihood appears first. Degradation into response suppression follows when the rejected sample has already been driven to very low likelihood and the chosen sample no longer receives a comparably strong learning signal. The dispersion effect on unseen responses then arises because probability mass does not move cleanly from rejected to chosen outputs; instead, it can leak into outputs absent from the preference dataset.

The same paper argues that these behaviors are moderated by the distribution of preference data. Four scenarios are analyzed: chosen on-policy with rejected on-policy, chosen off-policy with rejected on-policy, chosen on-policy with rejected off-policy, and chosen off-policy with rejected off-policy. The most stable case is the fully on-policy setting, because the rejected response begins with higher likelihood and therefore takes longer to collapse. Empirically, this pattern is reported on MATH and SuperCLUE-Math, while instruction-following experiments on poem and slogan generation show that DPO improves over the base model in some settings but that RLHF-PPO still performs better overall on stricter metrics. The same study proposes Flexible-$\beta$ DPO and SFT regularization as stabilizers, while emphasizing that these alleviate rather than eliminate the underlying asymmetry.

A common misconception is therefore that DPO’s reward-model-free form makes it automatically stable or universally preferable to PPO. The literature does not support that claim. Instead, it suggests that DPO’s simplicity comes with structural optimization risks, and that more holistic methods should be read partly as attempts to manage those risks.

3. System-level DPO in compound AI systems

The most explicit formalization of Holistic DPO appears in System-level DPO, or SysDPO, which extends DPO from a single policy to a compound AI system represented as a directed acyclic graph (DAG). The DAG contains the user input $\pi_\theta(y\mid x)$0, intermediate outputs $\pi_\theta(y\mid x)$1, and final outputs $\pi_\theta(y\mid x)$2, with full system output

$\pi_\theta(y\mid x)$3

The system likelihood factorizes over the graph as

$\pi_\theta(y\mid x)$4

where $\pi_\theta(y\mid x)$5 denotes parent nodes. Preference data are then defined over whole system outputs, $\pi_\theta(y\mid x)$6, and the DPO-style objective is lifted to the joint system distribution:

$\pi_\theta(y\mid x)$7

The defining claim is that alignment should operate on complete trajectories, including all intermediate outputs, final outputs, and component interactions, rather than on local component outputs in isolation (Wang et al., 24 Feb 2025).

This formulation is motivated by three obstacles in compound systems: non-differentiable component interactions, the failure of system-level preferences to decompose cleanly into component-level preferences, and the fact that available data are often holistic system-level preference pairs rather than labeled preferences for internal substeps. The paper gives two main examples: an LLM plus diffusion pipeline, and a Mixture-of-Agents collaboration system. In the LLM-plus-diffusion case, the system factorization becomes

$\pi_\theta(y\mid x)$8

and the resulting system-level DPO objective combines text likelihood ratios with diffusion likelihood surrogates.

Because diffusion likelihoods are not directly tractable, the paper derives a tractable upper bound using DDPM-style denoising losses. That move yields the operational distinction between SysDPO-Direct, which defines the objective directly at the system level, and SysDPO-Sampling, which estimates it via sampled timesteps, noisy states, and denoising-loss differences. A plausible implication is that “holistic” here denotes not only broader supervision but also a broader probabilistic object: the joint distribution induced by the entire computational graph.

The multimodal experiment in the paper evaluates sequence generation tasks in which a scene attribute varies progressively across three images. Preference data are built over whole sequences, with four generated sequences per prompt, six comparison pairs, and a total of 6000 comparison pairs. The preference score is

$\pi_\theta(y\mid x)$9

favoring correct order and smooth progression. The reported results are as follows:

Method	Pref. Score	OC Ratio
SysDPO (Proposed)	0.25	70%
System Before Alignment	-0.20	32%
Best-of-Sampling	0.16	67%
Only Train LLM	0.23	65%
Only Train Diffusion Model	-0.03	35%

These results are interpreted as evidence that merely combining strong pretrained components does not guarantee coordinated system behavior, and that joint system-level alignment can outperform component-wise training.

4. Holistic evaluation of DPO across alignment axes

A second major sense of Holistic DPO is evaluative rather than algorithmic. The five-dimensional alignment evaluation framework compares PPO, DPO, ORPO, and KTO on factuality, safety, conciseness, proactivity, and diversity, using both in-distribution and out-of-distribution datasets. The main evaluator is an LLM-as-a-Judge protocol with LLaMA-3.1-70B, comparing candidate responses against gold responses. The principal scalar measure is the win-tie rate

$\pi^+ = \pi_\theta(a^+\mid x)$0

and robustness is summarized through a generalisation gap between in-distribution and out-of-distribution settings. Factuality on the TLDR summarization benchmark is measured separately with HHEM-2.1-Open, while safety is decomposed into FAR, FRR, and a Harmlessness metric; proactivity is measured only on unsafe prompts and normalized by the rate of correct refusals; and diversity is computed by averaging SentBERT, NLI, and EAD over 16 responses per prompt (Janiak et al., 16 Sep 2025).

The paper’s DPO instantiation uses LLaMA-7B, starts from an SFT checkpoint trained on AlpacaFarm instruction-following data, and performs DPO alignment on a combined dataset comprising general instruction-following examples and safety-focused data. The optimization details are 5 epochs, learning rate 1e-6, linear scheduler, AdamW with β1=0.9, β2=0.999, and DPO-specific parameter β = 0.1.

Within this framework, DPO exhibits a clear behavioral profile. It is reported to be among the strongest methods for factuality, especially out of distribution; to demonstrate the highest safety performance in both in-distribution and out-of-distribution settings; and to achieve very low FAR across PKU-SafeRLHF, BeaverTails Evaluation, and DataAdvisor. The same study also states that “DPO provides significantly stronger generalisation in terms of proactivity compared to other methods,” indicating that DPO tends to do more than issue bare refusals on unsafe prompts.

The main weakness is conciseness. The paper explicitly states that “DPO and KTO models are frequently less concise than SFT,” and further links DPO’s strong proactivity to its “very low score for conciseness,” because DPO-trained models tend to generate long responses. PPO is identified as the method achieving the best balance between proactivity and conciseness. DPO also does not emerge as the diversity leader; the paper notes more generally that alignment tends to reduce diversity relative to SFT and that temperature tuning matters more for diversity than method choice.

The resulting picture is not one of universal superiority. It is instead a trade-off profile: DPO is strong on factuality, safety, and proactivity, especially under distribution shift, but pays for that with verbosity and weaker conciseness. This multidimensional profile is central to the holistic evaluation viewpoint because it rejects one-dimensional claims of method dominance.

5. Fine-grained preferences, DMPO, and conflict-aware selection

A third sense of Holistic DPO departs from the assumption that each pair of responses can be labeled reliably by one global “better-than” judgment. The fine-grained preference literature argues that diverse human values are better represented through multiple sub-preferences such as helpfulness, honesty, instruction-following, and truthfulness. The dataset is then written as

$\pi^+ = \pi_\theta(a^+\mid x)$1

From a multi-reward objective, the paper derives Direct Multi-Preference Optimization (DMPO):

$\pi^+ = \pi_\theta(a^+\mid x)$2

where

$\pi^+ = \pi_\theta(a^+\mid x)$3

and $\pi^+ = \pi_\theta(a^+\mid x)$4 is the Preference Divergence (PD) term (Zhang et al., 11 Aug 2025).

The PD term is defined by

$\pi^+ = \pi_\theta(a^+\mid x)$5

or equivalently

$\pi^+ = \pi_\theta(a^+\mid x)$6

Its role is to quantify inter-aspect preference conflict. Negative PD indicates consensus with other aspects; positive PD indicates conflict with them. The paper’s central operational claim is that PD should not primarily be optimized directly. Instead, it should guide data selection before ordinary DPO training.

The resulting rule is to select a subset

$\pi^+ = \pi_\theta(a^+\mid x)$7

that is, the samples with the most negative PD values. The theoretical argument is that this high-consensus subset minimizes upper and lower bounds on the DMPO loss under a subset-selection budget. To estimate PD in practice, the paper trains per-aspect proxy reward models via Bradley–Terry loss, applies quantile scaling and clipping to make pseudo-gaps comparable across aspects, and introduces two length-bias mitigations: length-balanced sampling and a length penalty in the reward objective.

The empirical evaluation is conducted on UltraFeedback with three conflict levels: 30%, 20%, and 10%. The main DPO model is Llama-3.1-8B, the proxy reward model is Llama-3.2-3B, both are initialized from SFT on OpenHermes-2.5, the reward model uses only 30% of the training data for proxy learning, and DPO is trained for one epoch. Baselines include Overall, Oracle, ALL, DMPO, RAND, HIGH, MID, and the PD-selection method. The paper reports “over 10% relative improvement” against both standard holistic preference relabeling and the stronger Oracle baseline using aggregated preference signals, while also improving training efficiency. In the 30% conflict setting, the reported AW values are 1.00 for Overall, 1.15 for Oracle, and 1.21 for the proposed method.

A common misconception is that holistic alignment necessarily requires holistic annotation. The fine-grained preference work argues the opposite: holistic alignment may be better achieved by collecting easier aspect-specific judgments, estimating inter-aspect consensus, and selecting the most reliable training pairs rather than forcing every example into a single aggregated label.

6. Synthesis, controversies, and disambiguation

Across these strands, Holistic DPO is not a single standardized algorithmic name. It is more accurately a research direction in which DPO is broadened along one or more axes: the object of optimization becomes the whole system rather than a single model; the supervision signal becomes fine-grained and conflict-aware rather than a single global label; and the evaluation target becomes a five-axis behavioral profile rather than a single preference score. This broader view also incorporates a negative lesson from the 3D-properties analysis: local pairwise preference optimization can conceal unstable sequence-level dynamics.

Several controversies and misunderstandings recur in the literature. One is the claim that DPO is simply a cheaper substitute for PPO. The evidence is mixed: DPO avoids reward-model training and can be very strong on safety and factuality, but PPO is reported to provide the best balance between proactivity and conciseness, while the optimization analysis argues that PPO-based RLHF does not exhibit the same 3D-properties. Another misconception is that better holistic labels automatically solve multi-objective alignment. The fine-grained preference results instead suggest that conflict-aware selection may outperform both standard holistic labeling and even an oracle that aggregates aspect scores. A third misconception is that component-wise alignment is enough in compound systems. SysDPO is explicitly built on the claim that system-level preferences do not decompose cleanly into component-level preferences.

The acronym DPO is also overloaded across research areas. In portfolio theory, DPO can denote “Distributional Portfolio Optimization,” a unified framework over distributions of weights, returns, and parameters (Alonso, 28 May 2026). In graph transformation, “DPO-based” can refer to the double-pushout approach to term graph rewriting (Kahl et al., 2019). Those usages are unrelated to Direct Preference Optimization in language-model alignment and should be kept distinct.

The most defensible general conclusion is therefore narrow but technically significant. Holistic DPO, in the alignment sense, denotes a move from isolated pairwise preference fitting toward broader optimization and assessment regimes in which preference learning is treated as a property of entire systems, multidimensional behaviors, and structured supervision signals. The literature does not show that DPO dominates all alternatives. It does show that when DPO is situated in a holistic framework, its strengths, failure modes, and appropriate use cases become substantially clearer.