MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment

Abstract: Aligning LLMs to desirable human values requires balancing multiple, potentially conflicting objectives such as helpfulness, truthfulness, and harmlessness, which presents a multi-objective optimisation challenge. Most alignment pipelines rely on a fixed scalarisation of these objectives, which can introduce procedural unfairness by systematically under-weighting harder-to-optimise or minority objectives. To promote more equitable trade-offs, we introduce MGDA-Decoupled, a geometry-based multi-objective optimisation algorithm that finds a shared descent direction while explicitly accounting for each objective's convergence dynamics. In contrast to prior methods that depend on reinforcement learning (e.g., GAPO) or explicit reward models (e.g., MODPO), our approach operates entirely within the lightweight Direct Preference Optimisation (DPO) paradigm. Experiments on the UltraFeedback dataset show that geometry-aware methods -- and MGDA-Decoupled in particular -- achieve the highest win rates against golden responses, both overall and per objective.

• The paper introduces MGDA-Decoupled, a geometry-based method that decouples gradient normalization to balance multiple objectives in DPO LLM alignment.
• It leverages loss-normalized gradients to steer updates, achieving significantly faster convergence on a synthetic toy problem and higher net win rates in alignment experiments.
• The approach ensures fairness by enforcing balanced trade-offs among objectives, robustly enhancing performance on hard-to-optimize safety and ethical criteria.

Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment: The MGDA-Decoupled Approach

Motivation: Fair Multi-Objective Alignment in Preference Optimization

LLM alignment increasingly faces the challenge of balancing multiple normative objectives such as helpfulness, honesty, truthfulness, and harmlessness within a unified fine-tuning framework. Traditional alignment pipelines often collapse preference data into a single objective, either via static scalarisation or implicit biases in annotator distribution. This leads to "procedural unfairness," i.e., domination of majority or easy-to-optimize objectives, marginalizing those that are more difficult or under-represented but still critical for safety or social acceptability.

Recent work suggests maintaining per-objective preference datasets and reconstructing alignment as a genuine multi-objective optimization (MOO) problem. The paper introduces MGDA-Decoupled, an algorithm that leverages geometric properties of gradients to ensure balanced improvements across all objectives, strictly within the Direct Preference Optimization (DPO) paradigm without resorting to reinforcement learning, auxiliary reward models, or hyperparameter-sensitive control mechanisms (2604.20685).

Figure 1: Overview of the DPO-based multi-objective fine-tuning framework, with per-objective datasets and dynamic gradient-combinator mediation.

Formalization and Methodological Landscape

At each training step, standard DPO constructs losses per objective ($L_i$) based on preference data and computes the corresponding gradients. The core multi-objective update reduces to finding aggregation coefficients $c_i$ such that the combined update $\sum_i c_i \nabla L_i$ achieves desirable properties according to various algorithms:

• Uniform: Static, equal weighting of gradients; theoretically and empirically prone to dominance by objectives with larger gradient magnitudes or more abundant data.
• GroupDRO: Dynamically upweights higher-loss objectives according to a softmax rule, agnostic to geometric relationships among gradients.
• MGDA-Normalised: Computes coefficients using normalized per-objective gradient directions (removing absolute scale), then applies them to raw gradients.
• MGDA-Decoupled: Computes coefficients based on loss-normalised gradients ($\nabla L_i / L_i$) to capture objective-specific convergence states, applies those coefficients to the raw gradients—thus using the loss as a proxy for optimization distance, but without discarding all scale information.

The geometric rationale is to recover a direction that improves all objectives, avoiding shortsighted trade-offs or stalls due to vanishing gradients on nearly-saturated objectives. This calibration is crucial in ensuring that "hard" objectives, which may be minority-valued but high-stakes (such as harmlessness or fairness criteria), are not systematically neglected.

The geometric intuition is illustrated by the relative orientation and norm of per-objective gradients: standard approaches collapse to the mean, overfitting to majority clusters; geometry-aware methods such as MGDA seek true Pareto-improving directions.

Empirical Analysis: Toy Problem and Real-World Evaluation

Controlled 2D MOO Toy Problem

To elucidate algorithmic differences, the paper presents a two-objective quadratic/exponential synthetic landscape with a shared unique optimum. MGDA-Decoupled achieves the fastest convergence, requiring 265 steps to reach the optimum, compared to 319 (MGDA-Normalised), 823 (GroupDRO), and 1201 (Uniform); standard MGDA fails to converge, stalling due to imbalance in gradient norm magnitude.

Figure 2: Optimisation trajectories for a 2D multi-objective problem, highlighting the accelerated convergence of MGDA-Decoupled relative to other strategies.

Alignment Experiments on LLMs

Two small instruction-tuned foundation models are used: Gemma-2-2b-it (2.6B) and Qwen2.5-0.5B-Instruct (0.5B). The UltraFeedback dataset is used with objective decomposition into Helpfulness, Honesty, Instruction Following, and Truthfulness. All methods leverage the same DPO infrastructure, differing solely in the update rule.

Key Results:

• MGDA-Decoupled consistently achieves the highest overall net win rates against golden human-rated responses compared to all baselines, both on aggregate and in most per-objective evaluations.
• On Gemma-2-2b-it, MGDA-Decoupled shows a statistically significant increase in net win rate (+2.3%) over the instruction-finetuned reference.
• On Qwen2.5-0.5B-Instruct, all dynamic algorithms surpass reference performance, with MGDA-Decoupled again achieving the maximal overall gain.

  Figure 3: Overall net win rates against golden responses for Gemma-2-2b-it, showcasing MGDA-Decoupled as the only algorithm with statistically significant improvement over the reference model.

  

  Figure 4: Overall net win rates against golden responses for Qwen2.5-0.5B-Instruct, with MGDA-Decoupled leading improvement among multi-objective optimizers.

Notably, in absolute terms, the improvement over well-tuned instruction-following references is moderate, reflecting saturation of major alignment axes. Nevertheless, the dominance of geometry-aware methods is robust across multiple objectives and model scales.

Theoretical and Practical Implications

Theoretical Considerations:

• MGDA-Decoupled introduces a principled, geometry-based approach to distributing model capacity that is sensitive to the convergence state of each objective, hence improving fairness without the need for RL loop instability or reward model overhead.
• The decoupled normalization tightly aligns with Polyak step size intuition, effectively prioritizing objectives further from their optima in a loss-geometry-aware manner.

Practical Impact:

• The computational overhead for MGDA-based algorithms is quadratic in the number of objectives due to Gram matrix construction but is negligible for typical alignment scenarios ($k\leq5$).
• The design is amenable to further generalization: for more complex or more numerous objectives (e.g., safety-critical environments), geometry-aware dynamic scalarisation yields substantial improvements in worst-case metrics.
• The method preserves DPO's simplicity and efficiency advantages, making it attractive for large-scale industrial alignment practice where compute constraints and robustness to control settings are critical.

Comparison to Prior Art:

• MGDA-Decoupled is distinct from RLHF-based geometric approaches (e.g., GAPO), advancing geometry-aware methods in the low-overhead, stable DPO context and empirically outperforming both static and dynamic alternatives that do not leverage gradient geometry.
• Unlike MODPO and safe-DPO-style constrained optimizers, it avoids explicit reward modeling or auxiliary constraints while still promoting multi-dimensional value realization.

Future Directions

• Extension to higher-variance or more adversarial alignment datasets, especially those featuring pronounced trade-off gradients between hard safety objectives, can empirically validate geometry-awareness in the presence of dense conflict.
• Combining geometry-aware aggregation with controllable inference or steerability strategies could further enable on-demand value regime traversals, crucial for AGI safety and customization to pluralistic stakeholder values.
• Direct integration of multi-level human preference calibration and adaptive user-facing control over scalarisation weights may become critical as foundation models serve increasingly diverse user bases with incommensurable utility and disutility axes.

Conclusion

MGDA-Decoupled demonstrates that geometry-aware multi-objective optimization is not only tractable within the DPO paradigm but also confers tangible improvements in both aggregate and worst-case alignment. By leveraging the loss-normalized gradient structure, it robustly outperforms uniform, dynamic loss-based, and basic geometry-aware approaches, concretely advancing fairness in LLM alignment without the computational burden of reinforcement learning or auxiliary reward models. This work has significant implications for scalable, fair, and pluralistic alignment in future LLM development and deployment (2604.20685).