Multi-Objective Training

• Multi-objective training is a learning paradigm that optimizes multiple, often competing, loss objectives to produce Pareto-optimal solutions.
• Key methodologies include scalarization, dynamic loss aggregation using hypervolume maximization, and collaborative learning to balance diverse performance metrics.
• Evaluation metrics such as hypervolume, IGD, and spread validate these methods, with applications spanning GANs, robust networks, multi-task reinforcement learning, and engineering design.

Multi-objective training refers to any learning paradigm in which models or algorithms are exposed to multiple, often competing or non-comparable, objectives during the optimization process. In contrast to single-objective formulations where a scalar loss function suffices, multi-objective training treats the learning problem as a vector optimization over several loss terms, demanding specialized architectures, training algorithms, and evaluation protocols to produce Pareto-optimal or otherwise well-balanced solutions.

1. Mathematical Foundations and Pareto Optimality

Formally, a multi-objective optimization problem (MOP) is defined as minimizing (or maximizing) a vector-valued objective over a feasible set: $$\min_{x\in\mathcal X} \;F(x) = (f_1(x),\,f_2(x),\,\dots,\,f_m(x))^T\,, \quad \mathcal X\subset\mathbb R^n,\ m\ge2$$ A point $x^*$ is Pareto-optimal if there does not exist any $y \in \mathcal X$ such that $f_i(y)\le f_i(x^*)$ for all $i$ with at least one strict inequality. The set of all such non-dominated designs is the Pareto set (PS); its image in objective space is the Pareto front (PF) (Shang et al., 2024).

These concepts carry over directly to machine learning, deep learning, and reinforcement learning settings—whether the goals are accuracy vs. robustness, trade-offs among fairness/diversity metrics, or coverage of diverse behaviors in RL.

2. Core Multi-objective Training Methodologies

2.1 Scalarization and Preference Conditioning

A prevalent practical approach is scalarization: forming a weighted sum (or another combination) of objectives,

$\ell(x|u) = \sum_{i=1}^m u_i f_i(x)$

where $u\in\Delta^{m-1}$ is a preference vector on the simplex. In Pareto Set Learning (PSL), a neural network learns a mapping $x_\theta:u\mapsto x$ such that $x_\theta(u)$ approximates the Pareto-optimal point for weights $u$ (Shang et al., 2024). This is generalized in RL and MARL by passing $x^*$0 into policy or value networks to enable conditioning over trade-off spectra (Hu et al., 28 Feb 2026).

2.2 Dynamic Loss Aggregation and Hypervolume Maximization

Weighted-sum optimization is effective only on convex Pareto fronts. To achieve a uniform or diverse coverage—including concave regions—dynamic loss aggregation methods maximize performance metrics that directly encode front spread and trade-off coverage. A central approach is dominated hypervolume maximization: $x^*$1 where $x^*$2 is a set of objective vectors and $x^*$3 is a reference point (Deist et al., 2021, Su et al., 2020, Grewal et al., 2024). The gradient of the hypervolume operator, with respect to losses or predictions, provides dynamic, per-objective weights, enabling Pareto-diverse, uniformly spread solutions.

2.3 Collaborative Learning Across Multiple Problems

Collaborative Pareto Set Learning (CoPSL) extends PSL by introducing shared representations across multiple MOPs. Here, a common encoder captures preference features useful to all tasks while individual decoders (task-specific heads) specialize these representations for each MOP. The total loss is a sum over all tasks,

$x^*$4

with per-task objectives and architectures optimized jointly, leveraging synergies across related (or even dissimilar) optimization problems (Shang et al., 2024).

3. Model Architectures and Algorithmic Implementations

3.1 Shared/Task-specific Layered Networks

In collaborative multi-objective settings, architectures often comprise a trunk of shared layers $x^*$5 (e.g., preference encoders) and $x^*$6 sets of task-specific layers $x^*$7, one per MOP (Shang et al., 2024): $x^*$8 This hard parameter sharing simultaneously reduces model size and enforces information sharing.

3.2 Loss Functions and Training Objectives

Loss formulations vary by application and optimization domain:

• Linear sum: $x^*$9
• Tchebycheff: $y \in \mathcal X$0
• Modified Tchebycheff: $y \in \mathcal X$1
• Cosine penalty: addition of $y \in \mathcal X$2 for aligning solutions with preferences

Per-task empirical losses are minimized via gradient-based optimization; shared and task-specific parameters are updated accordingly (Shang et al., 2024).

3.3 Dynamic and Adaptive Weighting

Practical systems require adaptivity in objective weighting to cope with changing model landscapes or to counteract interference. Recent methods dynamically update scalarization weights to maintain positive covariance between per-objective rewards and scalarized training signals, avoiding cross-objective interference and collapsing modes (Lu et al., 6 Feb 2026). Mechanisms such as Covariance Targeted Weight Adaptation (CTWA) compute running estimates of objective covariances and increase the weights of under-served objectives, thereby ensuring robust, multi-objective improvement.

4. Evaluation Metrics and Benchmarks

Multi-objective algorithms demand bespoke evaluation metrics:

Metric	Means of Measurement	Significance
Hypervolume	Lebesgue measure dominated by Pareto solutions	Simultaneously reflects convergence and spread
IGD	Distance between approximated and true Pareto fronts	Quantifies proximity to ideal/known front
Spread	Width or diversity of solution distribution	Ensures no collapse onto single-objective extrema
Efficiency	Wall-clock runtime, theoretical FLOPs, parameter count	Resource scaling and practical viability

Benchmarks include synthetic functions (e.g., F1–F6), engineering design problems, as well as high-dimensional real-world datasets (Shang et al., 2024).

5. Applications Across Machine Learning and Engineering Domains

Multi-objective training is pervasive in areas with inherently conflicting goals:

• Multi-criteria GANs: Hypervolume-based loss weighting balances adversarial, pixel, and perceptual losses without hand-tuned coefficients, yielding superior sample fidelity and better training stability (Su et al., 2020, Albuquerque et al., 2019).
• Robust and Sparse Neural Networks: Lexicographic multi-objective mixed-integer programming yields binarized/integer networks that are highly robust to input perturbations, enforce sparsity, and maintain high classification accuracy in low-data regimes (Bernardelli et al., 2022).
• Multi-task Reinforcement Learning: Preference-conditioned policies and dynamic objective weighting achieve full front coverage and avoid missing non-convex regions of the Pareto set—critical for online preference alignment and multi-agent RL (Hu et al., 28 Feb 2026, Lu et al., 14 Sep 2025).
• Computational Engineering: In CFD-driven model discovery, multi-output Gaussian process surrogates guide high-cost physical evaluations, enabling multiobjective model development at a fraction of ab initio simulation cost (Fang et al., 22 Dec 2025).
• Deformable Image Registration and PINNs: Dynamic hypervolume maximization facilitates simultaneous trade-offs among similarity, smoothness, and anatomical accuracy, producing solution sets covering the complete trade-off surface (Grewal et al., 2024, Lu et al., 2023, Bahmani et al., 2021).

6. Challenges, Limitations, and Future Directions

Addressing cross-objective interference has become central. Empirical findings establish that classic scalarization can induce negative improvements in some objectives unless covariance is explicitly controlled (Lu et al., 6 Feb 2026). As the number of objectives increases, computational demands for accurate hypervolume or multi-dimensional sorting become substantial. Methods must scale model architectures to facilitate sharing while retaining per-objective specificity.

Automated, dynamic weighting schemes—including those leveraging hypervolume derivatives or gradient covariance—represent the forefront of overcoming these limitations. In large-scale deep learning, such approaches have been empirically shown to yield faster, more robust convergence, and improved generalization across domains as diverse as LLM pre-training, audio-language representation learning, and physical simulation (Mei et al., 18 Jan 2026, Dang et al., 23 Jun 2026).

Future work is likely to further hybridize dynamic weighting, surrogate modeling, and collaborative architectures, with a focus on scaling to many objectives (NSGA-III class algorithms) and leveraging automated front analysis (hypervolume, IGD, diversity) for both selection and termination. These advances are poised to make multi-objective optimization an integral, scalable component of machine learning and scientific discovery pipelines.