Multi-Objective Parallel Losses

• Multi-objective parallel losses are optimization methods that handle several distinct loss functions simultaneously to approximate diverse points on a sample’s Pareto front.
• Techniques like hypervolume maximization convert multiple objectives into dynamic scalar losses, ensuring well-spread coverage and robust trade-offs in complex models.
• Gradient-based methods such as MGDA and EDM, along with parallel architectures, enable efficient joint optimization while accommodating varying loss scales and computational constraints.

Searching arXiv for recent and foundational papers relevant to multi-objective parallel losses. arXiv_search tool not available in this environment. Proceeding by grounding the article strictly in the supplied arXiv records and details. Multi-objective parallel losses denote training or estimation settings in which multiple, potentially conflicting losses are handled as a multi-objective optimization problem rather than collapsed a priori into a single fixed scalar objective. In the cited literature, this appears in several forms: a bundle of $p$ networks trained so that their predictions jointly approximate distinct points on each sample’s Pareto front; high-dimensional penalized regression in which $K$ penalized loss functions are fit entirely in parallel and then aggregated by a vote procedure; multi-task and adversarial training in which several task or discriminator losses define a vector objective; and end-to-end decision-focused pipelines in which landscape loss, Pareto set loss, and decision loss are optimized jointly (Deist et al., 2021, Dai et al., 2020, Sener et al., 2018, Albuquerque et al., 2019, Li et al., 2024). Across these formulations, the common organizing idea is Pareto optimality: a solution is preferred when no other solution improves one objective without worsening another.

1. Pareto formulation and the meaning of “parallel” losses

A standard formulation collects $n$ losses for a sample $s$ into the vector

$$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$

A solution $\theta$ is Pareto-optimal for $s$ if there is no other $\theta'$ for which $L_{i}(\theta',s)\le L_{i}(\theta,s)$ for all $i$ and a strict inequality in at least one $K$0; the set of all such loss-vectors is the Pareto front for sample $K$1 (Deist et al., 2021). In multi-task learning, the same principle is written as minimizing the vector $K$2 “in the sense of Pareto-optimality” (Sener et al., 2018). In GAN training with multiple discriminators, the generator update is posed as

$K$3

again as a multi-objective problem (Albuquerque et al., 2019).

The “parallel” aspect has two distinct meanings in the literature. First, multiple losses can be optimized jointly but retained as separate objectives, as in multi-task learning, multi-discriminator GANs, and hypervolume-based Pareto-front prediction (Sener et al., 2018, Albuquerque et al., 2019, Deist et al., 2021). Second, the losses can be optimized separately in embarrassingly parallel fashion and only later aggregated. In penalized regression with multiple loss functions, each preliminary estimator

$K$4

depends only on $K$5, so one may fit the $K$6 penalized regressions entirely in parallel (Dai et al., 2020).

A recurrent contrast is with weighted linear combinations of losses. The cited works repeatedly state that a proxy objective based on a weighted sum is often inadequate when objectives compete or when one seeks broad coverage of the Pareto front rather than one compromise point (Sener et al., 2018, Deist et al., 2021). This suggests that “parallel losses” are not merely multiple addends in a scalar objective; they are losses whose separate structure is preserved during optimization or post hoc aggregation.

2. Hypervolume maximization as dynamic loss weighting

One influential approach is to convert multiple losses into dynamic scalar objectives using hypervolume-maximizing gradients. For a reference point $K$7 that dominates all loss-vectors of interest, the dominated hypervolume of a single loss-vector $K$8 is

$K$9

For $n$0 loss-vectors $n$1,

$n$2

Maximizing this hypervolume drives the points to be on and spread out along the Pareto front (Deist et al., 2021).

The gradient of the hypervolume with respect to network parameters is decomposed by the chain rule as

$n$3

and the partial derivatives with respect to each objective become dynamic weights,

$n$4

These define the scalar “joint” loss for network $n$5 on sample $n$6,

$n$7

followed by the stochastic-gradient update

$n$8

In practice, one first non-dominated sorts the $n$9 loss-vectors into fronts; only the first front has nonzero HV-gradients, trailing fronts are optimized in turn, and gradients are normalized so that $s$0 before being used as weights (Deist et al., 2021).

A closely related but simpler hypervolume construction appears in multi-discriminator GANs. With $s$1 discriminator losses $s$2 and a nadir point $s$3 such that $s$4, the hypervolume is

$s$5

equivalently minimizing

$s$6

Its gradient is

$s$7

so higher losses $s$8 get higher weight (Albuquerque et al., 2019).

The per-sample emphasis is central in Pareto-front prediction. The cited experiments and discussion argue that training processes to approximate Pareto fronts need to optimize on fronts of individual training samples instead of on only the front of average losses, because averaging HV-gradients across samples may collapse diversity and can fail on curved or asymmetric per-sample fronts (Deist et al., 2021).

3. Gradient geometry beyond fixed scalarization

Several works formulate multi-objective parallel losses directly in gradient space. A necessary Pareto-stationarity condition is the existence of coefficients in the simplex such that the weighted sum of gradients vanishes. In multi-task learning, the shared-parameter condition is

$s$9

and otherwise the minimizer of

$$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$0

gives a common descent direction that simultaneously reduces all $$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$1 losses (Sener et al., 2018). The same convex quadratic program appears in multiple-discriminator GANs as Multiple Gradient Descent (MGDA), where one solves for the smallest-norm convex combination of the discriminator-loss gradients and updates with the resulting common descent direction (Albuquerque et al., 2019).

The Equiangular Direction Method (EDM) modifies this geometry by normalizing each gradient,

$$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$2

then solving

$$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$3

For active coefficients, the resulting direction is equiangular to all gradients, and the method is designed to guarantee equal relative decrease of objective functions rather than equal absolute decrease (Katrutsa et al., 2020). The paper emphasizes that this is appropriate for multi-objective optimization problems with multi-scale losses (Katrutsa et al., 2020).

A different scalarization mechanism appears in parallel multi-objective Bayesian optimization for hyperparameter optimization. At each iteration, weights $$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$4 are drawn uniformly from the unit simplex by

$$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$5

where $$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$6, and the scalarized acquisition target is

$$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$7

Re-sampling $$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$8 each iteration encourages uniform coverage of different Pareto regions and avoids clustering on the PF (Egele et al., 2023). The same work normalizes each objective to $$\mathcal{L}(\theta,s)=\bigl[L_{1}(\theta,s),\,L_{2}(\theta,s),\dots,L_{n}(\theta,s)\bigr]^{\mathsf T}.$$9 via its empirical CDF,

$\theta$0

a “Quantile–Uniform” mapping that is order-preserving and guarantees Pareto-set invariance (Egele et al., 2023).

Taken together, these methods show that multi-objective parallel losses are not tied to one update rule. The literature includes hypervolume-based weighting, min-norm common descent, equiangular descent based on normalized gradients, and randomized scalarization over normalized objectives (Deist et al., 2021, Sener et al., 2018, Katrutsa et al., 2020, Egele et al., 2023).

4. Parallel architectures and algorithmic realizations

Parallelism enters both at the level of loss-specific optimization and at the level of system architecture. In penalized regression with multiple loss functions, a generic coordinate-descent or proximal-gradient loop is run independently for each loss $\theta$1, and modern implementations exploit warm starts, active-set updates, and are easily “forked” over $\theta$2 (Dai et al., 2020). The outputs are then aggregated through the vote-aggregated support

$\theta$3

so a predictor enters the final model if it is selected by at least $\theta$4 out of $\theta$5 fits (Dai et al., 2020).

In Pareto-front prediction with a bundle of networks $\theta$6, the high-level training algorithm for each minibatch is explicit: compute each loss-vector $\theta$7, stack the $\theta$8 vectors, non-dominated sort into fronts, compute HV-gradients for each front, form weighted scalar losses for each network and sample, then backpropagate the sum of weighted losses over the minibatch and update all $\theta$9 (Deist et al., 2021). This is a parallel-loss construction because the $s$0 networks jointly approximate $s$1 distinct points on each sample’s front (Deist et al., 2021).

For hyperparameter optimization, the architecture is decentralized and asynchronous. Each of $s$2 workers independently reads newly completed evaluations from shared storage, normalizes objectives by ECDF, adds penalties for soft upper-bound violations, samples a new simplex weight vector, updates its local RF surrogate, proposes the next configuration using an LCB acquisition rule with decaying exploration parameter, evaluates it in parallel, and writes the result back to shared storage (Egele et al., 2023). The reported speed-up arises because all $s$3 workers continuously evaluate BO suggestions without waiting for a central manager (Egele et al., 2023).

M-HOF-Opt frames multiplier adaptation itself as a closed loop. The surrogate single-objective penalty loss is

$s$4

with time-varying multipliers updated by

$s$5

$s$6

The method dispatches the multi-objective descent goal hierarchically into constrained subproblems with shrinking bounds according to Pareto dominance, and adds only $s$7 scalar operations per epoch beyond a standard single-objective optimizer (Sun et al., 2024).

In end-to-end decision-focused learning, three losses are combined,

$s$8

and gradients are propagated through a differentiable solver using entropic-OT for the landscape term and the implicit function theorem for the decision and Pareto-set terms (Li et al., 2024). This is parallel in a different sense: multiple losses are retained simultaneously and differentiated through the same pipeline.

5. Statistical and decision-theoretic roles of multiple losses

In high-dimensional linear models, multiple loss functions are used both to select relevant predictors and to estimate parameters. Typical convex losses include least-squares, least-absolute-deviation, and quantile check losses (Dai et al., 2020). The vote procedure is intended to reduce false positives in variable selection, while aggregated estimators on the true support

$s$9

have asymptotic normality, and the minimal variance is achieved by

$\theta'$0

under the stated assumptions (Dai et al., 2020). In the quantile-regression special case, with $\theta'$1, the asymptotic variance approaches the Fisher information and hence semiparametric efficiency as $\theta'$2 (Dai et al., 2020).

In decision-focused learning, multiple losses serve different semantic roles. Landscape loss compares the shape of the predicted objective landscape to the true landscape through sRMMD on a cached solution set. Pareto-set loss measures Euclidean solution-space distance between the single predicted solution and samples from the true Pareto set. Decision loss measures average true-objective regret of the representative solution extracted from the predicted problem after a uniform weighted-sum scalarization (Li et al., 2024). The three terms therefore distinguish objective-space geometry, decision-space proximity, and decision quality (Li et al., 2024).

A plausible implication is that “multi-objective parallel losses” need not always mean “one optimizer, many gradients.” In some settings, such as penalized regression, the methodological gain comes from running several penalized problems separately and aggregating them statistically; in others, the gain comes from maintaining multiple losses throughout an end-to-end differentiable training loop (Dai et al., 2020, Li et al., 2024).

6. Empirical behavior across domains

The empirical literature emphasizes diversity of trade-offs, robustness to asymmetry, and computational efficiency. In Pareto-front prediction, experiments on toy regression, multi-observer medical segmentation, neural style transfer, and multi-style transfer showed that hypervolume maximization returns outputs that are well-spread across different trade-offs on the approximated Pareto front without requiring the trade-off vectors to be specified a priori; the added value was highlighted especially in asymmetric Pareto fronts (Deist et al., 2021). On the toy regression problem, all methods performed similarly on symmetric fronts, but as soon as one loss was an $\theta'$3 or scaled MSE, only HV-maximization maintained uniform coverage, with statistically significant improvement in HV (Deist et al., 2021).

In multi-discriminator GANs, multiple gradient descent achieved the best median FID on MNIST but ran about $\theta'$4 slower, while hypervolume maximization reached FID close to MGDA without a QP subproblem (Albuquerque et al., 2019). On upscaled CIFAR-10 with $\theta'$5 discriminators, the reported values were $\theta'$6 and $\theta'$7, compared with SNGAN values of about $\theta'$8 and $\theta'$9; on stacked MNIST, HV with 16 discriminators covered about $L_{i}(\theta',s)\le L_{i}(\theta,s)$0 modes with reverse KL about $L_{i}(\theta',s)\le L_{i}(\theta,s)$1 (Albuquerque et al., 2019). The paper attributes part of the appeal of HV to a better compromise between sample quality and computational cost (Albuquerque et al., 2019).

For parallel multi-objective Bayesian optimization on the Combo benchmark, scaling from $L_{i}(\theta',s)\le L_{i}(\theta,s)$2 GPUs reduced the time to reach the final 40-worker Pareto quality by about $L_{i}(\theta',s)\le L_{i}(\theta,s)$3, improved area-under-HVI-curve from $L_{i}(\theta',s)\le L_{i}(\theta,s)$4, and increased final hypervolume from $L_{i}(\theta',s)\le L_{i}(\theta,s)$5 (Egele et al., 2023). NSGA-II also benefited from parallelism, but the reported comparison stated that model-based MoBO shows higher iteration efficiency, whereas MoTPE and random search scale poorly in solution quality (Egele et al., 2023).

In multi-task learning, MGDA-UB improved over uniform weighting, uncertainty weighting, and GradNorm on MultiMNIST, CelebA, and Cityscapes, while substantially reducing overhead compared with full MGDA; for the 40-task multi-label setting, the reported training time dropped from $L_{i}(\theta',s)\le L_{i}(\theta,s)$6 without UB to $L_{i}(\theta',s)\le L_{i}(\theta,s)$7 with UB (Sener et al., 2018). EDM was reported to preserve high accuracy under severe loss-scale imbalance on MultiMNIST when the second loss was multiplied by $L_{i}(\theta',s)\le L_{i}(\theta,s)$8, whereas MGDA collapsed (Katrutsa et al., 2020). In domain generalization with DIVA on PACS, M-HOF-Opt yielded median accuracy about $L_{i}(\theta',s)\le L_{i}(\theta,s)$9–$i$0 with small spread and was far more stable across sampled $i$1 and $i$2 than manual warm-up or fixed-multiplier baselines (Sun et al., 2024). In decision-focused learning, the proposed MoDFL method beat two-stage and all single-objective DFL baselines on every reported metric—GD, MPFE, HAR, and average $i$3regret—on web-advertisement allocation and citation-graph bipartite matching (Li et al., 2024).

7. Limitations, misconceptions, and open directions

A common misconception is that any weighted sum of losses constitutes a satisfactory multi-objective treatment. The cited works repeatedly qualify this: weighted sums produce one compromise, can be dominated by scale differences, and may fail to recover diverse regions of the Pareto front or behave poorly on asymmetric or nonconvex fronts (Sener et al., 2018, Katrutsa et al., 2020, Deist et al., 2021). Another misconception is that averaging losses over samples suffices for front approximation; the per-sample hypervolume work explicitly argues that only per-sample HV maximization guarantees well-spread coverage for each sample’s front (Deist et al., 2021).

The methods also have clear limitations. Exact hypervolume is exponential in the number of objectives $i$4, although for moderate $i$5 and $i$6 the reported overhead is small compared to backpropagation (Deist et al., 2021). Per-sample HV gradients permit different networks to occupy different trade-off positions per sample, so the ordering of networks along the front may vary across samples; in applications requiring consistent ordering at inference, the cited remedies are average-loss HV or post-sorting the outputs (Deist et al., 2021). In GANs, MGDA requires solving a QP each batch and becomes prohibitive for large networks, whereas increasing the number of discriminators improves FID, IS, and sample diversity but increases compute roughly linearly (Albuquerque et al., 2019). In hyperparameter optimization, higher worker count increases exploration but raises the risk of redundant evaluations, penalty bounds must be set with domain insight, and overly tight bounds can rule out useful regions (Egele et al., 2023). In decision-focused learning, a finite sample of the Pareto set may miss some frontier modes, and weighted-sum scalarization only recovers the convex hull of the Pareto front; the paper therefore notes Chebyshev- or epsilon-constraint–based alternatives as possible extensions (Li et al., 2024). For M-HOF-Opt, a full formal proof of global convergence is left to future work (Sun et al., 2024).

The open directions named in the cited works are correspondingly diverse: approximate hypervolume by random scalarizations or Monte-Carlo HV estimates; coupling networks through shared layers to reduce computation; alternative differentiable multi-objective solvers; adaptive scalarization weights per instance; and extensions to many-objective, mixed-integer, nonlinear, or sequential-decision settings (Deist et al., 2021, Li et al., 2024). This suggests that multi-objective parallel losses are best understood not as one algorithm, but as a family of formulations in which multiple losses remain structurally present throughout optimization, aggregation, or both.