Dual-Balancing Multi-Task Learning

• Dual-Balancing Multi-Task Learning (DB-MTL) is a framework that harmonizes discrepancies in loss scales, gradient magnitudes, and structural heterogeneity across tasks.
• Key methodologies include logarithmic loss transformation, gradient normalization, bi-level optimization, and dual-encoder frameworks, ensuring balanced and efficient task learning.
• Empirical evaluations on benchmarks like NYU-v2 and Cityscapes show that DB-MTL techniques reduce error rates and computation time compared to traditional multi-task approaches.

Dual-Balancing Multi-Task Learning (DB-MTL) refers to a class of methods in multi-task learning (MTL) that jointly address the challenges of task balancing by leveraging dual strategies—optimizing both loss/gradient scale and structural heterogeneity—across tasks. Recent work demonstrates that DB-MTL encompasses approaches ranging from bi-level optimization for gradient balancing to architectures designed for heterogeneous shared/specific representations, and to explicit loss and gradient magnitude normalization. This article synthesizes the major methodological advances, mathematical formulations, and empirical insights from key sources on DB-MTL (Chen et al., 8 Mar 2026, Sui et al., 30 May 2025, Lin et al., 2023).

1. Dual-Balancing: Conceptual Foundations

Dual-Balancing in MTL addresses the core problem that learning simultaneously across tasks can be compromised by disparities in loss scales, gradient magnitudes, or underlying data/model heterogeneity. Three major, complementary forms of "duality" emerge in the literature:

• Loss and Gradient Duality: Simultaneous normalization of per-task loss scales (typically via transformations such as logarithms) and per-task gradient magnitudes (typically ensuring equal contribution in norm to the joint parameter update) (Lin et al., 2023).
• Structural Duality: Explicit modeling of both task-shared and task-specific latent representations, often through dual-encoder architectures (Sui et al., 30 May 2025).
• Optimization Hierarchy Duality: Formulating MTL as a bi-level optimization problem, with upper-level variables (task weights) and lower-level model parameters, capturing both task importance/competition and model fitting (Chen et al., 8 Mar 2026).

This dual perspective enables more impartial convergence and knowledge sharing by harmonizing distinct sources of imbalance or heterogeneity across tasks.

2. Core Methodologies

2.1 Loss-Scale and Gradient-Magnitude Balancing

The "dual balancing" scheme of (Lin et al., 2023) implements two key techniques:

• Logarithmic Loss Transformation: Each task loss $\ell_t$ is replaced by $L_t = \log(\ell_t + \varepsilon)$, with $\varepsilon$ ensuring numerical stability. This compresses the dynamic range of loss scales and, at the gradient level, scales gradients by $1/\ell_t$, automatically counteracting disparities.
• Gradient Normalization: After the log transform, instantaneous per-task gradients $g_{t,k}$ are computed, with exponential moving average filtering. Each is normalized to unit norm and rescaled by the maximal gradient norm $\alpha_k = \max_t \| \hat{g}_{t,k} \|_2$, enforcing equal contribution in shared updates:

$$\tilde{g}_k = \alpha_k \sum_{t=1}^T \frac{\hat{g}_{t,k}}{\|\hat{g}_{t,k}\|_2 + \varepsilon}$$

Pseudocode for this procedure is provided and requires no additional learnable parameters or complex optimization (Lin et al., 2023).

2.2 Bi-level Optimization for Gradient Balancing

The MARIGOLD algorithm (Chen et al., 8 Mar 2026) formalizes MTL gradient balancing as a bi-level (nested) optimization problem:

• Lower Level (LL): Model parameter update for fixed task weights, optimizing the weighted sum of per-task losses.

$$\theta^*(\lambda) = \arg\min_\theta \sum_{i=1}^m \lambda_i f_i(\theta)$$

• Upper Level (UL): Task-weight update minimizing the worst-case or aggregate loss degradation after a one-step update:

$$\min_{\lambda \in \Delta^m} \max_{\rho \in \Delta^m} \sum_{i=1}^m \rho_i [f_i(A(\lambda, \theta^*(\lambda))) - f_i(\theta^*(\lambda))]$$

A notable innovation is the use of zeroth-order (gradient-free) hypergradient estimates, allowing the method to collapse the need for $O(md)$ computation to $O(d)$ per iteration by probing only one random direction in the simplex. This yields substantial empirical speed-ups without loss of performance (Chen et al., 8 Mar 2026).

2.3 Dual-Encoder Framework for Heterogeneity

DB-MTL in heterogeneous MTL settings (Sui et al., 30 May 2025) constructs distinct representations for each task:

• Task-Specific Encoder $S_r$: Captures features unique to task $r$.
• Task-Shared Encoder $C$: Extracts commonalities across tasks.
• Predictive Model: $$E_r[Y_r] = \alpha_r^\top S_r(X_r) + \beta_r^\top C(X_r)$$

Adaptive $\ell_2$ penalties enforce similarity or divergence in coefficient mappings, and an orthogonality penalty minimizes redundancy between shared and task-specific spaces. Optimization alternates between encoder updates and convex coefficient updates, enabling both distribution and posterior heterogeneity to be handled within a unified empirical risk minimization framework.

3. Mathematical Formulation and Algorithms

The following table summarizes the three principal DB-MTL paradigms and their algorithmic strategies:

Approach	Key Balancing Mechanism	Optimization Procedure
Log+Norm (Lin et al., 2023)	Logarithmic loss + gradient normalization	SGD/Adam with normalization
MARIGOLD (Chen et al., 8 Mar 2026)	Bi-level (task-weights / model)	Alternating, with zeroth-order hypergradients
Dual Encoder (Sui et al., 30 May 2025)	Task-shared/task-specific separation	Block coordinate descent (encoders, coefficients)

In each approach, alternating or coupled updates are central, whether in parameter and weight space (MARIGOLD), or between network blocks and coefficient vectors (dual encoder).

4. Theoretical Properties

• Log+Norm DB-MTL: The log-transformation is theoretically motivated as a scale-invariant mapping; gradient normalization ensures no single task dominates the update direction. Empirical ablations confirm the necessity of both components for top performance (Lin et al., 2023).
• Bi-level DB-MTL (MARIGOLD): Lemma 4.1 guarantees the unbiasedness (up to smoothing) of zeroth-order hypergradient estimates. Under smoothness and bounded variance, first- or zeroth-order alternating optimization converges to stationarity at $O(1/\sqrt{K})$ rates (Chen et al., 8 Mar 2026).
• Dual Encoder DB-MTL: Generalization bounds are provided via local Rademacher complexity. The risk bound for the empirical minimizer interpolates between $O(1/\sqrt{N})$ and $O(1/N)$, where $N$ is total sample size. Under mild conditions, the method contracts excess risk relative to single-task or non-heterogeneity-aware baselines (Sui et al., 30 May 2025).

5. Empirical Evaluation and Results

MARIGOLD (Chen et al., 8 Mar 2026):

• Outperforms MGDA, FAMO, and others on NYU-v2 and Cityscapes (dense vision) in mean IoU, pixel accuracy, depth error, and mean rank.
• Achieves 20–50% per-epoch speed-ups over all-task-gradient methods, reducing time from 375s (MGDA) to 152s (MARIGOLD) on NYU-v2.
• In industrial tasks, hyperparameter tuning yields up to 0.14% relative improvement over equal-weighted baselines.

Log+Norm DB-MTL (Lin et al., 2023):

• On NYU-v2 and Cityscapes, achieves positive $\Delta_p$ improvement, the only MTL method to do so consistently.
• Classification and molecular prediction benchmarks show either the highest or second-highest aggregate scores.
• Ablation: Both log-transform and gradient normalization are necessary for optimal gains.

Dual Encoder DB-MTL (Sui et al., 30 May 2025):

• Simulation studies (R=2–5 tasks) show 15–35% reduction in RMSE versus baselines under covariate and/or posterior shift.
• On 5-cancer PDX data, achieves 5–11% lower RMSE than single-task learning, with substantial advantage over other MTL methods.
• t-SNE visualizations and coefficient distance analysis confirm interpretable representation of distribution and mapping similarities.

6. Practical Considerations and Limitations

• Hyperparameters: Log+Norm DB-MTL primarily requires $\beta$ (EMA decay) and learning rate tuning; MARIGOLD requires smoothing radius for zeroth-order estimation; Dual Encoder DB-MTL involves network configurations and $\ell_2$ shrinkage weights (Lin et al., 2023, Chen et al., 8 Mar 2026, Sui et al., 30 May 2025).
• Efficiency: MARIGOLD and Log+Norm DB-MTL are markedly more efficient than all-task-gradient approaches, scaling well to large task counts.
• Interpretability: Dual encoder architectures afford direct diagnosis of covariate and posterior heterogeneity, with clusterable latent factors and coefficient shrinkage trajectories.
• Limitations: Log+Norm DB-MTL does not resolve gradient direction conflicts; bi-level methods' non-convex-concave convergence remains an open area; log transforms may be unstable for very small losses. Extensions to classifier, sequence, and hierarchical task structures are plausible but require adaptation.

7. Extensions and Future Directions

• Conflict Resolution: Augmenting magnitude balancing with sign-based or directionally-aware methods (e.g., PCGrad, GradVac) to resolve antagonistic gradients (Lin et al., 2023, Chen et al., 8 Mar 2026).
• Hierarchical and Structured Sharing: Expanding dual-encoder methodology to partial sharing graphs or trace-norm penalties, enabling discovery of complex inter-task relationships (Sui et al., 30 May 2025).
• Zeroth-Order Methods: Generalizing MARIGOLD's random-projection approach to other nested learning paradigms (meta-learning, RL, differentiable architecture search) (Chen et al., 8 Mar 2026).
• Theoretical Tightening: Further analysis of smoothing errors and convergence rates for zeroth-order and non-convex bi-level optimization.
• Per-Layer/Block Balancing: Applying dual balancing at finer granularity within deep or multi-stream architectures to enhance transfer and performance.

Dual-Balancing Multi-Task Learning, as formalized across these works, establishes a foundation for robust, impartial, and heterogeneous-aware MTL. Through rigorous algorithmic innovation and theoretical underpinning, DB-MTL achieves both practical improvements and methodological unification across the spectrum of contemporary multi-task challenges.