Scale-Adaptive Loss (SAL)

• Scale-Adaptive Loss (SAL) is a technique that dynamically adjusts loss weights in multi-task learning to address scale imbalances from heterogeneous data and objectives.
• SAL methods compute adaptive weights using approaches such as softmax-rate scaling and physics-informed normalization to ensure balanced gradient distribution and stable optimization.
• Empirical results demonstrate that SAL enhances training stability and performance in applications like interatomic potentials, multi-physics PDEs, and computer vision.

Scale-Adaptive Loss (SAL) encompasses a class of techniques for dynamic rescaling or weighting of loss components during the training of machine learning models. These strategies address objective imbalance arising from heterogeneous task magnitudes, data characteristics, measurement units, or other sources of multi-scale variation in multi-objective and multi-task learning contexts. The primary aim is to improve stability, convergence, and generalization by dynamically calibrating the loss landscape, typically in a data-driven and sometimes physically-motivated manner.

1. Problem Motivation and Core Principles

SAL frameworks are motivated by the challenge of training with composite losses where constituent terms have disparate physical units, inherent magnitudes, or convergence behaviors. In neural interatomic potential models, pivotal objectives—such as total potential energy (meV/atom), atomic forces (meV/Å), and stress components (MPa)—exhibit differences in scale spanning several orders of magnitude, complicating manual optimization of static weighting coefficients (Ocampo et al., 26 Mar 2024). Analogously, in multi-objective PDE systems or computer vision, regression and classification objectives have inherently mismatched scales, leading to the dominance of select tasks or instability in gradient-based methods (Xu et al., 27 Oct 2024).

The key principle underlying SAL methodologies is to encode scale-awareness into the loss function through dynamic, context-sensitive weights computed based on property magnitudes, convergence rates, or task-specific statistics. The overarching goal is to promote a balanced optimization trajectory across tasks or objectives by equitably distributing gradient magnitude and learning capacity.

2. SAL Methodologies: Mathematical Frameworks

SAL approaches instantiate in a variety of mathematical forms, each tailored to their application domain and underlying imbalance. Representative frameworks include:

2.1 Softmax-Rate-Based SAL

For composite losses $\mathcal{L} = \sum_k w_k(t) \ell_k(t)$, the weights $w_k(t)$ are derived from the rate of change of the individual loss components. Specifically, for interatomic potentials:

$$w_k(t) = \frac{\exp\left[\beta\,\Delta \ell_k(t)\right]}{\sum_{m=1}^3 \exp\left[\beta\,\Delta \ell_m(t)\right]},\quad\text{where}\quad \Delta\ell_k(t) = \ell_k(t) - \ell_k(t-1)$$

Here, the hyperparameter $\beta$ controls the adaptation speed: larger $\beta$ assigns higher weight to the slowest-converging component, automatically steering the optimizer towards lagging objectives (Ocampo et al., 26 Mar 2024).

2.2 Physics-Informed Scale-Driven SAL

In multi-physics PDE or poroelastography models, each residual is decomposed as

$$\ell_i(\vartheta, v) = \sum_{j=1}^{N_j} f_{ij}(\vartheta) D_{ij}(v)$$

where $f_{ij}(\vartheta)$ carry the physical scales and $D_{ij}(v)$ are differential operators. The corresponding scale for each residual is estimated via

$$e_{ij} = \text{round}\left(\log_{10}|f_{ij}|\right) + \text{round}\left(\log_{10}\langle|D_{ij}|\rangle_\xi\right),\; \beta_i = \frac{1}{N_j}\sum_{j=1}^{N_j} e_{ij},\; w_i = 10^{-\beta_i}$$

The total SAL loss is

$\mathcal{L} = \sum_{i=1}^N w_i(t)\,\ell_i^2$

which ensures each term is order-unity, maintaining uniformly-scaled gradients throughout optimization (Xu et al., 27 Oct 2024).

2.3 Multi-Scale Loss for Detection and Metric Learning

SAL variants in object detection adapt weights per scale-level or per-object, employing statistics such as variance decay (Luo et al., 2021) or area-based attenuation (Li et al., 13 Nov 2025). For metric learning, learnable per-class scale parameters (e.g., $\alpha_c$, $\beta_c$) are incorporated into the loss via exponential or logarithmic functions, with constraints for stability (Jung et al., 2022).

3. Algorithmic Implementation

The algorithmic structure of principal SAL methods can be distilled as follows:

Step	Softmax-Rate SAL (Ocampo et al., 26 Mar 2024)	Physics-driven SAL (Xu et al., 27 Oct 2024)
1. Forward Pass	Compute $\ell_k(t)$	Compute residuals $\ell_i$
2. Compute Scaling	$\Delta\ell_k=\ell_k-\ell_k'$	$w_i=10^{-\beta_i}$
3. Update Weights	$w_k\leftarrow$ softmax ($\beta\Delta\ell_k$)	$w_i$ from algebraic scale
4. Form Total Loss	$L=\sum_k w_k\,\ell_k$	$\mathcal{L}=\sum_i w_i\,\ell_i^2$
5. Backpropagation	Compute $\nabla_\theta L$	Compute $\nabla_w \mathcal{L}$

For physics-based SAL, the weights are computed from physical magnitudes and operator averages without a sub-optimization loop, whereas rate-based SAL updates rely on the observed loss dynamics over time.

4. Theoretical Rationale and Gradient Analysis

The theoretical foundation of SAL methods lies in improved conditioning of the loss landscape and balanced optimization dynamics:

• Gradient Uniformity: By algebraically rescaling each loss component, the gradient with respect to network parameters becomes uniformly distributed across tasks:

$$\frac{\partial \mathcal{L}}{\partial w} = \sum_{i=1}^N 2 w_i\,\ell_i\,\frac{\partial \ell_i}{\partial w} + \ell_i^2\,\frac{\partial w_i}{\partial w}$$

If $f_{ij}$ are Lipschitz in parameters, $w_i, \ell_i = O(1)$, ensuring none of the loss components dominates the gradient flow (Xu et al., 27 Oct 2024).

• Responsiveness Control: Hyperparameters such as $\beta$ in the Softmax-Rate SAL tune responsiveness. Small $|\beta|$ yields slow adaptation (almost uniform weighting), while high $\beta$ leads to rapid, potentially oscillatory reallocation among objectives (Ocampo et al., 26 Mar 2024).
• Automatic Scale Normalization: SAL methods obviate the need for manual grid search or hand-crafted weighting heuristics by leveraging observed statistics or physical dimensionality directly.

5. Empirical Performance and Robustness

Empirical studies corroborate the efficacy of SAL, particularly in settings with severe objective imbalance:

• In neural network interatomic potentials, SAL achieved simultaneous minimization of energy, force, and stress RMSEs—matching or surpassing the best fixed-weight specialization in all objectives with a single model. Performance was robust with respect to initialization and rapidly converged to balanced weights over training epochs (Ocampo et al., 26 Mar 2024).
• In multi-scale PDE inversion, SAL ensured that each physics residual contributed comparably to optimization, enabling stable training in regimes where prior approaches—such as fixed manual weights or heuristics—struggled with convergence or led to gradient domination (Xu et al., 27 Oct 2024).
• Across computer vision and metric learning applications, adaptive scale-driven losses corrected class or object-level imbalances and yielded consistently stronger or more stable discrimination metrics than corresponding fixed hyperparameter baselines (Li et al., 13 Nov 2025, Jung et al., 2022).

6. Comparison to Related Adaptive Balancing Methods

SAL can be viewed in relation to prominent alternatives:

• GradNorm: Applies a sub-optimization to enforce gradient norm parity across objectives, requiring additional computational cost and potentially missing physically meaningful scaling (Xu et al., 27 Oct 2024).
• SoftAdapt: Allocates weight based on per-task loss decay rates via a softmax; while automatic and lightweight, it can conflate scale with convergence speed and is agnostic to physical units (Xu et al., 27 Oct 2024, Ocampo et al., 26 Mar 2024).
• Manual or Static Weighting: Lacks responsiveness to dynamic training behaviors or scale disparities, often resulting in inefficient learning and poor task trade-offs.

A comparative implication is that SAL, especially under physics-driven scale inference, achieves immediate and principled normalization rooted in domain knowledge, without auxiliary optimization or strong inductive biases.

7. Domain-Specific Variants and Future Extensions

SAL paradigms admit several context-driven constructions:

• Physics-based Normalization: Extends naturally to multi-physics and multi-scale systems, with scaling layers mapping MLP outputs to heterogeneous real-world quantities (Xu et al., 27 Oct 2024).
• Loss Landscape Smoothing and Metric Alignment: Reinforcement learning schemas can meta-learn loss parameterizations to align better with downstream evaluation metrics, facilitating smoother optimization and enhanced sample efficiency in metric learning (Huang et al., 2019).
• Mixed Precision Training: Layerwise adaptive loss scaling suppresses FP16 underflow/overflow, yielding hyperparameter-free, robust performance in resource-constrained training regimes (Zhao et al., 2019).
• Object Detection and Segmentation: Object- or region-scale-aware SALs improve detection accuracy by redressing object-size-induced bias, with negligible computational overhead (Li et al., 13 Nov 2025).
• Multi-head or Anytime Networks: SAL balances early and late classifier heads by weighing loss terms inversely proportional to estimated running averages, optimizing geometric mean error and consistently enhancing early-exit performance (Hu et al., 2017).

A plausible implication is that future work will synthesize scale-adaptive losses with automatic task weighting, differentiable architecture search, and hybrid priors, further automating and stabilizing composite-task learning across scales and modalities.