ASR Loss: Accuracy, Stability & Robustness

• Accuracy–Stability–Robustness (ASR) loss is a composite loss function that integrates predictive accuracy, input/output stability, and model robustness, with clear mechanisms for margin control.
• It unifies SP losses, TRADES trade-offs, and Lyapunov-based analyses to enforce principled boundary geometry and provide robust stability against adversarial perturbations.
• Optimization employs methods like boundary sampling and PGD-style local supremum to effectively balance clean accuracy with enhanced defensive performance.

An Accuracy–Stability–Robustness (ASR) loss is a composite loss function designed to jointly optimize predictive accuracy, input/output stability, and model robustness—particularly in the face of adversarial perturbations. Rooted in recent advances linking robust deep learning, boundary geometry, and dynamical stability, the ASR loss formalism integrates insights from stationary-point (SP) losses, theoretically principled tradeoffs between adversarial error and classification error, and Lyapunov-based stability analysis for neural ODEs. ASR losses provide explicit mechanisms for margin maximization, control of loss landscape sharpness, and boundary regularization, and are formulated to address the intrinsic tension among accuracy, stability, and robustness in modern deep learning (Gao et al., 2023, Zhang et al., 2019, Luo et al., 26 Sep 2025).

1. Mathematical Formalizations and Key Families

Three dominant constructions of ASR loss appear in the literature, each corresponding to a different modeling principle and stability guarantee.

Stationary Point (SP) Losses

The SP loss family modifies standard cross-entropy (CE) or focal losses by adding a regularizer $\mathcal{R}(\xi)$ to create one or more stationary points $\xi^*$ with $\xi_{y}^*>1/K$. In particular: $$\mathcal{L}_{\rm SP}(\xi, y) = L_{\rm CE}(\xi, y) + \eta\,\mathcal{R}(\xi)$$ For instance, the SP–CE loss

$$\mathcal{L}_{\rm SPCE}(\xi, y) = -\ln\xi_y + \eta(\|\xi\|^2 + (1-\xi_y)^2),\,\quad\eta>0.5$$

guarantees a stationary point in the correct classification regime. Similarly, the SP–focal loss

$$\mathcal{L}_{\rm SPFL}(\xi, y) = -\alpha(1-\xi_y)^\gamma\ln\xi_y + \eta\|\xi\|^2,\quad\alpha,\gamma,\eta>0$$

ensures the same. The stationary point regularizer prevents divergence of last-layer weights and enforces the boundary to pass through inter-class midpoints, maximizing margin (Gao et al., 2023).

TRADES and Surrogate-Based ASR Losses

In the TRADES framework, the robust (adversarial) error is exactly decomposed as

$$\varepsilon_{\rm rob}(f) = \varepsilon_{\rm nat}(f) + \varepsilon_{\rm bdy}(f)$$

The TRADES objective directly trades off accuracy and stability: $$\mathbb{E}_{(X,Y)}\Big[ L(f(X), Y) + \frac{1}{\lambda} \max_{X'\in B(X, \varepsilon)} L(f(X), f(X')) \Big]$$ where $L$ is a classification-calibrated surrogate (e.g., cross-entropy), and the second term encourages local consistency near each input, thus discouraging boundary proximity to the data manifold and enhancing stability (Zhang et al., 2019).

Dynamical and Lyapunov-Based ASR Losses

In the Zubov-Net paradigm for neural ODEs, tripartite losses are organized as: $$\mathcal{L}_{\rm ASR} = \mathcal{L}_{\rm cla} + \lambda_1 \mathcal{L}_{\rm FC} + \lambda_2 \mathcal{L}_{\rm con} + \lambda_3 \mathcal{L}_{\rm sep}$$ Here, $\mathcal{L}_{\rm FC}$ is standard classification (e.g., CE), $\mathcal{L}_{\rm cla}$ is a Lyapunov-induced stability-guaranteeing loss, $\mathcal{L}_{\rm con}$ aligns prescribed and true regions of attraction (RoAs) using Zubov’s equation, and $\mathcal{L}_{\rm sep}$ penalizes boundary overlap between classes—thereby enforcing geometric separation and robustness (Luo et al., 26 Sep 2025).

2. Theory: Stationary Points, Margins, and Boundary Geometry

A unifying property of ASR losses is the explicit control of decision boundary geometry via stationary points and margin regularization.

• Stationary Point Losses: CE has no stationary point in the correct classification regime ($\xi_y>1/K$), causing overconfident sharpening at the expense of margin width. The addition of a sufficiently strong regularizer $\eta\,\mathcal{R}(\xi)$ creates a finite stationary point where the loss gradient vanishes, leading to convergence of weights and enlarged margins.
• Margin Maximization: In both binary and multiclass settings, an SP-loss-trained classifier’s boundary passes exactly through the midpoint in feature space between class representatives—provably maximizing the margin, as shown by direct analysis of the global optima (Gao et al., 2023).
• Boundary Error Decomposition: TRADES formalizes this: robust error equals the sum of natural error and the probability mass near the boundary; minimizing both components via calibrated surrogates and a stability term offers the tightest differentiable upper bound on adversarial risk (Zhang et al., 2019).

3. Structure of the Composite Accuracy–Stability–Robustness Loss

An abstract accuracy–stability–robustness loss can be written as: $$\mathcal{L}_{\rm ASR} = L_{\rm acc}(\xi, y) + \lambda_s \|\nabla_x L_{\rm acc}\|^2 + \lambda_r \mathcal{R}_{\rm SP}(\xi)$$

• $L_{\rm acc}$: accuracy term (CE or focal loss).
• $\|\nabla_x L_{\rm acc}\|^2$: stability penalty, controlling local sensitivity to input perturbation.
• $\mathcal{R}_{\rm SP}(\xi)$: SP regularizer, enforcing margin and robust boundary.

In Zubov-Net, this loss is further split into:

• $\mathcal{L}_{\rm FC}$ and $\mathcal{L}_{\rm cla}$: accuracy and Lyapunov-induced classification.
• $\mathcal{L}_{\rm con}$: trajectory-level stability via Zubov consistency.
• $\mathcal{L}_{\rm sep}$: geometric separation via boundary regularization. Parameter trade-offs among these terms (via $\lambda$’s) directly modulate the accuracy-robustness-stability spectrum (Gao et al., 2023, Luo et al., 26 Sep 2025).

4. Optimization Algorithms and Practical Construction

Optimization typically involves stochastic gradient descent (SGD) variants over the composite loss. Notable pipeline components include:

• Boundary Sampling: Zubov-Net samples boundary points on PRoA surfaces by parallel multi-ray binary search, inserting them into $\mathcal{L}_{\rm sep}$ to explicitly widen margins via convex Lyapunov functionals.
• PGD-Style Local Supremum: Adversarial/trajectory states are used in $\mathcal{L}_{\rm con}$ to maximize alignment between prescribed and true RoAs.
• Attention and Convexity: Lyapunov energy functions are implemented with input-attention convex NNs (IACNNs), with an $\ell_2$ term to guarantee strong convexity and stable separation.
• Hyperparameter Tuning: The balance between clean accuracy and adversarial robustness is set by $\lambda_s$, $\lambda_r$, or the trade-off parameter $\lambda$ in TRADES: larger values shift training from prioritizing accuracy to favoring stability and robustness (Gao et al., 2023, Luo et al., 26 Sep 2025).

5. Theoretical Guarantees

Rigorous analytical results underpin each major ASR component:

• SP Losses: Lemma 1 and Theorem 2 in (Gao et al., 2023) prove that stationary-point minimization enforces maximal margin, with boundaries bisecting inter-class features.
• Robust Error Decomposition: TRADES (Zhang et al., 2019) proves that minimizing the ASR surrogate achieves the tightest possible differentiable upper bound on adversarial error. No uniformly better surrogate exists.
• Lyapunov/ODE Stability: Zubov-Net (Luo et al., 26 Sep 2025) offers:
  • Consistency (Prop. 2): vanishing $\mathcal{L}_{\rm con}$ yields exact prescribed RoA alignment.
  • Non-overlap (Prop. 3) and trajectory stability (Prop. 4).
  • Convex separability in high dimensions (Prop. Convex Separability), supporting the use of convex Lyapunov networks for improved boundary separation and class discrimination.
• Control of Sensitivity: SP and TRADES losses result in sharper and deeper basins in the loss landscape, reducing sensitivity to input and parameter perturbations.

6. Empirical Properties and Observed Trade-Offs

Experimental results across benchmarks highlight the following:

• Accuracy Retention: SP and TRADES models maintain near-CE accuracy on clean data.
• Robustness Increase: SP–focal loss improves adversarial accuracy by 20–50 points over CE/focal at $\varepsilon=8/255$ on standard benchmarks; Zubov-Net offers major gains against noise/adversarial perturbations (Gao et al., 2023, Luo et al., 26 Sep 2025).
• Stability: SP and Lyapunov-based networks evidence smaller changes in logits under small input or parameter changes, quantifying higher stability.
• Loss Landscape: SP/ASR loss landscapes are sharper but deeper, with a pronounced minimum around robust solutions, favoring generalization under adversarial shift.
• Sample Imbalance: SP boundaries remain at class midpoints under severe imbalance, mitigating sampling bias—unlike standard CE which shifts toward the minority class.

Loss Family	Accuracy (Clean)	Robustness (Adversarial)	Stability
CE/Focal	High	Low	Low (sharp/confident)
SP/ASR	High (~CE)	High	High (smooth/robust)
Zubov-Net	High	High	High (Lyapunov stable)

7. Extensions, Open Challenges, and Current Directions

Current challenges and future directions include:

• Adaptive Weighting: Determining optimal schedules for $\lambda$’s during training remains open; dynamic or data-dependent strategies are underexplored (Gao et al., 2023).
• Stochastic Optimization Guarantees: Precise convergence properties of ASR losses under mini-batch SGD are not fully characterized.
• Scalability: Behavior and scalability in regimes with very high number of classes (e.g., ImageNet-1k) require further empirical and theoretical scrutiny.
• Generalization of Regularizers: Exploration of alternative SP/robustness regularization, possibly leveraging SVM/bi-tempered margin terms, beyond simple $\ell_2$ (Gao et al., 2023).
• Dynamical Models: Lyapunov-based geometric control is specific to neural ODEs; broader applicability to conventional architectures is a subject of ongoing study (Luo et al., 26 Sep 2025).

A plausible implication is that ASR losses, by unifying concepts from geometric regularization, calibration theory, and dynamical stability, provide a principled, adaptable framework for balancing accuracy, stability, and robustness for safety-critical, adversarially exposed, and imbalanced machine learning domains.