Two-Phase Loss Function in Neural Networks

• Two-phase loss function is a structured neural optimization technique that decomposes the loss into two non-negative components to stabilize training and enforce auxiliary constraints.
• It alternates between phases, such as restoration and optimization, leveraging different loss regimes (e.g., SSE to CE, PINN strategies) to improve convergence dynamics.
• Empirical studies show that two-phase strategies yield enhanced accuracy, faster convergence, and improved constraint satisfaction over conventional single-phase loss methods.

A two-phase loss function refers to a structured approach in neural network optimization where the loss is explicitly decomposed into two distinct components, each minimized through an alternating or staged regimen. This paradigm appears in multiple settings, including constrained neural optimization (such as Physics-Informed Neural Networks, PINNs), hybrid loss regimes in classification, and multi-scale mechanisms that regulate training focus across data subsets. Two-phase strategies are designed to capitalize on complementary optimization dynamics, stabilize training, or prioritize model generalization and constraint satisfaction.

1. Mathematical Formulation and Scope

The canonical formulation presents the total loss as a sum of non-negative terms: $L(\theta) = f_1(\theta) + f_2(\theta)$ where $f_1(\theta) \geq 0$, $f_2(\theta) \geq 0$ for all $\theta \in \mathbb{R}^n$, and no explicit coupling constraints are imposed beyond nonnegativity (López, 2024). The objective is typically to find $\theta^*$ that (locally) minimizes $L(\theta)$, with an implicit feasibility requirement $f_2(\theta) \to 0$ (e.g., enforcing satisfaction of a PDE or other auxiliary condition).

This two-term loss-splitting generalizes to:

• Hybridizations between loss classes (e.g., SSE and cross-entropy) in classification (Dickson et al., 2022).
• Multi-scale loss adaptation for focusing gradient energy on low-confidence data examples (Berlyand et al., 2021).

2. Inexact Restoration and Alternating Descent

The most formal two-phase structure is instantiated via Inexact Restoration (IR), adapted from constrained nonlinear programming: $$\min_{x} f(x) \quad \text{subject to} \quad h(x) = 0$$ The neural variant equates $f(x)$ with $f_1(\theta)$, and the constraint $f_1(\theta) \geq 0$0 with $f_1(\theta) \geq 0$1. The algorithm alternates two procedures at each iterate $f_1(\theta) \geq 0$2 (López, 2024):

• Phase I (Restoration): Reduce the infeasibility $f_1(\theta) \geq 0$3 by finding $f_1(\theta) \geq 0$4 such that $f_1(\theta) \geq 0$5 for some $f_1(\theta) \geq 0$6.
• Phase II (Optimization): Accept steps only if both the primary loss decreases and the penalty/merit function

$f_1(\theta) \geq 0$7

is strictly reduced, with proper update rule for $f_1(\theta) \geq 0$8 that ensures monotonicity.

This method provides descent guarantees on either $f_1(\theta) \geq 0$9 or $f_2(\theta) \geq 0$0 at every step; limit points are KKT solutions of the constrained problem. Algorithmic details, pseudocode structure, and parameter settings are documented in the cited work, with Phase alternations and acceptance criteria (steps 1–4) explicitly controlling convergence and stability.

3. Hybrid Loss Regimes in Classification

An alternative to simultaneous summation is an explicit two-phase temporal regime where optimization switches from one loss to another based on training progress indicators (Dickson et al., 2022). For example, the reactive SSE→CE strategy:

• Phase I: Train with Sum Squared Error (SSE) for a fixed warm-up (e.g., 20 epochs).
• Phase II: Switch to Cross-Entropy (CE) when progress stalls (patience window, usually 3 epochs with no accuracy gain).

This method exploits the landscape properties of SSE (wider, flatter minima) for exploration, then transitions to CE for exploiting sharp minima. Empirical benchmarks demonstrate that this regime consistently achieves higher or statistically indistinguishable best test accuracy on several benchmark datasets compared to either loss alone.

Regime	Cancer	Glass	Diabetes	MNIST	Fashion-MNIST
CE-only	0.9782	0.6443	0.7585	0.9140	0.8600
SSE-only	0.9757	0.6594	0.7671	0.9177	0.8630
SSE→CE (reactive)	0.9780	0.6689	0.7708	0.9310	0.8655

Patience and warm-up hyperparameters require tuning per problem; no formal landscape adaptation guarantees are proven.

4. Multi-Scale and Task-Adaptive Loss Mechanisms

Two-phase concepts extend to multi-scale loss construction, explicitly targeting subsets of data points that are less confidently classified (Berlyand et al., 2021). Here, the per-example loss is: $f_2(\theta) \geq 0$1 where $f_2(\theta) \geq 0$2 splits data into low-confidence and high-confidence sets, and $f_2(\theta) \geq 0$3 accelerates "ignoring" well-classified examples (loss quickly zeroes), focusing updates on ambiguity regions. Training both scales is preferable to static assignment; updates inherently enforce $f_2(\theta) \geq 0$4 as the optimizer proceeds.

Empirical results show heightened train and test accuracy versus baseline CE, especially on MNIST and CIFAR-10, with no degradation on CIFAR-100. Selection of $f_2(\theta) \geq 0$5 as the typical early-training confidence gap is recommended.

5. PINN Training and Two-Term Feasibility–Optimality Balance

The two-phase method is immediately relevant for PINNs, with:

• $f_2(\theta) \geq 0$6 MSE data loss (initial, boundary, supervised terms)
• $f_2(\theta) \geq 0$7 MSE physics loss (PDE residual)

In this regime:

• Restoration phase: Apply Adam/SGD on $f_2(\theta) \geq 0$8 until the physics residual drops by preset factor.
• Optimization phase: Train weighted sum $f_2(\theta) \geq 0$9, with empirical tuning (e.g., $\theta \in \mathbb{R}^n$0) until both terms meet descent conditions.

This structure stabilizes physics residuals, prevents oscillatory dynamics, and yields monotone reduction in infeasibility ($\theta \in \mathbb{R}^n$1), with convergence typically faster or more stable than naïve Adam. Assignment symmetry (data ↔ physics) does not degrade performance; phase caps and hyperparameters are documented.

6. Convergence Properties and Empirical Observations

Analytical properties provided by the IR framework (López, 2024):

• Monotonic convergence of penalty sequence $\theta \in \mathbb{R}^n$2 within (0,1).
• Existence of feasible search directions for both phases.
• Limit points are KKT solutions for the constrained reformulation.

Empirically, two-phase PINN training suppresses loss spikes and catastrophic degradation observed in single-phase Adam. Competitive approximations are attainable after a few thousand partial-phase iterations. For hybrid or multi-scale losses, the two-phase approach consistently outperforms baseline single-term losses in both accuracy and stability (see (Dickson et al., 2022) and (Berlyand et al., 2021)).

7. Practical Considerations and Limitations

Parameter tuning (restoration factor $\theta \in \mathbb{R}^n$3, penalty/merit weights, patience windows) is crucial for optimal performance. Most studies employ shallow architectures, raising open questions regarding scalability to deep models (CNNs, RNNs, transformers) and robustness of switch criteria. No formal lower bounds or guarantees are provided on landscape navigation for hybridization regimes. Measuring sharpness and flatness of minima post-switch remains an open research direction.

A plausible implication is that two-phase strategies, by controlling either loss summands or optimization temporalities, provide direct regularization of training dynamics and allow more effective exploitation of the loss landscape—via guided exploration, stabilization, or reallocation of gradient energy—across supervised, constrained, and task-adaptive neural learning settings.