Mitigating Gradient Pathology in PINNs through Aligned Constraint

Abstract: While Physics-Informed Neural Networks (PINNs) are powerful for solving Partial Differential Equations (PDEs), their training is often paralyzed by gradient pathology. The gradients from the PDE residuals and boundary constraints oppose each other, trapping the model in local minima. Current solutions, such as adaptive weighting or hard constraints, either fail to fundamentally resolve this ill-conditioning or are limited to simple geometries. In this study, we systematically analyze the possible causes of this gradient pathology from the perspectives of loss landscapes and optimization dynamics. Based on the obtained conclusion, we propose Constraint-Aligned loss with Manifold Lifting (CAML). By reformulating all zeroth-order terms into aligned constraints, our method effectively mitigates gradient conflicts. In addition, we introduce a delay factor to help the optimizer skip the high-curvature area. Experiments demonstrate that our CAML significantly enhances numerical stability and efficiency in highly complex PINN problems. Our code is open-sourced on https://github.com/YichenLuo-0/CAML.

• The paper introduces CAML, a novel framework that realigns conflicting PDE residuals and boundary gradients to overcome gradient pathologies in PINNs.
• It employs aligned constraints and manifold lifting to expand feasible solution sets and shorten optimization trajectories, achieving lower relative L2 errors and fewer iterations.
• The framework is robust across architectures and optimizers, offering enhanced efficiency and stability for complex PDEs and boundary conditions.

Summary of "Mitigating Gradient Pathology in PINNs through Aligned Constraint" (2605.25001)

Motivation and Problem Analysis

Physics-Informed Neural Networks (PINNs) have become an essential paradigm for meshless numerical solutions of PDEs across scientific and engineering domains. However, PINN optimization is frequently hampered by gradient pathologies: PDE residuals and boundary constraint gradients often oppose each other, becoming trapped in non-optimal local minima. Existing remedies—adaptive weighting, hard constraints, conflict-aware optimizers—offer only partial alleviation, failing to structurally address the ill-conditioning driving slow convergence and instability, especially on complex geometries or composite boundary conditions.

This paper systematically characterizes the loss landscape pathology in PINNs via geometric analysis. Crucially, operator non-uniqueness of PDEs induces a connected manifold of global minimizers (loss valley) in function space for the residual term, which, after neural parameterization, is distorted and non-convex in parameter space. The boundary-condition-imposed loss landscape typically intersects this manifold but rarely aligns directionally in parameter space, causing persistent gradient conflicts and inefficient optimization.

Theoretical Contributions

The paper formally proves that:

• The PDE residual loss admits a flat, connected solution manifold in function space due to intrinsic operator invariances (additive constants and other modes). This manifold is mapped to a highly distorted, high-dimensional valley in parameter space, verified by infinite Hessian condition numbers and low subspace similarity across trained solutions.
• Within this loss valley, the boundary condition gradients are rarely tangent; the residual and boundary gradients generically point in opposite normal directions, yielding a negative inner product and consistent gradient conflict during PINN training.

The loss valley is fundamentally distinct from regression "flat minima" as it arises from structural non-uniqueness rather than parameter redundancy. Even with a near-zero residual, locating the boundary-compatible solution requires traversing a potentially long, highly curved path in parameter space.

Proposed Method: CAML (Constraint-Aligned loss with Manifold Lifting)

Addressing these insights, the authors introduce the CAML framework:

• Aligned Constraints: Reformulate all zeroth-order PDE and boundary terms by introducing a solvable additive offset $c$, allowing explicit translation along operator-invariant directions during training. This significantly enlarges the intersection (feasible region) between PDE and boundary constraint manifolds, minimizing incompatible minima and reducing residual-induced search distance.
• Manifold Lifting: For linear cases, $c$ is computed in closed-form per iteration; for nonlinear terms, Newton or gradient steps are used. This analytic (or low-dimensional) per-step alignment is superior to learnable bias terms due to active trajectory correction rather than passive adaptation.
• Delay Factor for Residual Loss: A time-dependent gating function $\lambda(t)$ is introduced for the residual term, allowing the network to first satisfy boundary conditions before residuals dominate, thereby bypassing high-curvature regions in the residual valley and improving entry-point quality.

Experimental Results and Numerical Analysis

The CAML framework is benchmarked against state-of-the-art PINN alternatives—including classic loss, adaptive weights, hard constraint architectures, and conflict-aware optimizers—across four PDE benchmarks (heat conduction, Poisson, Navier-Stokes, Helmholtz) and three body architectures (MLP, PirateNets, PINNsFormer).

Key quantitative results:

• CAML consistently achieves the lowest or second-lowest relative $L_2$ errors and requires fewer training iterations across all benchmarks and architectures.
• CAML dramatically increases the fraction of training iterations with positive gradient cosine similarity (i.e., reduced conflict), sometimes by an order of magnitude compared to prior loss formulations.
• Optimization trajectories with CAML are shorter in parameter space and less sensitive to initialization, confirming theoretical enlargement and smoothing of feasible regions.
• CAML is robust across optimizers: conflict-aware first-order methods (DCGD, ConFIG) accelerate convergence, while second-order L-BFGS achieves best precision in final refinement. The recommended practical strategy is conflict-aware first-order optimization followed by L-BFGS.

Ablation studies reveal that aligned constraint lifting (AC) is most impactful where boundary conditions are not strongly nonlinear; for composite or nonlinear boundaries, integrating delay-residual scheduling is essential for best performance.

Practical and Theoretical Implications

The structural enlargement of solution sets via manifold lifting translates to significant gains in efficiency, stability, and robustness for physics-driven deep learning. CAML's approach is orthogonal to architecture modifications and optimizer enhancements, and seamlessly integrates with advanced solvers and sampling strategies.

CAML is most effective under the following conditions:

• PDEs or boundary conditions contain at least one nontrivial zeroth-order term (Dirichlet, Robin).
• PDE coefficients are large, causing residual gradients to dominate early trajectories.
• Boundary conditions are composite or strongly nonlinear.

It is less beneficial for well-posed problems with pure Neumann constraints or benign loss landscapes, as shown in failure and sensitivity studies.

Future Directions

Extending CAML to time-dependent PDEs with initial conditions will require a time-dependent offset $c(t)$, potentially solved analytically for linear equations or approximated with low-dimensional temporal parameterizations for nonlinear systems. Structural characterization of additional alignment modes beyond the additive constant may yield further improvements, especially for PDEs with rich operator invariances.

Conclusion

This work provides a rigorous geometric and optimization-centric framework for understanding and mitigating gradient pathologies in PINNs, introducing CAML to align constraints and lift solution manifolds. Extensive empirical evidence establishes its broad applicability and superiority in efficiency and stability, making it a strong candidate for a new standard in physics-informed neural network training.

Future research is expected to advance this framework for dynamic PDEs and explore deeper structural alignment mechanisms in scientific deep learning.