Conditionally Adaptive Penalty Update (CAPU)

• The paper presents CAPU as a method that updates penalty parameters based on constraint violations to improve enforcement and convergence.
• CAPU assigns individual penalty parameters using an exponential moving average of squared violations and RMSprop-style updates to handle heterogeneous constraints.
• CAPU’s application in physics-informed neural networks demonstrates improved numerical accuracy and stability in solving challenging PDE problems.

The Conditionally Adaptive Penalty Update (CAPU) is an algorithmic strategy for dynamically adjusting penalty parameters in augmented Lagrangian and related constraint optimization methods. Its defining principle is that penalty parameters are updated based on the magnitude of constraint violations at each iteration, with the core objective that “larger violations incur stronger penalties.” CAPU is constructed to enhance constraint enforcement selectively, accelerate convergence of Lagrange multipliers in difficult cases, and manage heterogeneous constraints efficiently. Recent extensions, notably within the PECANN framework for physics-informed neural networks, generalize CAPU to support multiple independent constraints, mini-batch training regimes, and challenging PDE solvers.

1. Algorithmic Formulation of CAPU

CAPU is implemented by assigning a unique penalty parameter $\mu_i$ to each constraint $\mathcal{C}_i$ in an augmented Lagrangian setting. The standard loss function takes the form: $$\mathcal{L}(\theta, \lambda; \mu) = \mathcal{J}(\theta) + \lambda^T\mathcal{C}(\theta) + \frac{1}{2}\mu\|\mathcal{C}(\theta)\|_2^2$$ where $\mathcal{J}(\theta)$ is the primary objective, $\lambda$ the vector of Lagrange multipliers, and $\mathcal{C}(\theta)$ the stack of constraints.

The CAPU strategy proceeds by:

• Maintaining an exponential moving average $v_i$ of squared constraint violations for each constraint:

$$\bar{v}_i \leftarrow \zeta \bar{v}_i + (1-\zeta)[\mathcal{C}_i(\theta)]^2$$

where $\zeta$ is typically 0.99.

• Computing an “RMSprop-style” candidate penalty parameter:

$$\mu_i^{\text{RMSprop}} \leftarrow \eta_i/\sqrt{\bar{v}_i + \epsilon}$$

where $\eta_i$ is a scaling factor and $\epsilon$ a small constant for stability.

• Ensuring penalty growth is only upwards (as opposed to a naïve RMSprop penalty reduction):

$$\mu_i \leftarrow \max\{\mu_i, \mu_i^{\text{RMSprop}}\}$$

• Updating each Lagrange multiplier as:

$$\lambda_i^e \leftarrow \lambda_i^{e-1} + \mu_i^{e-1} \mathcal{C}_i(\theta^e)$$

• Performing updates conditionally, e.g. only when the constraint loss does not decrease sufficiently (often using a decrease threshold $\omega$).

This strategy allows each constraint to be enforced independently and robustly during training (Hu et al., 21 Aug 2025).

2. Conditional Adaptivity and Enforcement Principle

A defining property of CAPU is that the penalty strength for each constraint cannot inadvertently decrease during optimization when a constraint is persistently violated. The adaptive penalty is always increased (or left unchanged), never reduced when the moving average of the violation rises. This principle ensures that selectively challenging constraints receive higher penalty weights, resulting in more aggressive updates of the corresponding dual variables.

The broader implication is that CAPU avoids pathologies of earlier monotonic or global penalty update methods, such as MPU (which multiplies the global penalty parameter by a fixed factor) or CPU (which updates conditionally but uniformly across constraints). CAPU’s individual adaptive weights are more responsive to the “difficulty” of each constraint, yielding well-behaved Lagrange multiplier distributions and improved constraint satisfaction (Hu et al., 21 Aug 2025).

3. Comparison to Previous Augmented Lagrangian Penalty Updates

Traditional ALM implementations use a single global $\mu$ and update it in a monotonic, often exponential fashion. These approaches can be too aggressive for some constraints and too passive for others, leading to suboptimal or unstable training when constraints are heterogeneous (Basir et al., 2023).

CAPU differs by:

• Assigning penalty parameters per constraint,
• Adapting each $\mu_i$ independently using its violation statistics,
• Safeguarding penalty growth (never decreasing during persistent violation),
• Conditionally triggering multiplier updates based on loss progress.

Empirical comparisons show that methods such as MPU and CPU may result in significant oscillations and deviations in the solution or constraint satisfaction, while CAPU achieves stable and accurate solutions with well-controlled constraint enforcement (Hu et al., 21 Aug 2025).

4. Application to Physics-Informed Neural Networks and PDE Learning

In the PECANN framework for solving forward and inverse partial differential equations, CAPU’s multi-penalty adaptive mechanism addresses the challenge of enforcing numerous constraints arising from physical laws, boundary conditions, initial conditions, and calibrated data (Hu et al., 21 Aug 2025). Key advances enabled by CAPU include:

• Robust enforcement of PDE residuals and constraints at scale,
• Efficient mini-batch training via expectation-based constraint terms,
• Enhanced learning on problems with multi-scale or oscillatory solutions (e.g., high-wavenumber Helmholtz, transonic rarefaction in Burgers’ equation),
• Time-windowing for long-time evolution problems with continuity constraints.

Numerical experiments across various PDE benchmarks demonstrate that PECANN-CAPU achieves competitive accuracy with faster convergence compared to established methods such as Kolmogorov-Arnold-networks/cPIKAN, especially in regimes where constraint enforcement is challenging.

5. Mathematical Guarantee and Practical Safeguards

The mathematical structure of CAPU ensures that penalty parameters always respect the principle “larger violation $\rightarrow$ stronger penalty.” The candidate update

$$\mu_i \leftarrow \max\{\mu_i, \eta_i/\sqrt{\bar{v}_i + \epsilon}\}$$

prevents penalty reduction if the constraint violation increases, and avoids penalty overgrowth in the absence of significant constraint violation.

Further practical safeguards are:

• Performing dual/multiplier updates only when primal progress stalls (using thresholds such as loss drop by fraction $\omega$),
• Smoothing penalty parameter growth and adapting RMSprop scaling factors $\eta_i$ to optimizer choice (smaller for Adam, larger for quasi-Newton),
• Guaranteeing individualized penalty evolution without dominating the primal loss, supported by empirical distributions of multiplier values post-training.

6. Empirical Impact and Accuracy on Challenging Problems

Key empirical findings include:

• On the transonic rarefaction problem in Burgers' equation, CAPU reduces relative error from order $10^{-1}$ (seen in previous penalty update methods) to $10^{-2}$.
• On high-wavenumber Helmholtz problems employing Fourier feature mappings, CAPU achieves relative errors as low as $10^{-3}$, outperforming baseline PINN methods.
• Lagrange multiplier distributions under CAPU are more concentrated and stable, indicating proper individualized penalty scaling even in highly heterogeneous constraints.

CAPU’s adaptive update thus substantially enhances training stability, numerical accuracy, and constraint enforcement efficacy in both forward and inverse PDE learning tasks (Hu et al., 21 Aug 2025).

7. Broader Significance and Applicability

CAPU represents a methodological advance in large-scale constrained optimization, applicable not only to physics-informed neural networks but also to any domain requiring reliable enforcement of heterogeneous constraints through adaptive penalization. This includes machine learning, scientific computing, networked distributed optimization, and robust large-scale regression.

Its principled construction—conditional, individualized, and responsive—marks it as a foundational technique for adaptive penalty regulation in modern augmented Lagrangian frameworks. Its effectiveness in practice has been demonstrated through competitive accuracy and efficient convergence across canonical and challenging problem spaces.