Sequential Penalty Methods

• Sequential penalty methods are algorithmic strategies that convert constrained optimization problems into a sequence of unconstrained or less-constrained problems using increasing penalty terms.
• They employ adaptive mechanisms, including spatially varying penalties and single-loop updates, to enforce constraints effectively and guarantee convergence under proper conditions.
• Applications span nonlinear programming, quadratic optimization, deep learning, and sparse recovery, offering robust solutions with practical error control and improved computational efficiency.

A sequential penalty method is an algorithmic strategy for solving constrained optimization problems by converting the original problem into a sequence of unconstrained or less-constrained problems, using a penalty function whose weight increases over iterations. This approach is central to the theory and practice of nonlinear programming, variational inequalities, changepoint detection, nonconvex and non-Lipschitz optimization, quadratic programming, and deep learning under sample-wise constraints. Recent advances have materialized in adaptive coefficient tuning, spatially varying penalties, single-loop schemes, and rigorous convergence guarantees.

1. Mathematical Foundations of Sequential Penalty Methods

Given a generic constrained optimization problem

$$\min_{x\in\mathbb R^n} \, f(x) \quad \text{subject to}\quad g_i(x) \ge 0,\;\; i=1,\dots,m$$

the sequential penalty method introduces a scalar penalty parameter $\sigma_k \to+\infty$ and a penalty function $\psi:\mathbb R\to\mathbb R_+$ (commonly $\psi(t)=\max\{-t,0\}^p$, $p=1$ or $2$, or a smooth approximation thereof). The penalized objective at iteration $k$ is

$$P_k(x) = f(x) + \sigma_k \sum_{i=1}^m \psi(g_i(x)).$$

The algorithm approximately solves $\min_x P_k(x)$ for growing $\sigma_k$, forming a sequence $\{x^{(k)}\}$. As $\sigma_k \to \infty$, under constraint qualification, limit points of the sequence satisfy the KKT conditions of the original problem, and the penalty residual $r(x)=\sum_{i=1}^m \max\{-g_i(x),0\}$ converges to zero (Boon et al., 2022).

For composite structures or non-Lipschitz settings, penalties may act on subsets of constraints (partial penalization), or employ smoothings (e.g., Huber or smoothed plus functions) to enable efficient optimization (Nedich et al., 2020, Chen et al., 2014). Error bounds connecting constraint violation with penalized terms are critical for establishing rate and global convergence properties (Chen et al., 2014).

2. Algorithmic Variants and Adaptive Penalty Schemes

Classical Framework

The generic sequential penalty framework proceeds via outer iterations:

1. For fixed $\sigma_k$, minimize $P_k(x)$ approximately.
2. Increase $\sigma_k$ according to a prescribed schedule.
3. Stop when constraint violation residuals are below tolerance.

The penalized stationarity condition is

$$\nabla f(x^{(k)}) + \sigma_k \sum_i \psi'(g_i(x^{(k)})) \nabla g_i(x^{(k)}) = 0.$$

This is the basis for analyses of convergence to KKT points (Boon et al., 2022).

Adaptive Penalty Methods

Recent innovations replace the scalar parameter with a spatially varying “penalty field” $\rho(x)$, computed via auxiliary PDEs that regularize responses to the residual (Boon et al., 2022): $$\rho - \varepsilon \Delta \rho = \gamma\,|M(f-Au,\,u-g)|, \quad \tfrac{\partial\rho}{\partial n}=0$$ where $M$ models a complementarity mapping, $\varepsilon$ regularizes smoothing, and $\gamma$ tunes penalty removal speed. The method solves at each iterate an auxiliary PDE for $\rho$, calculates an active-set indicator via a smoothed plus function, and applies a quasi-Newton iteration: $$((I-\alpha^k)A + \alpha^k c)\,\delta u = M(f-Au^k,\,u^k-g)$$ yielding exact constraint imposition as the residual shrinks. This scheme recovers the primal-dual active-set method (semi-smooth Newton) for variational inequalities in the limit (Boon et al., 2022).

Single-Loop and Matrix-Free Penalty Updates

A single-loop proximal-conditional-gradient penalty method “proxCG$_{1\ell}^{\text{pen}}$” recasts penalty parameter updates so that no inner subproblem is solved to high accuracy before increasing the penalty. One step of proximal-gradient and one conditional-gradient operation are performed per outer loop, followed by explicit updates of the penalty parameter and step size: $$\beta_t = \beta_0 (t+1)^\delta, \quad \alpha_t = 2/(t+2), \quad H_{t+1} = \max\{ H_0, \tfrac{2M_f}{\mu+1} \}(t+1)^{1-\mu}$$ with both feasibility and objective residuals decaying sublinearly in $t$ (Zhang et al., 2024).

3. Convergence Theory and Rate Estimates

Theoretical analyses anchor convergence of sequential penalty methods to KKT points under appropriate constraint qualifications, strong convexity, and differentiability. For quadratic penalty functions and strongly convex objectives with linear constraints, using a power-law scheduling for penalty and smoothing parameters, sublinear rates $O(1/\sqrt{k})$ for the expected error are established for incremental single-loop schemes (Nedich et al., 2020).

In convex-separable settings, a single-loop penalized method with explicit parameter updates achieves objective value deviation decay at rate $t^{-\min\{\mu,\nu,1/2\}}$ and feasibility violations at $O(t^{-1/2})$, matching or improving on classical approaches (Zhang et al., 2024).

For monotone quasi-variational inequalities (QVI), monotonic convergence is proven under penalty parameter continuation, with theoretical error bounds of $O(\rho^{-\sigma}\ln \rho)$, or $O(\rho^{-\sigma})$ under further regularity (Reisinger et al., 2018).

In nonconvex, non-Lipschitz setups, partial exact penalization yields limit points solving the original KKT system; smoothing and continuation guarantee bounded iterates, global limit points, and practical error control (Chen et al., 2014).

4. Sequential Penalty Methods in Large-Scale and Nonlinear Optimization

Quadratic Programming and SQP

Inexact sequential quadratic optimization (SQP) with dynamic penalty updates incorporates parameter reduction “inside” the subproblem solver, leveraging matrix-free approaches. The penalty parameter is reduced only when optimality, feasibility, and complementarity ratios cross thresholds, minimizing unnecessary reductions and enhancing efficiency. This strategy, DUST (Dynamic Updating Strategy), guarantees ultimate convergence to feasible stationary points whenever one exists, and rapid infeasibility detection otherwise (Burke et al., 2018, Liu et al., 2023).

D-Stationary Theory and Rapid Infeasibility Detection

An exact penalty SQP method enables convergence to D-stationary, DL-stationary (KKT), and DZ-stationary points, encompassing both feasible and infeasible constrained problems. The penalty parameter need not tend to zero for rapid infeasibility detection; termination at large violation certifies stationarity of the regularized problem (Liu et al., 2023).

5. Applications in Deep Learning and Large-Scale Data Fitting

The sequential penalty method has been implemented for sample-wise constrained learning, such as in image processing and medical data. Instead of arbitrary penalty weights, strict constraints on each data sample (e.g., maximum distortion per image) are directly encoded and solved. The method employs an outer loop of penalty escalation and inner stochastic optimization (SGD), guaranteeing convergence to feasible solutions under modified KKT conditions (Lanzillotta et al., 23 Jan 2026).

Empirical results on MNIST with autoencoder-based constraints and chest X-ray watermarking demonstrates that the sequential penalty method robustly enforces per-sample requirements, avoids ad hoc weight tuning, and maintains main-task accuracy, outperforming fixed penalty approaches and providing human-interpretable constraint thresholds.

6. Sequential Penalty Schemes for Non-Lipschitz Optimization and Sparse Recovery

Sequential or partial penalty methods are applicable to constraints given by the intersection of polyhedra and noise-tolerance ellipsoids, and to nonconvex, non-Lipschitz objectives inducing sparsity. The exact penalty approach assures, for sufficiently large penalty parameters, that local (and under global Lipschitz, global) minimizers of the original constrained problem are preserved in the penalized formulation (Chen et al., 2014).

Numerical evidence indicates that continuation in the penalty parameter produces sparser solutions and tighter recovery errors than SPGL1 (ℓ₁–BPDN) or unconstrained quadratic penalization, with moderate computational cost.

7. Practical Guidelines, Advantages, and Limitations

Key practical recommendations include:

• Use inexact solves for penalty subproblems, leveraging warm starts, coordinate descent, and matrix-free solvers.
• Increase penalty parameters gradually (e.g., multiplicative schemes $\tau_{k+1} = \gamma\,\tau_k$, $\gamma>1$), stop when constraint violation is sufficiently small.
• In adaptive spatial schemes, set regularization/smoothing parameters proportional to mesh size, tune penalty field removal speed, solve auxiliary penalty PDEs at coarse resolution for efficiency (Boon et al., 2022).

Advantages include:

• Robust convergence to exact solution or infeasibility certificate.
• Avoidance of trial-and-error penalty weight selection.
• Applicability to large-scale, nonlinear, nonconvex, and non-Lipschitz regimes.
• Efficient computation, especially with single-loop parameter updates.

Limitations and open issues:

• Full global convergence proofs may require additional compactness or regularity conditions.
• Theoretical rates depend on problem data, regularity, and constraint structure.
• Spatially adaptive schemes require solving auxiliary PDEs per iteration, which may be computationally nontrivial in high dimensions.

Sequential penalty methods thus constitute a rigorous, efficient, and widely applicable framework for constrained optimization across a range of modern applications, including nonlinear programming, variational inequalities, dynamic systems, statistical learning, and high-dimensional inverse problems.