Augmented Lagrangian Fast Projected Gradient Method

• AL-FPGM is an accelerated optimization framework combining augmented Lagrangian with fast projected gradient methods to efficiently solve composite, linearly constrained problems.
• It employs a bi-level scheme with accelerated inner iterations (FISTA or Nesterov-style) and dual updates, achieving optimal nonergodic convergence rates.
• Applications in machine learning and control demonstrate AL-FPGM’s scalability and effectiveness in handling large-scale convex, weakly convex, and even nonconvex problems.

The Augmented Lagrangian Fast Projected Gradient Method (AL-FPGM) encompasses a family of first-order accelerated algorithms for solving linearly constrained optimization problems, particularly when those problems are of composite, nonsmooth, convex or weakly convex (sometimes even nonconvex) structure. AL-FPGM refers to methods that combine the augmented Lagrangian (AL) framework with variants of accelerated projected or proximal gradient methods (typically FISTA-type or Nesterov-style extrapolation) to achieve optimal or near-optimal convergence rates on both primal–dual gap and feasibility errors. These algorithms are widely used in large-scale optimization, machine learning, and control, where scalability, parallelizability, and provable rates are critical.

1. Problem Structures and the Augmented Lagrangian

AL-FPGM algorithms are designed for problems of the form: $$\min_{x\in \mathbb{R}^n} \; \varphi(x) := f(x) + p(x) \quad \text{subject to} \quad A x = b$$ where:

• $f: \mathbb{R}^n \to \mathbb{R}$ is smooth, $L_f$-Lipschitz, possibly nonconvex but $m_f$-weakly convex,
• $p: \mathbb{R}^n \to (-\infty, +\infty]$ is closed, proper, convex, and "simple" (proximal operator is efficiently computable),
• $A \in \mathbb{R}^{l \times n}$, $b \in \mathbb{R}^l$.

The classical augmented Lagrangian for $c > 0$ and $\lambda \in \mathbb{R}^l$ is: $$L_c(x, \lambda) := f(x) + p(x) + \langle \lambda, Ax - b \rangle + \frac{c}{2} \|Ax - b\|^2$$ This transforms the constrained problem into an unconstrained but penalized and dualized saddle-point problem amenable to first-order methods (Melo et al., 2020, Bot et al., 2021).

2. Core Algorithmic Structure

AL-FPGM employs an outer–inner bi-level iteration:

1. Outer iteration: Update the dual variables (multipliers) and, possibly, AL penalty parameters.
2. Inner iteration: Approximately minimize the AL subproblem in $f: \mathbb{R}^n \to \mathbb{R}$0 for current $f: \mathbb{R}^n \to \mathbb{R}$1, using a fast (accelerated) projected or composite gradient-type method.

A typical outer iteration is:

• Given $f: \mathbb{R}^n \to \mathbb{R}$2,
• Form the regularized subproblem:

$f: \mathbb{R}^n \to \mathbb{R}$3

(with $f: \mathbb{R}^n \to \mathbb{R}$4 ensuring strong convexity of the subproblem).

• Update dual variables:

$f: \mathbb{R}^n \to \mathbb{R}$5

• Repeat until approximate KKT conditions:

$f: \mathbb{R}^n \to \mathbb{R}$6

The inner minimization invokes an accelerated projected gradient or composite gradient method, often based on Nesterov's extrapolation or FISTA updates (Melo et al., 2020, Nedelcu et al., 2013).

3. Convergence Complexity and Rate Results

AL-FPGM schemes achieve provable convergence rates depending on problem class and assumptions:

• For weakly convex composite $f: \mathbb{R}^n \to \mathbb{R}$7 and linear constraints, with bounded domain and Slater-type conditions, AL-FPGM (IAIPAL variant) returns an $f: \mathbb{R}^n \to \mathbb{R}$8 approximate KKT tuple in

$f: \mathbb{R}^n \to \mathbb{R}$9

accelerated composite gradient steps (Melo et al., 2020).

• In fully convex regimes, AL-FPGM variants with inertial (Nesterov, Chambolle-Dossal, Attouch–Cabot) sequences achieve the optimal nonergodic rate

$L_f$0

for the primal–dual gap, feasibility, and function value error (Bot et al., 2021, Zhou et al., 21 Nov 2025, Nedelcu et al., 2013).

A key technical result is that maintaining fast rates for both the objective and feasibility under linear constraints is possible with only simple outer AL multiplier steps and accelerated first-order inner solves:

• The proof involves Lyapunov (energy function) descent arguments, adapted to include both primal and dual errors.
• The complete iteration complexity depends on the problem's smoothness/weak convexity, the proximal regularization, and the penalty parameter update policy.

4. Notable Algorithmic Variants and Inertial Schemes

Several variants within the AL-FPGM landscape have emerged:

• IAIPAL (Inexact Accelerated Inexact Proximal Augmented Lagrangian): Exploits FISTA for the inner strongly convex proximal AL subproblems, achieves $L_f$1 complexity with no feasibility assumption at initialization (Melo et al., 2020).
• Accelerated Linearized ALM: Applies Nesterov/Chambolle–Dossal/Attouch–Cabot inertial weights to primal and (in some recent work) dual updates, achieving nonergodic $L_f$2 rates (Bot et al., 2021).
• Güler-type Accelerated Linearized ALM (GLALM): Constructs Güler-style extrapolation using negative correction terms to simultaneously accelerate both primal and dual iterates, maintaining optimal rates and improved empirical efficiency (Zhou et al., 21 Nov 2025).
• Inexact Dual Fast Gradient Methods: For constraint applications like MPC, the dual outer AL-FPGM loop is itself an accelerated projected gradient, with explicit error allocation between inner primal and outer dual phases (Nedelcu et al., 2013).

A compact overview, identifying leading AL-FPGM classes and their rates:

Method Variant	Inner Algorithm	Rate (primal–dual gap)
IAIPAL (Melo et al., 2020)	ACG/FISTA	$L_f$3
Convex Inertial AL-FPGM	Nesterov-style	$L_f$4
Güler-accelerated GLALM	Güler-extrap.	$L_f$5
AL-FPGM for Dual (IDFGM)	FISTA (dual AG)	$L_f$6

5. Applications in Machine Learning and Control

AL-FPGM has demonstrated effectiveness for high-dimensional and large-scale problems where first-order methods are essential:

• One-Class SVM Training: AL-FPGM for box-constrained quadratic programs with sum-to-one equality, exploiting FPGM for inner minimization and rapid dual updates. Empirical results on kernel SVMs document superior speed and accuracy compared to standard solvers for large datasets (Yowetu et al., 2023).
• Embedded Model Predictive Control (MPC): AL-FPGM/IDFGM is applied to constrained MPC problems with box/ball constraints and quadratic costs. All matrix operations are amenable to embedded hardware implementations, and convergence can be tightly certified (Nedelcu et al., 2013).
• Sparse Regression, Quadratic Programming, and Compressive Sensing: Güler-type accelerated AL-FPGM variants outperform conventional ALM and classic FISTA-type methods in terms of iteration count and wall-clock time in $L_f$7-regularized logistic regression, QPs, and TV-based reconstruction (Zhou et al., 21 Nov 2025).

6. Algorithmic Comparison and Implementation Properties

AL-FPGM outperforms classic ALM and ergodic primal–dual methods under smoothness/weak convexity by:

• Providing fast, nonergodic convergence rates with simple per-iteration computational cost (one gradient and one simple projection/proximal step).
• Allowing flexibility for both primal and dual acceleration, and the possibility of simultaneously accelerating both components (cf. Güler-style and GLALM) (Zhou et al., 21 Nov 2025).
• Accommodating inexpensive parameter tuning. Only penalty and step-size must be chosen based on smoothness modulus and operator norm.

Typical resource requirements:

• Dominated by gradient evaluations and projections/proximal steps.
• For SVM or dense-kernel settings, memory is $L_f$8.
• For control (MPC), computation is dominated by structure-exploiting matrix–vector products.

Practical implications include ease of implementation, generality across composite/smooth/nonsmooth classes, and amenability to both parallelization and acceleration on structured problems (e.g., large sparse $L_f$9).

7. Future Directions and Extensions

Ongoing research investigates:

• Stochastic accelerated AL-FPGM, with potential for data-mining and online ML applications (Zhou et al., 21 Nov 2025).
• Extensions to nonconvex settings beyond weakly convex, with complexity guarantees that generalize the $m_f$0 result.
• Unified frameworks for extrapolated AL-type methods that generalize both Nesterov and Güler-style schemes.
• Efficient implementations for distributed and federated optimization settings, exploiting AL-FPGM’s decomposability.

A plausible implication is that AL-FPGM will remain a central technique in large-scale optimization, particularly for settings where exact interior-point methods are infeasible and accelerated first-order methods yield state-of-the-art performance.