---
title: Momentum-Accelerated Proximal AGD (AltGDAm)
url: https://www.emergentmind.com/topics/momentum-accelerated-proximal-agd-altgdam
type: topic
---

# Momentum-Accelerated Proximal AGD (AltGDAm)

Momentum-Accelerated Proximal AGD (AltGDAm), also referred to as accelerated block coordinate proximal gradient with adaptive momentum, denotes a class of block-coordinate algorithms for nonconvex, nonsmooth composite optimization that incorporate Nesterov-type, adaptive, or generalized momentum into the proximal gradient or proximal linear framework. These methods combine block-wise splitting, adaptive extrapolation, and monotonicity enforcement within the structure of the Kurdyka-Łojasiewicz (KL) convergence analysis, yielding provable global convergence and, under appropriate settings, local linear rates for a broad range of high-dimensional statistical learning and estimation problems. The approach subsumes and extends the classical accelerated proximal gradient (APG/FISTA) updates to block-separable, nonconvex, and even minimax or saddle-point settings.

## 1. Formal Problem Statement and Assumptions

Momentum-accelerated proximal AGD (AltGDAm) addresses composite minimization problems of the form
\[
\min_{x\in\mathbb{R}^n}\; F(x) := f(x) + \sum_{i=1}^s g_i(x_i),
\]
where the variable $x$ is partitioned into $s$ blocks, $f$ is continuous and differentiable (possibly nonconvex), and each $g_i$ is proper, lower semicontinuous, block-separable, and possibly nonsmooth or nonconvex [1710.05338].

The key assumptions are:
- $F$ is bounded below and admits at least one critical point $x^*$, $0\in\partial F(x^*)$.
- Each block partial gradient $x_i\mapsto\nabla_{x_i}f(x_{\neg i}, x_i)$ is $L_i$-Lipschitz in $x_i$.
- $F$ satisfies the KL property at every cluster point.
- Every block is updated at least once within any window of $T$ successive steps.

This framework accommodates important regularized regression objectives (Lasso, group Lasso, capped $\ell_1$, SCAD), matrix/tensor factorization with $\ell_0$ penalties, robust minimax learning, and regularized image reconstruction.

## 2. Algorithmic Structure and Update Rules

AltGDAm iterates alternating, block-wise, momentum-accelerated proximal-gradient steps. The generic iteration includes:

**Block selection** (Gauss–Southwell, uniform, or cyclic rules):
\[
b_k = \arg\max_{i} \|x_i^{(k-1)} - \prox_{g_i/\gamma_i}(x_i^{(k-1)} - \gamma_i \nabla_{x_i}f(x^{(k-1)}))\|.
\]

**Momentum Extrapolation** (for each $i$):
\[
\widehat{x}_i^{(k)} = x_i^{(k-1)} + \beta_i^{(k-1)} (x_i^{(k-1)} - x_i^{(k-2)}).
\]

**Proximal-Gradient Step** (current block $b_k$):
\[
x_i^{(k)} =
  \begin{cases}
    \prox_{\gamma_i g_i}\left(\widehat{x}_i^{(k)} - \gamma_i \nabla_{x_i}f(x_{\ne i}^{(k-1)}, \widehat{x}_i^{(k)})\right), & i = b_k \\
    x_i^{(k-1)}, & i \ne b_k.
  \end{cases}
\]

**Accelerated candidate**:
\[
v_i^{(k)} =
  \begin{cases}
    x_i^{(k)} + \beta_i^{(k-1)} (x_i^{(k)} - x_i^{(k-1)}), & i = b_k \\
    v_i^{(k-1)}, & i \ne b_k.
  \end{cases}
\]

**Momentum Adaptation** (monotonicity enforcement):
\[
\beta_i^{(k)} =
  \begin{cases}
    \min\{\beta_i^{(k-1)}/t, 1\}, & i=b_k, \ F(x^{(k)})\leq F(v^{(k)}) \\
    t\beta_i^{(k-1)}, & i=b_k, \ F(x^{(k)}) > F(v^{(k)}) \\
    \beta_i^{(k-1)}, & i\neq b_k
  \end{cases}
\]
The non-decreasing or shrinkage rule for the block-momentum parameter $t\in(0,1)$ prevents divergence due to momentum overshoot.

Typical step size selection is $\gamma_i = c/L_i$, $c\in(0,1]$ [1710.05338].

## 3. Theoretical Guarantees: Global and Local Rates

Under the above assumptions, AltGDAm exhibits the following convergence properties [1710.05338, 2308.12126, 1705.04925]:

- **Global convergence**: The iterates converge to a critical point $\bar{x}$, i.e., $x^{(k)} \to \bar{x}$ and $0\in\partial F(\bar{x})$.
- **Subgradient residual decay**: There exists $K>0$ such that $\mathrm{dist}(0,\partial F(x^{(k)})) \leq K \|x^{(k)}-x^{(k-1)}\|$, and the telescoping argument yields $\sum_k \|x^{(k)} - x^{(k-1)}\| < \infty$.
- **KL-based local convergence rate**: If $F$ satisfies the KL property at $\bar{x}$ with exponent $\theta=1/2$ (as for Lasso, group Lasso, SCAD), then $F(x^{(k)}) - F^* \leq C\alpha^k$ for some $\alpha\in(0,1)$, i.e., local $R$-linear convergence [1710.05338]. For $\theta\in(0,1/2)$, sublinear rates $O(k^{-\frac{\theta}{1-2\theta}})$ are implied [2308.12126, 1705.04925].

| Regime        | KL Exponent $\theta$ | Rate        |
|---------------|----------------------|-------------|
| Finite Steps  | $1$                  | Finite      |
| Linear        | $[1/2,1)$            | $O(\alpha^k)$ |
| Sublinear     | $(0,1/2)$            | $O(k^{-\frac{\theta}{1-2\theta}})$ |

In block-minimax contexts, momentum-accelerated Alternating GDA with proximal steps achieves improved complexity $O(\kappa^{11/6} \epsilon^{-2})$ for nonconvex-strongly-concave saddle point problems compared to prior $O(\kappa^2\epsilon^{-2})$ alternating schemes [2112.11663].

## 4. Connections, Variants, and Adaptive Extensions

Momentum-accelerated proximal AGD has several algorithmic variants and connections:

- **Adaptive Restart and Momentum Scheduling**: Algorithms such as APG-restart couple momentum with flexible restart based on function or gradient mapping tests; this prevents momentum-induced oscillations and enhances empirical convergence, maintaining global $O(1/K)$ rates even in the nonconvex, nonsmooth regime [2002.11582].
- **Generalized Nesterov Momentum**: Extensions to $\omega$-power momentum schemes (parametric interpolation between classical and slower-growth Nesterov momentum) provide improved robustness and high-order convergence in convex smooth preconditioned settings [2409.13344].
- **Block-Coordinate Generality**: AltGDAm encompasses both cyclic and random block ordering (see ABPL$^+$ [2308.12126]) and applies to fully nonconvex, nonsmooth matrix/tensor factorization, where adaptivity and monotonicity are preserved without loss of convergence properties.
- **Monotonicity Enforcement**: Monotonicity checks on the objective (or surrogate) after extrapolation—accepting only if the function value decreases—are universal, analytically critical tools [1710.05338, 1705.04925].

## 5. Empirical and Practical Behavior

Numerical exemplars emphasize AltGDAm's advantage:

- **High-dimensional sparse regression**: For $\ell_1$-LS, group Lasso, capped-$\ell_1$, and SCAD (all with $m = 1000, n = 5000, s = 5$–$10$ blocks), AltGDAm (with Gauss–Southwell or random block selection) consistently outperforms APG, mAPG, BPL, or APGnc in both early and late stages of convergence. The Gauss–Southwell rule, in particular, yields the fastest empirical decay of optimality gap [1710.05338].
- **NMF/Tensor Decomposition**: ABPL$^+$, a practical AltGDAm implementation, demonstrates superiority over PALM, iPALM, and related block methods for $\ell_0$-constrained nonnegative matrix and tensor factorization [2308.12126].
- **Adversarial Learning**: In robust classification minimax setups (e.g., Wasserstein robust MNIST), momentum-proximal AltGDAm accelerates both objective reduction and robust accuracy relative to GDA, prox-AltGDA, and primal-dual baselines [2112.11663].
- **Preconditioned Variants**: In image reconstruction, AltGDAm with EM-type preconditioning and general Nesterov momentum achieves order-optimal rates $o(1/k^{2\omega})$ in objective, robust to aggressive or conservative momentum parameters [2409.13344].

## 6. Extensions and Analytical Framework

- **Generalization to Multiple Nondifferentiable Terms**: The methodology naturally extends to settings with multiple nonsmooth separable regularizers, via block-wise fixed-point and proximal splitting [2409.13344].
- **Inexact Proximal Computations**: The proof template accommodates $\epsilon$-approximate proximal updates, with all main results unchanged provided the error is dominated by the update distance [2308.12126, 1705.04925].
- **KL-based Theory**: All main convergence theorems fundamentally depend on the KL property, which bridges nonconvexity and non-smoothness; the method exploits sufficient decrease and subgradient bounds to invoke KL-based Lyapunov analysis, leading to precise asymptotic rates [1710.05338, 2308.12126, 1705.04925].

## 7. Comparative Perspective

Relative to classic APG/FISTA frameworks, AltGDAm distinctively enables scalable, provably convergent optimization of high-dimensional nonconvex problems by:
- Supporting block-wise and adaptive update schedules.
- Allowing nonsmooth, nonconvex, and separable regularizers without bounded-domain constraints.
- Unifying monotonicity enforcement and adaptive momentum via restart/shrinkage without complex tuning requirements [1710.05338, 2002.11582, 2308.12126].
- Delivering improved empirical and provable convergence over standard block proximal/alternating and non-accelerated block methods, particularly in ill-conditioned or overparameterized regimes.

The method therefore constitutes a rigorous foundation for large-scale, sparse, and structured learning in contemporary computational statistics and machine learning.

Source: https://www.emergentmind.com/topics/momentum-accelerated-proximal-agd-altgdam