Nesterov’s Accelerated Gradient Method

Updated 25 November 2025

Nesterov’s Accelerated Gradient Method is a first-order optimization technique that uses a two-step extrapolation-correction scheme with momentum.
It achieves an optimal O(1/k²) convergence rate in smooth convex settings and robust linear rates in strongly convex regimes.
Extensions of NAG cover Euclidean, Riemannian, stochastic, and nonconvex problems, playing a key role in large-scale machine learning.

Nesterov’s Accelerated Gradient Method (NAG) is a foundational family of first-order optimization algorithms designed to achieve provably accelerated convergence compared to standard gradient descent, with extensions covering Euclidean, Riemannian, stochastic, and non-convex regimes, as well as continuous- and discrete-time dynamics. The method uses a two-step extrapolation-correction structure that adds momentum—a carefully tuned combination of current and past iterates—resulting in optimal complexity for smooth convex minimization and fundamental impacts on large-scale machine learning, including deep and over-parameterized neural networks.

1. Algorithm Structure and Theoretical Foundations

The canonical NAG (convex) algorithm seeks to minimize an $L$ -smooth convex function $f:\mathbb R^d\to\mathbb R$ . Given $x_0=y_0$ and iterates $\{x_k\}$ , $\{y_k\}$ , for step size $s=1/L$ :

$\begin{aligned} x_{k+1} &= y_k - s\,\nabla f(y_k), \ y_{k+1} &= x_{k+1} + \frac{t_k - 1}{t_{k+1}}(x_{k+1} - x_k), \ t_{k+1} &= \frac{1+\sqrt{1+4t_k^2}}{2},\quad t_0=1. \end{aligned}$

The strongly convex version (“NAG-SC”) uses constant momentum parameter $\beta = (\sqrt\kappa-1)/(\sqrt\kappa+1)$ with $\kappa=L/\mu$ , yielding

$x_{k+1} = y_k - s\,\nabla f(y_k),\qquad y_{k+1} = x_{k+1} + \beta(x_{k+1} - x_k)$

with step size $s=1/L$ (Liu, 24 Feb 2025).

The method’s acceleration is captured by Lyapunov or potential-function arguments, often using a quadratic plus a scaled function gap, e.g.,

$V_k = \|p_k + x_k - x^*\|^2 + 2\alpha a_k^2(f(x_k) - f^*)$

for an appropriately chosen sequence $\{a_k\}$ (Liu, 24 Feb 2025). For strongly convex $f$ , potentials include mixed “kinetic+potential” energy, such as

$W_k = f(x_k) - f^* + \frac{\mu}{2}\|v_k - x^*\|^2,$

where $v_k$ is an auxiliary sequence depending on $\sqrt{\kappa}$ (Liu, 24 Feb 2025).

These constructions yield non-increasing discrete energies, guaranteeing function-value convergence rates.

2. Acceleration Mechanisms: Discrete and Continuous-Time Perspectives

NAG can be interpreted both as a discretized second-order ODE and as a finite-difference integrator for gradient flow:

In the convex regime, the continuous limit corresponds to the ODE [Su–Boyd–Candès]:

$\ddot{x}(t) + \frac{3}{t}\dot{x}(t) + \nabla f(x(t)) = 0.$

For strongly convex problems, the ODE becomes:

$\ddot{x}(t) + 2\sqrt{\mu}\dot{x}(t) + \nabla f(x(t)) = 0$

(Kim et al., 2023).

Recent unified frameworks provide a Lagrangian formalism that interpolates between convex and strongly convex cases, offering time-dependent friction coefficients and yielding a single family of methods with convergence rates continuously dependent on the strong convexity parameter $\mu$ (Kim et al., 2023).

Variable-step-size linear multistep (VLM) interpretations represent NAG as an optimal member of consistent, absolutely stable two-step VLM schemes under certain parameterizations (Nozawa et al., 16 Apr 2024).

3. Convergence Theory: Polynomial and Linear Rates, Point Convergence

NAG achieves:

Sublinear $O(1/k^2)$ function-value decay for smooth convex objectives. Formally,

$f(x_k) - f^* \leq \frac{C}{(k+1)^2}$

for constant $C$ (Liu, 24 Feb 2025, Jang et al., 27 Oct 2025).

Linear (exponential) convergence for smooth strongly convex objectives with known $\mu$ :

$f(x_k) - f^* \leq C\rho^k,\quad \rho = 1 - 1/\sqrt{\kappa}$

(Liu, 24 Feb 2025, Fu et al., 18 Dec 2024, Bao et al., 2023). If $\mu$ is not built into the momentum, original NAG retains global R-linear convergence for strongly convex $f$ , resolving a longstanding question (Bao et al., 2023).

Pointwise convergence: The sequence $\{x_k\}$ converges to a minimizer $x_\infty$ , under standard assumptions and the canonical schedule $t_{k+1} = (1+\sqrt{1+4t_k^2})/2$ (Jang et al., 27 Oct 2025).

Lyapunov methods extend to composite problems (e.g., FISTA), with nonincreasing function gaps and, for monotonic modifications (M-NAG), robust linear rates independent of strong convexity parameters (Fu et al., 18 Dec 2024).

4. Extensions: Non-Euclidean, Stochastic, Ill-posed, and Nonconvex Settings

Riemannian optimization: RNAG generalizes NAG to geodesically convex and strongly convex functions on Riemannian manifolds, with analogous iteration complexity (up to curvature-dependent constants) and use of exponential/logarithm maps and parallel transport; the required metric-distortion lemmas manage curvature-induced discrepancies in quadratic bounds (Kim et al., 2022).
Noisy-gradient regimes: AGNES extends NAG to the multiplicative noise model, achieving $O(1/n^2)$ and $O(\rho^n)$ rates in convex and strongly convex settings, respectively, for arbitrarily high noise, unlike classical NAG, which is unstable for large noise-to-gradient ratios (Gupta et al., 2023).
Ill-posed inverse problems: NAG is provably effective for nonlinear inverse problems with a locally convex residual, using metric-projection and “discrepancy” stopping principles, yielding $O(1/k^2)$ residual convergence and regularization properties (Hubmer et al., 2018).
Nonconvex optimization: Variable-momentum NAG avoids strict saddle points almost surely and offers nearly optimal local rates after escaping nonconvex regions, with exit time from saddle neighborhoods scaling as $O(\log(1/\epsilon)/\sqrt{|\lambda_{\min}|})$ . Suitable choice of momentum parameter allows trade-off between escape efficiency and local convergence (Dixit et al., 2023).

5. Practical Impact and Large-Scale Machine Learning

NAG and its momentum principles are pervasive in deep learning and large-scale applications due to their robustness and empirical acceleration. Recent theoretical advances address over-parameterized and nonconvex models, particularly deep networks:

Over-parameterized deep linear and nonlinear networks: Under high-width and NTK conditions, NAG converges at an $(1-\Theta(1/\sqrt{\kappa}))^t$ rate, outperforming gradient descent $(1-\Theta(1/\kappa))^t$ ; this is established for fully connected and ResNet-style deep architectures (Liu et al., 2022).
Two-layer ReLU networks: NAG, via high-resolution ODE analysis and NTK theory, achieves provable acceleration over heavy-ball (HB) momentum, with linear convergence exponent strictly larger than HB’s, and empirical superiority on standard learning datasets (Liu et al., 2022).
Rectangular matrix factorization and nonconvex problems: Under suitable unbalanced initialization, NAG achieves $O(\kappa\log(1/\epsilon))$ iteration complexity, improving over GD’s $O(\kappa^2\log(1/\epsilon))$ in nonconvex settings, with only minimal overparameterization and no SVD-based initialization required (Xu et al., 12 Oct 2024).

6. Methodological Innovations, Stability, and Parametric Advances

Variable and higher-order momentum: By refining the momentum schedule (e.g., NAG- $\alpha$ with adaptive $a_k$ coefficients), convergence rates can be tuned to arbitrary inverse-polynomial $O(1/k^{2\alpha})$ decay for all $\alpha>0$ at the critical step size, including for monotonic and composite algorithms (M-NAG- $\alpha$ , FISTA- $\alpha$ ) (Fu et al., 17 Jan 2025).
Stability and step-size regimes: From a numerical analysis perspective, NAG is a variable-step-size linear multistep (VLM) method, optimal within a large class of absolutely stable two-step schemes. Higher-order VLMs (e.g., SAG) can extend NAG’s absolute stability region and allow for larger step sizes under the same Lipschitz constraints, directly improving empirical performance on ill-conditioned or large-scale problems (Feng et al., 2021, Nozawa et al., 16 Apr 2024).
Monotonic and modified NAG/FISTA: Lyapunov constructions eliminating standalone kinetic energy yield both NAG and its monotonic variants (M-NAG, M-FISTA) with global linear rates under strong convexity—robust to noise, step size, and model specification (Fu et al., 18 Dec 2024).

7. ODE Frameworks, High-Resolution Dynamics, and Sampling Applications

High-resolution ODEs: Recent analyses move beyond low-resolution ODEs (which predict only polynomial decay) by incorporating gradient-correction terms that accurately simulate discrete NAG's inertia and acceleration. These analyses support continuous dependence of rates on the momentum parameter, optimal $O(1/k^2)$ at critical damping ( $r=2$ ), and fully characterize the underdamped regime ( $r<2$ ) (Chen et al., 2023).
Unified Lagrangian perspectives: A Lagrangian viewpoint parallels optimal control insights, revealing deep connections between Bregman divergences, kernel symmetries, and acceleration mechanisms. These frameworks encompass both function-value and gradient-norm trajectories and extend to higher-order (tensor) optimization (Kim et al., 2023).
Markov Chain Monte Carlo and diffusion-based sampling: Discretized high-resolution NAG-inspired ODEs, with additional noise and modified splitting schemes, yield provably accelerated convergence in Wasserstein distances for log-concave sampling, outperforming underdamped Langevin algorithms in both theory and practice (Li et al., 2020).

Summary Table: Convergence Rates and Notable Regimes

Method/Setting	Convergence Rate	References
Convex, $L$ -smooth (canonical NAG)	$O(1/k^2)$	(Liu, 24 Feb 2025, Jang et al., 27 Oct 2025)
Strongly convex (known $\mu$ )	$O((1 - 1/\sqrt{\kappa})^k)$	(Liu, 24 Feb 2025, Fu et al., 18 Dec 2024, Bao et al., 2023)
Strongly convex (unknown $\mu$ )	$O(\rho^k)$ (R-linear)	(Bao et al., 2023)
Over-param deep nets (NTK regime)	$O((1-1/\sqrt{\kappa})^t)$	(Liu et al., 2022, Liu et al., 2021, Liu et al., 2022)
Matrix factorization (nonconvex)	$O(\kappa\log(1/\epsilon))$	(Xu et al., 12 Oct 2024)
Riemannian NAG	$O(1/k^2),\,O(\rho^k)$	(Kim et al., 2022)
Ill-posed/inverse problems	$O(1/k^2),\,k_*=O(1/\delta)$	(Hubmer et al., 2018)
NAG- $\alpha$ (tunable polynomial)	$O(1/k^{2\alpha})$	(Fu et al., 17 Jan 2025)
Monotonic NAG/M-NAG/M-FISTA	Linear (strongly convex), $O(1/k^2)$ (convex)	(Fu et al., 18 Dec 2024)
NAG under multiplicative noise	$O(1/n^2),\,O(\rho^n)$	(Gupta et al., 2023)

Nesterov’s Accelerated Gradient Method and its generalizations thus form a cornerstone of modern large-scale optimization, balancing optimal complexity, broad applicability, and deep connections to dynamical systems and geometry. The ongoing refinement and extension of NAG's theory and algorithms continue to address the demands of increasingly complex models and data regimes.