Stochastic Gradient Descent with Momentum

• Stochastic Gradient Descent with Momentum is an optimization method that incorporates past gradients to stabilize updates and accelerate convergence.
• It aggregates gradients via a momentum term, reducing noise and improving performance in convex, strongly convex, and nonconvex settings.
• Its effectiveness relies on careful tuning of learning rates, momentum coefficients, and batch-size schedules, with variants addressing heavy-tailed noise and saddle-point dynamics.

Stochastic Gradient Descent with Momentum (SGDM), often called heavy-ball momentum, augments plain stochastic gradient descent by introducing a velocity or momentum state that aggregates past stochastic gradients. In its standard role, this moving average stabilizes the search direction, filters gradient noise, and can accelerate progress along directions of persistent descent (Liu et al., 2020). Modern analyses treat SGDM not as a single update rule but as a family of closely related methods, including Polyak’s heavy-ball form, normalized heavy-ball variants, and generalized formulations that also recover Nesterov-type schemes. Across smooth convex, strongly convex, and nonconvex optimization, its behavior is now understood to depend sharply on the learning-rate schedule, momentum coefficient, batch-size schedule, geometry of the objective, and the distributional structure of gradient noise (Kamo et al., 15 Jan 2025, Wang et al., 26 Feb 2026).

1. Algorithmic forms and equivalent representations

The canonical heavy-ball form maintains a momentum vector $v_t$ and updates

$$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$

where $g_t$ is a stochastic or mini-batch gradient, $\eta>0$ is the learning rate, and $\beta\in[0,1)$ is the momentum weight. Unrolling the recursion gives an exponentially weighted moving average,

$v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$

which makes explicit that SGDM emphasizes recent gradients while retaining a geometrically decaying memory of earlier ones (Liu et al., 2020).

Equivalent position-only forms are also standard. In deterministic heavy-ball gradient descent,

$$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$

and the stochastic analogue replaces $\nabla E(x^k)$ by a mini-batch gradient $g_k=\nabla E_k(x^k)$ (Ghosh et al., 2023). In empirical risk minimization over a dataset $S=\{z_i\}_{i=1}^n$, the mini-batch gradient is commonly written

$$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$0

with unbiasedness and variance scaling

$$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$1

under the standard assumptions used in recent nonconvex analyses (Kamo et al., 15 Jan 2025).

A normalized heavy-ball variant writes

$$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$2

This formulation is equivalent to the heavy-ball version up to a rescaling of $$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$3 by $$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$4, and several recent convergence results are stated for both forms (Kamo et al., 15 Jan 2025).

A broader unifying parameterization writes

$$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$5

This generalized SGDM recovers vanilla SGD when $$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$6, Polyak’s heavy-ball momentum when $$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$7, and Nesterov momentum through a particular coupling of $$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$8, $$v_{t+1} = \beta\, v_t + g_t,\qquad w_{t+1} = w_t - \eta\, v_{t+1},$$9, and $g_t$0. The same framework also contains stochastic unified momentum, quasi-hyperbolic momentum, and MASS as special cases (Wang et al., 26 Feb 2026).

2. Convergence theory across convex, strongly convex, and nonconvex regimes

For smooth nonconvex objectives with unbiased stochastic gradients of bounded variance, SGDM can match the standard SGD stationarity guarantees. An improved analysis of heavy-ball SGDM establishes

$g_t$1

under explicit step-size restrictions depending on $g_t$2 and $g_t$3, thereby removing the $g_t$4 degradation that appeared in earlier SGDM analyses (Liu et al., 2020). In the strongly convex case, the same analysis yields

$g_t$5

so SGDM converges linearly to a noise-dominated neighborhood (Liu et al., 2020).

A more general recent framework proves ergodic $g_t$6 convergence for smooth convex objectives with constant parameters, accelerated $g_t$7 rates at intermediate points under a time-varying parameter coupling in the deterministic case, and a stochastic counterpart of order $g_t$8. In the nonconvex setting it gives sublinear convergence to stationary points,

$g_t$9

and under the Polyak–Łojasiewicz condition it establishes linear convergence to a neighborhood whose radius is proportional to $\eta>0$0 (Wang et al., 26 Feb 2026).

In strongly convex mini-batch settings, SGDM can improve the transient rate relative to mini-batch SGD without changing the order of the steady-state bias. For a linearized analysis based on the spectral radius of a $\eta>0$1 matrix $\eta>0$2, mini-batch SGDM converges geometrically to a neighborhood of the optimum, permits a broader range of learning rates through the condition

$\eta>0$3

and, in quadratic problems, attains the accelerated factor

$\eta>0$4

for appropriate $\eta>0$5 and momentum weight $\eta>0$6 (Tang et al., 2023).

This acceleration is not universal. On high-dimensional random least-squares models with small-batch sampling and fixed momentum parameter, stochastic heavy-ball momentum yields no actual performance improvement over SGD once the effective learning rate is adjusted correctly. In that regime, the operative parameter is

$\eta>0$7

and fixed-$\eta>0$8 SGDM is equivalent, in average-case loss dynamics, to SGD run at that effective learning rate (Paquette et al., 2021).

A separate limitation arises in nonconvex deep-network training with constant $\eta>0$9, constant $\beta\in[0,1)$0, and constant batch size $\beta\in[0,1)$1. Under standard smoothness and bounded-variance assumptions, mini-batch SGDM obeys

$\beta\in[0,1)$2

so the expected full-gradient norm does not vanish as $\beta\in[0,1)$3; a persistent variance floor remains (Kamo et al., 15 Jan 2025).

3. Hyperparameters, stagewise schedules, and increasing batch size

The dependence of SGDM on $\beta\in[0,1)$4, $\beta\in[0,1)$5, and $\beta\in[0,1)$6 is now understood to be structural rather than merely heuristic. A multistage heavy-ball analysis establishes the first convergence guarantee for stagewise SGDM with coupled parameters satisfying

$\beta\in[0,1)$7

and shows that the stagewise strategy is beneficial relative to fixed parameters: large early learning rates improve the optimization term, while later smaller learning rates reduce the noise floor (Liu et al., 2020).

The most explicit recent batch-size result concerns mini-batch SGDM with constant $\beta\in[0,1)$8 and $\beta\in[0,1)$9. With a constant batch size $v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$0, SGDM retains the variance floor above. By contrast, with an exponentially increasing schedule

$v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$1

over training stages and the condition $v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$2, the expected gradient norm decays as

$v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$3

even though the learning rate and momentum are kept constant. The mechanism is that the variance term inside the momentum buffer decreases quickly enough to offset the geometric weighting $v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$4 of past noise (Kamo et al., 15 Jan 2025).

A further Lyapunov-based analysis unifies three practical scheduling strategies for SGDM: constant batch size with decaying learning rate, increasing batch size with decaying learning rate, and increasing batch size with increasing learning rate. It shows a strict hierarchy. Constant batch size with decaying learning rate does not guarantee convergence of the expected gradient norm. Increasing batch size with decaying learning rate does. Increasing both batch size and learning rate also does, and moreover achieves a faster decay rate than the other two strategies; for a phase-based exponential schedule, the bound scales as $v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$5 in the number of update phases $v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$6. The same work reports that a warm-up schedule empirically outperformed all other strategies in convergence behavior (Kondo et al., 5 Aug 2025).

Practical prescriptions extracted from these analyses are correspondingly concrete. Increasing the mini-batch geometrically while keeping $v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$7 and $v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$8 fixed is theoretically justified, with doubling or quadrupling as representative choices; for $v_t = \sum_{i=1}^{t} \beta^{t-i} g_i,$9, $$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$0 already satisfies $$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$1. When physical memory is insufficient, gradient accumulation can emulate larger batches, and the batch size should still be capped by dataset size and hardware throughput (Kamo et al., 15 Jan 2025).

4. Continuous-time limits, implicit regularization, and saddle-point dynamics

Continuous-time analysis has become a major tool for understanding SGDM. One line of work derives a hyperparameter-dependent kinetic SDE whose overdamped reduction depends on the effective noise scale

$$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$2

with $$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$3 the learning rate and $$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$4 the momentum coefficient. In that framework, the linear convergence rate is governed by a Kramers–Fokker–Planck spectral gap whose barrier exponent depends on $$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$5; for $$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$6, momentum accelerates convergence relative to SGD, but the stationary gap grows with the same scale $$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$7 (Shi, 2021).

A complementary deterministic and stochastic viewpoint shows that heavy-ball momentum follows a continuous trajectory minimizing a modified loss. In the deterministic case,

$$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$8

so the implicit gradient regularization induced by heavy-ball momentum is stronger than that of gradient descent by the factor

$$x^{k+1} = x^k - h\,\nabla E(x^k) + \beta\,(x^k-x^{k-1}),$$9

In the stochastic setting, the expected implicit regularization remains stronger than for SGD, while the covariance of the momentum-driven force is reduced by the factor $\nabla E(x^k)$0 (Ghosh et al., 2023).

For a particular time-varying momentum schedule,

$\nabla E(x^k)$1

SGDM converges to the deterministic ODE

$\nabla E(x^k)$2

as the step size goes to zero. This continuous-time representation supports an anytime high-probability guarantee: for smooth convex $\nabla E(x^k)$3, there exists a constant $\nabla E(x^k)$4 such that

$\nabla E(x^k)$5

for any confidence parameter $\nabla E(x^k)$6 (Feng et al., 2023).

Momentum also changes second-order dynamics. In nonconvex stochastic optimization, heavy-ball SGDM has been shown to escape saddle points faster than SGD under a Correlated Negative Curvature assumption on the momentum. The core mechanism is multiplicative amplification of motion along negative-curvature directions through curvature-weighted products $\nabla E(x^k)$7, and the resulting escape-time threshold improves by a factor of $\nabla E(x^k)$8 relative to SGD (Wang et al., 2021).

When the gradient noise is heavy-tailed rather than approximately Gaussian, the usual underdamped Langevin surrogate of SGDM can become biased: the optima of its stationary distribution need not match the minima of the cost function. A fractional underdamped Langevin dynamics corrects this by replacing the standard kinetic structure so that the invariant law becomes Gibbs again; its Euler discretization has close algorithmic connections to natural-gradient-like preconditioning and gradient clipping (Şimşekli et al., 2020).

5. Generalization, algorithmic stability, and heavy-tailed effects

The relation between momentum and generalization has been especially subtle. A classical uniform-stability analysis shows that there exists a convex loss for which the stability gap of multi-epoch heavy-ball SGDM becomes unbounded, and more generally that for suitable convex constructions the stability gap can scale as $\nabla E(x^k)$9. The same work introduces SGD with early momentum (SGDEM), in which momentum is used only in early iterations, and proves uniform-stability guarantees for smooth Lipschitz losses under broad decaying step-size schedules. In the strongly convex case, it also identifies a parameter range in which standard SGDM run for multiple epochs is stable (Ramezani-Kebrya et al., 2018).

Recent work closes much of this gap in smooth convex learning by shifting from uniform stability to on-average model stability. For a generalized SGDM framework encompassing both Polyak and Nesterov momentum, one can prove tight on-average stability bounds for any $g_k=\nabla E_k(x^k)$0 without assuming Lipschitz losses. Combined with optimization bounds, this yields optimal excess population risk bounds for both heavy-ball and Nesterov SGDM in smooth convex problems (Lei et al., 27 May 2026).

Heavy-tailed noise changes the picture again. For quadratic losses under $g_k=\nabla E_k(x^k)$1-stable Lévy perturbations, SGDM admits a looser Wasserstein stability bound than SGD because the relevant spectral quantity $g_k=\nabla E_k(x^k)$2 is always no larger than the corresponding SGD quantity $g_k=\nabla E_k(x^k)$3. The same analysis proves a uniform-in-time discretization error bound

$g_k=\nabla E_k(x^k)$4

for the discrete approximation of the Lévy-driven SDE and shows that, in this heavy-tailed regime, momentum can be harmful for generalization (Dang et al., 2 Feb 2025).

6. Variants, comparisons, and unresolved boundaries

SGDM now sits inside a much broader design space of momentum-based stochastic optimizers. Generalized SGDM contains Polyak, Nesterov, stochastic unified momentum, quasi-hyperbolic momentum, and MASS as parameter specializations (Wang et al., 26 Feb 2026). Other momentum constructions alter the weighting kernel itself: an inverse-decay momentum scheme replaces exponential weights by polynomially decaying weights $g_k=\nabla E_k(x^k)$5, retains $g_k=\nabla E_k(x^k)$6 time and memory complexity, and empirically outperforms both SGDM and Adam on several MNIST models (Liu et al., 2020).

At the same time, several comparisons delimit where classical SGDM should not be expected to dominate. Under non-uniform smoothness, Adam attains convergence rates matching lower bounds for first-order stochastic methods, whereas there exist instances in which SGDM fails to converge in expectation for any learning rate and any momentum parameter (Wang et al., 2024). In strongly convex proximal settings, stochastic proximal point with momentum admits a wider stable range of $g_k=\nabla E_k(x^k)$7 and more favorable dependence on problem constants than SGDM, while still converging linearly to a neighborhood (Kim et al., 2021).

Theoretical limits also remain model-dependent. On random least-squares quadratics, fixed-momentum stochastic heavy-ball is effectively just SGD with a rescaled learning rate, and genuine acceleration requires dimension- or iteration-dependent momentum schedules such as sDANA (Paquette et al., 2021). In nonconvex settings, convergence proofs still typically rely on smoothness and bounded-variance assumptions, and in heavy-tailed or small-data regimes increasing batch size or momentum need not help in the manner predicted by bounded-variance theory (Kamo et al., 15 Jan 2025, Dang et al., 2 Feb 2025).

Taken together, these results portray SGDM not as a uniformly accelerated replacement for SGD, but as a structured family of stochastic inertial methods whose behavior is governed by a precise interaction among memory, noise, geometry, and scheduling. In smooth convex and many smooth nonconvex regimes, momentum enlarges useful parameter ranges, improves transient dynamics, strengthens implicit regularization, and can accelerate escape from saddles. In small-batch quadratic problems, non-uniformly smooth objectives, or heavy-tailed-noise regimes, the same mechanism can create variance floors, stability degradation, or bias, motivating increasing-batch schedules, early-momentum schedules, generalized momentum parameterizations, or alternative corrected dynamics (Kondo et al., 5 Aug 2025, Lei et al., 27 May 2026).