---
title: Stochastic Heavy Ball (SHB) Method
url: https://www.emergentmind.com/topics/stochastic-heavy-ball-shb
type: topic
---

# Stochastic Heavy Ball (SHB) Method

Stochastic Heavy Ball (SHB) is the stochastic counterpart of Polyak’s heavy-ball method, obtained by replacing exact gradients with stochastic gradient estimates while retaining an inertial term. In its canonical discrete form, SHB updates
$$
x_{k+1}=x_k-\alpha\,G(x_k;\xi_k)+\beta(x_k-x_{k-1}),
$$
or, equivalently,
$$
v_{k+1}=\beta v_k-\alpha\,G(x_k;\xi_k), \qquad x_{k+1}=x_k+v_{k+1},
$$
with step size $\alpha>0$ and momentum parameter $\beta\in[0,1)$. The method is studied across smooth and nonsmooth optimization, quadratic regression, deep learning, inverse problems, consensus, and stochastic dynamical systems, with a recurring theme: inertial averaging can alter stability, bias decay, and landscape traversal, but its benefits are strongly regime-dependent rather than universal [1808.10396] [1609.04228].

## 1. Algorithmic forms and unified parametrizations

In the smooth empirical-risk setting, SHB is typically posed for
$$
f(x)=\frac{1}{n}\sum_{i=1}^n \ell(x,q_i),
$$
with unbiased stochastic gradients $G(x_k;\xi_k)$ satisfying $\mathbb{E}[G(x_k;\xi_k)]=\nabla f(x_k)$. The position form
$$
x_{k+1}=x_k-\alpha G(x_k;\xi_k)+\beta(x_k-x_{k-1})
$$
and the velocity form
$$
v_{k+1}=\beta v_k-\alpha G(x_k;\xi_k), \qquad x_{k+1}=x_k+v_{k+1}
$$
are treated as exactly equivalent. A widely used unification is the stochastic unified momentum (SUM) method
$$
y_{k+1}=x_k-\alpha G(x_k),\qquad y^s_{k+1}=x_k-s\alpha G(x_k),\qquad x_{k+1}=y_{k+1}+\beta(y^s_{k+1}-y^s_k),
$$
where SHB is recovered by $s=0$, stochastic Nesterov’s accelerated gradient (SNAG) by $s=1$, and stochastic gradient (SG) by $s=1/(1-\beta)$; in that parametrization, SG appears as SGD with effective step size $\alpha/(1-\beta)$ [1808.10396].

A second important representation is the Iterate Moving Average (IMA) form. There SHB is rewritten through auxiliary iterates
$$
z_{k+1}=z_k-\eta_k \nabla f_{v_k}(x_k),\qquad
x_{k+1}=\frac{\lambda_{k+1}}{\lambda_{k+1}+1}x_k+\frac{1}{\lambda_{k+1}+1}z_{k+1},
$$
which is equivalent to the heavy-ball recursion when
$$
\alpha_k=\frac{\eta_k}{1+\lambda_{k+1}},\qquad \beta_k=\frac{\lambda_k}{1+\lambda_{k+1}}.
$$
This viewpoint is central in almost-sure last-iterate analyses and in Polyak-type adaptive step-size rules for momentum, because it makes the effective scaling by $(1-\beta)$ explicit [2006.07867] [2406.04142].

More general time-varying momentum schemes subsume SHB by allowing $\mu_t$, $\alpha_t$, and related coupling coefficients to vary with time. In one such formulation,
$$
y_{t+1}=y_t+a_t v_t-b_t\alpha_t g_{t+1},\qquad
v_{t+1}=\mu_t v_t-\alpha_t g_{t+1},
$$
and classical SHB is recovered from
$$
x_{t+1}=x_t+\mu_t(x_t-x_{t-1})-\alpha_t g_{t+1}
$$
by setting $v_t=x_t-x_{t-1}$ and $y_t=x_t$. This embedding is designed to cover both SHB and SNAG under a single stochastic-approximation analysis with time-varying parameters [2506.11904].

## 2. Smooth convergence theory across nonconvex, convex, and strongly convex regimes

For smooth nonconvex optimization, a central result is that SHB, SNAG, and SG all attain the same order of convergence in the training objective under standard assumptions: $L$-smoothness, bounded gradient norm, and bounded stochastic-gradient variance. In the SUM framework, if
$$
\alpha=\min\left\{\frac{1-\beta}{2L},\frac{C}{\sqrt{t+1}}\right\},
$$
then
$$
\min_{k=0,\dots,t}\mathbb{E}\|\nabla f(x_k)\|^2 = O\!\left(\frac{1}{\sqrt{t}}\right).
$$
The variant-dependent term enters only through $((1-\beta)s-1)^2$, giving a $\beta^2$ contribution for SHB and a $\beta^4$ contribution for SNAG, but the overall conclusion is explicit: SHB and SNAG have no advantage over SG in convergence rate for nonconvex training objectives under the stated assumptions [1808.10396].

Earlier stochastic-heavy-ball analyses place the method in the broader theory of stochastic approximations to second-order dissipative dynamics. In the formulation
$$
X_{n+1}=X_n-\gamma_{n+1}Y_n,\qquad
Y_{n+1}=Y_n+\gamma_{n+1}r_n\big(\nabla f(X_n)-Y_n\big)+\gamma_{n+1}r_n\Delta M_{n+1},
$$
with exponential memory $r_n=r$ or polynomial memory $r_n=r/\Gamma_n$, Lyapunov arguments yield almost sure boundedness, convergence of $Y_n/\sqrt{r_n}$ to zero, and critical-point convergence of $X_n$ under smooth coercive objectives. Under additional ellipticity or sub-Gaussian conditions on the noise, convergence to local minima is established almost surely. In strongly convex settings, the same framework gives non-asymptotic rates such as
$$
\mathbb{E}\big[\|X_n-x^\star\|^2+\|Y_n\|^2\big]\lesssim \gamma_n
$$
for exponential memory and $\beta<1$, together with sharper $n^{-1}$-type behavior when $\beta=1$ and an explicit spectral condition is met [1609.04228].

Almost-sure last-iterate theory was later developed directly for SHB. In the convex, smooth case, with IMA parameters
$$
\lambda_k=\frac{\sum_{t=0}^{k-1}\eta_t}{4\eta_k},\qquad 0<\eta_k\le \frac{1}{8L},
$$
the last iterate satisfies
$$
f(x_k)-f_* = o\!\left(\frac{1}{\sum_{t=0}^{k-1}\eta_t}\right)\quad \text{a.s.},
$$
and $x_k\to x_*\in X_*$ almost surely. This yields
$$
f(x_k)-f_*=o(k^{-1/2+\epsilon}) \quad \text{a.s.}
$$
for $\eta_k=\eta/k^{1/2+\epsilon}$, and
$$
f(x_k)-f_*=o(1/k) \quad \text{a.s.}
$$
in the overparametrized case with constant $\eta\in(0,1/(4L))$. A related line of work gives last-iterate almost-sure and high-probability rates for SHB under globally convex or nonconvex objectives with $\gamma$-Hölder gradients, including
$$
\min_{s\le t}\|\nabla F(w_s)\|^2=o(t^{p-1}) \quad \text{a.s.}
$$
in the nonconvex case and
$$
F(w_{T+1})-F_* = O\!\left(T^{\max(p-1,-2p+1)}\log^2\frac{T}{\delta}\right)
$$
with probability at least $1-\delta$ in the convex Lipschitz-gradient case [2006.07867] [2507.07281].

The time-varying unified theory extends these guarantees to biased stochastic gradients and conditional variances that may grow unboundedly with time. There the sufficient conditions are natural generalizations of Robbins–Monro and Kiefer–Wolfowitz–Blum conditions:
$$
\sum_t \alpha_t^2<\infty,\qquad \sum_t \alpha_t B_t<\infty,\qquad \sum_t \alpha_t^2 M_t^2<\infty,
$$
with $\sum_t\alpha_t=\infty$ giving
$$
\liminf_{t\to\infty}\|\nabla J(x_t)\|=0 \quad \text{a.s.}
$$
Under KL′-type assumptions one further gets $J(y_t)\to 0$ and $v_t\to 0$ almost surely, and under PL one obtains rates of the form
$$
\|\nabla J(x_t)\|=o(t^{-\varepsilon}),\qquad J(y_t)=o(t^{-\varepsilon}),
$$
for every $\varepsilon$ below an explicit threshold determined by the bias and variance decay exponents [2506.11904].

## 3. Acceleration, sample complexity, and the setting-dependence of momentum gains

A major strand of the SHB literature argues that, on stochastic quadratics, momentum does not generically improve sample complexity. One lower-bound analysis for quadratic optimization shows that SHB and ASG require
$$
n_0\in \tilde{\Theta}\!\left(\frac{K}{\epsilon\,\lambda_{\min}(A)^2}\right)
$$
iterations to achieve $\mathbb{E}\|x_n-x_*\|^2\le \epsilon$ under persistent martingale-difference noise, and that vanilla SGD matches the same order. A separate random-matrix analysis of large random least-squares problems gives an exact deterministic characterization of the loss trajectory and proves that small-batch SHB with fixed momentum parameter $\theta$ is equivalent, at the level of average-case loss dynamics, to SGD with effective step size $\gamma/(1-\theta)$. In that model, fixed-momentum SHB provides no actual performance improvement over SGD when step sizes are adjusted correctly [2110.15547] [2106.03696].

Positive acceleration results appear only under more specific structural assumptions. For strongly convex quadratic objectives under anisotropic gradient noise,
$$
\mathbb{E}[n_t n_t^\top]\preceq \sigma^2 H,
$$
and a multistage geometric step-decay schedule
$$
\eta'_\ell=\frac{1}{L}\,C^{-(\ell-1)},\qquad
K=\frac{T}{\log_C(T\sqrt{\kappa})},
$$
SHB with
$$
\beta=\left(1-\frac{1}{\sqrt{\kappa}}\right)^2
$$
achieves last-iterate excess risk
$$
\mathbb{E}[f(x_T)-f(x_\star)]
\le \mathbb{E}[f(x_0)-f(x_\star)]\exp\!\left(-\tilde{\Omega}\!\left(\frac{T}{\sqrt{\kappa}}\right)\right)
+\tilde{O}\!\left(\frac{d\sigma^2}{MT}\right),
$$
yielding $\tilde{\mathcal O}(\sqrt{\kappa})$ acceleration of the bias term while preserving near-minimax variance scaling. A related analysis shows that for strongly convex quadratics, SHB with the deterministic heavy-ball parameters
$$
\eta=\frac{a}{L},\qquad \beta=\left(1-\frac{\sqrt{a}}{2\sqrt{\kappa}}\right)^2,
$$
accelerates when the mini-batch size exceeds an explicit $\kappa$-dependent threshold
$$
b^*:=n\cdot \max\left\{\frac{1}{1+n/(C\kappa^2)},\frac{1}{1+(na)/3}\right\}.
$$
Under $b\ge b^*$, one gets accelerated neighborhood convergence
$$
\mathbb{E}\|x_T-x^*\|_2
\le 6\sqrt{2}\sqrt{a\kappa}\exp\!\left(-\frac{\sqrt{a}}{4}\frac{T}{\sqrt{\kappa}}\right)\|x_0-x^*\|_2
+\frac{12\sqrt{a}}{\mu}\min\left\{1,\frac{\zeta}{\sqrt{a}}\right\}\chi,
$$
and a multistage schedule yields
$$
O\!\left(\exp\!\left(-\frac{T}{\sqrt{\kappa}}\right)+\frac{\sigma}{\sqrt{T}}\right)
$$
in expectation for the $\ell_2$ distance to the minimizer on quadratics, alongside
$$
O\!\left(\exp\!\left(-\frac{T}{\kappa}\right)+\frac{\sigma^2}{T}\right)
$$
for general smooth strongly convex objectives in squared norm [2312.14567] [2401.06738].

Taken together, these results suggest that claims about SHB “acceleration” are inseparable from the noise model, batch regime, stepsize schedule, and objective class. In the stochastic nonconvex deep-learning analysis of SUM, acceleration over SG is absent for training convergence [1808.10396]; in quadratic sample-complexity analyses, fixed-momentum gains disappear after effective-step normalization [2110.15547] [2106.03696]; in contrast, anisotropic noise, large batches, or carefully designed stagewise schedules can recover accelerated bias decay [2312.14567] [2401.06738].

## 4. Stability, generalization, and landscape properties

Although SHB does not improve nonconvex training rates over SG in the unified deep-learning analysis, it does improve uniform stability. For two runs on neighboring datasets, the stability recursion depends on cumulative gradient weights
$$
\eta_k^t=\frac{1}{1-\beta}-\beta^{t-k+1}\frac{1-s(1-\beta)}{1-\beta}.
$$
Plugging in the special cases gives
$$
\eta_k^t(\mathrm{SHB})<\eta_k^t(\mathrm{SNAG})<\eta_k^t(\mathrm{SG}),
$$
hence
$$
\Delta_t(\mathrm{SHB})\le \Delta_t(\mathrm{SNAG})\le \Delta_t(\mathrm{SG}),
$$
so SHB is the most stable of the three and SG the least stable. Under identical iteration counts, adding momentum in SHB or SNAG reduces the uniform-stability coefficient and can improve generalization even though the training convergence rate remains $O(1/\sqrt t)$. In the reported CIFAR-10 experiments, the best initial step size was $0.001$ for SHB and $0.01$ for SNAG and SG; with the same initial $\alpha=0.001$, training-error curves were very close across methods, but the absolute train–test gap was smaller for SHB and SNAG than for SG [1808.10396].

A mean-field treatment on wide two- and three-layer neural networks adds a distinct notion of stability. In that setting SHB is modeled as a second-order mean-field dynamics, and the finite-width trajectory is compared to a kinetic mean-field limit with dimension-free non-asymptotic bounds. This yields dropout-stability bounds of order
$$
\epsilon_D \le K(\gamma,T)\left[\frac{\sqrt{\log n}+\delta}{\sqrt n}
+\sqrt{\varepsilon}\,(\sqrt{D+\log n}+\delta)\right]
$$
for two-layer networks, with analogous three-layer bounds, and also establishes low-loss connectivity between SHB solutions. Under universal approximation, full-support initialization, and a stated convergence mode, the three-layer mean-field heavy-ball dynamics converges to the global minimum of the unregularized population risk [2210.06819].

These analyses separate two frequently conflated questions. One concerns optimization speed, where SHB may or may not improve rates depending on regime. The other concerns the geometry and robustness of the attained solution, where the literature records smaller train–test gaps, stronger uniform stability, dropout-stability, and low-loss connectivity under SHB dynamics [1808.10396] [2210.06819].

## 5. Nonsmooth analysis and continuous-time stochastic dynamics

SHB has also been analyzed beyond smooth Euclidean gradients. For nonsmooth nonconvex objectives
$$
F(w)=\mathbb{E}_{\xi\sim P}[f(w,\xi)],
$$
one formulation uses
$$
w_{k+1}=w_k-\mu_k\,v(w_k,\xi_k)+\nu_k(w_k-w_{k-1}),
$$
or equivalently
$$
w_{k+1}=w_k-\alpha_k y_k,\qquad
y_{k+1}=\beta_k v(w_{k+1},\xi_{k+1})+(1-\beta_k)y_k,
$$
under diminishing stepsizes, bounded iterates, and an exponential-memory regime $\alpha_k/\beta_k\to r>0$. The analysis is built on semialgebraic or definable assumptions, conservative gradients, and the differential inclusion
$$
\dot w=-r\,y,\qquad \dot y\in D_F(w)-y,
$$
with Lyapunov function
$$
E(w,y)=F(w)+\frac{r}{2}\|y\|^2.
$$
Essential accumulation points and minimal-energy accumulation points satisfy
$$
0\in D_F(w^*),\qquad y^*=0
$$
almost surely, and under definable distributional assumptions every accumulation point has this property while $F(w_k)$ converges. The same framework identifies “artificial critical points” arising from conservative calculus and proves that randomized initializations avoid them almost surely, so convergence is to Clarke critical points on a full-measure set of initial conditions [2304.13328].

Continuous-time stochastic SHB has been studied through diffusion approximations of small-step heavy-ball recursions. In one formulation,
$$
dx(t)=v(t)\,dt+\sigma_1(x(t),v(t))\,dW_t^{(1)},\qquad
dv(t)=-(\alpha v(t)+\nabla f(x(t)))\,dt+\sigma_2(x(t),v(t))\,dW_t^{(2)},
$$
and after rescaling time,
$$
dX^\epsilon(t)=V^\epsilon(t)\,dt+\sqrt{\epsilon}\,\sigma_1\,dW_t^{(1)},\qquad
dV^\epsilon(t)=(-\alpha V^\epsilon(t)-\nabla f(X^\epsilon(t)))\,dt+\sqrt{\epsilon}\,\sigma_2\,dW_t^{(2)}.
$$
For Morse functions with the strict saddle and strong saddle properties, the expected exit time from a saddle neighborhood scales as
$$
\mathbb{E}\tau^\epsilon_{(x,v)} \lesssim \frac{1}{2\mu_0}\log(\epsilon^{-1}),
$$
where
$$
\mu_0=\frac{-\alpha+\sqrt{\alpha^2-4\lambda_{\min}}}{2},
$$
and a chain of $k$ saddles yields
$$
\limsup_{\epsilon\to 0}\frac{\mathbb{E}T_x}{\epsilon^{-1}\log(\epsilon^{-1})}
\le \frac{k}{4\gamma_1}\big(\sqrt{\alpha^2+4\gamma_1}+\alpha\big).
$$
This suggests fast saddle escape in the diffusion model [1712.05733].

A separate Hilbert-space theory studies stochastic Heavy Ball systems directly at the trajectory level. For convex minimization with Lipschitz gradient,
$$
dY(t)=X(t)\,dt,\qquad
dX(t)=(-\lambda X(t)-b(t)\nabla f(Y(t)))\,dt+\sigma_Y(t)\,dW(t),
$$
square-integrable diffusion and
$$
\sup_{t\ge t_0}\frac{\dot b(t)}{b(t)}<\lambda
$$
imply almost sure weak convergence of $Y(t)$ to a minimizer, $X(t)\to 0$, and
$$
f(Y(t))-\inf_H f = o(1/b(t)) \quad \text{a.s.}
$$
With time rescaling, this system becomes a stochastic Su–Boyd–Candès-type vanishing-damping dynamics and yields
$$
f(Z(s))-\inf_H f = o(1/s^2)\quad \text{a.s.}
$$
for $\alpha>3$. The same paper extends the analysis to monotone equations via a corrected SHB system and shows equivalence to a stochastic Fast OGDA-type second-order dynamics with residual and gap rates
$$
\|V(Z(s))\| = o(1/s),\qquad \langle Z(s)-z^*,V(Z(s))\rangle=o(1/s)\quad \text{a.s.}
$$
[2510.02951].

## 6. Linear algebra, PCA, and distributed consensus

In linear systems, SHB admits a sketch-and-project interpretation. For the stochastic reformulation of a consistent system $Ax=b$,
$$
f_S(x)=\frac{1}{2}\|S^\top(Ax-b)\|_2^2,\qquad \nabla f_S(x)=A^\top S S^\top(Ax-b),
$$
the adaptive SHBM update
$$
x^{k+1}=x^k-\alpha_k \nabla f_{S_k}(x^k)+\beta_k(x^k-x^{k-1})
$$
chooses $\alpha_k$ and $\beta_k$ by projecting the minimum-norm solution onto the affine span generated by the current stochastic gradient and the previous step. This yields linear convergence in expectation with a contraction factor strictly tighter than the corresponding basic stochastic Polyak method whenever the geometric gain $\gamma_k>0$. In the deterministic case, the same recursion becomes a conjugate-gradient-type method; in fact, for $\Omega_k=\{I\}$ it reduces to CGNE, and the stochastic generalization induces a stochastic conjugate-gradient framework [2305.05482].

For PCA, the heavy-ball idea appears in variance-reduced power iteration. Given covariance
$$
C=\frac{1}{n}AA^\top,
$$
the variance-reduced heavy-ball power update uses
$$
g_t=Cw_t+(C_t-C)Pw_t
$$
and
$$
w_{t+1}=2\big((1-\eta)w_t+\eta g_t\big)-\beta w_{t-1},
$$
with normalization after each step. The resulting method enjoys global linear convergence in expectation to the leading eigenvector, with a per-epoch contraction factor that splits into a deterministic accelerated term and a stochastic variance term. The analysis emphasizes that large mini-batches reduce the covariance variance parameter $K$, allowing larger $\eta$ and making acceleration especially visible when the eigengap is small [1901.08179].

SHB also acts as a randomized gossip method for average consensus. If $A$ is an incidence matrix encoding the equalities $x_i=x_j$ on graph edges, then the sketch-and-project SHB update
$$
x^{k+1}=x^k-\omega A^\top H_k(Ax^k-b)+\beta(x^k-x^{k-1})
$$
becomes a distributed momentum gossip protocol. In the pairwise case, when edge $(i,j)$ is selected,
$$
x_i^{k+1}=\frac{2-\omega}{2}x_i^k+\frac{\omega}{2}x_j^k+\beta(x_i^k-x_i^{k-1}),
$$
$$
x_j^{k+1}=\frac{2-\omega}{2}x_j^k+\frac{\omega}{2}x_i^k+\beta(x_j^k-x_j^{k-1}),
$$
while nonselected nodes perform only the momentum update. In the block case, every node in a sampled connected component moves toward the component average plus the same heavy-ball term. The protocol preserves the network average when $x^0=x^1$, and the expected error contracts linearly under explicit spectral conditions on the mean sketch matrix $W=\mathbb{E}[A^\top H A]$ [1809.08657].

## 7. Inverse problems, adaptive step-sizes, and regularization viewpoints

For linear ill-posed inverse problems, one stochastic heavy-ball scheme uses
$$
x_{n+1}^\delta=x_n^\delta-\alpha_n \eta_{i_n}A_{i_n}^*(A_{i_n}x_n^\delta-y_{i_n}^\delta)+\beta_n(x_n^\delta-x_{n-1}^\delta),
$$
with
$$
\alpha_n=\frac{1}{n+2},\qquad \beta_n=\frac{n}{n+2}.
$$
Its iterate-moving-average representation
$$
z_{n+1}^\delta=z_n^\delta-\eta_{i_n}A_{i_n}^*(A_{i_n}x_n^\delta-y_{i_n}^\delta),\qquad
x_{n+1}^\delta=\frac{n+1}{n+2}x_n^\delta+\frac{1}{n+2}z_{n+1}^\delta
$$
enables regularization analysis in expectation. If $0<\eta_i<1/\|A_i\|^2$ and the stopping index satisfies $n_\delta\to\infty$ with $\delta^2 n_\delta\to 0$, then
$$
\mathbb{E}\|x_{n_\delta}^\delta-x^\dagger\|^2\to 0.
$$
Under the benchmark source condition
$$
x^\dagger-x_0=A^*w^\dagger,
$$
one obtains
$$
\mathbb{E}\|x_n^\delta-x^\dagger\|^2
\le \frac{2pM_0}{c_0(n+1)}+\frac{2\bar\eta\,n\delta^2}{c_0p},
$$
and the a priori choice $(n_\delta+1)/p\asymp \delta^{-1}$ gives
$$
\mathbb{E}\|x_{n_\delta}^\delta-x^\dagger\|^2 = O(\delta)
$$
[2406.16814].

A more general adaptive SHB for nonlinear ill-posed systems incorporates a strongly convex penalty $\Theta$, dual variables $\xi_n^\delta$, an adaptive step size, an adaptive momentum coefficient, and a discrepancy-inspired stopping rule. The update reads
$$
\xi_{n+1}^\delta=\xi_n^\delta-t_n^\delta g_n^\delta+\beta_n^\delta m_n^\delta,\qquad
x_{n+1}^\delta=\arg\min_x\{\Theta(x)-\langle \xi_{n+1}^\delta,x\rangle\},
$$
with
$$
m_n^\delta=\xi_n^\delta-\xi_{n-1}^\delta,
$$
and $t_n^\delta$ chosen from a residual-based minimum formula involving $\mu_0,\mu_1,\eta,\tau$. The momentum parameter is computed by minimizing a quadratic upper bound,
$$
\beta_n^\delta=\min\left\{\max\left\{0,\frac{t_n^\delta\langle g_n^\delta,m_n^\delta\rangle-2\sigma\tilde\gamma_n^\delta}{\|m_n^\delta\|^2}\right\},\beta\right\},
$$
subject to stability thresholds. The resulting Bregman-descent inequality
$$
D_{\xi_{n+1}^\delta}\Theta(\hat x,x_{n+1}^\delta)-D_{\xi_n^\delta}\Theta(\hat x,x_n^\delta)
\le -c_0 t_n^\delta\|r_n^\delta\|^r-\frac{1}{2}\upsilon_1\beta_n^\delta\|m_n^\delta\|^2
$$
yields almost sure finite termination of the discrepancy-style stopping rule, almost sure convergence for exact data, and regularization in both almost-sure and expectation senses for noisy data [2605.13144].

Adaptive step-size design has also been transplanted from stochastic Polyak step-sizes to SHB. Using the IMA viewpoint and the identity $\eta_t=(1-\beta)\gamma_t$, three momentum-aware rules are proposed:
$$
\gamma_t=(1-\beta)\min\left\{\frac{f_{S_t}(x^t)-\ell_{S_t}^*}{c\|\nabla f_{S_t}(x^t)\|^2},\gamma_b\right\}
$$
for MomSPS$_{\max}$,
$$
\gamma_t=\min\left\{(1-\beta)\frac{f_{S_t}(x^t)-\ell_{S_t}^*}{c_t\|\nabla f_{S_t}(x^t)\|^2},\frac{\gamma_{t-1}c_{t-1}}{c_t}\right\}
$$
for MomDecSPS, and
$$
\gamma_t=\min\left\{(1-\beta)\frac{f_{S_t}(x^t)-\ell_{S_t}^*}{c\|\nabla f_{S_t}(x^t)\|^2\sqrt{\sum_{s=0}^t(f_{S_s}(x^s)-\ell_{S_s}^*)}},\gamma_{t-1}\right\}
$$
for MomAdaSPS. MomSPS$_{\max}$ converges to a neighborhood for convex smooth problems without interpolation, and to the true solution at a fast rate matching deterministic HB under interpolation. MomDecSPS and MomAdaSPS are described as the first adaptive step-sizes for SHB that guarantee convergence to the exact minimizer without a priori problem parameters and without interpolation [2406.04142].

Across these application domains, SHB functions not only as a momentum heuristic but as a regularization mechanism, a projection-based accelerator, and a continuous-time inertial model. The common algebraic skeleton is simple, but the provable behavior depends sharply on how stochasticity enters, how parameters are scheduled, and whether the ambient problem is a smooth objective, a nonsmooth definable landscape, a spectral estimation task, or an inverse problem with explicit regularization [2406.16814] [2605.13144] [2406.04142].

Source: https://www.emergentmind.com/topics/stochastic-heavy-ball-shb