Double-Clipped NSGD-MVR

• The paper introduces double-clipped NSGD-MVR, an algorithm that extends NSGD-MVR with dual-level gradient clipping to secure high-probability convergence under heavy-tailed noise conditions.
• It utilizes momentum-variance reduction combined with tailored clipping strategies to balance bias and variance while achieving near-optimal sample complexity under relaxed smoothness assumptions.
• The framework provides rigorous high-probability convergence guarantees that improve on expectation-based methods, notably achieving optimal rates for canonical parameter settings (p=q=2).

Double-Clipped NSGD-MVR is an algorithmic framework designed for stochastic nonconvex optimization under heavy-tailed noise conditions, where the underlying gradient noise is characterized by bounded $p$-th central moment ($p$-BCM) for $p \in (1,2]$. The method extends the Normalized Stochastic Gradient Descent with Momentum Variance Reduction (NSGD-MVR) by employing dual-level gradient clipping and tailored high-probability analysis to deliver near-optimal complexity guarantees even in regimes where conventional assumptions do not hold. Double-Clipped NSGD-MVR provides explicit high-probability convergence rates under weak smoothness—relaxed mean-squared smoothness ($q$-WAS, $q\in[1,2]$) and $(q,\delta)$-similarity—broadening applicability and improving upon previous lower bounds for stochastic optimization in heavy-tailed scenarios (Fradin et al., 21 Dec 2025).

1. Algorithmic Structure and Update Rules

The Double-Clipped NSGD-MVR algorithm operates on iterates $x_t$ over $T$ steps, with primary parameters consisting of step-size $\gamma$, momentum parameter $\alpha\in(0,1)$, clipping thresholds $\lambda_1,\lambda_2>0$, and access to a stochastic gradient oracle $\nabla f(x,\xi)$:

• Gradient Clipping at $\lambda_2$:

$$\tilde g_t = \mathrm{clip}(\nabla f(x_t,\xi_t),\lambda_2) = \min\Big\{1,\,\frac{\lambda_2}{\|\nabla f(x_t,\xi_t)\|}\Big\}\nabla f(x_t,\xi_t)$$

• Momentum-Variance-Reduction (MVR) Update with Difference Clipping at $\lambda_1$:

$$\bar\Delta_t = \mathrm{clip}\big(\nabla f(x_t,\xi_t)-\nabla f(x_{t-1},\xi_t),\,\lambda_1\big)$$

$$g_t = (1-\alpha)\big(g_{t-1}+\bar\Delta_t\big) + \alpha\,\tilde g_t$$

• Gradient Normalization and Descent:

$x_{t+1} = x_t - \gamma\frac{g_t}{\|g_t\|}$

The composite update ensures stability under heavy-tailed noise by constraining the magnitude of both raw and difference gradients through dual clipping, followed by normalization.

2. Analytical Assumptions and Heavy-Tailed Regime

Double-Clipped NSGD-MVR is analyzed under the following formal model assumptions:

• Bounded $p$-th Central Moment (Heavy-Tailed Noise):

$$E_\xi[\nabla f(x,\xi)] = \nabla F(x), \quad E_\xi\|\nabla f(x,\xi)-\nabla F(x)\|^p \le \sigma_1^p$$

where $p\in(1,2]$ quantifies departure from sub-Gaussianity.

• Relaxed Mean-Squared Smoothness ($q$-WAS, $q\in[1,2]$):

$$E_\xi\left\|\nabla f(x,\xi)-\nabla f(y,\xi)\right\|^q \le \bar L^q\,\|x-y\|^q$$

• $(q,\delta)$-Similarity Plus $L_1$-Smoothness:

$$\|\nabla F(x)-\nabla F(y)\| \le L_1\|x-y\|,\qquad E_\xi\Big\|\big[\nabla f(x,\xi)-\nabla f(y,\xi)\big]-\big[\nabla F(x)-\nabla F(y)\big]\Big\|^q \le \delta^q\,\|x-y\|^q$$

This generalized smoothness expands the analysis to non-standard exponents and similarity regimes, surpassing prior works that restricted to $q=2$ or necessitated stricter gradient/Hessian regularity.

3. High-Probability Convergence Guarantees

Double-Clipped NSGD-MVR supports high-probability convergence to stationary points. Setting

$$\begin{align*} \alpha &= \max\left\{T^{-\frac{p}{2p-1}},\,T^{-\frac{pq}{p(2q+1)-2q}}\right\} \ \lambda_2 &= 4\sqrt{\bar L\Delta_1}\vee\sigma_1\alpha^{-1/p} \ \lambda_1 &= 2\gamma\bar L\,\alpha^{-1/q} \ \gamma &= O\left(\min\left\{\sqrt{\frac{\Delta_1}{\bar L T}},\,\alpha\sqrt{\frac{\Delta_1}{\bar L}},\,\frac{1}{\alpha T\ln(T/\beta)}\sqrt{\frac{\Delta_1}{\bar L}}\right\}\right) \end{align*}$$

with $g_0=0$, the algorithm achieves:

$$\frac{1}{T}\sum_{t=0}^{T-1}\|\nabla F(x_t)\| = O\left(\frac{\sqrt{\bar L\Delta_1}+\sigma_1}{T^{\min\left\{\frac{p-1}{2p-1},\,\frac{q(p-1)}{p(2q+1)-2q}\right\}}}\ln\frac{T}{\beta}\right)$$

with probability at least $1-\beta$. To guarantee $\min_{t\le T}\|\nabla F(x_t)\|\le\epsilon$, one requires

$$T = \widetilde{O}\left(\epsilon^{-\max\left(\frac{2p-1}{p-1},\,\frac{p(2q+1)-2q}{q(p-1)}\right)}\right)$$

For the canonical parameter choice $p=q=2$, this simplifies to $T=\widetilde{O}(\epsilon^{-3})$.

4. Comparative Analysis with Expectation-Only NSGD-MVR

Expectation-only convergence for NSGD-MVR yields $\mathbb{E}[\|\nabla F(\bar x)\|]=O(\epsilon)$ after $$T=O(\epsilon^{-2}-\text{or-}\epsilon^{-2-\frac{p}{q(p-1)}})$$ iterations. Double-Clipped NSGD-MVR strengthens this result, establishing the same rates (modulo logarithmic factors) with high probability, i.e., the bounds hold for “all tail events" up to probability $1-\beta$ rather than merely in expectation. The dual clipping parameters $\lambda_1,\lambda_2$ must be set as functions of $\beta$, incurring a $\ln(T/\beta)$ penalty, but enabling strong probabilistic control.

Algorithm	Guarantee Type	Required Iterations ($T$)
NSGD-MVR	Expectation	$O(\epsilon^{-2})$ or $O(\epsilon^{-2-\frac{p}{q(p-1)}})$
Double-Clipped NSGD-MVR	High probability	$\widetilde{O}(\epsilon^{-3})$ (for $p=q=2$), with log factors

5. Key Theoretical Lemmas and Analytical Techniques

The convergence analysis relies on several critical results:

• Zero‐Chain Progress (Lemma A.4): Clipping in each momentum update can cause zero progress in certain directions (“zero-chain effect”), implying that, with constant probability, some coordinates remain unchanged, important for tail control.
• Martingale Deviation Control (Lemma A.12): Freedman’s inequality is used on sums of clipped noise terms, $\sum\theta_t$ and $\sum\omega_t$, to manage the influence of heavy-tailed gradient fluctuations.
• Bias-Variance Decomposition for MVR Error (Lemma B.3): Decomposes the momentum error into a geometrically decaying bias (in $\alpha$) and residual terms bounded via $\gamma, \lambda_1, \lambda_2$.
• Refined Descent Lemmas (Lemmas B.1–B.2): These address the interplay between gradient clipping, variance reduction, and progression towards stationary points under weak smoothness.

The parameter selection ($\alpha,\gamma,\lambda_1,\lambda_2$) is optimized to balance geometric decay, control of heavy-tail effects, and overall smoothness, leading to high-probability Lyapunov descent by $\Delta_1$ in $O(\epsilon^{-\text{exponent}})$ steps.

6. Context and Significance in Stochastic Nonconvex Optimization

Double-Clipped NSGD-MVR represents a methodological advance for stochastic nonconvex optimization in heavy-tailed regimes. It includes and improves upon earlier analyses—both sample complexity lower bounds and algorithmic upper bounds—under broad smoothness and noise conditions. The dual clipping strategy, gradient normalization, and momentum-variance reduction yield a method that is both simple in implementation and rigorous in theoretical guarantees. Furthermore, the approach generalizes beyond standard $L_2$ smoothness, enabling applications to settings previously considered out-of-scope due to heavy-tailed, non-Gaussian gradient noise (Fradin et al., 21 Dec 2025).