Double-Clipped NSGD-MVR

• The paper introduces double-clipped NSGD-MVR, an algorithm that extends NSGD-MVR with dual-level gradient clipping to secure high-probability convergence under heavy-tailed noise conditions.
• It utilizes momentum-variance reduction combined with tailored clipping strategies to balance bias and variance while achieving near-optimal sample complexity under relaxed smoothness assumptions.
• The framework provides rigorous high-probability convergence guarantees that improve on expectation-based methods, notably achieving optimal rates for canonical parameter settings (p=q=2).

Double-Clipped NSGD-MVR is an algorithmic framework designed for stochastic nonconvex optimization under heavy-tailed noise conditions, where the underlying gradient noise is characterized by bounded $p$-th central moment ($p$-BCM) for $p \in (1,2]$. The method extends the Normalized Stochastic Gradient Descent with Momentum Variance Reduction (NSGD-MVR) by employing dual-level gradient clipping and tailored high-probability analysis to deliver near-optimal complexity guarantees even in regimes where conventional assumptions do not hold. Double-Clipped NSGD-MVR provides explicit high-probability convergence rates under weak smoothness—relaxed mean-squared smoothness ($q$-WAS, $q\in[1,2]$) and $(q,\delta)$-similarity—broadening applicability and improving upon previous lower bounds for stochastic optimization in heavy-tailed scenarios (Fradin et al., 21 Dec 2025).

1. Algorithmic Structure and Update Rules

The Double-Clipped NSGD-MVR algorithm operates on iterates $x_t$ over $T$ steps, with primary parameters consisting of step-size $\gamma$, momentum parameter $\alpha\in(0,1)$, clipping thresholds $p$0, and access to a stochastic gradient oracle $p$1:

• Gradient Clipping at $p$2:

$p$3

• Momentum-Variance-Reduction (MVR) Update with Difference Clipping at $p$4:

$p$5

$p$6

• Gradient Normalization and Descent:

$p$7

The composite update ensures stability under heavy-tailed noise by constraining the magnitude of both raw and difference gradients through dual clipping, followed by normalization.

2. Analytical Assumptions and Heavy-Tailed Regime

Double-Clipped NSGD-MVR is analyzed under the following formal model assumptions:

• Bounded $p$8-th Central Moment (Heavy-Tailed Noise):

$p$9

where $p \in (1,2]$0 quantifies departure from sub-Gaussianity.

• Relaxed Mean-Squared Smoothness ($p \in (1,2]$1-WAS, $p \in (1,2]$2):

$p \in (1,2]$3

• $p \in (1,2]$4-Similarity Plus $p \in (1,2]$5-Smoothness:

$p \in (1,2]$6

This generalized smoothness expands the analysis to non-standard exponents and similarity regimes, surpassing prior works that restricted to $p \in (1,2]$7 or necessitated stricter gradient/Hessian regularity.

3. High-Probability Convergence Guarantees

Double-Clipped NSGD-MVR supports high-probability convergence to stationary points. Setting

$p \in (1,2]$8

with $p \in (1,2]$9, the algorithm achieves:

$q$0

with probability at least $q$1. To guarantee $q$2, one requires

$q$3

For the canonical parameter choice $q$4, this simplifies to $q$5.

4. Comparative Analysis with Expectation-Only NSGD-MVR

Expectation-only convergence for NSGD-MVR yields $q$6 after $q$7 iterations. Double-Clipped NSGD-MVR strengthens this result, establishing the same rates (modulo logarithmic factors) with high probability, i.e., the bounds hold for “all tail events" up to probability $q$8 rather than merely in expectation. The dual clipping parameters $q$9 must be set as functions of $q\in[1,2]$0, incurring a $q\in[1,2]$1 penalty, but enabling strong probabilistic control.

Algorithm	Guarantee Type	Required Iterations ($q\in[1,2]$2)
NSGD-MVR	Expectation	$q\in[1,2]$3 or $q\in[1,2]$4
Double-Clipped NSGD-MVR	High probability	$q\in[1,2]$5 (for $q\in[1,2]$6), with log factors

5. Key Theoretical Lemmas and Analytical Techniques

The convergence analysis relies on several critical results:

• Zero‐Chain Progress (Lemma A.4): Clipping in each momentum update can cause zero progress in certain directions (“zero-chain effect”), implying that, with constant probability, some coordinates remain unchanged, important for tail control.
• Martingale Deviation Control (Lemma A.12): Freedman’s inequality is used on sums of clipped noise terms, $q\in[1,2]$7 and $q\in[1,2]$8, to manage the influence of heavy-tailed gradient fluctuations.
• Bias-Variance Decomposition for MVR Error (Lemma B.3): Decomposes the momentum error into a geometrically decaying bias (in $q\in[1,2]$9) and residual terms bounded via $(q,\delta)$0.
• Refined Descent Lemmas (Lemmas B.1–B.2): These address the interplay between gradient clipping, variance reduction, and progression towards stationary points under weak smoothness.

The parameter selection ($(q,\delta)$1) is optimized to balance geometric decay, control of heavy-tail effects, and overall smoothness, leading to high-probability Lyapunov descent by $(q,\delta)$2 in $(q,\delta)$3 steps.

6. Context and Significance in Stochastic Nonconvex Optimization

Double-Clipped NSGD-MVR represents a methodological advance for stochastic nonconvex optimization in heavy-tailed regimes. It includes and improves upon earlier analyses—both sample complexity lower bounds and algorithmic upper bounds—under broad smoothness and noise conditions. The dual clipping strategy, gradient normalization, and momentum-variance reduction yield a method that is both simple in implementation and rigorous in theoretical guarantees. Furthermore, the approach generalizes beyond standard $(q,\delta)$4 smoothness, enabling applications to settings previously considered out-of-scope due to heavy-tailed, non-Gaussian gradient noise (Fradin et al., 21 Dec 2025).