Momentum-Variance-Reduced Update

• Momentum-variance-reduced update is an optimization method that combines momentum and variance reduction to enhance convergence in nonconvex and composite problems.
• It employs a recursive mix of current and past stochastic gradients with control-variates to effectively reduce estimation noise without requiring full batch computations.
• Variants like PStorm, SHARP, and Gluon-MVR demonstrate its practical benefits, improving sample efficiency and robustness in deep learning and reinforcement learning applications.

A momentum-variance-reduced update is a class of optimization algorithms that combine momentum techniques with stochastic variance reduction in order to improve convergence rates and reduce the variance of stochastic gradient estimators, particularly for large-scale nonconvex, composite, or non-smooth optimization problems commonly encountered in machine learning. These updates generalize and unify a series of approaches: they exploit temporal gradient correlations through momentum while leveraging variance-reduced estimators such as SVRG, SARAH, SPIDER, STORM, and control-variates, often achieving accelerated or optimal oracle/sample complexity and enhanced empirical robustness.

1. Algorithmic Principles and Update Mechanisms

Momentum-variance-reduced (MVR) updates build upon two pillars: (a) leveraging past gradients for acceleration (momentum) and (b) reducing finite-sample noise in gradient estimates (variance reduction). Fundamentally, MVR updates recursively mix current stochastic gradients with previous momentum states, employing a control-variates correction to shrink estimator variance without full-batch computations. A generic MVR recursion for a gradient surrogate $g^k$ is: $$g^{k+1} = \nabla f_{\xi^{k+1}}(x^{k+1}) + (1-\alpha_{k+1})\left[ g^k - \nabla f_{\xi^{k+1}}(x^k) \right]$$ where $\alpha_{k+1} \in (0,1)$ is the momentum/variance-reduction parameter, and $\nabla f_{\xi^{k+1}}$ denotes the stochastic gradient on batch/sample $\xi^{k+1}$, evaluated at iterates $x^{k+1}$ and $x^k$ (Xu et al., 2020, Qian et al., 18 Dec 2025, Luo et al., 2023).

This update can be embedded in proximal, coordinate, or composite optimization frameworks, with additional momentum extrapolations. Notably, in the context of linearly constrained/structured settings (e.g., ADMM), or for LMO-based optimizers in deep learning, the momentum and variance reduction are incorporated at the level of primal or dual variables, with layer-wise or data-cluster-wise correction (Qian et al., 18 Dec 2025, Tondji et al., 2021, Liu et al., 2017).

2. Variants: Algorithms and Structural Recursions

Multiple distinct instantiations of MVR have been developed:

• PStorm: For stochastic composite nonconvex optimization, PStorm uses the update

$g^{k+1} = u^{k+1} + (1-\beta_k)(g^k - v^{k+1})$

where $u^{k+1}$ and $v^{k+1}$ are mini-batch gradient estimates at $x^{k+1}$ and $x^k$ (Xu et al., 2020).

• SHARP: In reinforcement learning, SHARP introduces a recursive estimator incorporating Hessian-vector products,

$$v_t = (1-\alpha_t)\left[ v_{t-1} + B(\tau_t^b; \theta_t^b)(\theta_t-\theta_{t-1}) \right] + \alpha_t g(\tau_t; \theta_t)$$

where $B$ is an unbiased Hessian estimator and $\alpha_t$ decays as $t^{-2/3}$ (Salehkaleybar et al., 2022).

• Gluon-MVR: In deep neural network optimization, layer-wise MVR is realized either directly on the momentum buffer or through an explicit gradient-surrogate. For example, Gluon-MVR-2 defines

$$g_i^k = \nabla_i f_{\xi^k}(X^k) + (1-q)\left[ g_i^{k-1} - \nabla_i f_{\xi^k}(X^{k-1}) \right]$$

then applies classical momentum

$M_i^k = \beta M_i^{k-1} + (1-\beta) g_i^k$

followed by an LMO step (Qian et al., 18 Dec 2025).

• Cluster-Momentum (Discover): For deep learning with structured data, momentum buffers are maintained per cluster (e.g., per class), and the update uses control-variates correction (Tondji et al., 2021).
• Proximal/Coordinate Extensions: Many algorithms combine the above recursions with proximal, coordinate, or constraint-aware updates, for instance in ASVRCD (Hanzely et al., 2020), ASVRG-ADMM (Liu et al., 2017), and MVRC (Chen et al., 2020).

3. Theoretical Guarantees and Oracle Complexity

MVR updates optimally exploit low-variance gradient surrogates, yielding sample complexity and convergence rates that match oracle lower bounds for various problem classes. Notable complexity results include:

Algorithmic Instance	Setting	Sample/Iteration Complexity	Rate Type
PStorm (Xu et al., 2020)	Nonconvex, composite	$O(\varepsilon^{-3})$	Stationarity in expectation
SHARP (Salehkaleybar et al., 2022)	RL, nonconvex	$O(\varepsilon^{-3})$ trajectories	First-order stationary point
MVRC (Chen et al., 2020)	Nonconvex comp.	$O(n+\sqrt{n}\varepsilon^{-2})$, $O(\varepsilon^{-3})$ online	Stationarity/generalized gradient
ASVRCD (Hanzely et al., 2020)	Strong convexity	$\widetilde O(\sqrt{\rho/\mu})$	Oracle (gradient calls)
Gluon-MVR (Qian et al., 18 Dec 2025)	Nonconvex, LMO	$O(K^{-1/3})$	Layer-wise gradient norm
ASVRG-ADMM (Liu et al., 2017)	Strong/gen. convex	Linear or $O(1/T^2)$	Primal-gap, objective

These results consistently demonstrate that MVR augments the classical momentum paradigm to achieve theoretically optimal or near-optimal rates, without large-batch requirements or storage costs typical in classical variance-reduced methods.

4. Empirical Observations and Robustness

MVR-based methods have been empirically validated across a range of problem settings:

• In nonconvex neural network training, PStorm and Gluon-MVR outperform their vanilla momentum or Adam-type baselines in both sample efficiency and asymptotic loss (Xu et al., 2020, Qian et al., 18 Dec 2025).
• For deep learning on structured datasets (CIFAR, ImageNet), Discover-type algorithms demonstrate greater robustness to label noise and augmentation schemes than standard momentum-SGD (Tondji et al., 2021).
• In reinforcement learning, SHARP achieves fast variance decay ($O(1/t^{2/3})$) and improved data efficiency over existing policy-gradient techniques (Salehkaleybar et al., 2022).
• In decentralized and federated optimization, mini-batch MVR updates (as in DSE-MVR) effectively suppress stochastic noise and data heterogeneity effects, yielding noise-independent leading-order convergence rates for large batches or infrequent communication (Luo et al., 2023).

5. Connections to and Distinctions from Related Approaches

MVR occupies a conceptual and practical continuum with classical variance reduction and momentum approaches:

• Versus SVRG/SAGA/SARAH/SPIDER: Classical variance reduction requires periodic large-batch/epoch computation or table storage. MVR updates only require $O(1)$ storage and per-iteration batch sizes, bypassing the scaling bottlenecks in large-scale or online regimes (Xu et al., 2020, Hanzely et al., 2020).
• Versus vanilla momentum: Standard Polyak/Nesterov momentum achieves geometric averaging, but the steady-state variance remains $O(\sigma^2)$. MVR recursions reduce this to $O(\alpha\sigma^2)$, and in the case of STORM-type updates, decay the bias geometrically (Qian et al., 18 Dec 2025, Luo et al., 2023).
• Control-Variates Perspective: Multi-momentum (cluster-wise) updates and Kalman-filter–based approaches further generalize the variance-reduction effect. Both maintain auxiliary variables that adaptively track "local" expectations, eliminating between-cluster or temporally correlated gradient noise (Tondji et al., 2021, Vuckovic, 2018).

6. Implementation Considerations and Hyperparameter Regimes

Several key design choices are central to realizing the benefits of MVR updates:

• Momentum/variance reduction parameter: The contractive parameter $\alpha$ (or $q$) is typically chosen $O(1/K^{2/3})$ or decreasing with $t^{-\gamma}$, balancing variance decay with estimator stability (Xu et al., 2020, Qian et al., 18 Dec 2025, Salehkaleybar et al., 2022).
• Batch sizes: Practical performance is robust for $m=1$ to $m=64$; small-batch updates retain theoretical guarantees (Xu et al., 2020).
• Structured buffers: When cluster or layer structure is known, maintaining separate momentum surrogates can boost noise reduction (multi-momentum Discover, Gluon-MVR).
• Proximal and trust-region steps: For composite, constrained, or LMO-based settings, the combined use of a single-proximal step per iteration and an MVR-corrected surrogate is key to both convergence rates and computational efficiency (Chen et al., 2020, Qian et al., 18 Dec 2025).
• Parameter tuning: Hyperparameters such as $\alpha$, learning rates, and LMO radii can be chosen adaptively or by decaying schedules; explicit knowledge of smoothness constants can be replaced by cross-validation in practice (Xu et al., 2020).

7. Significance and Future Perspectives

Momentum-variance-reduced updates represent an overview that (i) preserves the tractability and scalability of momentum methods, (ii) inherits optimal sample or oracle complexity from advanced variance reduction, and (iii) exhibits adaptations to decentralized, federated, layer-wise, and structured-data scenarios. They underpin a range of recent advances, including layer-wise optimizers for deep neural networks (Qian et al., 18 Dec 2025), robust training under data heterogeneity or in the presence of structural dataset clusters (Tondji et al., 2021), accelerated policy optimization in RL (Salehkaleybar et al., 2022), and fast nonconvex composite optimization without large-batch computation (Xu et al., 2020).

A continuing direction is the systematic unification of these approaches within general frameworks (e.g., Gluon), the optimization of meta-parameters for complex models, and the transfer of theoretical rates into large-scale, distributed, and online learning systems.

---

References

• "Momentum-based variance-reduced proximal stochastic gradient method for composite nonconvex stochastic optimization" (Xu et al., 2020).
• "Muon is Provably Faster with Momentum Variance Reduction" (Qian et al., 18 Dec 2025).
• "Momentum-Based Policy Gradient with Second-Order Information" (Salehkaleybar et al., 2022).
• "Variance Reduction in Deep Learning: More Momentum is All You Need" (Tondji et al., 2021).
• "Decentralized Local Updates with Dual-Slow Estimation and Momentum-based Variance-Reduction for Non-Convex Optimization" (Luo et al., 2023).
• "Variance Reduced Coordinate Descent with Acceleration: New Method With a Surprising Application to Finite-Sum Problems" (Hanzely et al., 2020).
• "Momentum with Variance Reduction for Nonconvex Composition Optimization" (Chen et al., 2020).
• "Accelerated Variance Reduced Stochastic ADMM" (Liu et al., 2017).
• "Kalman Gradient Descent: Adaptive Variance Reduction in Stochastic Optimization" (Vuckovic, 2018).
• "Exploiting Uncertainty of Loss Landscape for Stochastic Optimization" (Bhaskara et al., 2019).