---
title: Stochastic Variance Reduced Gradient (SVRG)
url: https://www.emergentmind.com/topics/stochastic-variance-reduced-gradient-svrg
type: topic
---

# Stochastic Variance Reduced Gradient (SVRG)

Stochastic Variance Reduced Gradient (SVRG) is a stochastic optimization algorithm for minimizing finite-sum composite objectives, of the form
$$
F(x) = \frac{1}{n} \sum_{i=1}^n f_i(x) + g(x)
$$
where $f_i$ are typically smooth, possibly convex or nonconvex functions, and $g$ is a regularization term (possibly non-smooth but simple). SVRG was developed to address the high variance in stochastic gradient descent (SGD) and thereby accelerate convergence, particularly for large-scale machine learning problems.

## 1. Algorithmic Principle and Standard SVRG Workflow

SVRG operates in an outer-inner loop structure. At the start of each epoch (outer loop), a "snapshot" point $\tilde x$ is selected, and the full gradient $\mu = \nabla f(\tilde x) = \frac{1}{n}\sum_{i=1}^n \nabla f_i(\tilde x)$ is computed. Within the inner loop of $m$ steps, SVRG alternates between accessing fresh gradients at the current iterate and the stored gradients at the snapshot, to form a variance-reduced estimator:
$$
\tilde{\nabla} f_{i}(x) = \nabla f_{i}(x) - \nabla f_{i}(\tilde x) + \mu
$$
This estimator is unbiased with respect to $\nabla f(x)$ and has drastically reduced variance as $x \to \tilde x$. The update in the smooth case is
$$
x_{k+1} = x_k - \eta \left[\tilde{\nabla} f_{i_k}(x_k) + \nabla g(x_k) \right]
$$
and in the non-smooth case, a proximal step is used:
$$
x_{k+1} = \operatorname{prox}_{\eta g} \left(x_k - \eta \tilde{\nabla} f_{i_k}(x_k)\right)
$$
At the end of the inner loop, SVRG either sets the snapshot to the last iterate ("Option I") or the average over the inner iterates ("Option II", as in Prox-SVRG).

Key properties:
- Each epoch requires one full gradient computation (cost $O(nd)$), but inner steps are $O(d)$.
- The algorithm attains geometric convergence in the strongly convex setting, with complexity $O((n + L/\mu)\log(1/\epsilon))$ for $L$-smooth, $\mu$-strongly convex $F$.

[1802.09932], [1704.04966], [1908.02725]

## 2. Variants and Extensions of SVRG

Several SVRG variants address practical, statistical, and computational bottlenecks:
- **VR-SGD** modifies snapshot and starting-point selection by using the average and last iterate of the previous epoch, respectively. This choice allows much larger step sizes (e.g., up to $1/L$ rather than the more conservative $1/(10L)$), leading to faster variance decay per epoch. The variance bound
  $$
  \mathbb{E}\left[\|\tilde{\nabla} f_{i}(x) - \nabla f(x)\|^2\right] \le 4L [f(x) - f(x^*) + f(\tilde x) - f(x^*)]
  $$
  decreases more quickly when $\tilde x$ is an average, facilitating larger learning rates without loss of stability [1802.09932], [1704.04966].

- **Sufficient Decrease SVRG (SVRG-SD)** introduces a scaling parameter at each inner iterate to guarantee sufficient decrease in the objective, even with noisy stochastic gradients. This parameter is computed by solving a scalar minimization problem at each step, with closed-form expressions for Lasso and ridge regression. SVRG-SD achieves reduced effective data passes to given accuracy (up to $2-5\times$ fewer) and often surpasses accelerated methods in wall-clock time [1802.09933], [1703.06807].

- **Loopless SVRG/L-SVRG** eliminates explicit epoch structure by randomly deciding at each step whether to refresh the snapshot, enabling more aggressive update schedules and improving practical speed [1908.02725].

- **SVRG with Barzilai–Borwein (BB) Hessian Approximation (SVRG-2BB)** incorporates scalar second-order information, further reducing variance and allowing larger stable step sizes with minimal additional computational cost [2208.11075].

- **CheapSVRG** replaces exact full gradients by cheap stochastic surrogates (computed on small subsamples) at each epoch. This builds in a bias-variance-complexity trade-off and yields linear convergence up to a controllable error floor, with empirical gains in large-scale regimes [1603.06861].

## 3. Theoretical Guarantees and Convergence Analysis

Under $L$-smoothness and $\mu$-strong convexity, classic SVRG achieves
$$
\mathbb{E}[F(x^s) - F(x^*)] \le \rho^s [F(x^0) - F(x^*)]
$$
for some $\rho < 1$, with the number of required gradients $O((n + L/\mu)\log(1/\epsilon))$. The variance reduction is quantified by:
$$
\mathbb{E}\left[\|\tilde{\nabla} f_{i}(x) - \nabla f(x)\|^2\right] \le 2L [F(x) - F(x^*) + F(\tilde x) - F(x^*)]
$$
and further improved in variants like VR-SGD and SVRG-SD. In convex but not strongly convex cases, convergence is sublinear $O(1/S)$, while momentum-accelerated SVRG variants and VR-SGD with extrapolation (Algorithm 3 in [1802.09932]) achieve the $O(1/S^2)$ rate of optimal first-order methods.

Extensions to non-convex settings (including deep learning and matrix factorization) preserve convergence to first-order stationary points, in some cases with optimal sample complexity $O(N + N^{2/3}/\epsilon)$ as in the trust-region SVRG (TRSVR) method [2601.14647].

[1802.09932], [1704.04966], [2208.11075], [1802.09933], [2510.14759], [2108.04429], [2601.14647]

## 4. Computational and Algorithmic Enhancements

**Step-Size Rules and Batch Strategies:**  
- VR-SGD and SVRG-2BB admit much larger and sometimes adaptive step sizes compared to textbook SVRG.
- Mini-batching and arbitrary sampling are analyzed in generality for modern SVRG, with closed-form expressions for optimal batch sizes ($b^*$ often in $[8,128]$) [1908.02725].
- Loopless and asynchronous updates offer superior empirical wall-clock performance by removing explicit outer loops and supporting variable inner-loop lengths.

**Second-Order and Curvature Information:**  
- SVRG-2BB approximates local Hessians using Barzilai–Borwein secants in a scalar form, enabling curvature-adaptive steps at $O(d)$ per-iteration cost, with robust performance across ill-conditioned problems [2208.11075].
- TRSVR leverages SVRG as the inner engine of a stochastic trust-region method with adaptability to nonconvex landscapes. This leads to fast, reliable convergence even in ill-conditioned and nonconvex regimes [2601.14647].

**Distributed and Heterogeneous Data:**  
- Adaptive Sampling Distributed SVRG (ASD-SVRG) addresses the bottleneck in distributed settings due to heterogeneity by sampling machines according to local smoothness, reducing the iteration complexity dependence from the worst-case to the average Lipschitz constant across machines [2002.08528].

[1908.02725], [2208.11075], [2601.14647], [2002.08528]

## 5. Applications across Domains

**Supervised Learning and Empirical Risk Minimization:**  
SVRG and its variants have been thoroughly tested on convex risk minimization objectives, including ridge regression, Lasso, elastic-net, and logistic regression, showing superior or comparable performance to SAGA, accelerated methods (Catalyst, Katyusha), and plain SGD, with very low sensitivity to learning rate choices when using variance-reduction enhancements [1802.09932], [1704.04966].

**Reinforcement Learning:**  
- **Deep Q-Learning:** SVRG techniques, embedded in Deep Q-networks (DQN), produce significantly lower gradient variance and substantially faster convergence than vanilla DQN+Adam in Atari benchmarks [1905.08152].
- **Policy Gradient Estimation:** Trust-region policy optimization methods equipped with SVRG-based gradient estimators achieve reduced sample complexity and higher performance in MuJoCo continuous control benchmarks [1710.06034].
- **Policy Evaluation:** Novel batching and SCSG-inspired SVRG variants make high-accuracy value estimation achievable in fewer data passes, critically important for resource-limited RL pipelines [1906.03704].

**Inverse Problems and Regularization:**  
- SVRG matches or exceeds the classical order-optimal regularization rates for (possibly infinite-dimensional) linear inverse problems under source conditions, with provably smaller variance than SGD, even under noise, and with extensions to built-in regularization via truncated SVD [2510.14759], [2108.04429].
- SVRG's flexibility in incorporating regularization directly via proximal steps is theoretically justified and improves convergence for standard Tikhonov and Lasso setups [1511.01942].

**Semidefinite Optimization:**  
- The low-rank SVRG approach with submanifold convergence guarantees (using Option I / last-iterate snapshot) achieves global linear convergence under restricted strong convexity, surpassing alternative variance-reduced nonconvex methods for SDPs [2101.00236].

## 6. Practical Recommendations and Modern Trends

**Parameter Selection:**  
- Step sizes: VR-SGD/VR-variants enable step sizes close to $1/L$. Empirical tuning is simplified—robust performance for wide ranges, most notably for VR-SGD in $\eta \in [0.1/L, 0.4/L]$.
- Mini-batch sizes and inner loop lengths can be chosen via formulas based on problem smoothness and strong convexity parameters.
- Loopless SVRG and decaying-step-size schemes offer algorithmic stability for a wide range of model and data scales [1908.02725].

**Deep Learning Caveats:**  
- Classical SVRG is often ineffective or even detrimental in nonconvex deep learning due to the staleness of the snapshot correction and misalignment of the control variate. Empirically, a decaying coefficient for the control variate ($\alpha$-SVRG) restores consistent variance reduction in deep models; component-wise or epoch-wise decay schedules match the empirically optimal control variate strength [2311.05589].

**Heuristics:**  
- Skipping gradient computations for "inactive" examples (support vector exploitation), growing-batch strategies in early epochs, and sufficient decrease heuristics can yield sizable practical gains [1511.01942], [1603.06861], [1802.09933].

[1908.02725], [1802.09932], [1704.04966], [2311.05589], [1511.01942], [1802.09933]

## 7. Impact and Comparison to Alternative Approaches

SVRG lies at the foundation of modern finite-sum variance-reduction methods. Its core methodology—periodic full-gradient anchoring with control-variates correction—provides geometric convergence absent in plain SGD, while retaining low per-iteration complexity and minimal memory demands (unlike SAGA). Accelerated methods (Katyusha, Catalyst) offer $O(1/T^2)$ convergence via momentum and extrapolation, but VR-SGD and SVRG-SD often outperform these in effective runtime due to simpler iterates and larger allowable step sizes.

Variance-reduced variants are now the reference optimization backbone for a broad spectrum of convex and certain nonconvex machine learning, signal processing, and control problems, but need careful adaptation for deep neural networks to avoid detrimental variance amplification.

[1802.09932], [1704.04966], [1511.01942], [2311.05589]

---

**References**:  
- [1802.09932] VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning  
- [1704.04966] Larger is Better: The Effect of Learning Rates Enjoyed by Stochastic Optimization with Progressive Variance Reduction  
- [1802.09933] Guaranteed Sufficient Decrease for Stochastic Variance Reduced Gradient Optimization  
- [1703.06807] Guaranteed Sufficient Decrease for Variance Reduced Stochastic Gradient Descent  
- [1511.01942] Stop Wasting My Gradients: Practical SVRG  
- [1603.06861] Trading-off variance and complexity in stochastic gradient descent  
- [2208.11075] A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information  
- [1908.02725] Towards closing the gap between the theory and practice of SVRG  
- [2002.08528] Adaptive Sampling Distributed SVRG  
- [2101.00236] On Stochastic Variance Reduced Gradient Method for Semidefinite Optimization  
- [2601.14647] TRSVR: An Adaptive Stochastic Trust-Region Method with Variance Reduction  
- [2510.14759] On the convergence of stochastic variance reduced gradient for linear inverse problems  
- [2108.04429] An Analysis of Stochastic Variance Reduced Gradient for Linear Inverse Problems  
- [1710.06034] Stochastic Variance Reduction for Policy Gradient Estimation  
- [1905.08152] Stochastic Variance Reduction for Deep Q-learning  
- [2311.05589] A Coefficient Makes SVRG Effective  
- [1906.03704] SVRG for Policy Evaluation with Fewer Gradient Evaluations

Source: https://www.emergentmind.com/topics/stochastic-variance-reduced-gradient-svrg