---
title: Variance-Reduced SGHMC
url: https://www.emergentmind.com/topics/variance-reduced-sghmc
type: topic
---

# Variance-Reduced SGHMC

Variance-reduced Stochastic Gradient Hamiltonian Monte Carlo (VR-SGHMC) refers to a class of methods that enhance the efficiency and scalability of Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) by incorporating variance reduction schemes, primarily those developed within the stochastic optimization literature. These algorithms enable efficient Bayesian inference for large datasets by addressing the inherent variance in stochastic gradient approximations, thereby improving both theoretical convergence guarantees and empirical performance in high-dimensional or nonconvex Bayesian models [1802.04791, 1803.11159, 1709.01180, 2102.04613].

## 1. Theoretical Background and Motivation

SGHMC algorithms are discretizations of underdamped Langevin dynamics or Hamiltonian systems used for posterior sampling. Conventional HMC employs the exact gradient of the negative log-posterior $\nabla f(\theta)$, which becomes infeasible for large-scale problems. SGHMC replaces the full gradient with a stochastic estimator $\widetilde \nabla$, generally computed over minibatches. However, naive stochastic gradients exhibit high variance, resulting in slow mixing, poor approximation to the posterior, and degraded empirical risk.

Variance-reduction methods, such as SVRG and SAGA, originally developed for stochastic optimization, construct control variates by leveraging either a full-gradient snapshot or a table of component-wise gradients. In the SGHMC context, these methods produce unbiased or biased gradient estimates with guaranteed lower variance, directly improving the quality of the Markov chain both in theory and practice [1802.04791, 1803.11159, 1709.01180, 2102.04613].

## 2. Variance-Reduction Schemes in SGHMC

Variance-reduced SGHMC methods typically employ one of four estimators:

| Estimator  | Bias     | Mechanism                          |
|------------|----------|------------------------------------|
| SVRG       | Unbiased | Full-gradient snapshot, epoch-wise |
| SAGA       | Unbiased | Per-component table, per-iteration |
| SARAH      | Biased   | Recursive single-pass estimator    |
| SARGE      | Biased   | Per-component recursive memory     |

- **SVRG**: Maintains a reference (snapshot) point $\tilde{x}$ at which the full gradient $\nabla f(\tilde{x})$ is computed. Within each epoch, stochastic gradients are formed as
  $$
  \widetilde\nabla = \nabla f(\tilde{x}) + \frac{N}{b} \sum_{i \in B_k} [\nabla f_i(x_k) - \nabla f_i(\tilde{x})],
  $$
  where $B_k$ is a minibatch of size $b$ [2102.04613, 1803.11159, 1802.04791].

- **SAGA**: Maintains a table of most recent component gradients $\phi^i$ and, at each step, updates entries for a random batch while forming
  $$
  \widetilde\nabla = \left(\frac{N}{b}\sum_{i \in B_k} [\nabla f_i(x_k) - \phi^i]\right) + \sum_{i=1}^N \phi^i.
  $$
  [2102.04613, 1803.11159].

- **SARAH/SARGE**: Deploy recursive estimators with single-pass or table-based memory. They are biased in general but can exhibit smaller mean-squared-error (MSE) in certain regimes [2102.04613].

These estimators are seamlessly embedded in the SGHMC integrator, replacing the naive minibatch gradient.

## 3. Algorithmic Frameworks

The core dynamics for variance-reduced SGHMC are based on the discretized underdamped Langevin SDE:
$$
\begin{aligned}
d\theta &= p\,dt, \\
dp &= -\nabla f(\theta)\,dt - Dp\,dt + \sqrt{2D}\,dW_t,
\end{aligned}
$$
where $D$ is the friction parameter and $W_t$ is a $d$-dimensional Wiener process. Naive Euler discretization with a variance-reduced gradient $\widetilde\nabla$ at each step leads to updates of the form:
$$
\begin{aligned}
p_{t+1} &= (1-Dh)p_t - h\,\widetilde\nabla_t + \sqrt{2Dh}\,\xi_t, \\
\theta_{t+1} &= \theta_t + h\,p_{t+1}.
\end{aligned}
$$
$\xi_t$ is a standard normal vector.

Higher-order splitting schemes, notably symmetric splitting, further decrease discretization bias, as established in [1803.11159]. In symmetric splitting, each iteration is decomposed as:
$$
\begin{aligned}
\theta^{(1)} &= \theta + \frac{h}{2}p, \\
p^{(1)} &= e^{-Dh/2}p, \\
p^{(2)} &= p^{(1)} - h\,\widetilde\nabla(\theta^{(1)}) + \sqrt{2Dh}\,\xi, \\
p^+ &= e^{-Dh/2}p^{(2)}, \\
\theta^+ &= \theta^{(1)} + \frac{h}{2}p^+,
\end{aligned}
$$
which improves the bias from $O(h^2)$ to $O(h^4)$ in the MSE bound [1803.11159].

## 4. Convergence Theory and Gradient Complexity

Central to the analysis of VR-SGHMC are precise non-asymptotic bounds on mean-square error (MSE) and 2-Wasserstein distance to the target posterior. The bias and variance terms are governed by the properties of the gradient estimator—particularly through the Mean-Squared-Error-Bias (MSEB) property [2102.04613].

- **Unbiased estimators (SVRG, SAGA):** Achieve, for strongly log-concave posteriors, gradient complexity of
  $$
  \widetilde{O}(N + \kappa^{2}d^{1/2}\varepsilon^{-1} + \kappa^{4/3} d^{1/3} N^{2/3} \varepsilon^{-2/3})
  $$
  to reach $\varepsilon$-accuracy in 2-Wasserstein distance, substantially improving over standard SGHMC and full-gradient HMC in practical regimes [1802.04791, 2102.04613].

- **Biased estimators (SARAH, SARGE):** Reduce the $N$-dependence in gradient complexity, achieving
  $$
  \widetilde{O}(N + \sqrt{N} \kappa^2 d^{1/2} \varepsilon^{-1})
  $$
  but with a weaker $\varepsilon^{-1}$ dependency [2102.04613].

The table summarizes regime differences:

| Method           | Gradient Complexity                                               | Bias    | Best Regime                |
|------------------|------------------------------------------------------------------|---------|----------------------------|
| SVRG-HMC, SAGA   | $\widetilde{O}(N + \kappa^2 d^{1/2} \varepsilon^{-1} + N^{2/3} \kappa^{4/3} d^{1/3} \varepsilon^{-2/3})$ | Unbiased | High-precision, moderate $N$ |
| SARAH, SARGE     | $\widetilde{O}(N + \sqrt{N} \kappa^2 d^{1/2} \varepsilon^{-1})$  | Biased  | Moderate-accuracy, large $N$ |

The bounds depend explicitly on the smoothness $L$, strong convexity $m$, minibatch size $b$, and tuning (snapshot interval $p$) [1802.04791, 2102.04613].

## 5. Hyperparameter Selection and Practical Aspects

Empirical and theoretical recommendations for hyperparameters are as follows:

- **Minibatch size ($b$):** Modest, e.g., $b=10$ is typical for SVRG/SAGA schemes [1803.11159].
- **Snapshot interval ($p$, $m$):** For SVRG, $p=N/b$, typically $m=10$, $n_1 \gg n_2$ for two-batch control variates [1709.01180].
- **Step size ($h$):** Symmetric splitting allows larger $h$; $D h < 1$ must be satisfied [1803.11159].
- **Friction ($D$, $\gamma$):** $1 \leq D \leq 10$; larger $D$ leads to more rapid velocity dissipation.
- **Control variate batch ($n_1$) and online batch ($n_2$):** $n_2 = 10$–$100$, $n_1=10 \, n_2$, with update every $m=10$ iterations [1709.01180].

Pragmatic implementation replaces the SGHMC gradient with the control-variate estimate; memory and computation scale with the size of auxiliary tables for SAGA-type methods.

## 6. Empirical Performance and Applications

Experimental results demonstrate uniform acceleration of convergence and reduction in estimator variance for VR-SGHMC methods compared to standard SGHMC and Langevin approaches. In large-scale Bayesian regression, classification, and neural network inference tasks, SVRG2nd-HMC and its SAGA/HMC variants exhibit the fastest convergence in both training and test metrics. Key metrics include test mean-squared error (MSE), test negative log-likelihood, and root-MSE for Bayesian neural networks.

- On the UCI “concrete” dataset: after 5 data-passes,
  - SGHMC test MSE $\approx 6.3$
  - SVRG-HMC test MSE $\approx 5.2$
  - SVRG2nd-HMC test MSE $\approx 4.8$
- On the “protein” dataset (BNN, after 2 passes): 
  - SGHMC test RMSE $\approx 5.8$
  - SVRG2nd-HMC test RMSE $\approx 4.9$ [1803.11159].

Speedups of $2\times$–$10\times$ in convergence are observed across Bayesian regression, classification, deep neural networks (MLP, CNN, ResNet), and language models [1709.01180, 1803.11159]. Variance reduction also yields smoother learning curves and more robust out-of-sample performance.

## 7. Comparison and Extensions

Variance-reduced SGHMC variants outperform traditional stochastic gradient MCMC methods (SGHMC, SGLD, VR-SGLD) across a wide range of regimes in both theory and practice.

- For strongly convex log-posteriors, VR-SGHMC achieves a mixed regime complexity strictly better than standard HMC or SGHMC except in rare limits (i.e., extremely large $N$) [1802.04791].
- Higher-order symmetric splitting can further accelerate mixing and reduce discretization error to $O(h^4)$ [1803.11159].
- For general (non-strongly convex) log-concave targets, extensions use quadratic regularization and achieve comparable bounds modulo $d$-dependent terms [1802.04791].

Unbiased (SVRG/SAGA) and biased (SARAH/SARGE) VR-SGHMC schemes entail a trade-off between asymptotic bias and mean-square error; unbiased methods excel in high-precision regimes, while biased estimators are more attractive when moderate precision or very large data is paramount [2102.04613].

## References

- "Stochastic Variance-Reduced Hamilton Monte Carlo Methods" [1802.04791]
- "Stochastic Gradient Hamiltonian Monte Carlo with Variance Reduction for Bayesian Inference" [1803.11159]
- "A Convergence Analysis for A Class of Practical Variance-Reduction Stochastic Gradient MCMC" [1709.01180]
- "A New Framework for Variance-Reduced Hamiltonian Monte Carlo" [2102.04613]

Source: https://www.emergentmind.com/topics/variance-reduced-sghmc