---
title: 'LS-SGLD: Laplacian Smoothing for Bayesian Sampling'
url: https://www.emergentmind.com/topics/laplacian-smoothing-sgld-ls-sgld
type: topic
---

# LS-SGLD: Laplacian Smoothing for Bayesian Sampling

Laplacian Smoothing Stochastic Gradient Langevin Dynamics (LS-SGLD) is a Markov Chain Monte Carlo (MCMC) method designed to address the high-variance bottleneck of stochastic gradient Langevin dynamics (SGLD) in Bayesian sampling. By introducing Laplacian smoothing (LS) as a preconditioner, it achieves provably reduced discretization error in 2-Wasserstein distance across both log-concave and non-log-concave targets, with negligible computational overhead relative to standard SGLD. The LS-SGLD algorithm demonstrates improved empirical performance in posterior sampling, Bayesian logistic regression, and Bayesian convolutional neural network (CNN) training, with robust variance reduction and larger stable step sizes [1911.00782].

## 1. Motivation and Conceptual Foundations

LS-SGLD is motivated by the variance limitations of SGLD, which is based on discretizing the continuous-time overdamped Langevin stochastic differential equation (SDE):

$$ d\theta_t = -\nabla U(\theta_t)\, dt + \sqrt{2}\, dB_t, $$

where the target distribution is $\pi(\theta) \propto \exp(-U(\theta))$. In practice, $\nabla U(\theta)$ is approximated by a mini-batch stochastic gradient $g_k$, resulting in the update:

$$ \theta_{k+1} = \theta_k - \eta g_k + \sqrt{2\eta}\, \xi_k, \quad \xi_k \sim \mathcal{N}(0, I). $$

High variance in $g_k$ compels the use of small step-sizes $\eta$. Laplacian smoothing, previously introduced for stochastic gradient descent (SGD) [Osher et al. '18], proposes a preconditioning operator:

$$ H := (I - \sigma \Delta)^{-1}, $$

where $\Delta$ is the Laplacian (or, more generally, a graph Laplacian) and $\sigma \ge 0$ is the smoothing parameter. This operator enforces local averaging across parameter coordinates, reducing stochastic variance "on the fly" without extra storage requirements.

## 2. Algorithmic Specification

### 2.1 Smoothing Operator Construction

For $\theta \in \mathbb{R}^d$, define $L \in \mathbb{R}^{d \times d}$ as the circulant Laplacian matrix with periodic boundary conditions. The smoothing and preconditioning operators are:

- $A_\sigma := I - \sigma L$
- $H := A_\sigma^{-1}$
- $H^{1/2} := A_\sigma^{-1/2}$

Both $H$ and $H^{1/2}$ are circulant, admitting efficient evaluation via Fast Fourier Transform (FFT):

$$ Hv = \text{ifft} \left( \frac{\text{fft}(v)}{\text{fft}(a)} \right), $$

where $a$ is the first column of $A_\sigma$, and $\text{fft}$, $\text{ifft}$ denote discrete Fourier (inverse Fourier) transforms. The same transformation applies to $H^{1/2}$.

### 2.2 Continuous and Discrete-Time Dynamics

The LS-Langevin SDE generalizes the drift and diffusion structure:

$$ d\Theta_t = -H \nabla U(\Theta_t)\, dt + \sqrt{2H}\, dB_t, $$

maintaining $\pi(\theta) \propto \exp(-U(\theta))$ as the unique invariant measure.

The discrete-time LS-SGLD Euler–Maruyama update is:

$$
\theta_{k+1} = \theta_k - \eta H g_k + \sqrt{2\eta}\, H^{1/2} \xi_k,
$$

with
$$
g_k = \frac{1}{B} \sum_{i \in \mathcal{C}_k} \nabla f_i(\theta_k), \qquad \xi_k \sim \mathcal{N}(0, I) .
$$

Efficient implementation uses FFT for both $H g_k$ and $H^{1/2} \xi_k$ at each step.

***LS-SGLD Algorithm at a Glance***

| Step | Operation                          | Notes                               |
|------|------------------------------------|-------------------------------------|
| 1    | Sample mini-batch, compute $g_k$   | $g_k = \frac{1}{B} \sum_{i \in \mathcal{C}_k} \nabla f_i$ |
| 2    | Compute $H g_k$ via FFT            | $v_1 = \text{ifft}(\text{fft}(g_k)/\text{fft}(a))$    |
| 3    | Generate $\xi_k$, compute $H^{1/2} \xi_k$ via FFT | $v_2 = \text{ifft}(\text{fft}(\xi_k)/\sqrt{\text{fft}(a)})$ |
| 4    | Update: $\theta_{k+1} = \theta_k - \eta v_1 + \sqrt{2\eta} v_2$ |      |

## 3. Theoretical Guarantees

The convergence of LS-SGLD is established under the following assumptions:

- **Dissipativity:** $\nabla U(\theta) \cdot \theta \ge m \|\theta\|^2 - b$
- **Smoothness:** Each $f_i$ is $M$–Lipschitz gradient
- **Variance bound:** $\mathbb{E}\|\nabla f_i(\theta) - \nabla U(\theta)\|^2 \le d \omega^2$
- **Log-concavity (optional):** U convex

Key metrics are reported in terms of the $2$-Wasserstein distance $W_2$ and log-Sobolev constant $\lambda$.

### 3.1 Strongly Convex (Log-Concave) Targets

The total error splits as:

$$
W_2(\mathrm{Law}(\theta_K), \pi) \leq \underbrace{W_2(\mathrm{Law}(\theta_K), \mathrm{Law}(\Theta_{K\eta}))}_{\text{Discretization}} + \underbrace{W_2(\mathrm{Law}(\Theta_{K\eta}), \pi)}_{\text{Ergodic term}}
$$

The ergodic term decays exponentially in time, with decay rate constant $c_0 \in [\|A_\sigma\|^{-1}, 1]$.

Crucially, LS-SGLD exhibits strictly smaller discretization error than vanilla SGLD due to variance-reducing factors $\gamma_1, \gamma_2 < 1$, where:

- $\gamma_1 \in [\|A_\sigma\|^{-2}, 1]$
- $\gamma_2 = \frac{1}{d} \sum_{j=1}^d \left[1 + 2\sigma - 2\sigma \cos(2\pi j/d)\right]^{-1} < 1$

### 3.2 General (Non-Log-Concave) Targets

For general non-convex $U$, analogous error bounds are derived using a coupled SDE argument and Girsanov’s theorem, showing that LS-SGLD achieves reduced discretization terms (scaled by $\gamma_1$, $\gamma_2$) while preserving exponential ergodic decay as above, though with slightly reduced mixing rate.

A plausible implication is that for a broad class of sampling tasks, LS-SGLD offers a stringent trade-off: notably improved accuracy per iteration at the cost of marginally slower mixing in continuous time.

## 4. Computational Aspects and Hyperparameter Choices

The dominant additional cost over SGLD is two FFTs of length $d$ per iteration, resulting in $O(d \log d)$ overhead, which is negligible in high-dimensional settings. The smoothing parameter $\sigma$ regulates both the strength of preconditioning and the potential step-size, with larger $\sigma$ yielding smaller $\gamma_2$ and increased maximal stable step-sizes. The effective step-size in practice can be set proportional to $(1+4\sigma)^{1/4}$ in smooth quadratic landscapes, reflecting robust stability gains observed in experiments. For arbitrary high-dimensional graph Laplacians, sparse FFT or polynomial preconditioning may be employed.

## 5. Empirical Performance

LS-SGLD demonstrates consistent empirical improvements across multiple paradigms:

### 2D Gaussian and Gaussian Mixture

- With a 2D Gaussian target $N(0, \Sigma)$ ($\Sigma_{12}=0.9$), LS-SGLD recovers the covariance structure with reduced autocorrelation time and support for step-sizes up to $(1+4\sigma)^{1/4}$ times larger than SGLD.
- In a bimodal 2D Gaussian mixture, both LS-SGLD and the LS-preconditioned pSGLD achieve near-constant $W_2$ distance to ground-truth Metropolis-Hastings samples ($\sim$0.42), with SGLD/pSGLD showing larger bias and fluctuations.

### Bayesian Logistic Regression (UCI “a3a”)

- With batch size 5 and grid-searched $\eta$, LS-SGLD yields lower test negative log-likelihood, higher classification accuracy, and reduced stochastic gradient variance compared to baselines.

### Bayesian Convolutional Neural Networks

- On MNIST using a two-layer convnet (batch size 100, $\sigma=0.5$), LS-SGLD and LS-pSGLD converge more rapidly and marginally outperform SGLD/pSGLD in both training density and generalization accuracy.

| Task                        | SGLD                     | LS-SGLD             |
|-----------------------------|--------------------------|---------------------|
| 2D Gaussian, $\gamma_2$     | 1.00                     | 0.149–0.268 (σ=1–3) |
| GMM, $W_2$ to MH samples    | Variable, large bias     | $\sim$0.42 constant |
| Logistic Reg. (a3a), variance | Higher                  | Lower               |
| Bayesian CNN, test accuracy | Lower, slower convergence| Slightly higher, faster|

## 6. Interpretations and Concluding Principles

Laplacian smoothing functions as a lightweight, variance-reducing preconditioner for SGLD, leading to provable gains in discretization error (in $W_2$) at the expense of a modest reduction in mixing speed. This effect is robust across both simple and deep Bayesian target distributions. The FFT-based implementation ensures negligible computational overhead. Empirical findings across synthetic and real-world machine learning benchmarks corroborate the theoretical results and uptake of LS-SGLD as an effective Bayesian inference technique [1911.00782].

Source: https://www.emergentmind.com/topics/laplacian-smoothing-sgld-ls-sgld