---
title: Local Reparameterization Trick
url: https://www.emergentmind.com/topics/local-reparameterization-trick
type: topic
---

# Local Reparameterization Trick

The local reparameterization trick (LRT) is a variance reduction technique for stochastic gradient-based variational inference in neural networks with latent Gaussian (or certain discrete) parameterizations. LRT achieves substantial variance reduction in gradient estimators by shifting stochasticity from model parameters ("global noise") to layer activations ("local noise"), enabling efficient, highly parallelizable, and lower-variance training for Bayesian neural networks and quantized network variants. The trick builds on the central limit theorem and Rao-Blackwellisation, providing both theoretical clarity and practical speedups in scalable Bayesian deep learning [1506.02557][1710.07739][2506.07687][2307.01683].

## 1. Variational Inference and Gradient Variance in Neural Networks

Stochastic gradient variational Bayes (SGVB) introduces a variational posterior $q_\phi(w)$ over neural network weights $w$, with the evidence lower bound (ELBO)
$$
\mathcal{L}(\phi) = \mathbb{E}_{w \sim q_\phi}[ \sum_{i=1}^N \log p(y^i|x^i, w)] - \mathrm{KL}(q_\phi(w) \parallel p(w)).
$$
Typically, global reparameterization samples $w$ once per minibatch using $w = \mu + \sigma \odot \epsilon$, $ \epsilon \sim \mathcal{N}(0,1)$. The resulting gradients with respect to variational parameters $\phi$ are
$$
\nabla_\phi \widehat{\mathcal{L}}_{\mathrm{SGVB}} = \frac{N}{M} \sum_{m=1}^M \nabla_\phi \log p(y^{i_m}|x^{i_m}, w),
$$
where high covariance between minibatch elements induces an irreducible variance floor that does not diminish as the minibatch size $M$ increases [1506.02557].

## 2. Local Reparameterization Trick: Noise Transfer from Parameters to Activations

LRT exploits the property that, if the posterior in a linear layer is fully factorized Gaussian, the pre-activations $b_{m,j} = \sum_{i} A_{m,i} w_{ij}$ (for batch input $A$ and weights $W$) are marginally Gaussian:
$$
b_{m,j} \sim \mathcal{N}(\gamma_{m,j}, \delta_{m,j}),
\qquad
\gamma_{m,j} = \sum_{i} A_{m,i} \mu_{ij},
\qquad
\delta_{m,j} = \sum_{i} A_{m,i}^2 \sigma_{ij}^2.
$$
Instead of sampling all weights $w_{ij}$ for each example in the minibatch, LRT samples one independent $\zeta_{m,j} \sim \mathcal{N}(0,1)$ per activation and reconstructs $b_{m,j} = \gamma_{m,j} + \sqrt{\delta_{m,j}}\,\zeta_{m,j}$. This procedure ensures that noise is local to each data point and neuron, rendering covariance between data points zero. Consequently,
$$
\mathrm{Var}[\sum_{m=1}^{M} L_{m}] = N^2 \frac{\mathrm{Var}[L_i]}{M},
$$
which scales the gradient variance inversely with batch size and eliminates the variance floor seen in global parameter sampling [1506.02557][2506.07687].

## 3. Computational Efficiency and Parallelization

The LRT reduces computational complexity and memory footprint. Global reparameterization requires $O(MKL)$ random samples and matrix multiplications for a batch size $M$, input dimension $K$, and output dimension $L$. LRT only requires $O(ML)$ standard normal samples and a single batched matrix multiplication, as noise is injected post-aggregation at the activation level. This local noise structure makes the approach fully parallelizable, fitting well with BLAS routines and GPU computation [1506.02557]. Empirical results showed a $\sim 200\times$ speedup on GPU versus per-datapoint weight sampling, reducing epoch time from $\sim$1,600\,s to $\sim$7.4\,s in large-scale experiments.

## 4. Rao-Blackwellisation and Theoretical Foundations

The variance reduction property of LRT can be formalized through the Rao-Blackwellised reparameterization gradient estimator (R2-G2) [2506.07687]. In linear Gaussian models, conditioning on the pre-activations (sufficient statistics for each data point) provides a closed-form conditional expectation. The R2-G2 estimator computes
$$
\widehat{\nabla_\theta \ell}^{\mathrm{R2-G2}} = \mathbb{E}[\widehat{\nabla_\theta \ell}^{\mathrm{RT}} \mid z],
$$
where $z$ is the pre-activation, and $\widehat{\nabla_\theta \ell}^{\mathrm{RT}}$ is the standard reparameterization gradient. This conditioning collapses multi-dimensional global noise into independent local scalar noise per activation, provably reducing variance with minimal increase in computation. For models with linear layers and independent Gaussian weights, LRT coincides exactly with the R2-G2 estimator [2506.07687].

## 5. Applications to Discrete Weights and Activations

The LRT has been extended to settings with discrete weights and activations, facilitating low-precision neural network training suited for hardware efficiency [1710.07739][2307.01683]. For binary or ternary weights, each weight $w_{ij}$ is treated as a discrete random variable. By the central limit theorem, the pre-activation $z_i = \sum_j w_{ij} h_j$ is Gaussian for large input dimension, and the layer output is sampled as $z_i = m_i + v_i \epsilon_i$, where $m_i$ and $v_i^2$ are the mean and variance of $z_i$ under the discrete distribution. This approach provides a differentiable, low-variance surrogate objective and significantly reduces gradient estimator variance compared to naïve REINFORCE or Gumbel-softmax relaxations.

LRT has also been combined with the Gumbel-softmax trick for binarized activations. For an activation $h_i = \operatorname{sign}(z_i)$, the probability $P(h_i = +1) = 1 - \Phi(-m_i / v_i)$ allows for a continuous relaxation in backpropagation, further extending the variance reduction and computational benefits to the binarized activations regime [2307.01683].

## 6. Empirical Effects and Variational Dropout

Empirical evaluation has demonstrated the effectiveness of LRT for both model performance and efficiency. Local reparameterization attains an order of magnitude lower gradient variance than global sampling, leading to much faster convergence. On benchmarks such as MNIST and CIFAR-10, variational dropout trained with LRT matches or outperforms traditional dropout or fixed-variance Gaussian dropout, especially in resource-limited model regimes [1506.02557]. For low-precision discrete networks, LRT-based methods approach full-precision performance with significant computational and storage reductions [1710.07739][2307.01683].

Variational dropout extends these principles by learning dropout rates as variational parameters, yielding a posterior $q_\phi(w_{ij}) = \mathcal{N}(\theta_{ij}, \alpha_{ij} \theta_{ij}^2)$ with the ELBO simplified under a log-uniform prior. This enables model- and layer-specific noise injection rates, optimized via SGD for maximal generalization [1506.02557].

## 7. Connections to Dropout and Practical Implementation

Gaussian dropout can be interpreted as a special case of LRT with a fixed variance, where the corresponding variational posterior is $q(w_{ij}) = \mathcal{N}(\theta_{ij}, \alpha \theta_{ij}^2)$ and the KL-divergence regularization with respect to the prior can be made scale-invariant via a log-uniform prior. This renders the dropout objective equivalent to maximizing the evidence lower bound for the corresponding variational formulation [1506.02557].

Implementation of LRT typically involves precomputing mean and variance at the activation layer, sampling standard Gaussians per activation, forming noisy activations, and propagating these through the rest of the network. Backpropagation proceeds via the chain rule through both mean and variance with gradients scaling according to injected noise. Practical recommendations include using LRT whenever possible in Bayesian neural networks and VAE decoders, particularly in multilayer structures; on single-layer encoders, excessive variance reduction may be suboptimal for representational dynamics [2506.07687]. In binarized settings, entropy-regularizing penalties on logits preserve stochasticity essential for effective Gaussian approximation [2307.01683].

In summary, the local reparameterization trick is a foundational technique for scalable, low-variance variational inference in modern neural architectures, offering theoretical guarantees via Rao-Blackwellisation and broadening the landscape for efficient Bayesian deep learning and quantized network deployment [1506.02557][1710.07739][2506.07687][2307.01683].

Source: https://www.emergentmind.com/topics/local-reparameterization-trick