---
title: Variational Lower Bound and Negative ELBO
url: https://www.emergentmind.com/topics/variational-lower-bound-negative-elbo
type: topic
---

# Variational Lower Bound and Negative ELBO

A variational lower bound, typically called the Evidence Lower Bound (ELBO), is a central objective in variational inference for probabilistic models, particularly in variational autoencoders (VAEs) and Bayesian latent variable models. The negative ELBO—often referred to as the variational free energy—serves as a minimization target and quantifies the gap between an approximate posterior and the true posterior. In modern research, the development and analysis of negative ELBO-based objectives have become essential for model design, gradient estimator theory, and understanding information-theoretic properties of generative models.

## 1. Formal Definition and Standard Decomposition

The ELBO for observed data $x$ and latent variable $z$, under a model $p_\theta(x, z)$ and variational distribution $q_\phi(z)$ (or $q_\phi(z|x)$ for amortized inference), is defined as:
\[
\mathrm{ELBO}(\phi) = \mathbb{E}_{q_\phi(z)}\left[\log p_\theta(x, z) - \log q_\phi(z)\right]
\]
Its negative form—minimized in practice—is
\[
-\mathrm{ELBO}(\phi) = \mathbb{E}_{q_\phi(z)}\left[\log q_\phi(z) - \log p_\theta(x, z)\right]
\]
Minimizing $-\mathrm{ELBO}$ is equivalent to minimizing the Kullback–Leibler divergence to the true posterior:
\[
-\mathrm{ELBO}(\phi) = \mathrm{KL}\big(q_\phi(z)\,\|\,p_\theta(z|x)\big) - \log p_\theta(x)
\]
where $\log p_\theta(x)$ is constant with respect to $\phi$ [1805.11183][1810.11859]. In amortized settings (e.g., VAEs), one commonly writes:
\[
-\mathrm{ELBO}(x; \theta, \phi) = \mathrm{KL}\big(q_\phi(z|x)\,\|\,p_\theta(z)\big) - \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)]
\]
This decomposition tracks rate (KL term) and distortion (reconstruction term) as in rate–distortion theory [1711.00464][1904.06395].  

## 2. Information-Theoretic Interpretations and Structural Decomposition

At stationary points for a wide class of models, the ELBO can be expressed as a sum (and differences) of entropies:
\[
\mathrm{ELBO}^* = \underbrace{\mathbb{E}_n H\big[q_\phi(z|x_n)\big]}_{\text{average posterior entropy}}
- \underbrace{H\big[p_\theta(z)\big]}_{\text{prior entropy}}
- \underbrace{\mathbb{E}_n \mathbb{E}_{q_\phi(z|x_n)} H\big[p_\theta(x_n|z)\big]}_{\text{expected decoder entropy}}
\]
For standard exponential family models (notably Gaussian VAEs), this reduces to a tractable, closed-form expression in terms of parameterized variances and means [2010.14860][2209.03077]. This entropy sum characterization enables efficient, variance-free evaluation of the ELBO at convergence, and provides principled diagnostics for phenomena such as posterior collapse. The negative ELBO, $-\mathrm{ELBO}$, thus admits an interpretation as the aggregate mismatch and compression cost imposed by approximate inference.

## 3. Advanced Variational Families and Sandwich Bounds

Generative models increasingly employ variational families for which the marginal density $q_\phi(z)$ is intractable. Semi-implicit variational inference (SIVI) constructs two-level mixtures:
\[
q_\phi(z) = \int q_\phi(z|\psi) q_\phi(\psi) \,d\psi
\]
where both $q_\phi(z|\psi)$ and $q_\phi(\psi)$ are reparameterizable but not necessarily tractable jointly [1805.11183]. The SIVI framework defines Monte Carlo-based lower and upper bounds that sandwich the true ELBO:
\[
L^L_K(\phi) \leq \mathrm{ELBO}(\phi) \leq L^U_K(\phi)
\]
where $K$ is the number of auxiliary samples. Both bounds converge monotonically to the true ELBO as $K \to \infty$. SIVI provides an unbiased surrogate objective whose gradient can be estimated stochastically, and which, for finite $K$, is always a valid lower bound. This approach generalizes to doubly semi-implicit settings, where both prior and variational distributions are semi-implicit mixtures, preserving the sandwich property [1810.02789].

## 4. Gradient Estimators and Optimization Strategies

Gradient-based optimization of the negative ELBO typically employs two classes of estimators:

- **Score-function (REINFORCE) estimators:** Use the identity
  \[
  \nabla_\phi \mathcal{L} = \mathbb{E}_{q_\phi}\big[(\log p_\theta(x, z) - \log q_\phi(z)) \nabla_\phi \log q_\phi(z)\big]
  \]
  but suffer from high variance [2011.02271].

- **Reparameterization (pathwise) estimators:** For $z = h_\phi(\epsilon)$, $\epsilon \sim p(\epsilon)$, leverage
  \[
  \nabla_\phi \mathcal{L} = \mathbb{E}_\epsilon \left[\nabla_\phi (\log p_\theta(x, h_\phi(\epsilon)) - \log q_\phi(h_\phi(\epsilon)|x))\right]
  \]
  When available, reparameterization yields lower-variance, unbiased estimates.

Variance reduction can be achieved by deterministic quasi-Monte Carlo or quantization schemes; for instance, Quantized Variational Inference (QVI) replaces Monte Carlo with optimal cubature over quantized support points, yielding zero-variance (but biased) gradients. The bias decays polynomially with the number of quantization points, and Richardson extrapolation can further reduce bias [2011.02271].

For the VR-IWAE class of bounds, the choice of estimator (reparameterized or doubly-reparameterized) affects signal-to-noise ratio (SNR) scaling with sample size $N$ and model class [2410.12035]. In high-dimensional regimes, importance weight collapse may nullify SNR gains unless $N$ is exponentially large in $d$.

## 5. Extensions: Rate–Distortion, Thermodynamic, and Discrete Variants

Interpreting the negative ELBO through the lens of rate–distortion theory, minimizing $-\mathrm{ELBO}$ corresponds to the sum $I_Q(X;Z) + \mathbb{E}[-\log \ell(X|Z)]$, where the first term measures information encoding cost and the second term, distortion [1904.06395][1711.00464]. This framework clarifies tradeoffs and motivates alternative objectives—e.g., enforcing minimum mutual information or rate lower bounds (free bits) to prevent posterior collapse [1711.00464]. Thermodynamic Variational Objectives (TVO) further generalize the ELBO via path integration over interpolations between the variational posterior and the model joint, yielding tighter bounds via Riemann sum approximations [1907.00031].

For graphical models with discrete latent variables, entropy and expectations under expressive distributions (e.g., selective-SPNs) can be computed exactly, circumventing the limitations of sampling-based estimators and enabling direct optimization of the negative ELBO [2010.11446].

## 6. Practical Implementation and Model Selection

During training, the negative ELBO serves as the objective for stochastic gradient methods. Its minimization encourages the variational family to approach the true posterior while maximizing the marginal likelihood lower bound. The variance properties of the chosen gradient estimator, the tractability of the variational family, and the potential for bound tightness (e.g., via importance weighting or surrogate bounds) directly impact practical learning dynamics [1805.11183][2011.02271][2410.12035].

For model selection, penalized ELBO approaches have been shown to yield consistent estimators even under model misspecification, provided suitable prior mass conditions [1810.11859]. Closed-form entropy decompositions enable more efficient post-training model diagnostics and facilitate interpretable control over different components of the inference objective [2010.14860][2407.06797].

## 7. Geometric, Asymptotic, and Theoretical Perspectives

Recent work situates the negative ELBO as a Bregman divergence—specifically $D_A(\phi^* \,\|\,\phi)$ with respect to the exponential family log-partition function $A$. This geometric perspective underpins rigorous convergence bounds for gradient-based algorithms, with convergence rates governed by spectral properties of the Fisher information matrix [2510.15548].

In entropy-sum formulations, the negative ELBO at stationary points can be fully characterized in terms of entropies and cross-entropies of the variational distribution, the prior, and the conditional model distribution. These results extend to generalized exponential families and remain valid under broad practical conditions (finite/infinite data, deep networks, saddle or local optima) [2010.14860][2209.03077].

---

**Selected references:**  
- Semi-Implicit Variational Inference [1805.11183]  
- Quantized Variational Inference [2011.02271]  
- On the Convergence of the ELBO to Entropy Sums [2209.03077]  
- The ELBO of Variational Autoencoders Converges to a Sum of Three Entropies [2010.14860]  
- Fixing a Broken ELBO [1711.00464]  
- Geometric Convergence Analysis of Variational Inference via Bregman Divergences [2510.15548]  
- Learning with Importance Weighted Variational Inference [2410.12035]  
- Doubly Semi-Implicit Variational Inference [1810.02789]  
- ED-VAE: Entropy Decomposition of ELBO in Variational Autoencoders [2407.06797]  
- Information Theoretic Lower Bounds on Negative Log Likelihood [1904.06395]

Source: https://www.emergentmind.com/topics/variational-lower-bound-negative-elbo