---
title: Importance Weighted VI
url: https://www.emergentmind.com/topics/importance-weighted-variational-inference-vi
type: topic
---

# Importance Weighted VI

Importance Weighted Variational Inference (IWVI) refers to a class of variational inference methods in which the traditional evidence lower bound (ELBO) is tightened using importance sampling with multiple samples. This approach, which originated with the Importance-Weighted Autoencoder (IWAE) framework, increases the accuracy of marginal likelihood estimation for latent variable models and has motivated a rich line of research extending both theoretical analysis and practical algorithms.

## 1. Foundations of Importance-Weighted Variational Inference

Given a latent-variable model $p_\theta(x,z) = p_\theta(z) p_\theta(x|z)$ and a variational posterior $q_\phi(z|x)$, the classical ELBO is:
\[
\mathcal{L}(x) = \mathbb{E}_{z \sim q_\phi(z|x)} \left[ \log \frac{p_\theta(x, z)}{q_\phi(z|x)} \right].
\]
This is a lower bound on the log marginal likelihood $\log p_\theta(x)$ due to Jensen’s inequality. Burda et al. (2015) replaced this single-sample expectation with a $K$-sample importance sampling estimate:
\[
\mathrm{IWELBO}_K(x) = \mathbb{E}_{z_{1:K} \sim q_\phi} \left[ \log \frac{1}{K} \sum_{k=1}^K \frac{p_\theta(x, z_k)}{q_\phi(z_k|x)} \right].
\]
$\mathrm{IWELBO}_K(x)$ forms a sequence of bounds that increase monotonically with $K$ and approach $\log p_\theta(x)$ as $K \rightarrow \infty$ [1907.10477, 1808.09034]. This tightening occurs because the log is applied to an unbiased estimator of $p_\theta(x)$, and Jensen’s inequality is reversed in averaging over more samples.

Table: Hierarchy of Variational Bounds

| Bound                   | Equation                                                                                                    | Recoverable Special Cases    |
|-------------------------|-------------------------------------------------------------------------------------------------------------|-----------------------------|
| ELBO                    | $\mathbb{E}_{q_\phi}[\log w(z)]$                                                                           | $K=1$ in IWELBO             |
| IWELBO (IWAE)           | $\mathbb{E}_{q_\phi^{K}} [ \log (1/K \sum_{k=1}^K w(z_k) ) ]$                                              | $K>1$                       |
| VR-IWAE                 | $\frac{1}{1-\alpha} \mathbb{E}_{q_\phi^K} \left[\log \frac{1}{K} \sum_{k=1}^K w(z_k)^{1-\alpha} \right]$ | $\alpha=0$ is IWAE; $K=1$ is VR [2210.06226] |



## 2. Algorithmic Implementations and Gradient Estimation

Optimization of IWVI objectives requires gradients with respect to the variational parameters $\phi$. For reparameterizable models, the standard estimator is:
\[
\nabla_\phi\, \mathrm{IWELBO}_K = \mathbb{E}_{\varepsilon_{1:K}} \left[ \sum_{k=1}^K \tilde w_k \nabla_\phi \log w \big(f(\varepsilon_k; \phi, x)\big) \right],
\]
where $\tilde w_k$ are the normalized importance weights. However, Rainforth et al. (2018) demonstrated that the signal-to-noise ratio (SNR) of this gradient estimator vanishes as $K$ increases, causing inference network gradients to become ineffective [1907.10477, 2602.04272].

Two prominent remedies are:
- **Sticking-the-landing (STL):** Drops high-variance score-function terms, yielding a biased but low-variance estimator [1907.10477].
- **Doubly-reparameterized (DReG):** Uses an identity to construct an unbiased, low-variance estimator:
\[
\sum_{k=1}^K \tilde w_k^{2} \nabla_\phi \log w_k
\]
This approach preserves unbiasedness while mitigating SNR collapse [1907.10477].

When reparameterization is not available (e.g., for discrete latents), REINFORCE-type (score-function) estimators are used; however, these also suffer from SNR decay as $O(1/\sqrt{N})$ as the sample size $N$ increases [2602.01412]. The VIMCO family of estimators introduced leave-one-out baselines to reduce variance, but recent analysis (e.g., VIMCO-$\star$) demonstrated that with optimal baselining, SNR can scale as $\sqrt{N}$ [2602.01412].

Table: SNR Scaling Regimes

| Gradient Estimator      | SNR behavior with $K$                               | Key References   |
|------------------------|-----------------------------------------------------|------------------|
| Naive reparam          | $O(1/\sqrt{K})$ (vanishing)                         | [1907.10477, 2602.04272] |
| STL/DReG               | $O(\text{const})$ or $\sqrt{K}$ under assumptions   | [1907.10477]     |
| VIMCO                  | $O(1/\sqrt{N})$                                     | [2602.01412]     |
| VIMCO-$\star$          | $O(\sqrt{N})$                                       | [2602.01412]     |

In Bures-Wasserstein geometry, the Wasserstein natural gradient for IW-ELBO attains SNR scaling as $\Omega(\sqrt{K})$, outperforming the Euclidean parameterization for large $K$ [2602.04272].

## 3. Extensions: Hierarchical, Structured, and Ensemble Variants

IWVI supports multiple extensions to hierarchical and structured models:

- **Hierarchical proposals:** Rather than $K$ i.i.d. proposals, H-IWAE uses a "meta-sample" $z_0$ to induce negative correlation among conditionally independent proposals, reducing the estimator’s variance beyond the $1/K$ i.i.d. scaling [1905.04866].
- **Conditionally structured Gaussian approximations:** Partitioning variables into global and local blocks, conditionally structured variational approximations exploit conditional independence—enabling scalable IWVI for, e.g., GLMMs and state space models [1904.09591].
- **Locally enhanced bounds:** Importance weighting applied to blocks of latent variables separately (e.g., per data group) allows unbiased minibatch gradients and lower variance in hierarchical models [2203.04432].
- **Multiple importance sampling ELBO (MISELBO):** Uses deep ensembles of variational distributions to further tighten the bound, exploiting the Jensen-Shannon divergence between proposals for additional gain [2202.10951].

In hierarchical models, the gap between true posterior and approximate inference is often dominated by variance in local blocks; thus, local IWVI or blockwise importance weighting is preferable to global IWVI [2203.04432].

## 4. Generalizations and Theoretical Guarantees

**Generalized objectives:** Importance weighting can be seen as a specific path in the thermodynamic variational objectives (TVO) framework associated with “geometric mean” interpolations. This viewpoint leads to Hölder-bounded VI, which improves discretization error by flattening the local-evidence curve, yielding one-step bounds that are potentially much tighter than IW-ELBO for a matched compute budget [2111.02947].

**Alpha-divergence (VR-IWAE):** The VR-IWAE objective generalizes IWAE and Rényi-ELBOs, parameterized by $\alpha$. This family smoothly interpolates between ELBO ($\alpha\to1$), IWAE ($\alpha=0$), and Rényi ($K \to \infty$ at fixed $\alpha$). For $0 < \alpha < 1$, the VR-IWAE achieves strictly better SNR in the encoder gradients than IWAE, at the cost of a bias governed by $\alpha$ [2210.06226, 2410.12035]. As dimension increases, however, $K$ must scale exponentially in $d$ to avoid weight collapse, limiting practical gains in high dimensions.

**Asymptotics:** With mild regularity, maximizing IWELBO as both the number of importance samples and dataset size $N$ grow yields consistent, asymptotically normal estimators attaining the statistical efficiency of maximum likelihood when the sample size grows faster than $N^{\delta/2}$, with $\delta$ depending on higher moments of the importance weights [2501.08477]. In practice, $K \approx 5$–20 is sufficient for most gains if importance weight variance is moderate [1904.09591, 1905.04866].

## 5. Variance Reduction, Robustness, and Practical Considerations

Naïve IWVI gradient estimators are subject to severe variance issues due to score-function terms and high-dimensional weight collapse. Recent developments address this through:
- **Variance-reduced U-statistics:** Overlapping batch averages of the base gradient estimator (using U-statistics) provably lower variance, with efficient computational approximations available and empirical reductions in wall-clock variance and improved performance in IWAEs [2302.13918].
- **Antithetic and hierarchical proposals:** Negative correlation in hierarchical proposals (e.g., H-IWAE) can reduce variance strictly below i.i.d. levels [1905.04866].
- **Elliptical variational families:** Employing heavy-tailed or elliptical $q$ can improve moment matching and robustness in both low and high dimensions [1808.09034].
- **Deep ensembles:** Utilizing ensembles of variational proposals with multiple importance sampling (MISELBO) outperforms single-model IWVI at fixed computational budget, as quantified in MNIST and phylogenetic inference experiments [2202.10951].

## 6. Empirical Applications and Domains

IWVI approaches have achieved state-of-the-art results in modern deep generative modeling (e.g., VAEs for image data), latent variable models, state-space models, deep Gaussian Processes, generalized linear mixed models, and combinatorial structures (e.g., Bayesian phylogenetics) [1905.04866, 1905.05435, 1904.09591, 2602.01412]. Extensions such as Annealed Importance Sampling Variational Inference (AIS-VI) further bridge the gap between VI and MCMC, providing even tighter bounds and better density estimation than IWAE for a given compute budget [1906.04904].

## 7. Limitations and Open Directions

Despite its tightness, IWVI suffers from fundamental high-dimensional collapse: as dimensionality increases, unless $K$ grows exponentially, the normalized importance weights concentrate on a single sample, and tighter bounds yield no improvement in SNR or parameter estimation. This limits the effectiveness of large $K$ in complex, high-dimensional models and motivates further work on variance reduction, alternative geometries (Wasserstein/Bures), and adaptive proposal design [2210.06226, 2602.04272, 2410.12035].

Ongoing developments include sharper nonasymptotic analyses, efficient exploitation of conditional independence in hierarchical architectures, adaptive weighting strategies, and hybridization with MCMC [2111.02947, 1906.04904]. The unification of IWVI with more general divergence minimization and information-geometric optimization (e.g., Wasserstein gradients) points toward robust, high-SNR algorithms for future deep probabilistic modeling [2602.04272].

Source: https://www.emergentmind.com/topics/importance-weighted-variational-inference-vi