Importance Weighted VI

• Importance Weighted Variational Inference is a method that tightens the ELBO using multiple importance samples to yield a more accurate estimation of the marginal likelihood.
• It employs techniques like sticking-the-landing (STL) and doubly-reparameterized gradients (DReG) to counteract gradient variance and the signal-to-noise ratio collapse.
• Extensions such as hierarchical proposals, structured variational families, and deep ensembles broaden its applications in high-dimensional latent variable models.

Importance Weighted Variational Inference (IWVI) refers to a class of variational inference methods in which the traditional evidence lower bound (ELBO) is tightened using importance sampling with multiple samples. This approach, which originated with the Importance-Weighted Autoencoder (IWAE) framework, increases the accuracy of marginal likelihood estimation for latent variable models and has motivated a rich line of research extending both theoretical analysis and practical algorithms.

1. Foundations of Importance-Weighted Variational Inference

Given a latent-variable model $p_\theta(x,z) = p_\theta(z) p_\theta(x|z)$ and a variational posterior $q_\phi(z|x)$, the classical ELBO is: $$\mathcal{L}(x) = \mathbb{E}_{z \sim q_\phi(z|x)} \left[ \log \frac{p_\theta(x, z)}{q_\phi(z|x)} \right].$$ This is a lower bound on the log marginal likelihood $\log p_\theta(x)$ due to Jensen’s inequality. Burda et al. (2015) replaced this single-sample expectation with a $K$-sample importance sampling estimate: $$\mathrm{IWELBO}_K(x) = \mathbb{E}_{z_{1:K} \sim q_\phi} \left[ \log \frac{1}{K} \sum_{k=1}^K \frac{p_\theta(x, z_k)}{q_\phi(z_k|x)} \right].$$ $\mathrm{IWELBO}_K(x)$ forms a sequence of bounds that increase monotonically with $K$ and approach $\log p_\theta(x)$ as $K \rightarrow \infty$ (Finke et al., 2019, Domke et al., 2018). This tightening occurs because the log is applied to an unbiased estimator of $p_\theta(x)$, and Jensen’s inequality is reversed in averaging over more samples.

Table: Hierarchy of Variational Bounds

Bound	Equation	Recoverable Special Cases
ELBO	$\mathbb{E}_{q_\phi}[\log w(z)]$	$K=1$ in IWELBO
IWELBO (IWAE)	$$\mathbb{E}_{q_\phi^{K}} [ \log (1/K \sum_{k=1}^K w(z_k) ) ]$$	$K>1$
VR-IWAE	$$\frac{1}{1-\alpha} \mathbb{E}_{q_\phi^K} \left[\log \frac{1}{K} \sum_{k=1}^K w(z_k)^{1-\alpha} \right]$$	$\alpha=0$ is IWAE; $K=1$ is VR (Daudel et al., 2022)

2. Algorithmic Implementations and Gradient Estimation

Optimization of IWVI objectives requires gradients with respect to the variational parameters $\phi$. For reparameterizable models, the standard estimator is: $$\nabla_\phi\, \mathrm{IWELBO}_K = \mathbb{E}_{\varepsilon_{1:K}} \left[ \sum_{k=1}^K \tilde w_k \nabla_\phi \log w \big(f(\varepsilon_k; \phi, x)\big) \right],$$ where $\tilde w_k$ are the normalized importance weights. However, Rainforth et al. (2018) demonstrated that the signal-to-noise ratio (SNR) of this gradient estimator vanishes as $K$ increases, causing inference network gradients to become ineffective (Finke et al., 2019, Jiang et al., 4 Feb 2026).

Two prominent remedies are:

• Sticking-the-landing (STL): Drops high-variance score-function terms, yielding a biased but low-variance estimator (Finke et al., 2019).
• Doubly-reparameterized (DReG): Uses an identity to construct an unbiased, low-variance estimator: $\sum_{k=1}^K \tilde w_k^{2} \nabla_\phi \log w_k$ This approach preserves unbiasedness while mitigating SNR collapse (Finke et al., 2019).

When reparameterization is not available (e.g., for discrete latents), REINFORCE-type (score-function) estimators are used; however, these also suffer from SNR decay as $O(1/\sqrt{N})$ as the sample size $N$ increases (Daudel et al., 1 Feb 2026). The VIMCO family of estimators introduced leave-one-out baselines to reduce variance, but recent analysis (e.g., VIMCO-$\star$) demonstrated that with optimal baselining, SNR can scale as $\sqrt{N}$ (Daudel et al., 1 Feb 2026).

Table: SNR Scaling Regimes

Gradient Estimator	SNR behavior with $K$	Key References
Naive reparam	$O(1/\sqrt{K})$ (vanishing)	(Finke et al., 2019, Jiang et al., 4 Feb 2026)
STL/DReG	$O(\text{const})$ or $\sqrt{K}$ under assumptions	(Finke et al., 2019)
VIMCO	$O(1/\sqrt{N})$	(Daudel et al., 1 Feb 2026)
VIMCO-$\star$	$O(\sqrt{N})$	(Daudel et al., 1 Feb 2026)

In Bures-Wasserstein geometry, the Wasserstein natural gradient for IW-ELBO attains SNR scaling as $\Omega(\sqrt{K})$, outperforming the Euclidean parameterization for large $K$ (Jiang et al., 4 Feb 2026).

3. Extensions: Hierarchical, Structured, and Ensemble Variants

IWVI supports multiple extensions to hierarchical and structured models:

• Hierarchical proposals: Rather than $K$ i.i.d. proposals, H-IWAE uses a "meta-sample" $z_0$ to induce negative correlation among conditionally independent proposals, reducing the estimator’s variance beyond the $1/K$ i.i.d. scaling (Huang et al., 2019).
• Conditionally structured Gaussian approximations: Partitioning variables into global and local blocks, conditionally structured variational approximations exploit conditional independence—enabling scalable IWVI for, e.g., GLMMs and state space models (Tan et al., 2019).
• Locally enhanced bounds: Importance weighting applied to blocks of latent variables separately (e.g., per data group) allows unbiased minibatch gradients and lower variance in hierarchical models (Geffner et al., 2022).
• Multiple importance sampling ELBO (MISELBO): Uses deep ensembles of variational distributions to further tighten the bound, exploiting the Jensen-Shannon divergence between proposals for additional gain (Kviman et al., 2022).

In hierarchical models, the gap between true posterior and approximate inference is often dominated by variance in local blocks; thus, local IWVI or blockwise importance weighting is preferable to global IWVI (Geffner et al., 2022).

4. Generalizations and Theoretical Guarantees

Generalized objectives: Importance weighting can be seen as a specific path in the thermodynamic variational objectives (TVO) framework associated with “geometric mean” interpolations. This viewpoint leads to Hölder-bounded VI, which improves discretization error by flattening the local-evidence curve, yielding one-step bounds that are potentially much tighter than IW-ELBO for a matched compute budget (Chen et al., 2021).

Alpha-divergence (VR-IWAE): The VR-IWAE objective generalizes IWAE and Rényi-ELBOs, parameterized by $\alpha$. This family smoothly interpolates between ELBO ($\alpha\to1$), IWAE ($\alpha=0$), and Rényi ($K \to \infty$ at fixed $\alpha$). For $0 < \alpha < 1$, the VR-IWAE achieves strictly better SNR in the encoder gradients than IWAE, at the cost of a bias governed by $\alpha$ (Daudel et al., 2022, Daudel et al., 2024). As dimension increases, however, $K$ must scale exponentially in $d$ to avoid weight collapse, limiting practical gains in high dimensions.

Asymptotics: With mild regularity, maximizing IWELBO as both the number of importance samples and dataset size $N$ grow yields consistent, asymptotically normal estimators attaining the statistical efficiency of maximum likelihood when the sample size grows faster than $N^{\delta/2}$, with $\delta$ depending on higher moments of the importance weights (Cherief-Abdellatif et al., 14 Jan 2025). In practice, $K \approx 5$–20 is sufficient for most gains if importance weight variance is moderate (Tan et al., 2019, Huang et al., 2019).

5. Variance Reduction, Robustness, and Practical Considerations

Naïve IWVI gradient estimators are subject to severe variance issues due to score-function terms and high-dimensional weight collapse. Recent developments address this through:

• Variance-reduced U-statistics: Overlapping batch averages of the base gradient estimator (using U-statistics) provably lower variance, with efficient computational approximations available and empirical reductions in wall-clock variance and improved performance in IWAEs (Burroni et al., 2023).
• Antithetic and hierarchical proposals: Negative correlation in hierarchical proposals (e.g., H-IWAE) can reduce variance strictly below i.i.d. levels (Huang et al., 2019).
• Elliptical variational families: Employing heavy-tailed or elliptical $q$ can improve moment matching and robustness in both low and high dimensions (Domke et al., 2018).
• Deep ensembles: Utilizing ensembles of variational proposals with multiple importance sampling (MISELBO) outperforms single-model IWVI at fixed computational budget, as quantified in MNIST and phylogenetic inference experiments (Kviman et al., 2022).

6. Empirical Applications and Domains

IWVI approaches have achieved state-of-the-art results in modern deep generative modeling (e.g., VAEs for image data), latent variable models, state-space models, deep Gaussian Processes, generalized linear mixed models, and combinatorial structures (e.g., Bayesian phylogenetics) (Huang et al., 2019, Salimbeni et al., 2019, Tan et al., 2019, Daudel et al., 1 Feb 2026). Extensions such as Annealed Importance Sampling Variational Inference (AIS-VI) further bridge the gap between VI and MCMC, providing even tighter bounds and better density estimation than IWAE for a given compute budget (Ding et al., 2019).

7. Limitations and Open Directions

Despite its tightness, IWVI suffers from fundamental high-dimensional collapse: as dimensionality increases, unless $K$ grows exponentially, the normalized importance weights concentrate on a single sample, and tighter bounds yield no improvement in SNR or parameter estimation. This limits the effectiveness of large $K$ in complex, high-dimensional models and motivates further work on variance reduction, alternative geometries (Wasserstein/Bures), and adaptive proposal design (Daudel et al., 2022, Jiang et al., 4 Feb 2026, Daudel et al., 2024).

Ongoing developments include sharper nonasymptotic analyses, efficient exploitation of conditional independence in hierarchical architectures, adaptive weighting strategies, and hybridization with MCMC (Chen et al., 2021, Ding et al., 2019). The unification of IWVI with more general divergence minimization and information-geometric optimization (e.g., Wasserstein gradients) points toward robust, high-SNR algorithms for future deep probabilistic modeling (Jiang et al., 4 Feb 2026).