---
title: 'MISELBO: Multiple Importance Sampling ELBO'
url: https://www.emergentmind.com/topics/multiple-importance-sampling-elbo-miselbo
type: topic
---

# MISELBO: Multiple Importance Sampling ELBO

The Multiple Importance Sampling Evidence Lower Bound (MISELBO) is a framework for variational inference (VI) that leverages ensembles of independently trained variational approximations to provide a strictly tighter lower bound on the marginal log-likelihood than standard approaches. MISELBO arises by combining variational proposals into a mixture and applying multiple importance sampling strategies, yielding improved estimator tightness, variance reduction, and empirical gains in density estimation and Bayesian inference tasks in both classical and amortized VI contexts [2202.10951][2406.07083].

## 1. Background: Variational Bounds and Ensembles

In VI, the marginal likelihood $\log p_\theta(x)$ of an observed variable $x$ under a latent variable model $p_\theta(x, z)$ is typically lower bounded by the evidence lower bound (ELBO):
\[
L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).
\]
The importance-weighted ELBO (IWELBO) generalizes this using $L$ samples:
\[
L_L = \mathbb{E}_{z_{1:L} \sim q_\phi}\left[ \log\left( \frac{1}{L} \sum_{\ell=1}^L w_\ell\right)\right], \quad w_\ell = \frac{p_\theta(x, z_\ell)}{q_\phi(z_\ell|x)}.
\]
Both bounds suffer limitations when the variational approximation $q_\phi(z|x)$ is unimodal or mismatched to the true posterior, motivating the use of ensembles of variational distributions and mixtures [2202.10951][2406.07083].

## 2. Definition and Derivation of MISELBO

Suppose an ensemble of $S$ independently inferred variational approximations $\mathcal Q_S = \{q_{\phi_s}(z|x)\}_{s=1}^S$, with mixture proposal
\[
q_{\rm mix}(z|x) = \frac{1}{S} \sum_{s=1}^S q_{\phi_s}(z|x).
\]
MISELBO replaces the individual proposal $q_\phi$ in IWELBO by the mixture, yielding (for $L$ samples per member):
\[
\mathcal{L}^L_{\rm MIS} = \frac{1}{S}\sum_{s=1}^S \mathbb{E}_{z_{s, 1:L} \sim q_{\phi_s}} \left[ \log \frac{1}{L} \sum_{\ell=1}^L \frac{p_\theta(x, z_{s, \ell})}{q_{\rm mix}(z_{s, \ell}|x)} \right].
\]
For single samples ($L=1$), this reduces to:
\[
\mathcal{L}^1_{\rm MIS} = \frac{1}{S}\sum_{s=1}^S \mathbb{E}_{q_{\phi_s}} \left[ \log \frac{p_\theta(x, z)}{q_{\rm mix}(z|x)} \right].
\]
This formulation is directly aligned with the deterministic mixture weights or balance heuristic of multiple importance sampling (MIS) literature and generalizes to mixture families in variational inference settings [2202.10951][2406.07083].

## 3. Tightness and Theoretical Guarantees

Let $\overline{L}_L = \frac{1}{S} \sum_{s=1}^S L_L(q_{\phi_s})$ denote the average of per-member IWELBOs. For $L=1$:
\[
\Delta_1 = \mathcal{L}^1_{\rm MIS} - \overline{L}_1 = \mathrm{JSD}(\mathcal{Q}_S) \in [0, \log S],
\]
where $\mathrm{JSD}$ is the Jensen–Shannon divergence across $\mathcal Q_S$. Thus, MISELBO is provably tighter than the average ELBO unless all proposals coincide, and the gap directly quantifies their diversity. For $L>1$, empirical results uniformly find $\mathcal{L}^L_{\rm MIS} \geq \overline{L}_L$. Moreover,
\[
\mathcal{L}^L_{\rm MIS} \geq \mathcal{L}^{L-1}_{\rm MIS} \geq \cdots \geq \mathcal{L}^1_{\rm MIS}
\]
and $\mathcal{L}^L_{\rm MIS} \nearrow \log p_\theta(x)$ as $L \rightarrow \infty$. Thus, MISELBO supplies a valid, monotonically tightening lower bound whose improvement is quantified by JSD [2202.10951].

## 4. Practical Implementation: Deep Ensembles and Efficient Estimators

### Classical and Amortized VI

- **Classical VI:** Independently optimized variational parameters $\{\phi_s\}$ for each ensemble member, yielding diversely supported proposals.
- **Amortized VI:** Multiple independent encoders are trained for a fixed decoder, e.g., in a VAE, by re-initializing each encoder separately and optimizing only encoder parameters, producing a "deep ensemble" with shared generation [2202.10951].

### Black-Box VI and Mixtures

Given a mixture of $A$ components,
\[
q_\phi(z|x) = \frac{1}{A} \sum_{a=1}^A q_{\phi_a}(z|x),
\]
the naive "All-to-All" (A2A) estimator for MISELBO requires $O(A^2)$ density evaluations:
\[
\widetilde{\mathcal{L}}_{\mathrm{A2A}} = \frac{1}{A} \sum_{a=1}^A \log \frac{p_\theta(x, z_a)}{\frac{1}{A} \sum_{a'=1}^A q_{\phi_{a'}}(z_a|x)}
\]
where $z_a \sim q_{\phi_a}(z|x)$. To address this inefficiency, sub-sampling based estimators are introduced:
- **Some-to-All (S2A):** Subsets $S \le A$ are evaluated using the full mixture in the denominator, yielding unbiased estimates at $O(SA)$ cost.
- **Some-to-Some (S2S):** Denominator summation is restricted to the subset, reducing cost to $O(S^2)$ but inducing a downward bias; expectation strictly less than $\mathcal{L}_{\mathrm{MIS}}$ [2406.07083].

#### Complexity Table

| Estimator | Cost per Data Point                                | Bias     |
|-----------|----------------------------------------------------|----------|
| A2A       | $A C_p + A^2 C_q$                                  | Unbiased |
| S2A       | $S C_p + S A C_q$                                  | Unbiased |
| S2S       | $S C_p + S^2 C_q$                                  | Downward |

*Here $C_p$ is the cost of evaluating $\log p_\theta(x, z)$, $C_q$ is for $\log q_{\phi_a}(z|x)$.*

Scaling $A$ is thus practical with S2A/S2S, unlocking the approximation power of large ensembles with minimal parameter overhead, especially with amortized Mixture VAEs [2406.07083].

## 5. Empirical Performance

### Density Estimation

- **Toy multimodal densities:** Ensembles of $S=2$ variational approximations capture all modes, yielding substantially lower KL divergence to the true density compared to single ELBO or IWELBO [2202.10951].
- **MNIST with NVAE (Nouveau VAE):** MISELBO with $S=2$, $L=1000$ achieves NLL $\approx 77.77$ versus $78.21$ for a single-model IWELBO, a $0.44$ nat improvement. MISELBO with $L=50$ outperforms IWELBO with $L=1000$ (90% fewer samples) and further gains are seen with $S=3$ [2202.10951]. In settings with $A=800$ (with $S=1$ S2A), NLL $\approx 74.07$ is achieved with slight increase in compute [2406.07083].
- **FashionMNIST and robust likelihoods:** S2A/S2S allow continued NLL improvement with mixture size increase, and S2S maintains wall-clock time irrespective of $A$ [2406.07083].

### Bayesian Phylogenetics

- VBPI-NF and VBPI-mixtures evaluated on six and eight real datasets show consistent substantial gains (0.3–0.6 nat improvements in marginal log-likelihood) and up to $4\times$ reduction in inference runtime via S2A while matching prior ELBOs [2202.10951][2406.07083].

## 6. Connections to Importance Sampling and Broader Context

MISELBO is derived by applying the balance-heuristic (deterministic mixture) weights from MIS theory (Veach & Guibas 1995; Elvira et al. 2019), which are optimal for reducing estimator variance when using diverse proposals. This connection elucidates how advances in the theory of importance sampling, such as adaptive proposal learning and variance reduction schemes, can be ported to VI via the MISELBO formalism. The gap between standard ensemble ELBOs and MISELBO is exactly the Jensen–Shannon divergence, providing an operational measure of mixture utility [2202.10951]. Following this perspective, recent works have introduced efficient amortized mixture learning (MISVAE) and highly scalable mixture estimators (S2A, S2S), increasing the practical impact on state-of-the-art density estimation and black-box variational inference [2406.07083].

## 7. Summary and Significance

MISELBO provides a rigorous unification of variational inference with multiple importance sampling, enabling the deployment of large, diverse ensembles (including deep ensembles in amortized VI) and offering principled estimator constructions whose statistical efficiency and tightness are quantifiable. Empirically, MISELBO delivers improved bounds, sample efficiency, and estimation speed across a range of models and tasks, including density estimation (MNIST, FashionMNIST), and Bayesian phylogenetic inference. By incorporating advances such as S2A and S2S estimators, MISELBO enables scalable generalization to large mixture families with minimal computational penalty and lays methodological groundwork for further integration of importance sampling theory with VI [2202.10951][2406.07083].

Source: https://www.emergentmind.com/topics/multiple-importance-sampling-elbo-miselbo