MISELBO: Multiple Importance Sampling ELBO

• The paper demonstrates that MISELBO yields a strictly tighter lower bound than standard ELBO by leveraging ensemble diversity quantified via the Jensen–Shannon divergence.
• It employs a mixture of variational proposals and novel estimators (A2A, S2A, S2S) to reduce estimator variance and computational cost in both classical and amortized inference settings.
• Empirical evaluations on benchmarks like MNIST and Bayesian phylogenetics reveal significant improvements in negative log-likelihood and runtime efficiency.

The Multiple Importance Sampling Evidence Lower Bound (MISELBO) is a framework for variational inference (VI) that leverages ensembles of independently trained variational approximations to provide a strictly tighter lower bound on the marginal log-likelihood than standard approaches. MISELBO arises by combining variational proposals into a mixture and applying multiple importance sampling strategies, yielding improved estimator tightness, variance reduction, and empirical gains in density estimation and Bayesian inference tasks in both classical and amortized VI contexts (Kviman et al., 2022, Hotti et al., 2024).

1. Background: Variational Bounds and Ensembles

In VI, the marginal likelihood $\log p_\theta(x)$ of an observed variable $x$ under a latent variable model $p_\theta(x, z)$ is typically lower bounded by the evidence lower bound (ELBO): $$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$ The importance-weighted ELBO (IWELBO) generalizes this using $L$ samples: $$L_L = \mathbb{E}_{z_{1:L} \sim q_\phi}\left[ \log\left( \frac{1}{L} \sum_{\ell=1}^L w_\ell\right)\right], \quad w_\ell = \frac{p_\theta(x, z_\ell)}{q_\phi(z_\ell|x)}.$$ Both bounds suffer limitations when the variational approximation $q_\phi(z|x)$ is unimodal or mismatched to the true posterior, motivating the use of ensembles of variational distributions and mixtures (Kviman et al., 2022, Hotti et al., 2024).

2. Definition and Derivation of MISELBO

Suppose an ensemble of $S$ independently inferred variational approximations $\mathcal Q_S = \{q_{\phi_s}(z|x)\}_{s=1}^S$, with mixture proposal

$$q_{\rm mix}(z|x) = \frac{1}{S} \sum_{s=1}^S q_{\phi_s}(z|x).$$

MISELBO replaces the individual proposal $x$0 in IWELBO by the mixture, yielding (for $x$1 samples per member): $x$2 For single samples ($x$3), this reduces to: $x$4 This formulation is directly aligned with the deterministic mixture weights or balance heuristic of multiple importance sampling (MIS) literature and generalizes to mixture families in variational inference settings (Kviman et al., 2022, Hotti et al., 2024).

3. Tightness and Theoretical Guarantees

Let $x$5 denote the average of per-member IWELBOs. For $x$6: $x$7 where $x$8 is the Jensen–Shannon divergence across $x$9. Thus, MISELBO is provably tighter than the average ELBO unless all proposals coincide, and the gap directly quantifies their diversity. For $p_\theta(x, z)$0, empirical results uniformly find $p_\theta(x, z)$1. Moreover,

$p_\theta(x, z)$2

and $p_\theta(x, z)$3 as $p_\theta(x, z)$4. Thus, MISELBO supplies a valid, monotonically tightening lower bound whose improvement is quantified by JSD (Kviman et al., 2022).

4. Practical Implementation: Deep Ensembles and Efficient Estimators

Classical and Amortized VI

• Classical VI: Independently optimized variational parameters $p_\theta(x, z)$5 for each ensemble member, yielding diversely supported proposals.
• Amortized VI: Multiple independent encoders are trained for a fixed decoder, e.g., in a VAE, by re-initializing each encoder separately and optimizing only encoder parameters, producing a "deep ensemble" with shared generation (Kviman et al., 2022).

Black-Box VI and Mixtures

Given a mixture of $p_\theta(x, z)$6 components,

$p_\theta(x, z)$7

the naive "All-to-All" (A2A) estimator for MISELBO requires $p_\theta(x, z)$8 density evaluations: $p_\theta(x, z)$9 where $$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$0. To address this inefficiency, sub-sampling based estimators are introduced:

• Some-to-All (S2A): Subsets $$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$1 are evaluated using the full mixture in the denominator, yielding unbiased estimates at $$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$2 cost.
• Some-to-Some (S2S): Denominator summation is restricted to the subset, reducing cost to $$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$3 but inducing a downward bias; expectation strictly less than $$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$4 (Hotti et al., 2024).

Complexity Table

Estimator	Cost per Data Point	Bias
A2A	$$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$5	Unbiased
S2A	$$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$6	Unbiased
S2S	$$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$7	Downward

Here $$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$8 is the cost of evaluating $$L_{\rm ELBO} = \mathbb{E}_{z \sim q_\phi(z|x)}\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] \leq \log p_\theta(x).$$9, $L$0 is for $L$1.

Scaling $L$2 is thus practical with S2A/S2S, unlocking the approximation power of large ensembles with minimal parameter overhead, especially with amortized Mixture VAEs (Hotti et al., 2024).

5. Empirical Performance

Density Estimation

• Toy multimodal densities: Ensembles of $L$3 variational approximations capture all modes, yielding substantially lower KL divergence to the true density compared to single ELBO or IWELBO (Kviman et al., 2022).
• MNIST with NVAE (Nouveau VAE): MISELBO with $L$4, $L$5 achieves NLL $L$6 versus $L$7 for a single-model IWELBO, a $L$8 nat improvement. MISELBO with $L$9 outperforms IWELBO with $$L_L = \mathbb{E}_{z_{1:L} \sim q_\phi}\left[ \log\left( \frac{1}{L} \sum_{\ell=1}^L w_\ell\right)\right], \quad w_\ell = \frac{p_\theta(x, z_\ell)}{q_\phi(z_\ell|x)}.$$0 (90% fewer samples) and further gains are seen with $$L_L = \mathbb{E}_{z_{1:L} \sim q_\phi}\left[ \log\left( \frac{1}{L} \sum_{\ell=1}^L w_\ell\right)\right], \quad w_\ell = \frac{p_\theta(x, z_\ell)}{q_\phi(z_\ell|x)}.$$1 (Kviman et al., 2022). In settings with $$L_L = \mathbb{E}_{z_{1:L} \sim q_\phi}\left[ \log\left( \frac{1}{L} \sum_{\ell=1}^L w_\ell\right)\right], \quad w_\ell = \frac{p_\theta(x, z_\ell)}{q_\phi(z_\ell|x)}.$$2 (with $$L_L = \mathbb{E}_{z_{1:L} \sim q_\phi}\left[ \log\left( \frac{1}{L} \sum_{\ell=1}^L w_\ell\right)\right], \quad w_\ell = \frac{p_\theta(x, z_\ell)}{q_\phi(z_\ell|x)}.$$3 S2A), NLL $$L_L = \mathbb{E}_{z_{1:L} \sim q_\phi}\left[ \log\left( \frac{1}{L} \sum_{\ell=1}^L w_\ell\right)\right], \quad w_\ell = \frac{p_\theta(x, z_\ell)}{q_\phi(z_\ell|x)}.$$4 is achieved with slight increase in compute (Hotti et al., 2024).
• FashionMNIST and robust likelihoods: S2A/S2S allow continued NLL improvement with mixture size increase, and S2S maintains wall-clock time irrespective of $$L_L = \mathbb{E}_{z_{1:L} \sim q_\phi}\left[ \log\left( \frac{1}{L} \sum_{\ell=1}^L w_\ell\right)\right], \quad w_\ell = \frac{p_\theta(x, z_\ell)}{q_\phi(z_\ell|x)}.$$5 (Hotti et al., 2024).

Bayesian Phylogenetics

• VBPI-NF and VBPI-mixtures evaluated on six and eight real datasets show consistent substantial gains (0.3–0.6 nat improvements in marginal log-likelihood) and up to $$L_L = \mathbb{E}_{z_{1:L} \sim q_\phi}\left[ \log\left( \frac{1}{L} \sum_{\ell=1}^L w_\ell\right)\right], \quad w_\ell = \frac{p_\theta(x, z_\ell)}{q_\phi(z_\ell|x)}.$$6 reduction in inference runtime via S2A while matching prior ELBOs (Kviman et al., 2022, Hotti et al., 2024).

6. Connections to Importance Sampling and Broader Context

MISELBO is derived by applying the balance-heuristic (deterministic mixture) weights from MIS theory (Veach & Guibas 1995; Elvira et al. 2019), which are optimal for reducing estimator variance when using diverse proposals. This connection elucidates how advances in the theory of importance sampling, such as adaptive proposal learning and variance reduction schemes, can be ported to VI via the MISELBO formalism. The gap between standard ensemble ELBOs and MISELBO is exactly the Jensen–Shannon divergence, providing an operational measure of mixture utility (Kviman et al., 2022). Following this perspective, recent works have introduced efficient amortized mixture learning (MISVAE) and highly scalable mixture estimators (S2A, S2S), increasing the practical impact on state-of-the-art density estimation and black-box variational inference (Hotti et al., 2024).

7. Summary and Significance

MISELBO provides a rigorous unification of variational inference with multiple importance sampling, enabling the deployment of large, diverse ensembles (including deep ensembles in amortized VI) and offering principled estimator constructions whose statistical efficiency and tightness are quantifiable. Empirically, MISELBO delivers improved bounds, sample efficiency, and estimation speed across a range of models and tasks, including density estimation (MNIST, FashionMNIST), and Bayesian phylogenetic inference. By incorporating advances such as S2A and S2S estimators, MISELBO enables scalable generalization to large mixture families with minimal computational penalty and lays methodological groundwork for further integration of importance sampling theory with VI (Kviman et al., 2022, Hotti et al., 2024).