IWMDE: Importance-Weighted Marginal Density Estimator

• IWMDE is a family of estimators that reweight samples from tractable distributions to recover intractable marginal densities in Bayesian models.
• It underpins methodologies in latent-variable modeling, evidence estimation, and Bayes factor sensitivity analysis by providing unbiased or reciprocal estimates.
• Practical applications include Variational Autoencoders, reciprocal importance sampling, and improved Bayesian model comparisons with lower variance estimates.

Importance-Weighted Marginal Density Estimator (IWMDE) denotes a class of estimators that recover an intractable marginal density or marginal data density by reweighting samples drawn from a tractable auxiliary distribution. The exact term is explicitly used in recent Bayes factor sensitivity analysis to estimate posterior density ordinates of a hyper-parameter from extended-model MCMC output (Bartoš et al., 23 Apr 2026). Closely related constructions appear earlier in latent-variable generative modeling, where the canonical estimator is the importance-sampling average

$$\hat p_K(x)=\frac{1}{K}\sum_{k=1}^K \frac{p_\theta(x,z_k)}{q_\phi(z_k\mid x)}, \qquad z_k\sim q_\phi(z\mid x),$$

which is an unbiased estimator of the marginal density $p_\theta(x)$ (Burda et al., 2015). The literature therefore supports a broad usage in which “IWMDE” refers not to a single universally standardized estimator, but to a family of importance-weighted constructions for marginal likelihoods, marginal densities, or posterior ordinates.

1. Terminology and conceptual scope

The most explicit contemporary use of the name arises in Bayes factor sensitivity analysis, where the IWMDE is used to estimate the posterior density ordinate

$p(\gamma^\ast \mid y, H_1^e)$

of a sensitivity hyper-parameter $\gamma$ under an extended model $H_1^e$ (Bartoš et al., 23 Apr 2026). In that setting, the estimator is not the Bayes factor itself; rather, it supplies the posterior density ratio

$$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$

which is then combined with an anchor Bayes factor $BF_{10}(\gamma_0)$ and a prior ratio to recover the full sensitivity curve (Bartoš et al., 23 Apr 2026).

Earlier papers do not generally use the exact label “Importance-Weighted Marginal Density Estimator,” but they provide the core estimators naturally associated with that term. In latent-variable models, “Importance Weighted Autoencoders” introduces the standard importance-sampling estimator of the marginal density $p_\theta(x)$ and optimizes the expectation of its logarithm (Burda et al., 2015). “Reinterpreting Importance-Weighted Autoencoders” makes the same estimator central and interprets the resulting bound through an implicit importance-weighted posterior family (Cremer et al., 2017). In Bayesian model comparison, “Accurate Computation of Marginal Data Densities Using Variational Bayes” develops a reciprocal-importance-sampling marginal data density estimator whose weighting density is a variational Bayes posterior approximation $q^*(\theta)$ (Hajargasht et al., 2018).

This suggests a useful taxonomy. In one branch, IWMDE refers to forward importance estimators of an intractable marginal such as $p_\theta(x)$. In another, it refers to reciprocal estimators of model evidence $p_\theta(x)$0. In a third, it refers to importance-weighted estimators of posterior density ordinates needed for downstream quantities such as Bayes factors. The common structure is the same: a target marginal object is estimated by reweighting samples from a tractable law.

2. Canonical latent-variable formulation

The canonical latent-variable setup writes

$p_\theta(x)$1

with posterior

$p_\theta(x)$2

Because the marginal integral is typically intractable, a tractable recognition or proposal distribution $p_\theta(x)$3 is introduced (Burda et al., 2015). Importance sampling then yields

$p_\theta(x)$4

Given $p_\theta(x)$5 i.i.d. samples $p_\theta(x)$6, define

$p_\theta(x)$7

The corresponding importance-weighted marginal density estimator is

$p_\theta(x)$8

Under the usual support and integrability conditions, this estimator is unbiased: $p_\theta(x)$9 (Burda et al., 2015).

A central distinction in the literature is between the estimator $p(\gamma^\ast \mid y, H_1^e)$0 itself and the training objective built from it. The IWAE objective is

$p(\gamma^\ast \mid y, H_1^e)$1

By Jensen’s inequality,

$p(\gamma^\ast \mid y, H_1^e)$2

so the expectation of the log of the IWMDE is a lower bound on the marginal log-likelihood (Burda et al., 2015). The special case $p(\gamma^\ast \mid y, H_1^e)$3 recovers the standard VAE ELBO, and the bound tightens monotonically with $p(\gamma^\ast \mid y, H_1^e)$4: $p(\gamma^\ast \mid y, H_1^e)$5 If $p(\gamma^\ast \mid y, H_1^e)$6 is bounded, then

$p(\gamma^\ast \mid y, H_1^e)$7

(Burda et al., 2015).

A common misconception is to conflate these objects. The estimator

$p(\gamma^\ast \mid y, H_1^e)$8

is unbiased for $p(\gamma^\ast \mid y, H_1^e)$9, whereas

$\gamma$0

is not unbiased for $\gamma$1; its expectation is downward biased, precisely because it is a lower bound (Burda et al., 2015). This distinction is central to any precise use of the term IWMDE in the latent-variable literature.

3. Implicit posterior geometry and variational interpretation

The IWAE literature does not treat the importance-weighted marginal estimator merely as a scalar approximation to $\gamma$2. It also induces an implicit posterior approximation. “Reinterpreting Importance-Weighted Autoencoders” defines the unnormalized importance-weighted distribution

$\gamma$3

and shows that the IWAE objective can be reinterpreted as the standard variational lower bound evaluated under this richer implicit distribution (Cremer et al., 2017).

Averaging over the auxiliary samples yields the normalized expected importance-weighted distribution

$\gamma$4

The paper proves the ordering

$\gamma$5

and establishes

$\gamma$6

(Cremer et al., 2017). This implies that the importance-weighted construction improves the effective posterior approximation, not just the scalar lower bound.

This suggests a broader interpretation of IWMDE-style estimators in latent-variable models. They do not only estimate a marginal quantity; they reshape the effective inference distribution toward the true posterior. In practical terms, the induced posterior can be sampled by sampling-importance-resampling: $\gamma$7 then drawing an index $\gamma$8 and returning $\gamma$9 (Cremer et al., 2017). A plausible implication is that, whenever IWMDE is used as a training primitive, its inferential consequences extend beyond likelihood approximation into posterior geometry and representation learning.

4. Reciprocal importance-weighted evidence estimators

A distinct but closely related lineage concerns marginal data density estimation in Bayesian model comparison. “Accurate Computation of Marginal Data Densities Using Variational Bayes” studies the marginal data density

$H_1^e$0

and starts from the reciprocal importance sampling (RIS) estimator

$H_1^e$1

Its proposal is to set the weighting density to the variational Bayes posterior approximation $H_1^e$2, yielding

$H_1^e$3

(Hajargasht et al., 2018).

In this formulation, the basic unbiased object is the reciprocal: $H_1^e$4 The paper proves

$H_1^e$5

and establishes that the reciprocal estimator is unbiased, consistent, and asymptotically normal under the stated assumptions (Hajargasht et al., 2018). The effective importance ratio is

$H_1^e$6

This reciprocal construction differs from the forward latent-variable estimator $H_1^e$7, but the underlying logic is the same: a marginal quantity is reconstructed by reweighting samples under a tractable law. The paper also proves a variance identity,

$H_1^e$8

which makes closeness of $H_1^e$9 to the posterior central for stability (Hajargasht et al., 2018). It further argues that the VB-based weighting density avoids the truncation often required in harmonic-mean-type methods because the posterior dominates the VB approximation under its assumptions (Hajargasht et al., 2018).

A frequent source of confusion is to treat all importance-weighted marginal estimators as forward Monte Carlo averages. The RIS-VB construction shows that an IWMDE can also be reciprocal: the estimator targets $$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$0 first and then inverts. The unbiasedness statement is therefore about the reciprocal, not about $$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$1 itself (Hajargasht et al., 2018).

5. IWMDE in Bayes factor sensitivity analysis

The 2026 sensitivity-analysis literature uses the name IWMDE explicitly. The framework considers Bayes factors

$$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$2

where $$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$3 indexes the prior under the alternative (Bartoš et al., 23 Apr 2026). Introducing an extended model $$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$4 with hyper-prior $$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$5 yields

$$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$6

Taking ratios at $$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$7 and $$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$8 gives

$$\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},$$9

Under a uniform hyper-prior, this simplifies to

$BF_{10}(\gamma_0)$0

(Bartoš et al., 23 Apr 2026).

The IWMDE in this setting estimates the posterior ordinate $BF_{10}(\gamma_0)$1 from joint posterior draws $BF_{10}(\gamma_0)$2: $BF_{10}(\gamma_0)$3 When the sensitivity parameter enters only through the prior on the model parameters, the likelihood cancels: $BF_{10}(\gamma_0)$4 Under a uniform hyper-prior, this becomes a simple prior-density ratio (Bartoš et al., 23 Apr 2026).

The paper identifies two important weighting choices. If

$BF_{10}(\gamma_0)$5

the IWMDE reduces to the conditional marginal density estimator

$BF_{10}(\gamma_0)$6

which the paper states is asymptotically variance-minimizing (Bartoš et al., 23 Apr 2026). A simpler uniform weighting function is also consistent but higher variance (Bartoš et al., 23 Apr 2026).

The empirical role of the estimator is also precise. It is applied to a univariate Bayesian $BF_{10}(\gamma_0)$7-test, a bivariate informed $BF_{10}(\gamma_0)$8-test, and Bayesian model-averaged meta-analysis, where it substantially outperforms kernel density estimation for posterior ordinate recovery across the sensitivity range (Bartoš et al., 23 Apr 2026). The paper reports that in the univariate $BF_{10}(\gamma_0)$9-test the IWMDE stays within about $p_\theta(x)$0 of the exact Bayes factor curve even at the boundaries, whereas KDE can deviate by roughly $p_\theta(x)$1 at extremes, and that IWMDE is already nearly exact at $p_\theta(x)$2 post-warmup draws (Bartoš et al., 23 Apr 2026).

One common misunderstanding is that the IWMDE here estimates Bayes factors directly. It does not. It estimates posterior density ordinates of $p_\theta(x)$3, or ratios of such ordinates, and these are then inserted into the Bayes factor identity above (Bartoš et al., 23 Apr 2026).

6. Related constructions, misconceptions, and limitations

The literature surrounding IWMDE-like ideas is broader than the three principal formulations above, but not every marginal estimator is an importance-weighted one. “Symmetry-Aware Marginal Density Estimation” introduces a Rao–Blackwellized estimator

$p_\theta(x)$4

for discrete structured probabilistic models with automorphism-group symmetries (Niepert, 2013). That estimator improves marginal estimation by conditioning on orbits and applying the Rao–Blackwell theorem, not by reweighting a proposal-target mismatch. It is therefore related by topic but not an IWMDE in the ordinary importance-sampling sense (Niepert, 2013).

A second misconception is that every method using importance weights is necessarily estimating a marginal density. “Learning Causal Models from Conditional Moment Restrictions by Importance Weighting” rewrites conditional moments as weighted unconditional moments using the density ratio

$p_\theta(x)$5

but its target is conditional moment estimation rather than a marginal density itself (Kato et al., 2021). By contrast, the IWAE, RIS-VB, and Bayes-factor-sensitivity constructions all directly target a marginal object: $p_\theta(x)$6, $p_\theta(x)$7, or $p_\theta(x)$8.

A third point concerns computational representation. Recent adaptive importance-sampling work trains a variational autoencoder from weighted samples using a weighted ELBO

$p_\theta(x)$9

thereby learning a marginal proposal density from weighted samples (Demange-Chryst et al., 2023). This suggests that IWMDE can also designate a learned marginal estimator driven by importance weights rather than a closed-form Monte Carlo ratio. A plausible implication is that the term will continue to broaden as deep generative models are integrated into adaptive importance sampling and sensitivity analysis.

Across formulations, the main limitations are stable and recurring. Proposal mismatch or poor overlap increases variance or induces unreliable density-ratio estimates (Burda et al., 2015, Hajargasht et al., 2018, Bartoš et al., 23 Apr 2026). In the IWAE setting, the raw marginal estimator can have high variance even though the log-bound is more stable (Burda et al., 2015). In reciprocal evidence estimation, unbiasedness and asymptotic normality attach to the reciprocal estimator, not to the evidence estimate after inversion (Hajargasht et al., 2018). In sensitivity analysis, density-ratio estimates deteriorate in posterior tails or higher-dimensional sensitivity spaces, so the method is best suited to low-dimensional hyper-parameter sensitivity with appreciable posterior mass in the evaluation region (Bartoš et al., 23 Apr 2026).

In this broader sense, IWMDE names a methodological pattern rather than a single fixed formula: identify a marginal quantity of interest, introduce a tractable law under which sampling is feasible, and construct an importance-weighted estimator whose expectation or induced lower bound recovers the desired marginal object.