---
title: 'IWMDE: Importance-Weighted Marginal Density Estimator'
url: https://www.emergentmind.com/topics/importance-weighted-marginal-density-estimator-iwmde
type: topic
---

# IWMDE: Importance-Weighted Marginal Density Estimator

Importance-Weighted Marginal Density Estimator (IWMDE) denotes a class of estimators that recover an intractable marginal density or marginal data density by reweighting samples drawn from a tractable auxiliary distribution. The exact term is explicitly used in recent Bayes factor sensitivity analysis to estimate posterior density ordinates of a hyper-parameter from extended-model MCMC output [2604.21596]. Closely related constructions appear earlier in latent-variable generative modeling, where the canonical estimator is the importance-sampling average
\[
\hat p_K(x)=\frac{1}{K}\sum_{k=1}^K \frac{p_\theta(x,z_k)}{q_\phi(z_k\mid x)},
\qquad z_k\sim q_\phi(z\mid x),
\]
which is an unbiased estimator of the marginal density \(p_\theta(x)\) [1509.00519]. The literature therefore supports a broad usage in which “IWMDE” refers not to a single universally standardized estimator, but to a family of importance-weighted constructions for marginal likelihoods, marginal densities, or posterior ordinates.

## 1. Terminology and conceptual scope

The most explicit contemporary use of the name arises in Bayes factor sensitivity analysis, where the IWMDE is used to estimate the posterior density ordinate
\[
p(\gamma^\ast \mid y, H_1^e)
\]
of a sensitivity hyper-parameter \(\gamma\) under an extended model \(H_1^e\) [2604.21596]. In that setting, the estimator is not the Bayes factor itself; rather, it supplies the posterior density ratio
\[
\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)},
\]
which is then combined with an anchor Bayes factor \(BF_{10}(\gamma_0)\) and a prior ratio to recover the full sensitivity curve [2604.21596].

Earlier papers do not generally use the exact label “Importance-Weighted Marginal Density Estimator,” but they provide the core estimators naturally associated with that term. In latent-variable models, “Importance Weighted Autoencoders” introduces the standard importance-sampling estimator of the marginal density \(p_\theta(x)\) and optimizes the expectation of its logarithm [1509.00519]. “Reinterpreting Importance-Weighted Autoencoders” makes the same estimator central and interprets the resulting bound through an implicit importance-weighted posterior family [1704.02916]. In Bayesian model comparison, “Accurate Computation of Marginal Data Densities Using Variational Bayes” develops a reciprocal-importance-sampling marginal data density estimator whose weighting density is a variational Bayes posterior approximation \(q^*(\theta)\) [1805.10036].

This suggests a useful taxonomy. In one branch, IWMDE refers to forward importance estimators of an intractable marginal such as \(p_\theta(x)\). In another, it refers to reciprocal estimators of model evidence \(p(\mathbf y)\). In a third, it refers to importance-weighted estimators of posterior density ordinates needed for downstream quantities such as Bayes factors. The common structure is the same: a target marginal object is estimated by reweighting samples from a tractable law.

## 2. Canonical latent-variable formulation

The canonical latent-variable setup writes
\[
p_\theta(x,z)=p_\theta(x\mid z)p(z),
\qquad
p_\theta(x)=\int p_\theta(x,z)\,dz,
\]
with posterior
\[
p_\theta(z\mid x)=\frac{p_\theta(x,z)}{p_\theta(x)}.
\]
Because the marginal integral is typically intractable, a tractable recognition or proposal distribution \(q_\phi(z\mid x)\) is introduced [1509.00519]. Importance sampling then yields
\[
p_\theta(x)=
\mathbb{E}_{q_\phi(z\mid x)}
\!\left[
\frac{p_\theta(x,z)}{q_\phi(z\mid x)}
\right].
\]

Given \(K\) i.i.d. samples \(z_{1:K}\sim q_\phi(z\mid x)\), define
\[
w_k=\frac{p_\theta(x,z_k)}{q_\phi(z_k\mid x)}.
\]
The corresponding importance-weighted marginal density estimator is
\[
\hat p_K(x)=\frac{1}{K}\sum_{k=1}^K w_k
=
\frac{1}{K}\sum_{k=1}^K \frac{p_\theta(x,z_k)}{q_\phi(z_k\mid x)}.
\]
Under the usual support and integrability conditions, this estimator is unbiased:
\[
\mathbb{E}[\hat p_K(x)]=p_\theta(x)
\]
[1509.00519].

A central distinction in the literature is between the estimator \(\hat p_K(x)\) itself and the training objective built from it. The IWAE objective is
\[
\mathcal{L}_K(x)=
\mathbb{E}\!\left[\log \hat p_K(x)\right]
=
\mathbb{E}_{z_{1:K}\sim q_\phi(z\mid x)}
\left[
\log
\frac{1}{K}\sum_{k=1}^K
\frac{p_\theta(x,z_k)}{q_\phi(z_k\mid x)}
\right].
\]
By Jensen’s inequality,
\[
\mathcal{L}_K(x)\le \log p_\theta(x),
\]
so the expectation of the log of the IWMDE is a lower bound on the marginal log-likelihood [1509.00519]. The special case \(K=1\) recovers the standard VAE ELBO, and the bound tightens monotonically with \(K\):
\[
\log p_\theta(x)\ge \mathcal{L}_{K+1}(x)\ge \mathcal{L}_K(x).
\]
If \(p_\theta(x,z)/q_\phi(z\mid x)\) is bounded, then
\[
\lim_{K\to\infty}\mathcal{L}_K(x)=\log p_\theta(x)
\]
[1509.00519].

A common misconception is to conflate these objects. The estimator
\[
\hat p_K(x)
\]
is unbiased for \(p_\theta(x)\), whereas
\[
\log \hat p_K(x)
\]
is not unbiased for \(\log p_\theta(x)\); its expectation is downward biased, precisely because it is a lower bound [1509.00519]. This distinction is central to any precise use of the term IWMDE in the latent-variable literature.

## 3. Implicit posterior geometry and variational interpretation

The IWAE literature does not treat the importance-weighted marginal estimator merely as a scalar approximation to \(p(x)\). It also induces an implicit posterior approximation. “Reinterpreting Importance-Weighted Autoencoders” defines the unnormalized importance-weighted distribution
\[
\tilde{q}_{IW}(z|x,z_{2:k})
=
\frac{ \frac{p(x,z)}{q(z|x)} }{ \frac{1}{k}\sum_{j=1}^k \frac{p(x,z_j)}{q(z_j|x)} } q(z|x)
=
\frac{p(x,z)}{\frac{1}{k}\left(\frac{p(x,z)}{q(z|x)}+\sum_{j=2}^k\frac{p(x,z_j)}{q(z_j|x)}\right)},
\]
and shows that the IWAE objective can be reinterpreted as the standard variational lower bound evaluated under this richer implicit distribution [1704.02916].

Averaging over the auxiliary samples yields the normalized expected importance-weighted distribution
\[
q_{EW}(z|x)=
E_{z_{2}...z_{k} \sim q(z|x)}
\left[
\tilde{q}_{IW}(z|x,z_{2:k})
\right].
\]
The paper proves the ordering
\[
\log p(x) \geq L_{VAE}[q_{EW}] \geq L_{IWAE}[q] \geq L_{VAE}[q]
\]
and establishes
\[
KL(q_{EW}||p) \leq KL(q||p)
\]
[1704.02916]. This implies that the importance-weighted construction improves the effective posterior approximation, not just the scalar lower bound.

This suggests a broader interpretation of IWMDE-style estimators in latent-variable models. They do not only estimate a marginal quantity; they reshape the effective inference distribution toward the true posterior. In practical terms, the induced posterior can be sampled by sampling-importance-resampling:
\[
\tilde w_i=\frac{w_i}{\sum_{i=1}^k w_i},
\]
then drawing an index \(j\sim \mathrm{Categorical}(\tilde{\mathbf w})\) and returning \(z_j\) [1704.02916]. A plausible implication is that, whenever IWMDE is used as a training primitive, its inferential consequences extend beyond likelihood approximation into posterior geometry and representation learning.

## 4. Reciprocal importance-weighted evidence estimators

A distinct but closely related lineage concerns marginal data density estimation in Bayesian model comparison. “Accurate Computation of Marginal Data Densities Using Variational Bayes” studies the marginal data density
\[
p(\mathbf y)=\int_{\Theta} p(\mathbf y\mid \theta)p(\theta)\,d\theta
\]
and starts from the reciprocal importance sampling (RIS) estimator
\[
\hat{p}_{RIS}(\mathbf{y}) =
\left(
\frac{1}{S}\sum_{s=1}^{S}
\frac{h\left(\theta^{(s)}\right)}
{p\left(\mathbf{y}|\theta^{(s)}\right)p\left(\theta^{(s)}\right)}
\right)^{-1},
\qquad
\theta^{(s)}\sim p(\theta\mid \mathbf y).
\]
Its proposal is to set the weighting density to the variational Bayes posterior approximation \(q^*(\theta)\), yielding
\[
\hat p_{RIS.VB}(\mathbf{y}) =
\left(
\frac{1}{S}\sum_{s=1}^{S}
\frac{q^*(\theta^{(s)})}
{p(\mathbf{y}|\theta^{(s)})p(\theta^{(s)})}
\right)^{-1}
\]
[1805.10036].

In this formulation, the basic unbiased object is the reciprocal:
\[
\hat p_{RIS.VB}(\mathbf y)^{-1}
=
\frac{1}{S}\sum_{s=1}^S
\frac{q^*(\theta^{(s)})}
{p(\mathbf y\mid \theta^{(s)})p(\theta^{(s)})}.
\]
The paper proves
\[
E_{p(\theta\mid \mathbf y)}
\!\left[
\frac{q^*(\theta)}
{p(\mathbf y\mid \theta)p(\theta)}
\right]
=
p(\mathbf y)^{-1},
\]
and establishes that the reciprocal estimator is unbiased, consistent, and asymptotically normal under the stated assumptions [1805.10036]. The effective importance ratio is
\[
\frac{q^*(\theta)}{p(\theta\mid \mathbf y)}.
\]

This reciprocal construction differs from the forward latent-variable estimator \(\hat p_K(x)\), but the underlying logic is the same: a marginal quantity is reconstructed by reweighting samples under a tractable law. The paper also proves a variance identity,
\[
Var_p\!\left(\frac{q^*}{p}\right)=E_q\!\left[\frac{q^*}{p}\right]-1,
\]
which makes closeness of \(q^*\) to the posterior central for stability [1805.10036]. It further argues that the VB-based weighting density avoids the truncation often required in harmonic-mean-type methods because the posterior dominates the VB approximation under its assumptions [1805.10036].

A frequent source of confusion is to treat all importance-weighted marginal estimators as forward Monte Carlo averages. The RIS-VB construction shows that an IWMDE can also be reciprocal: the estimator targets \(p(\mathbf y)^{-1}\) first and then inverts. The unbiasedness statement is therefore about the reciprocal, not about \(\hat p_{RIS.VB}(\mathbf y)\) itself [1805.10036].

## 5. IWMDE in Bayes factor sensitivity analysis

The 2026 sensitivity-analysis literature uses the name IWMDE explicitly. The framework considers Bayes factors
\[
BF_{10}(\gamma)=\frac{Z(\gamma)}{Z_0},
\qquad
Z(\gamma)=p(y\mid H_1(\gamma)),
\]
where \(\gamma\) indexes the prior under the alternative [2604.21596]. Introducing an extended model \(H_1^e\) with hyper-prior \(\pi(\gamma)\) yields
\[
p(\gamma \mid y, H_1^e)
=
\frac{Z(\gamma)\,\pi(\gamma)}{p(y \mid H_1^e)}.
\]
Taking ratios at \(\gamma_x\) and \(\gamma_0\) gives
\[
BF_{10}(\gamma_x)
=
BF_{10}(\gamma_0)
\times
\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)}
\times
\frac{\pi(\gamma_0)}{\pi(\gamma_x)}.
\]
Under a uniform hyper-prior, this simplifies to
\[
BF_{10}(\gamma_x)
=
BF_{10}(\gamma_0)
\times
\frac{p(\gamma_x \mid y, H_1^e)}{p(\gamma_0 \mid y, H_1^e)}
\]
[2604.21596].

The IWMDE in this setting estimates the posterior ordinate \(p(\gamma^\ast \mid y,H_1^e)\) from joint posterior draws \(\{(\gamma_i,\boldsymbol{\theta}_i)\}_{i=1}^n\):
\[
\hat{p}(\gamma^\ast \mid y, H_1^e)
=
\frac{1}{n}\sum_{i=1}^n
w(\gamma^\ast \mid \boldsymbol{\theta}_i)\,
\frac{p(\gamma^\ast,\boldsymbol{\theta}_i \mid y, H_1^e)}
{p(\gamma_i,\boldsymbol{\theta}_i \mid y, H_1^e)}.
\]
When the sensitivity parameter enters only through the prior on the model parameters, the likelihood cancels:
\[
\frac{p(\gamma^\ast,\boldsymbol{\theta}_i \mid y, H_1^e)}
{p(\gamma_i,\boldsymbol{\theta}_i \mid y, H_1^e)}
=
\frac{p(\boldsymbol{\theta}_i \mid \gamma^\ast)\,\pi(\gamma^\ast)}
{p(\boldsymbol{\theta}_i \mid \gamma_i)\,\pi(\gamma_i)}.
\]
Under a uniform hyper-prior, this becomes a simple prior-density ratio [2604.21596].

The paper identifies two important weighting choices. If
\[
w(\gamma \mid \boldsymbol{\theta},y,H_1^e)=p(\gamma \mid \boldsymbol{\theta},y,H_1^e),
\]
the IWMDE reduces to the conditional marginal density estimator
\[
\hat{p}(\gamma^\ast \mid y, H_1^e)
=
\frac{1}{n}\sum_{i=1}^n p(\gamma^\ast \mid \boldsymbol{\theta}_i, y, H_1^e),
\]
which the paper states is asymptotically variance-minimizing [2604.21596]. A simpler uniform weighting function is also consistent but higher variance [2604.21596].

The empirical role of the estimator is also precise. It is applied to a univariate Bayesian \(t\)-test, a bivariate informed \(t\)-test, and Bayesian model-averaged meta-analysis, where it substantially outperforms kernel density estimation for posterior ordinate recovery across the sensitivity range [2604.21596]. The paper reports that in the univariate \(t\)-test the IWMDE stays within about \(1\%-2\%\) of the exact Bayes factor curve even at the boundaries, whereas KDE can deviate by roughly \(10\%-20\%\) at extremes, and that IWMDE is already nearly exact at \(3{,}000\) post-warmup draws [2604.21596].

One common misunderstanding is that the IWMDE here estimates Bayes factors directly. It does not. It estimates posterior density ordinates of \(\gamma\), or ratios of such ordinates, and these are then inserted into the Bayes factor identity above [2604.21596].

## 6. Related constructions, misconceptions, and limitations

The literature surrounding IWMDE-like ideas is broader than the three principal formulations above, but not every marginal estimator is an importance-weighted one. “Symmetry-Aware Marginal Density Estimation” introduces a Rao–Blackwellized estimator
\[
\hat{\theta}^{\mathtt{rb}}_N
=
\frac{1}{N}\sum_{i=1}^{N}
P(\mathbf{\hat{X}=\mathbf{\hat{x}}\mid s_i^{\mathfrak{G}})
\]
for discrete structured probabilistic models with automorphism-group symmetries [1304.2694]. That estimator improves marginal estimation by conditioning on orbits and applying the Rao–Blackwell theorem, not by reweighting a proposal-target mismatch. It is therefore related by topic but not an IWMDE in the ordinary importance-sampling sense [1304.2694].

A second misconception is that every method using importance weights is necessarily estimating a marginal density. “Learning Causal Models from Conditional Moment Restrictions by Importance Weighting” rewrites conditional moments as weighted unconditional moments using the density ratio
\[
r^*(y,x\mid z)=\frac{p(y,x\mid z)}{p(y,x)},
\]
but its target is conditional moment estimation rather than a marginal density itself [2108.01312]. By contrast, the IWAE, RIS-VB, and Bayes-factor-sensitivity constructions all directly target a marginal object: \(p_\theta(x)\), \(p(\mathbf y)\), or \(p(\gamma^\ast\mid y,H_1^e)\).

A third point concerns computational representation. Recent adaptive importance-sampling work trains a variational autoencoder from weighted samples using a weighted ELBO
\[
\mathrm{wELBO}(\phi,\theta,\lambda)
=
\mathbb E_f\!\left[
\frac{\widetilde g^*(X)}{\widetilde f(X)}
\mathbb E_{q_\phi(\cdot\mid X)}[\log g_{\theta}(X\mid Z)]
\right]
-
\mathbb E_f\!\left[
\frac{\widetilde g^*(X)}{\widetilde f(X)}
D_{\mathrm{KL}}(q_\phi(Z\mid X)\|p_{\lambda,\phi}(Z))
\right],
\]
thereby learning a marginal proposal density from weighted samples [2310.09194]. This suggests that IWMDE can also designate a learned marginal estimator driven by importance weights rather than a closed-form Monte Carlo ratio. A plausible implication is that the term will continue to broaden as deep generative models are integrated into adaptive importance sampling and sensitivity analysis.

Across formulations, the main limitations are stable and recurring. Proposal mismatch or poor overlap increases variance or induces unreliable density-ratio estimates [1509.00519; 1805.10036; 2604.21596]. In the IWAE setting, the raw marginal estimator can have high variance even though the log-bound is more stable [1509.00519]. In reciprocal evidence estimation, unbiasedness and asymptotic normality attach to the reciprocal estimator, not to the evidence estimate after inversion [1805.10036]. In sensitivity analysis, density-ratio estimates deteriorate in posterior tails or higher-dimensional sensitivity spaces, so the method is best suited to low-dimensional hyper-parameter sensitivity with appreciable posterior mass in the evaluation region [2604.21596].

In this broader sense, IWMDE names a methodological pattern rather than a single fixed formula: identify a marginal quantity of interest, introduce a tractable law under which sampling is feasible, and construct an importance-weighted estimator whose expectation or induced lower bound recovers the desired marginal object.

Source: https://www.emergentmind.com/topics/importance-weighted-marginal-density-estimator-iwmde