Marginalized Importance Sampling Overview

• Marginalized Importance Sampling is a technique that reduces the variance of estimators by integrating out latent or nuisance variables using structured correction ratios.
• It leverages the Rao–Blackwell principle and operator-theoretic constructions to achieve provable variance reduction in high-dimensional and complex models.
• The method enhances efficiency in Bayesian models, reinforcement learning, and rare event simulations by transitioning from joint to marginal weight estimators.

Marginalized Importance Sampling (MIS) is a methodology that systematically reduces the variance of classical importance sampling estimators by analytically or algorithmically integrating out latent or nuisance variables, or by employing structured correction ratios over marginal distributions rather than path-wise distributions. This leads to substantial statistical and computational benefits in high-dimensional simulation, Bayesian inference, reinforcement learning, and rare event estimation. MIS frameworks leverage the Rao–Blackwell principle, operator-theoretic constructions, density-ratio estimation, and state abstraction to achieve provable variance reduction, consistent estimation, and greater computational efficiency relative to standard importance sampling.

1. Core Concepts in Marginalized Importance Sampling

Classical importance sampling (IS) seeks to estimate expectations under a target distribution $p(x)$ via samples from a proposal $q(x)$ using weights $w(x) = p(x)/q(x)$. However, naive IS weights can incur large variance, especially when attempting to estimate marginal probabilities, normalizing constants, or expected returns in rare event regimes or off-policy settings with distributional shift.

MIS mitigates this by constructing weights that relate marginal (rather than joint or path-wise) distributions. A typical MIS estimator takes the form

$\E_p[f(x)] = \E_q\bigl[w(x)\,f(x)\bigr],$

where $w(x)$ is often a marginal density ratio. Rao–Blackwellization is frequently leveraged to integrate over latent variables, yielding weights that are conditional expectations and thus strictly reduce variance.

In finite mixture models for evidence approximation, the marginal likelihood is expressed using allocations $z$ over latent component indices. Marginalizing $z$ out of the IS proposal—in practice via Rao–Blackwellization—produces the "dual" or Rao–Blackwellised importance function, leading to lower-variance estimators for model evidence and increasing effective sample size (Lee et al., 2013).

Analogously, in annealed importance sampling, marginalizing hidden variables out of the transition kernels produces marginalized AIS (mAIS), yielding estimators for the free energy with provably reduced variance and bias given appropriate kernel factorization (Yasuda et al., 2022).

In reinforcement learning, MIS replaces the product of trajectory-wise IS ratios with a single marginalized density ratio connecting the occupancy distribution of the target and behavior policy. This marginalization provides polynomial rather than exponential scaling in mean-square error with respect to horizon length, and can be generalized via operator-theoretic constructions (Xie et al., 2019, Tang et al., 2022).

2. Bayesian Models and Marginalized IS Design

Marginalized IS is especially relevant for inference in Bayesian latent variable models and mixture models. Let $x$ denote observed data, $\theta$ parameters, $z$ latent allocations, and $q(x)$0 the model likelihood. Traditional IS for model evidence is confounded by label-switching: the posterior is invariant to permutations of component labels, causing proposals that do not span all $q(x)$1 modes to be inefficient.

Lee and Robert (Lee et al., 2013) propose Rao–Blackwellized (dual) IS schemes:

• Construct a proposal by symmetrizing over all label permutations and averaging over latent variable samples from a posterior under the full model.
• Marginalize $q(x)$2 from the proposal: $q(x)$3.
• Compute IS weights as $q(x)$4.

This estimator is unbiased and has variance no greater than the non-marginalized alternative. Empirically, the dual IS scheme achieves 2–10x higher effective sample size and order-of-magnitude CPU speedups by truncating the sum over permutations while maintaining >99.999% of importance mass when modes are well separated.

In the context of Bayesian MCMC targeting approximate marginals, Vihola et al. (Vihola et al., 2016) present a two-phase strategy:

1. Run MCMC on a tractable approximate marginal posterior in the parameters.
2. For each sample $q(x)$5, draw conditionally independent latent variables from an auxiliary distribution, correct the bias by importance weighting (possibly via particle filtering), and average functionals with these weights.

Under minimal support, ergodicity, and unbiasedness, this yields strongly consistent estimators. The approach is also amenable to full parallelization and multilevel debiasing.

3. Variance Reduction via Rao–Blackwellization and Abstraction

MIS achieves variance reduction fundamentally by the Rao–Blackwell theorem. In annealed importance sampling, marginalized AIS (mAIS) collapses out subsets of variables (e.g., hidden units in RBMs), replacing path-wise weights $q(x)$6 with their conditional expectation $q(x)$7 given visible variables $q(x)$8 (Yasuda et al., 2022). Theorem 1 establishes that under factorized kernels,

$q(x)$9

and an analogous inequality holds for the bias of the log-partition estimator.

In reinforcement learning, state abstraction can further lower variance: mapping the state space $w(x) = p(x)/q(x)$0 to a lower-dimensional abstract space via a surjective $w(x) = p(x)/q(x)$1 aggregates distributional mass, and ratios computed in the abstract space necessarily have variance no greater than those in the ground space unless the ratios are constant within blocks (Pavse et al., 2022). Empirically, AbstractBestDICE achieves 2–10x lower MSE relative to ground-space density ratio estimators in high-dimensional off-policy evaluation tasks.

4. MIS in Reinforcement Learning and Control

MIS excels in off-policy evaluation (OPE) for RL. Classical IS re-weights trajectories using products of local likelihood ratios, incurring variance that grows exponentially with horizon. MIS estimates single-step density ratios between the discounted visitation distribution of the evaluation and behavior policies, yielding a dramatic improvement.

For episodic, tabular MDPs, the MIS estimator (Xie et al., 2019) decomposes estimates over each timestep $w(x) = p(x)/q(x)$2: $w(x) = p(x)/q(x)$3 where $w(x) = p(x)/q(x)$4 and $w(x) = p(x)/q(x)$5 are recursively estimated marginal state distributions, and $w(x) = p(x)/q(x)$6 are conditional reward averages. This yields an MSE bound scaling as $w(x) = p(x)/q(x)$7, matching Cramér–Rao lower bounds up to a factor $w(x) = p(x)/q(x)$8.

Tang et al. (Tang et al., 2022) introduce marginalized Bellman operators, parameterized by collections of TD weights that generalize multi-step and IS operators. If the true density-ratios are provided, the operator is exact and instantly contractive.

Recent advances include scalable estimators via the deep successor representation (SR-DICE) (Fujimoto et al., 2021), which computes density ratios using the successor features of the target policy, enabling MIS to scale to high-dimensional and continuous domains such as MuJoCo and Atari.

For off-environment policy evaluation (Sim2Sim, Sim2Real), MIS has been adapted by learning the correction ratio as a product: one factor estimated directly by supervised ratio estimation (from simulator and real data), and a second (close to one) estimated with lower variance via minimax methods (Katdare et al., 2023). The two-step factorization leads to improved sample complexity and robustness under domain shift.

5. Rare Event Simulation and Marginal Likelihood Ratios

In rare event simulation, such as overflow probabilities for tandem queues, pathwise IS weights are well known to have variance that explodes exponentially with the number of steps or the event rarity (Zhao et al., 18 Apr 2025). MIS reframes the problem by re-weighting only the terminal state via the marginal likelihood ratio: $w(x) = p(x)/q(x)$9 where $\E_p[f(x)] = \E_q\bigl[w(x)\,f(x)\bigr],$0 is the stationary law under an alternative process. Since $\E_p[f(x)] = \E_q\bigl[w(x)\,f(x)\bigr],$1 depends only on the terminal state, the variance grows only polynomially with the rarity parameter, in stark contrast to the exponential growth of traditional pathwise IS.

A machine learning estimator for $\E_p[f(x)] = \E_q\bigl[w(x)\,f(x)\bigr],$2 is constructed by characterizing the stationary distribution via a functional equation, and then fitting a neural network $\E_p[f(x)] = \E_q\bigl[w(x)\,f(x)\bigr],$3 to satisfy the stationarity constraint in expectation through an RKHS-based penalized loss. Empirically, this approach yields rMSE within a factor of 1.3–1.5 of the oracle, and is robust to increasing event rarity, provided a mixture of kernels is used in the loss.

6. Applications, Practical Considerations, and Empirical Results

MIS frameworks have attained notable practical significance:

• In Bayesian mixture models and evidence estimation, dual (Rao–Blackwellized) IS schemes efficiently address multimodality induced by label switching, with effective sample size gains and computational accelerations (Lee et al., 2013).
• In reinforcement learning, MIS and its variants (including abstraction-augmented and SR-based techniques) yield consistent and information-theoretically near-optimal OPE estimators, with polynomial dependence on horizon and robustness in high-dimensional, partially observed, and control environments (Xie et al., 2019, Fujimoto et al., 2021, Pavse et al., 2022).
• In rare event estimation for Markovian queues, the MLIS approach outperforms regenerative pathwise IS in MSE and computational time, especially as the event becomes rarer (Zhao et al., 18 Apr 2025).

Empirical studies repeatedly demonstrate that Rao–Blackwellization, marginalization, and abstraction either strictly reduce estimator variance or yield substantial practical improvements in sample efficiency, estimation stability, and computational tractability. A recurring pattern is that, whenever sufficient structure allows the construction of tractable marginal or abstract ratios and the computational cost remains sub-exponential, MIS yields dominant performance over classical IS.

7. Limitations, Assumptions, and Future Directions

The principal limitations of MIS concern:

• The availability of tractable marginalizations (exact or approximate), particularly in complex, non-factorized or non-bipartite models.
• The estimation challenges and possible bias in learning high-dimensional or abstracted density ratios, especially when distributional support does not overlap (coverage issues).
• The need for strong realizability assumptions—reward-equality and transition-similarity for abstraction, sufficient function capacity for neural density ratio estimation, or ergodicity and unbiasedness in the MCMC context.

Ongoing research directions include:

• Extending variance reduction via partial or hierarchical Rao–Blackwellization in deep latent models (Yasuda et al., 2022, Pavse et al., 2022).
• Developing consistent methods for learning state abstractions jointly with ratio estimation.
• Advancing confidence interval and uncertainty quantification for MIS-based OPE in nonstationary, real-world domains (Katdare et al., 2023).
• Multilevel techniques and improved debiasing in hierarchical or doubly-intractable settings (Vihola et al., 2016).

MIS continues to be central in advancing inference, evaluation, and simulation methods in statistics and machine learning, exploiting marginalization as a primary vehicle for variance control and computational efficiency.