The sample size required in importance sampling (1511.01437v3)

Abstract: The goal of importance sampling is to estimate the expected value of a given function with respect to a probability measure $\nu$ using a random sample of size $n$ drawn from a different probability measure $\mu$. If the two measures $\mu$ and $\nu$ are nearly singular with respect to each other, which is often the case in practice, the sample size required for accurate estimation is large. In this article it is shown that in a fairly general setting, a sample of size approximately $\exp(D(\nu||\mu))$ is necessary and sufficient for accurate estimation by importance sampling, where $D(\nu||\mu)$ is the Kullback-Leibler divergence of $\mu$ from $\nu$. In particular, the required sample size exhibits a kind of cut-off in the logarithmic scale. The theory is applied to obtain a general formula for the sample size required in importance sampling for one-parameter exponential families (Gibbs measures).

• The paper establishes that the sample size required for accurate importance sampling is approximately exponential in the Kullback-Leibler divergence between the target and sampling distributions.
• It identifies a cut-off phenomenon in the logarithmic scale for sample size, where increasing size beyond a threshold yields diminishing accuracy returns.
• The paper applies the theory to exponential families and proposes a new diagnostic method based on sample contribution, improving reliability over traditional variance estimates.

Overview of "The Sample Size Required in Importance Sampling"

The paper "The Sample Size Required in Importance Sampling" by Sourav Chatterjee and Persi Diaconis addresses a crucial problem in the field of Monte Carlo methods: the determination of the sample size needed for accurate importance sampling estimation. In scenarios where the target distribution and the sampling distribution are nearly singular with respect to each other, standard variance-based methods are often inadequate. This paper provides a rigorous mathematical framework for estimating the minimum sample size necessary for reliable estimations, utilizing the Kullback-Leibler (KL) divergence as a central metric.

Key Findings

1. Sample Size Criterion: The authors establish that the sample size required for accurate importance sampling is approximately exponential in the KL divergence between the target and sampling measures, denoted as $\exp(D(\nu||\mu))$. This result offers a more reliable measure compared to traditional variance-based methods, which tend to overestimate the required sample size.
2. Cut-off in Logarithmic Scale: The paper identifies a cut-off phenomenon in the logarithmic scale for the required sample size. It highlights that in many practical scenarios, once a threshold sample size is exceeded, further increases provide diminishing returns regarding accuracy improvement.
3. Applications to Exponential Families: The theory is applied specifically to Gibbs measures and one-parameter exponential families, where it is shown that the required sample size also depends on additional parameters such as the inverse temperature in statistical mechanics contexts.
4. Alternative Diagnostics: The authors propose an alternative diagnostic method to traditional variance estimations, which they demonstrate as unreliable in diagnosing importance sampling convergence. Their suggested method is based on estimating the maximum contribution of a sample to the sum of importance weights, providing a more justifiable indication of convergence.

Implications and Applications

The findings of this paper have significant implications for both theoretical developments and practical applications in statistical computing and simulation-based inference. The insights into the necessary sample size help in designing efficient algorithms for high-dimensional and complex systems where traditional Monte Carlo methods struggle. The results are especially pertinent in fields like statistical mechanics, where calculating partition functions via simulation is notoriously challenging.

Future Directions

The paper opens several avenues for future research. There is a need to further investigate the efficacy of the newly proposed diagnostic tool across various domains and to explore how these methods can be integrated into existing computational frameworks. Additionally, extending the analysis to other forms of dependencies and distributions beyond the scope of Gibbs measures could broaden the applicability of these results.

This work contributes to a deeper understanding of the interplay between information theory (via KL divergence) and Monte Carlo sampling, offering both a theoretical and practical advance in our ability to perform effective statistical estimation in challenging scenarios.