Asymptotic normality, concentration, and coverage of generalized posteriors (1907.09611v2)

Abstract: Generalized likelihoods are commonly used to obtain consistent estimators with attractive computational and robustness properties. Formally, any generalized likelihood can be used to define a generalized posterior distribution, but an arbitrarily defined "posterior" cannot be expected to appropriately quantify uncertainty in any meaningful sense. In this article, we provide sufficient conditions under which generalized posteriors exhibit concentration, asymptotic normality (Bernstein-von Mises), an asymptotically correct Laplace approximation, and asymptotically correct frequentist coverage. We apply our results in detail to generalized posteriors for a wide array of generalized likelihoods, including pseudolikelihoods in general, the Gaussian Markov random field pseudolikelihood, the fully observed Boltzmann machine pseudolikelihood, the Ising model pseudolikelihood, the Cox proportional hazards partial likelihood, and a median-based likelihood for robust inference of location. Further, we show how our results can be used to easily establish the asymptotics of standard posteriors for exponential families and generalized linear models. We make no assumption of model correctness so that our results apply with or without misspecification.

An Overview of Asymptotic Properties of Generalized Posteriors

The paper presents an in-depth examination of the asymptotic behavior of generalized posteriors derived from generalized likelihoods. The main focus is on the establishment of conditions under which these posteriors demonstrate concentration, asymptotic normality (also known as the Bernstein--von Mises theorem), an accurate Laplace approximation, and correct asymptotic frequentist coverage.

Key Contributions and Methodology

1. Sufficient Conditions for Generalized Posteriors: The paper provides a set of sufficient conditions for generalized posteriors to concentrate around true parameter values. The core results hinge on conditions that ensure the posterior adequately mirrors certainty as sample size increases. Specifically, the paper stipulates that for a generalized posterior to concentrate, the functions involved must be consistent with the true data-generating process, even when not perfectly specified.
2. Asymptotic Normality: The paper extends the Bernstein--von Mises theorem to generalized posteriors, traditionally applicable to standard Bayesian posteriors under i.i.d. settings. The asymptotic normality established here ensures that generalized posteriors behave in a manner similar to maximum likelihood estimators in large samples, capturing uncertainty through a normal distribution.
3. Laplace Approximation: The authors demonstrate the applicability of Laplace approximation to generalized likelihoods. Under sufficiently regular conditions, the approximation holds, simplifying inference by reducing reliance on computationally intensive methods.
4. Frequentist Coverage: An important contribution is the delineation of the conditions under which credible intervals derived from generalized posteriors maintain asymptotically correct frequentist coverage. This bridges Bayesian and frequentist perspectives by ensuring that intervals calculated under a Bayesian framework tend to enclose true parameter values as samples grow.
5. Applicability to Various Models: The theoretical findings are illustrated with applications to a variety of models, including pseudolikelihoods, exponential families, and generalized linear models. These applications demonstrate the versatility of the proposed framework across different statistical modeling contexts that often employ approximations to likelihood functions.

Implications for Practice and Theory

The theoretical contributions of this paper have significant implications for practical applications where exact likelihoods are intractable or computationally expensive. By establishing robust asymptotic properties of generalized posteriors, researchers can confidently use approximations without the loss of inferential reliability associated with many heuristic approaches.

From a theoretical perspective, the results provide a formal justification for the use of generalized likelihoods in obtaining Bayesian posteriors, expanding the scope of likelihood-based methods in statistical inference. Furthermore, this work opens avenues for future research into higher-dimensional parameter spaces and non-i.i.d. data structures, which are commonplace in contemporary statistical challenges.

Future Directions

While the paper lays a robust foundation, further investigations could extend these results to encompass more complex dependencies and irregularities in data. Particularly, exploring the relationships between generalized posteriors and machine learning techniques could yield fruitful insights as Bayesian methods continue to evolve in the context of modern data science.

In summation, the paper significantly advances our understanding of the asymptotic behavior of generalized posteriors, providing both theoretical rigor and practical guidance for their application in statistical inference. These contributions will likely influence both the development and application of sophisticated statistical methods in various fields of research.