Rates of Convergence of Generalised Variational Inference Posteriors under Prior Misspecification (2510.03109v1)

Abstract: We prove rates of convergence and robustness to prior misspecification within a Generalised Variational Inference (GVI) framework with bounded divergences. This addresses a significant open challenge for GVI and Federated GVI that employ a different divergence to the Kullback--Leibler under prior misspecification, operate within a subset of possible probability measures, and result in intractable posteriors. Our theoretical contributions cover severe prior misspecification while relying on our ability to restrict the space of possible GVI posterior measures, and infer properties based on this space. In particular, we are able to establish sufficient conditions for existence and uniqueness of GVI posteriors on arbitrary Polish spaces, prove that the GVI posterior measure concentrates on a neighbourhood of loss minimisers, and extend this to rates of convergence regardless of the prior measure.

• The paper establishes that GVI posteriors remain near the empirical loss minimizer despite severe prior misspecification.
• It proves existence, uniqueness, and convergence rates just slower than n⁻¹ for GVI posteriors even in infinite dimensional settings.
• The findings demonstrate GVI's practical robustness, offering reliable inference in high-dimensional and federated learning applications.

Rates of Convergence of Generalised Variational Inference Posteriors under Prior Misspecification

This paper examines the theoretical underpinnings and practical implications of Generalised Variational Inference (GVI), especially in the context of prior misspecification. The authors aim to address how GVI can ensure robustness and consistency of posterior distributions even when priors are misspecified. Their focus lies in establishing the rates of convergence for GVI posteriors and extending these results to infinite dimensional spaces, a setting highly relevant to modern machine learning applications.

Introduction to GVI and Prior Misspecification

The paper begins by framing the inherent challenges in Bayesian inference related to prior misspecification. Traditional Bayesian approaches rely heavily on the assumption that a well-specified prior leads to consistent posterior distributions; however, this is seldom the case in practice. GVI offers a framework that replaces the Bayesian approach with an optimization-based perspective, allowing the use of different divergences and loss functions other than the Kullback-Leibler divergence and log-likelihood, respectively.

By casting Bayesian updating into an optimization problem over a simpler class of measures, GVI provides flexibility in tackling prior and model misspecification. This results in a posterior that is theoretically robust irrespective of the prior's misspecification, a pivotal shift in methodology from classical Bayesian methods.

Theoretical Contributions

The core theoretical contributions of the paper are encapsulated in several key results:

1. Characterization of GVI Posteriors:

Under certain bounded divergence assumptions, the authors show that GVI posteriors remain within a specific neighborhood of the empirical loss minimizer, practically independent of the prior selection from a broad class. This characteristic ensures that even with severely misspecified priors, the posteriors do not deviate significantly from data-driven insights.

2. Existence and Uniqueness:

The paper extends the theory of GVI by proving the existence of minimizers to the GVI objective in infinite dimensional settings. The existence is guaranteed under conditions that the functional form of the divergence and loss are coercive and lower semi-continuous.

3. Asymptotic Consistency and Convergence Rates:

The authors demonstrate that GVI posteriors achieve asymptotic consistency, converging to sets containing minimizers of the loss function. They derive rates of convergence just slower than $n^{-1}$, where $n$ is the number of observations, under the assumptive structure provided, including bounded divergence measures. This establishes a formal basis for the robustness of GVI posteriors even under adversarial conditions.

Practical Implications and Applications

Practically, the implications of this research are significant for domains where model and prior misspecification is rampant, such as in complex, hierarchical, or high-dimensional data settings commonly seen in machine learning and data science. The results provide assurance that inference conducted via GVI remains stable and reliable, mitigating the risk of posterior inconsistency induced by poor prior choices.

This is particularly relevant in federated learning settings, where data privacy constrains the sharing of models rather than data itself. Federated GVI would accommodate the integration of diverse data distributions across clients without assuming centralized prior knowledge.

Conclusion

The paper significantly advances the theoretical foundations of GVI by addressing prior misspecification—a critical issue in Bayesian inference—and establishes concrete convergence rates for its posteriors. By demonstrating that robustness and consistency can be preserved without restrictive assumptions on the prior, this work lays the groundwork for practical applications in varied real-world data environments where traditional Bayesian models may underperform.

The research trajectory put forth suggests further exploration into the unbounded divergence scenarios remains an open area for future investigation, along with the development of algorithms to implement these theoretical insights into scalable solutions.