Bayesian Averaging Approach

• Bayesian averaging is a method that quantifies uncertainty by averaging predictions from multiple models weighted by their posterior probabilities.
• It integrates Bayesian model averaging with conformal prediction to achieve finite-sample validity and robust predictive coverage.
• Empirical results show that CBMA maintains reliable coverage and efficiency across simulated and real data, even under model misspecification.

A Bayesian averaging approach is a formal method for quantifying predictive, inferential, and decision-theoretic uncertainty by averaging over a set of probabilistic models, each weighted by its posterior probability given observed data. This paradigm is prominent in settings where model uncertainty is significant, such as in predictive inference, causal estimation, clustering, and classification. A contemporary instantiation of the Bayesian averaging approach is given by the Conformal Bayesian Model Averaging (CBMA) framework, which unites Bayesian model averaging (BMA) with the frequentist-robust guarantees of conformal prediction (Bhagwat et al., 21 Nov 2025).

1. Bayesian Model Averaging (BMA) Framework

BMA begins with a candidate model set $\{\mathcal{M}_1, \dots, \mathcal{M}_K\}$, each with its likelihood $p_{\theta_k}(y|x)$ and prior $\pi_k(\theta_k)$. For data $D$, the posterior predictive under model $k$ is

$$p_{\mathcal{M}_k}(y|x, D) = \int p_{\theta_k}(y|x)\, \pi_k(\theta_k|D) \, d\theta_k,$$

where $\pi_k(\theta_k|D)$ is the posterior for $\mathcal{M}_k$. BMA aggregates these to produce a mixture predictive,

$$p(y|x, D) = \sum_{k=1}^K p(\mathcal{M}_k|D)\, p_{\mathcal{M}_k}(y|x, D),$$

where $p(\mathcal{M}_k|D)$ is the posterior probability for $\mathcal{M}_k$, calculated via the marginal likelihood

$$p(\mathcal{M}_k|D) = \frac{m(D|\mathcal{M}_k)\,p(\mathcal{M}_k)}{\sum_{j=1}^K m(D|\mathcal{M}_j)\,p(\mathcal{M}_j)},$$

with $$m(D|\mathcal{M}_k) = \int \prod_{i=1}^n p_{\theta_k}(y_i|x_i)\,\pi_k(\theta_k)\,d\theta_k$$.

This weighted post-data model ensemble characterizes epistemic uncertainty and allows inference or prediction that incorporates the possibility of model misspecification. When BMA is used in predictive construction (such as in conformal inference), the integrated predictive reflects the combined uncertainty across all considered model structures (Bhagwat et al., 21 Nov 2025).

2. Integration with Conformal Prediction: The CBMA Methodology

The CBMA method merges BMA’s adaptability with conformal prediction’s marginal coverage guarantees. CBMA does not commit to a single best model; instead, it uses the hierarchical Bayesian mixture predictive as the nonconformity—or, more precisely, “conformity”—score within a permutation-invariant conformal algorithm.

Given training pairs $D = \{(x_i, y_i)\}_{i=1}^n$ and a new query $x_{n+1}$, for a candidate label $y$, the CBMA conformity score for each $i=1,\dots,n+1$ (after appending $(x_{n+1}, y)$ to $D$) is

$$\sigma_i^{\mathrm{CBMA}} = \sum_{k=1}^K p(\mathcal{M}_k|D \cup \{(x_{n+1}, y)\}) \cdot p_{\mathcal{M}_k}(Y_i | X_i, D \cup \{(x_{n+1}, y)\}).$$

The conformal $p$-value is then computed by ranking: $$r(y) = \frac{1}{n+1} \sum_{i=1}^{n+1} \mathbf{1}\left\{ \sigma_i^{\mathrm{CBMA}} \leq \sigma_{n+1}^{\mathrm{CBMA}} \right\},$$ and the $(1-\alpha)$ prediction set is

$C_{1-\alpha}(x_{n+1}) = \{ y : r(y) > \alpha \}.$

Permutation-invariance of the conformity scores ensures valid finite-sample coverage.

3. Algorithmic Implementation and Computational Strategy

CBMA is implemented algorithmically in three primary steps:

1. Posterior Quantities: For each $\mathcal{M}_k$, draw posterior samples $\{\theta_k^{(t)}\}_{t=1}^T$ from $\pi_k(\theta_k|D)$, compute estimated model weights $\hat{p}(\mathcal{M}_k|D)$.
2. Conformity Calculation for Prediction: For a grid of candidate $y$, for each $k$, compute add-one-in importance weights $w_k^{(t)} \sim p_{\theta_k^{(t)}}(y|x_{n+1})$, then compute

   $$\hat{\sigma}_i^{\mathcal{M}_k} = \sum_{t=1}^T \tilde{w}_k^{(t)}\, p_{\theta_k^{(t)}}(y_i | x_i),$$

   aggregate as

   $$\hat{\sigma}_i^{\mathrm{CBMA}} = \sum_{k=1}^K q_k\,\hat{\sigma}_i^{\mathcal{M}_k}$$

   where $$q_k \propto \hat{p}(\mathcal{M}_k|D)\, \hat{p}_{\mathcal{M}_k}(y|x_{n+1}, D)$$.

3. Set Construction: Compute ranks as above, and set $C_{1-\alpha}(x_{n+1}) = \{ y : r(y) > \alpha \}$.

This procedure is computationally efficient; the overhead of BMA (model averaging, multiple posterior draws) is dominated by the MCMC or sampling cost for individual models (Bhagwat et al., 21 Nov 2025).

4. Theoretical Properties: Validity and Efficiency

CBMA inherits several key theoretical properties:

• Marginal Coverage: For all distributions (so long as the hierarchical model assumptions are satisfied) and any finite $n$, CBMA sets achieve at least their nominal $1-\alpha$ marginal coverage, with at most $(1/(n+1))$ conservative bias due to possible ties in the conformity scores.
• Convergence and Optimality: If the true data-generating model is in the candidate set $\{\mathcal{M}_k\}$, as $n\to\infty$, the aggregated model weights $q_k$ concentrate on the true model $k^*$ and CBMA thus converges to the “full conformal Bayes” (oracle) efficiency—i.e., the minimal expected volume prediction set among all valid conformal sets at level $1-\alpha$. If the true model is not present, CBMA maintains valid coverage but may not be globally efficient, highlighting the value of model expansion (Bhagwat et al., 21 Nov 2025).
• Assumptions: Consistency and efficiency arguments rely on standard regularity conditions (posterior concentration, identifiability, continuous prior, Fisher information positive-definiteness, etc.).

5. Empirical Performance and Applications

CBMA’s empirical properties have been validated in multiple regimes:

• Simulated Regression (Well-specified): In a quadratic regression setting with quadratic, linear, and intercept-only models, CBMA achieves oracle (minimal) interval length matching the true model’s conformal Bayes set, preserving nominal coverage even in moderate sample sizes (e.g., $n=100,200$).
• Misspecified/Heteroskedastic Regimes: In a heteroskedastic Hermite-polynomial regression where the true model is outside the candidate list, CBMA maintains valid (e.g., $80\%$) coverage and yields substantially tighter average interval length than single-model conformal Bayes across all candidates.
• Real Data (California Housing): On low-dimensional real data (predicting house prices from two covariates at $n=50,100,150$), CBMA attains nominal coverage and the shortest intervals among all candidate conformal-Bayes or Bayes-alone sets, with essentially negligible additional computational burden over standard BMA.

These findings indicate that CBMA robustly adapts to model uncertainty across diverse settings. The approach balances the finite-sample robustness of conformal prediction with the asymptotic efficiency of Bayesian model averaging, providing reliable and sharp predictive sets (Bhagwat et al., 21 Nov 2025).

6. Broader Significance and Directions

The Bayesian averaging approach, exemplified by CBMA in predictive inference, offers a robust framework for uncertainty quantification. Its strengths are:

• Model Robustness: By aggregating across models, BMA mitigates the adverse effects of misspecification, avoiding overconfidence.
• Frequentist Validity: The integration with conformal prediction ensures valid finite-sample guarantees even when the Bayesian model is incorrect.
• Adaptivity: If model selection is successful asymptotically, BMA-based prediction, estimation, and decision procedures inherit oracle (optimal) properties.
• Feasibility: The marginal cost of BMA in this context is negligible compared to the dominant computational costs (e.g., MCMC), making this approach practically accessible for a wide range of applications.

This approach is especially relevant in scientific and technical domains where model selection uncertainty is inescapable and valid uncertainty quantification is demanded. It exemplifies the ongoing synthesis of Bayesian and frequentist reasoning at the frontier of machine learning and statistical inference (Bhagwat et al., 21 Nov 2025).