Credal Model Averaging (CMA)

• Credal Model Averaging (CMA) is a generalization of Bayesian Model Averaging that uses a set of priors to capture epistemic uncertainty and produce interval-valued predictions.
• It has been applied in statistical model selection, ensemble methods like SPODE/AODE, and deep learning frameworks to enhance decision robustness under uncertain conditions.
• CMA integrates rigorous sensitivity analysis with practical performance metrics such as indeterminacy rates and improved calibration, offering a principled approach for risk-averse predictions.

Credal Model Averaging (CMA) is a generalization of Bayesian Model Averaging (BMA) that replaces the single prior over model space with a set of priors (a "credal set") to rigorously address prior uncertainty and propagate epistemic uncertainty through to model predictions. CMA has been applied in classical statistical model selection, ensemble methods such as SPODEs, and deep learning architectures including Bayesian neural networks and deep ensembles. Its main objectives are reliable classification under model uncertainty and explicit quantification of the sensitivity of predictions to prior assumptions (Corani et al., 2014, Corani et al., 2012, Wang et al., 2024).

1. Formalization of Credal Model Averaging

Let $\mathcal{M} = \{m_1, ..., m_M\}$ denote a finite model space. Given data $D$, Bayesian Model Averaging predicts the probability of a class label $y$ as: $$P(y|x, D) = \sum_{m \in \mathcal{M}} P(y|x, m, D) P(m|D),$$ where $P(m|D) \propto P(m) P(D|m)$. $P(m)$ is the prior over models; $P(D|m)$ is the model marginal likelihood or Bayesian Evidence.

CMA replaces $P(m)$ with a credal set $\mathcal{K}$: $$\mathcal{K} = \{p(m): p(m) \ge 0,\, \sum_{m \in \mathcal{M}} p(m) = 1,\, p(m) \text{ satisfies bounds}\}.$$ Predictions become interval-valued: $$P_*(y|x, D) = \min_{p \in \mathcal{K}} \sum_{m} P(y|x,m,D)\,p(m|D)$$

$$P^*(y|x, D) = \max_{p \in \mathcal{K}} \sum_{m} P(y|x,m,D)\,p(m|D)$$

where $p(m|D) \propto p(m) P(D|m)$. The credible set of priors provides automatic sensitivity analysis and formalizes both prior-ignorance and domain-informed prior elicitation (Corani et al., 2014).

2. CMA Design in Classical and Ensemble Methods

CMA algorithms have been developed for various model classes, including logistic regression ensembles and SPODE-based ensembles.

Logistic Regression Ensembles (Corani et al., 2014)

• CMA$_{ib}$ (Independent-Bernoulli): The prior on model size parameter $\theta \in [\underline\theta,\bar\theta]$ specifies $$p_\theta(m) \propto \theta^{k_m} (1-\theta)^{k-k_m}$$, where $k_m$ is the number of covariates in model $m$. Near-ignorance is encoded with $\theta \in [\varepsilon, 1-\varepsilon]$.
• CMA$_{nb}$ (Non-identical Bernoulli): Each covariate $j$ has its own inclusion probability interval $[\underline\theta_j, \bar\theta_j]$, such that $$p_\theta(m) \propto \prod_{j\in m} \theta_j \prod_{j\notin m}(1-\theta_j)$$. Expert opinion is incorporated by narrowing these intervals per covariate.

SPODE/AODE Ensembles (Corani et al., 2012)

• In the SPODE context, class-posterior aggregation using BMA is dominated by a single model, degrading performance. To overcome this, compression-based averaging applies a logarithmic smoothing to the posteriors. CMA replaces the unique model prior with credal sets:

$$\mathcal{P}(S) = \{P(s_j) \geq \varepsilon\, \forall j, \sum_j P(s_j)=1\}$$

yielding interval-valued class posteriors and robust (maximality/interval-dominance based) decisions.

Deep Ensemble Credal Wrappers (Wang et al., 2024)

• Given $K$ decision-distribution outputs $\{p_1(y|x), ..., p_K(y|x)\}$ (e.g., deep ensemble predictions), the class interval is

$$\underline P(y=c|x) = \min_k p_k(y=c|x), \quad \overline P(y=c|x) = \max_k p_k(y=c|x).$$

The credal set is the convex polytope of all distributions within $[\underline P,\,\overline P]$ for each class.

3. Decision Rules and Prior-Dependent Instances

CMA supports interval-valued predictions and robust decisions.

• Interval-dominance rule (binary case): For classes $c_0$, $c_1$, predict $c_1$ if $\underline P(c_1|x,D)>1/2$, $c_0$ if $\overline P(c_1|x,D)<1/2$, otherwise remain indeterminate ($\{c_0,c_1\}$).
• Prior-dependent instance: Any $x$ for which $\arg\max_y P_*(y|x,D)\neq \arg\max_y P^*(y|x,D)$. In such cases, CMA returns the set of plausible classes, thereby suspending judgment where model-prior arbitrariness would produce brittle results (Corani et al., 2014, Corani et al., 2012).

In multiclass problems (e.g., SPODE/AODE), maximality rules and interval-dominance are used to define non-dominated class sets based on lower/upper bounds optimized over the credal set.

• Intersection-probability transformation in deep ensemble settings selects a single normalized distribution $P^*(y|x)$ lying on the line between the lower and upper corners of the credal polytope:

$$P^*(y=c|x) = \underline P(y=c|x) + \alpha(\overline P(y=c|x)-\underline P(y=c|x)),$$

where $\alpha$ normalizes the sum to one.

4. Elicitation of Prior Ignorance and Expert Knowledge

CMA schemes vary the width of the credal set to encode different sources of prior information.

• Prior-ignorance: In CMA$_{ib}$, using the widest feasible interval for $\theta$ ($[\varepsilon,1-\varepsilon]$) encodes minimal prior knowledge.
• Expert knowledge: In CMA$_{nb}$, expert-elicited intervals $[\underline\theta_j,\overline\theta_j]$ for each covariate sharply limit the credal set and enable more informative and less conservative inference.
• In ensemble contexts, the prior credal set can be defined by lower bounds $\varepsilon$ on each model or covariate inclusion probability (Corani et al., 2014, Corani et al., 2012).

A relevant implication is that the trade-off between robustness (wider credal sets) and informativeness (narrower credal sets) is tunable, and empirical results show that expert-informed priors yield sharper inferences for covariates strongly supported by data (Corani et al., 2014).

5. Reliability, Utility, and Performance Metrics

CMA metrics extend conventional ML evaluation to account for interval-valued and set-valued outputs (Corani et al., 2014, Corani et al., 2012, Wang et al., 2024):

• Indeterminacy rate: Fraction of instances for which CMA prediction is set-valued (i.e., prior-dependent).
• Accuracy on determinate instances: Performance where CMA gives a single class.
• Set-accuracy: Proportion of times the returned set of classes contains the true label.
• Quadratic utility ($u_{65}$, $u_{80}$): Rewards correct set-valued predictions based on their cardinality.
• AUC, recall, Brier loss: Determinate metrics for compatibility with BMA and standard classifiers.
• Expected Calibration Error (ECE), AUROC, AUPRC: Used especially in deep ensemble applications to measure calibration and OOD detection ability.

Key empirical findings:

• CMA's indeterminacy rate is $5$–$8$% for independent-Bernoulli priors, up to $20$% for non-identical Bernoulli priors; it decreases with sample size.
• On prior-dependent instances, BMA's accuracy degrades to random-guessing levels (≈$60$%), whereas CMA retains high reliability by abstaining or returning sets.
• Under conservative utility ($u_{65}$), CMA always outperforms BMA in risk-averse scenarios; under $u_{80}$, the most imprecise (broadest) CMA configuration is optimal when data is scarce.
• In deep learning, the credal wrapper improves EU-based AUROC for OOD detection by $2$–$4$ points and reduces ECE by $20$–$30$% compared to vanilla ensemble/BMA softmax averaging (Wang et al., 2024).

6. Algorithmic Procedure and Computational Considerations

In classical CMA for model ensembles:

• Posterior intervals for classes or models are computed by optimizing linear-fractional (BMA) or non-linear (compression-weighted) functions over the credal set, with constraints given by minimum prior weights.
• For logistic regression, independent-Bernoulli CMA reduces intervals to 1D optimizations, while non-identical Bernoulli CMA requires multi-dimensional (but tractable) optimization, commonly solved via NLopt (Corani et al., 2014).
• In credal-ensemble meta-algorithms (SPODEs), BMA-based CMA and compression-based CMA only add a classification-time overhead (for small optimization subproblems); overall computational complexity remains polynomial in the number of models and classes (Corani et al., 2012).

In deep learning, the credal wrapper calculation is efficient:

• Lower and upper class bounds are computed via $K$-wise minima/maxima.
• The intersection-probability transformation is closed-form and efficient for inference, although some measures of generalized uncertainty require more computational resources (Wang et al., 2024).

7. Applications, Extensions, and Theoretical Properties

CMA and its extensions have been deployed in diverse settings:

• Ecological inference: Predicting Alpine marmot burrow presence in a heterogeneous landscape (Corani et al., 2014).
• Tabular datasets: 40 UCI-like datasets, demonstrating reliability and calibration improvement over standard AODE, BMA, and COMP-AODE (Corani et al., 2012).
• Deep learning OOD detection: Tasks on CIFAR, ImageNet, and variants showing improved calibration and uncertainty quantification (Wang et al., 2024).

Theoretical properties:

• Reliability: CMA maintains high set-accuracy and reliability on prior-dependent instances, never making confident errors.
• Sensitivity analysis: CMA automates and formalizes the quantification of prior-sensitivity in model-based prediction.
• Complexity: Additional computational cost is confined to classification time but remains tractable for practical ensemble sizes and moderate feature/label cardinality.
• Convexity: The credal set forms a convex polytope in probability simplex, supporting principled operations such as the intersection-probability transform.

A plausible implication is that as high-dimensional and ensemble-based models become more prevalent, credal methods offer a principled framework for epistemic uncertainty management, with potential for seamless integration into risk-averse decision-making and active learning. Proposed future work includes efficient entropy approximation on credal polytopes, extending CMA to regression and structured outputs, and leveraging interval widths for active data acquisition (Wang et al., 2024).