- The paper shows that neither representation is universally superior: Wasserstein distance leads selective prediction, while Generalized Hartley dominates OOD detection, with results varying by task and benchmark.
- The study derives both representations from identical neural-network predictions across six model families and eight benchmarks, isolating the effects of uncertainty representation through repeated trials and Wilcoxon tests.
- The findings show that metric design matters substantially: entropy-difference measures often underperform, OOD detection reveals differences more clearly than selective prediction, and computational cost remains a trade-off for credal measures.
Motivation and problem statement
Epistemic uncertainty (EU) in neural networks is conventionally formalized through two second-order paradigms: distribution-based representations, exemplified by Bayesian neural networks (BNNs) and deep ensembles (DE), which place a probability distribution over predictive distributions; and set-based representations, exemplified by credal sets, which encode EU as convex sets of plausible probability distributions. These frameworks are widely regarded as non-comparable because they differ in semantics, mathematical structure, and evaluation practice. Moreover, existing empirical comparisons are confounded: predictors are typically derived from different learning algorithms, architectures, or training procedures, so observed performance differences cannot be attributed to the uncertainty representation itself rather than to predictive accuracy or optimization effects.
The paper under review addresses this gap with a controlled comparative study designed to isolate representational effects. Both representations are constructed from the same finite collection of M predictive distributions produced by a shared neural network: the distribution-based representation is a uniform distribution over this collection B={pm​}m=1M​, while the credal representation is derived from identical predictions via class-wise probability intervals (2602.22747). This design enables a like-for-like evaluation that prior work—restricted to within-paradigm comparisons or single metrics and tasks—does not provide.
Experimental design
The study spans six underlying predictive model families: stochastic variational inference (SVI), Monte Carlo Dropout (MCDO), deep ensembles, and three computationally efficient ensemble variants (BatchEns, MaskEns, PackEns), all using ResNet backbones. Three uncertainty measures are evaluated per representation:
- Distribution-based: mutual information (MI) via entropy decomposition, label-wise variance (LWV), and Wasserstein distance (WD).
- Credal-based: entropy difference (Hdiff​) between upper and lower entropies over the credal set, the Generalized Hartley (GH) measure computed via Möbius inversion of lower probabilities, and maximum mean imprecision (MMI).
Evaluation covers eight benchmarks across two downstream tasks—selective prediction (AUARC) and out-of-distribution (OOD) detection (AUROC)—on CIFAR10 (with SVHN and FMNIST as OOD sets), Camelyon17 histopathology data with scanner-induced domain shift, and a ship classification dataset (SeaShip, with corruption-based SeaShip-C and externally sourced SeaShip-O shifts). Each configuration is run 10 times independently, and pairwise one-sided Wilcoxon signed-rank tests at the 5% level yield net-win rankings for both intra- and inter-representation comparisons.
Key findings
No representation is uniformly superior; conclusions depend on the representation–measure interaction. On selective prediction, WD paired with the distribution-based representation ranks first by net wins on every benchmark (e.g., +21 to +30 inter-representation net wins versus −18 to −30 for MI and Hdiff​). On OOD detection, GH paired with the credal representation dominates (net wins up to +30, e.g., CIFAR10 vs. SVHN AUROC 0.8303 vs. 0.7657 for MI). The paper's central implication is that claims about uncertainty representations must be stated conditionally on the measure, task, model, and dataset.
OOD detection is more discriminative than selective prediction. Although Wilcoxon tests detect significant differences on selective prediction, average AUARC gaps remain small (e.g., 0.9633 for WD vs. 0.9594 for Hdiff​ on in-distribution Camelyon17). The authors attribute this to the evaluation protocol: distinct uncertainty estimates often induce similar rejection sets, and high baseline accuracy compresses observable gains. OOD detection, which depends directly on how uncertainty responds to distributional shift, exposes representational differences more clearly—a practical caution for papers validating new measures on selective prediction alone.
Within each representation, metric choice matters substantially. WD consistently outperforms MI and LWV, plausibly because it captures geometric dispersion around the barycenter on the simplex, whereas MI depends on global entropy shape and LWV ignores simplex geometry. Among credal measures, GH dominates while the popular Hdiff​ performs worst nearly everywhere—an effect the authors trace to its dependence solely on entropy-interval width: in a binary simplex, any credal set containing the center and a vertex yields identical Hdiff​ regardless of set shape. This is not an optimization artifact, since closed-form solutions exist in the binary case. The trade-off is computational: GH requires enumeration over 2K subsets, motivating MMI and its linear-time upper bound as alternatives.
Limitations
The authors are explicit about scope restrictions. Dirichlet-based models and random-set representations are excluded because they lack a general, representation-agnostic construction from arbitrary EU-aware predictors, precluding controlled comparison under a shared protocol. Similarly, only probability-interval credal sets induced from finite predictive samples are considered, excluding alternative credal constructions. The evaluation is also constrained computationally, and extensions to additional measures, datasets, predictors, and settings such as active learning remain open. A further implicit limitation is that the credal representation is itself a transformation of the same finite sample used by the distribution-based representation, so conclusions concern these particular induced formalisms rather than credal sets constructed by dedicated credal learning methods.
Conclusion
This study demonstrates that principled comparison between distribution-based and set-based second-order uncertainty representations is feasible when both are derived from a shared predictor, and that such comparison yields actionable conclusions: performance is jointly determined by representation and uncertainty measure, OOD detection is the more sensitive evaluation regime, and widely used measures such as Hdiff​ can materially underperform within their own paradigm. The results argue for conditional reporting of uncertainty evaluations—specifying measure, benchmark, model, and dataset—and for treating measure design as co-equal to representation design in advancing uncertainty-aware learning.