Empirical evaluation of uncertainty quantification methods in supervised learning

Develop empirical evaluation methodologies for assessing methods that quantify aleatoric, epistemic, and total predictive uncertainty in supervised learning, despite the absence of ground-truth uncertainty labels in typical datasets.

Background

The paper notes that, unlike target variables, datasets typically do not provide ground-truth uncertainty labels, which makes it difficult to directly assess the correctness of predicted aleatoric, epistemic, or total uncertainty. Consequently, most current evaluations are indirect, for example by measuring the usefulness of uncertainty estimates for improved prediction and decision-making (e.g., through set-valued prediction).

The authors mention accuracy–rejection curves as one such indirect assessment, where a classifier’s accuracy is plotted against the fraction of abstentions. However, they highlight that a principled, general methodology for empirical evaluation of uncertainty quantification methods remains to be established.

References

In addition to theoretical problems of this kind, there are also many open practical questions. This includes, for example, the question of how to perform an empirical evaluation {empirical evaluation} of methods for quantifying uncertainty, whether aleatoric, epistemic, or total.

— Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods  (1910.09457 - Hüllermeier et al., 2019) in Discussion and conclusion (Section 5)

Rather, the open issue is whether these methods can be integrated into a more general, process-aware but result-centered framework, analogous to the one developed in measurement science for the modeling, evaluation, and expression of uncertainty.

— Uncertainty Evaluation: Bridging Machine Learning and Measurement Science  (2610.18066 - Petri et al., 13 Oct 2026) in Section V, page 5

A limitation of using classifier model uncertainty for inspection is, the difficulty in determining whether high uncertainty reflects genuine ambiguity in the image itself or is instead an consequence of the model's architecture and/or training limitations.

— Illumination-Aware Active Perception for Spacecraft Inspection  (2610.22335 - Valluri et al., 15 Oct 2026) in Section Conclusion and Future Work