A Survey on Uncertainty Quantification Methods for Deep Learning (2302.13425v5)

Abstract: Deep neural networks (DNNs) have achieved tremendous success in making accurate predictions for computer vision, natural language processing, as well as science and engineering domains. However, it is also well-recognized that DNNs sometimes make unexpected, incorrect, but overconfident predictions. This can cause serious consequences in high-stake applications, such as autonomous driving, medical diagnosis, and disaster response. Uncertainty quantification (UQ) aims to estimate the confidence of DNN predictions beyond prediction accuracy. In recent years, many UQ methods have been developed for DNNs. It is of great practical value to systematically categorize these UQ methods and compare their advantages and disadvantages. However, existing surveys mostly focus on categorizing UQ methodologies from a neural network architecture perspective or a Bayesian perspective and ignore the source of uncertainty that each methodology can incorporate, making it difficult to select an appropriate UQ method in practice. To fill the gap, this paper presents a systematic taxonomy of UQ methods for DNNs based on the types of uncertainty sources (data uncertainty versus model uncertainty). We summarize the advantages and disadvantages of methods in each category. We show how our taxonomy of UQ methodologies can potentially help guide the choice of UQ method in different machine learning problems (e.g., active learning, robustness, and reinforcement learning). We also identify current research gaps and propose several future research directions.

Uncertainty Quantification for Deep Learning: A New Perspective

The continuous advancement of deep neural networks (DNNs) has driven substantial breakthroughs in fields such as computer vision, natural language processing, and domains reliant on scientific data analysis. However, the efficacy of these models is frequently undermined by their occasional propensity for erroneous yet overconfident predictions, a challenge particularly pronounced in applications where decisions bear significant consequences, such as autonomous driving and medical diagnostics. Addressing this issue requires methodologies that go beyond merely enhancing prediction accuracy by also quantifying the uncertainty associated with these predictions.

Uncertainty Quantification Sources

This paper proposes a systematic taxonomy of uncertainty quantification (UQ) methods for DNNs, categorizing them based on the type of uncertainty they address: data uncertainty and model uncertainty.

Data uncertainty, also known as aleatory uncertainty, arises from intrinsic randomness or noise in the data that is generally irreducible. This can originate from sensor inaccuracies or overlapping features in different classes. For instance, in medical imaging, conflicting annotations can create data uncertainty. Conversely, model uncertainty, or epistemic uncertainty, is due to incomplete knowledge about model parameters, suboptimal architecture choices, or insufficient training data. It implies that model uncertainty can potentially be reduced with additional data.

Taxonomy of UQ Methods

The paper divides UQ methodologies into three major categories:

1. Model Uncertainty Approaches:
   • Bayesian Neural Networks (BNNs) approximate the posterior distribution of model parameters to capture uncertainty, using techniques like variational inference or Monte Carlo dropout.
   • Ensemble Models harness diversity through architectural variations or bootstrap aggregations, estimating uncertainty via prediction variance.
   • Sample Density-Aware Networks leverage Gaussian processes or distance-aware embeddings to address uncertainty arising from sparse data regions.
2. Data Uncertainty Approaches:
   • Deep Discriminative Models employ parameterized predictive distributions to estimate uncertainty. These approaches extend to both classification and regression tasks, using distributions such as Gaussian or mixture models.
   • Deep Generative Models, including VAE and GAN-based frameworks, capture structured uncertainty by modeling output distributions conditioned on input features.
3. Combining Data and Model Uncertainty:
   • Hybrid approaches combine elements from both data and model uncertainty methods, though they often carry increased computational demands.
   • Evidential Deep Learning introduces a more integrated framework that uses a single network to estimate both uncertainty types efficiently, predicting evidence parameters for Dirichlet distributions to model uncertainties.

Practical Implications and Future Directions

This categorization not only facilitates the selection of appropriate UQ methods for specific applications but also highlights gaps in current research. The detailed assessment offers insights essential for tackling high-stakes machine learning domains like medical diagnosis, geosciences, and autonomous systems, where the cost of error is high.

The paper underscores unexplored areas, such as combining uncertainty quantification with explainability, structured uncertainty quantification, and uncertainty in physics-aware neural networks. These areas present intriguing opportunities for enhancing the trustworthiness and robustness of AI models.

By offering a nuanced perspective, this paper serves as a roadmap for researchers, guiding future directions in AI that prioritize not just accuracy, but the reliability of AI-driven insights.