A Survey of Uncertainty in Deep Neural Networks (2107.03342v3)

Abstract: Due to their increasing spread, confidence in neural network predictions became more and more important. However, basic neural networks do not deliver certainty estimates or suffer from over or under confidence. Many researchers have been working on understanding and quantifying uncertainty in a neural network's prediction. As a result, different types and sources of uncertainty have been identified and a variety of approaches to measure and quantify uncertainty in neural networks have been proposed. This work gives a comprehensive overview of uncertainty estimation in neural networks, reviews recent advances in the field, highlights current challenges, and identifies potential research opportunities. It is intended to give anyone interested in uncertainty estimation in neural networks a broad overview and introduction, without presupposing prior knowledge in this field. A comprehensive introduction to the most crucial sources of uncertainty is given and their separation into reducible model uncertainty and not reducible data uncertainty is presented. The modeling of these uncertainties based on deterministic neural networks, Bayesian neural networks, ensemble of neural networks, and test-time data augmentation approaches is introduced and different branches of these fields as well as the latest developments are discussed. For a practical application, we discuss different measures of uncertainty, approaches for the calibration of neural networks and give an overview of existing baselines and implementations. Different examples from the wide spectrum of challenges in different fields give an idea of the needs and challenges regarding uncertainties in practical applications. Additionally, the practical limitations of current methods for mission- and safety-critical real world applications are discussed and an outlook on the next steps towards a broader usage of such methods is given.

A Survey of Uncertainty in Deep Neural Networks

The paper "A Survey of Uncertainty in Deep Neural Networks" presents an extensive overview of the methods for estimating and quantifying uncertainty in neural network predictions. This work is particularly valuable given the critical role that uncertainty estimation plays in high-stakes applications such as medical imaging, autonomous driving, and earth observation.

The central concern addressed by the paper is the inability of standard deep neural networks (DNNs) to provide reliable uncertainty estimates. This limitation reduces the trustworthiness of DNNs in applications where the cost of errors is high. The paper categorizes the sources of uncertainty into reducible model uncertainty and irreducible data uncertainty, explaining their origins and effects on neural network predictions.

The authors present a detailed taxonomy of methods for uncertainty estimation in DNNs. These methods are classified into four primary categories:

1. Single deterministic methods
2. Bayesian methods
3. Ensemble methods
4. Test-time data augmentation methods

Single Deterministic Methods

Single deterministic methods estimate uncertainty using a single network evaluation, often leveraging internal or external mechanisms. Internal methods, such as Evidential Neural Networks, predict parameters for a distribution over outputs, enabling uncertainty quantification. External methods use additional models or tools to estimate uncertainty after the primary prediction. These methods are computationally efficient and can be applied to pre-trained networks but often lack the robustness provided by stochastic or multiple-model approaches.

Bayesian Methods

Bayesian neural networks (BNNs) provide a probabilistic approach to uncertainty estimation by modeling network parameters as distributions rather than fixed values. There are three notable approaches within BNNs:

• Variational Inference: Approximates the posterior distribution by optimizing within a tractable family of distributions.
• Sampling Methods: Use techniques like Markov Chain Monte Carlo (MCMC) to generate samples from the target distribution.
• Laplace Approximation: Simplifies the posterior by approximating it around a local mode with a Gaussian distribution.

Bayesian methods typically offer sound theoretical grounding and effective modeling of model uncertainty but come with significant computational overhead.

Ensemble Methods

Ensemble methods enhance model robustness by combining the predictions of multiple independently trained models. Varieties among ensemble members are introduced through different initialization, data shuffling, or architectural choices. While ensembles improve both prediction accuracy and uncertainty estimation, they require substantial computational and memory resources.

Test-Time Data Augmentation Methods

In test-time augmentation, several augmented versions of each input sample are evaluated, and the resulting predictions are used to estimate uncertainty. This method is simple to implement, as it does not necessitate changes to the original model, but it involves substantial computational costs due to the multiple evaluations required per input sample.

Practical Implications and Future Developments

The practical implications of effective uncertainty estimation are extensive. Accurate uncertainty measures enable better risk management in high-stakes applications by providing confidence levels for predictions. For active learning frameworks, uncertainty estimates guide the selection of informative samples, which can significantly reduce labeling costs. In reinforcement learning, these estimates can balance exploration and exploitation, improving learning efficiency.

From a theoretical perspective, further research could investigate the integration of domain-specific knowledge with deep learning models to enhance uncertainty predictions. Additionally, developing standardized protocols and benchmarks for evaluating uncertainty estimation methods is crucial for their broader adoption and comparison across different domains.

Conclusion

The paper meticulously surveys existing methodologies for uncertainty quantification in DNNs, delineating their strengths and limitations. By categorizing the methods and providing a comparative analysis, it offers a valuable resource for both researchers and practitioners aiming to enhance the reliability and robustness of deep learning applications. The authors also highlight several areas requiring further research, paving the way for future advancements in this vital area of AI.