Dirichlet-Based Monte Carlo Dropout for Uncertainty Estimation in Neural Networks

Abstract: Traditional neural networks provide deterministic predictions without inherent uncertainty estimates. While Bayesian Neural Networks (BNNs) offer a principled approach to uncertainty quantification, their computational complexity limits scalability. Monte Carlo (MC) Dropout, initially introduced as a regularization technique, has been shown to approximate Bayesian inference by enabling probabilistic modeling through multiple stochastic forward passes. In this work, we enhance uncertainty estimation in deep learning by integrating a Dirichlet-based framework within MC Dropout. Specifically, we leverage the formulation proposed by Sensoy et al. (2018), where class probabilities are modeled using a Dirichlet distribution, allowing for a more informative uncertainty representation. The proposed approach maintains the computational efficiency of MC Dropout while improving the quality of uncertainty estimates. We discuss the theoretical foundations of our method and compare it with existing uncertainty quantification techniques. The results highlight the effectiveness of the proposed method in producing well-calibrated uncertainty estimates, offering a practical solution for uncertainty-aware deep learning models.

• The paper introduces DBUE-Dropout, a novel integration of Monte Carlo Dropout with Dirichlet parameterization to achieve robust uncertainty quantification.
• The paper empirically estimates Dirichlet parameters from MC Dropout outputs, ensuring well-calibrated predictions for noisy and out-of-distribution data.
• The paper demonstrates superior performance through high accuracy on benchmark datasets and reliable uncertainty detection under distribution shifts.

Dirichlet-Based MC Dropout for Robust Uncertainty Estimation in Neural Networks

Motivation and Theoretical Foundations

Uncertainty estimation remains an essential aspect of deploying neural networks in safety-critical domains and scenarios with high distributional shift. Standard deterministic NNs lack explicit uncertainty quantification, while BNNs offer rigorous UQ at the cost of prohibitive computational demands. MC Dropout provides a scalable Bayesian approximation, yet classical MC Dropout only estimates uncertainty via standard deviation—a limited statistical proxy for full predictive distribution shape.

This paper proposes an integrated framework, Dirichlet-Based Uncertainty Estimation-MC Dropout (DBUE-Dropout), that leverages MC Dropout's computational efficiency but augments its UQ capabilities by parameterizing class probabilities via a Dirichlet distribution. The approach builds on the evidential formulation of Sensoy et al. (2018), using evidence-based Dirichlet parameterization, but introduces a novel empirical method to estimate Dirichlet parameters from MC Dropout outputs.

Given $L$ MC Dropout forward passes per input, the empirical mean and variance of class probabilities are directly computed. The Dirichlet concentration parameter $S$ is then defined as $$S = \frac{\mathbb{E}[p_k](1-\mathbb{E}[p_k])}{\text{Var}(p_k)} - 1$$, leveraging the variance formula for Dirichlet distributions. Uncertainty is quantified as $u = \frac{K}{S}$, reinforcing the theoretical link between evidence accumulation and UQ: high concentration (high $S$) corresponds to strong evidence and low uncertainty.

Methodology

The DBUE-Dropout algorithm requires a trained NN, dropout enabled at inference, and repeated stochastic forward passes. For each sample, the predicted class is determined via averaged MC predictions. The empirical mean and variance of predicted probabilities for that class are used to estimate Dirichlet distribution parameters, from which uncertainty is computed.

Unlike the DEC and IAD approaches, which map NN outputs to evidence with ReLU or Softplus, respectively, DBUE-Dropout derives evidence from MC Dropout statistics, providing rigorous Bayesian grounding and practical scalability. The empirical Dirichlet parameterization is justified by the Law of Large Numbers, ensuring robust estimation via sufficient MC samples.

Empirical Evaluation and Results

DBUE-Dropout is evaluated on tabular (Titanic) and image (MNIST) datasets under conditions of data shift, including noisy inputs and OOD samples. On the Titanic dataset, DBUE-Dropout yields the highest train accuracy (89.6%) among compared methods and demonstrates superior OOD uncertainty detection. Unlike DEC and IAD, DBUE-Dropout increases uncertainty for OOD cases as accuracy drops, accurately reflecting epistemic uncertainty and model confidence. DEC and IAD fail in OOD detection: their uncertainty measures decrease despite lower accuracy, implying unwarranted confidence on anomalous inputs.

On MNIST, the method achieves high training (99.95%) and test accuracy (99.25%), while showing sharply reduced accuracy (9.74%) for noisy inputs and (6.78%) for OOD cases. Uncertainty median rises to 0.53 (noisy) and 19.19 (OOD), properly correlating model calibration to distributional mismatch.

The rotational analysis of digit images further substantiates the method's capacity to track uncertainty. For rotation-invariant digits (e.g., 0), uncertainty remains low throughout, while ambiguity induced by rotation on other digits produces spikes in uncertainty—especially at visually indeterminate orientations.

Figure 1: Rotated images with corresponding uncertainty values, highlighting sensitivity to orientation and evidential ambiguity.

Practical and Theoretical Implications

The DBUE-Dropout algorithm presents both practical and theoretical advancements. Practically, it delivers well-calibrated UQ at the computational cost of MC Dropout—a scalable solution easily integrated into deep learning pipelines. Unlike ensemble-based methods, no additional models are required. Unlike evidential-only methods, DBUE-Dropout maintains Bayesian coherence and responds reliably to OOD and noisy data, robustly flagging incorrect predictions with high uncertainty.

Theoretically, the work establishes a bridge between Bayesian MC Dropout and Dirichlet evidential frameworks. It exploits LLN-driven empirical moments to robustly estimate predictive distributions while retaining connections to Bayesian principles and subjective logic in multiclass settings.

Future Directions and Speculation

Potential extensions include adaptation to imbalanced data regimes, optimization for robustness under heavy noise, and integration with causal inference methods for deeper uncertainty interpretability. Application to sequential models (RNNs, transformers) and multimodal domains promises broad impact. As deep evidential and Bayesian estimation methods converge, further theoretical investigation could refine concentration parameter estimation, leading to universally better-calibrated models.

Conclusion

The Dirichlet-Based MC Dropout methodology advances uncertainty quantification for deep learning classification. By empirically estimating Dirichlet parameters from MC Dropout outputs, the approach delivers well-calibrated, theoretically sound UQ, reliably identifying OOD and noisy scenarios. Comparative results accentuate its practical superiority and foundational rigor, suggesting broad applicability for uncertainty-aware AI systems.