Confidence-Aware Learning for Deep Neural Networks (2007.01458v3)

Abstract: Despite the power of deep neural networks for a wide range of tasks, an overconfident prediction issue has limited their practical use in many safety-critical applications. Many recent works have been proposed to mitigate this issue, but most of them require either additional computational costs in training and/or inference phases or customized architectures to output confidence estimates separately. In this paper, we propose a method of training deep neural networks with a novel loss function, named Correctness Ranking Loss, which regularizes class probabilities explicitly to be better confidence estimates in terms of ordinal ranking according to confidence. The proposed method is easy to implement and can be applied to the existing architectures without any modification. Also, it has almost the same computational costs for training as conventional deep classifiers and outputs reliable predictions by a single inference. Extensive experimental results on classification benchmark datasets indicate that the proposed method helps networks to produce well-ranked confidence estimates. We also demonstrate that it is effective for the tasks closely related to confidence estimation, out-of-distribution detection and active learning.

• The paper introduces a Correctness Ranking Loss (CRL) that orders class probabilities to produce robust confidence estimates in deep neural networks.
• It enhances model calibration by reducing expected calibration error and negative log likelihood without increasing computational costs.
• CRL-trained models outperform baselines in out-of-distribution detection and active learning, demonstrating practical viability for safety-critical applications.

Confidence-Aware Learning for Deep Neural Networks

This paper presents a novel approach aimed at mitigating the overconfidence problem in deep neural network (DNN) predictions, which poses significant challenges to their applicability in safety-critical domains. The authors introduce a method involving Correctness Ranking Loss (CRL), a new loss function designed to improve the quality of confidence estimates produced by DNNs. This loss function works by explicitly ordering class probabilities to be utilized as robust confidence measures. The CRL is proposed as a straightforward enhancement to standard DNN architectures that does not necessitate architectural modifications or substantially increased training and inference costs.

The authors emphasize the limitations of existing approaches, which often involve additional computational overhead or require specialized network architectures. The CRL, on the other hand, is easily integrable into any standard architecture and maintains the same computational requirements as traditional deep learning classifiers. Through a series of experiments on benchmark classification datasets (such as CIFAR-10, CIFAR-100, and SVHN), the authors show that CRL effectively improves the calibration of network confidences and enhances the reliability of DNNs for particular tasks beyond classification accuracy, including out-of-distribution (OOD) detection and active learning.

Key highlights from the experimental results include:

• Effective Confidence Ranking: CRL achieves significantly better performance in terms of area under risk-coverage curve (AURC) when compared to baseline models and other contemporary methods like Monte Carlo dropout and average early stopping ensembles.
• Calibration Improvement: Utilizing CRL yields enhanced expected calibration error (ECE) and negative log likelihood (NLL), indicating more reliable predictive probabilities from the networks.
• OOD Detection: DNNs trained with CRL are demonstrated to perform superiorly in OOD detection tasks, even outperforming approaches such as the ODIN method, which typically requires additional processing steps for confidence estimation.
• Active Learning Efficacy: When applied to active learning scenarios, CRL-trained models require fewer labeled examples to achieve high predictive accuracy, outperforming traditional active learning query strategies.

These findings suggest significant implications for both the practical deployment of DNNs and theoretical understanding. The method addresses a crucial reliability factor by ensuring that DNNs "know when they don't know," thus extending their usability in fields such as autonomous driving, medical diagnostics, and any domain where prediction errors can have severe consequences.

In terms of future directions, the integration of CRL in other DNN-based applications such as NLP could reveal further insights and extend its utility. Additionally, exploring its robustness against adversarial attacks could prove valuable, considering the growing concern over adversarial vulnerability in modern neural networks. Overall, the CRL technique illuminates a critical aspect of DNN reliability, making it a pivotal topic for further research and development within the AI community.