Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Preventing Arbitrarily High Confidence on Far-Away Data in Point-Estimated Discriminative Neural Networks (2311.03683v2)

Published 7 Nov 2023 in cs.LG

Abstract: Discriminatively trained, deterministic neural networks are the de facto choice for classification problems. However, even though they achieve state-of-the-art results on in-domain test sets, they tend to be overconfident on out-of-distribution (OOD) data. For instance, ReLU networks - a popular class of neural network architectures - have been shown to almost always yield high confidence predictions when the test data are far away from the training set, even when they are trained with OOD data. We overcome this problem by adding a term to the output of the neural network that corresponds to the logit of an extra class, that we design to dominate the logits of the original classes as we move away from the training data.This technique provably prevents arbitrarily high confidence on far-away test data while maintaining a simple discriminative point-estimate training. Evaluation on various benchmarks demonstrates strong performance against competitive baselines on both far-away and realistic OOD data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Understanding deep Neural Networks with rectified linear units. In International Conference on Learning Representations, 2018.
  2. Weight uncertainty in Neural Network. In International Conference on Machine Learning. PMLR, 2015.
  3. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718, 2018.
  4. Emnist: Extending MNIST to handwritten letters. In 2017 international joint conference on Neural Networks. IEEE, 2017.
  5. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning. PMLR, 2016.
  6. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
  7. On calibration of modern Neural Networks. In International Conference on Machine Learning. PMLR, 2017.
  8. Calibration of Neural Networks using Splines. In International Conference on Learning Representations, 2020.
  9. Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  10. Benchmarking Neural Network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2018.
  11. Deep anomaly detection with Outlier Exposure. In International Conference on Learning Representations, 2018.
  12. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013.
  13. Being Bayesian, even just a bit, fixes overconfidence in ReLU Networks. In International Conference on Machine Learning. PMLR, 2020.
  14. An infinite-feature extension for Bayesian ReLU nets that fixes their asymptotic overconfidence. Advances in Neural Information Processing Systems, 34, 2021.
  15. Posterior refinement improves sample efficiency in Bayesian Neural Networks. 2022a.
  16. Being a bit frequentist improves bayesian Neural Networks. In International Conference on Artificial Intelligence and Statistics. PMLR, 2022b.
  17. Krizhevsky, Alex et al. Learning multiple layers of features from tiny images. 2009.
  18. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. Advances in neural information processing systems, 32, 2019.
  19. Trainable calibration measures for Neural Networks from kernel mean embeddings. In International Conference on Machine Learning. PMLR, 2018.
  20. Simple and scalable predictive uncertainty estimation using Deep Ensembles. Advances in neural information processing systems, 30, 2017.
  21. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998.
  22. A simple unified framework for detecting out-of-distribution samples and Adversarial Attacks. Advances in Neural Information Processing Systems, 31, 2018.
  23. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, 2017.
  24. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33, 2020.
  25. Multiplicative normalizing flows for variational Bayesian Neural Networks. In International Conference on Machine Learning.
  26. Towards Neural Networks that provably know when they don’t know. In International Conference on Learning Representations, 2020.
  27. Revisiting the calibration of modern Neural Networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., and Vaughan, J. Wortman, editors, Advances in Neural Information Processing Systems, volume 34. Curran Associates, Inc., 2021.
  28. Deep deterministic uncertainty: A new simple baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  29. When does Label Smoothing help? Advances in neural information processing systems, 32, 2019.
  30. Reading digits in natural images with unsupervised feature learning. 2011.
  31. Deep Neural Networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015.
  32. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 32, 2019.
  33. On Mixup training: Improved calibration and predictive uncertainty for deep Neural Networks. Advances in Neural Information Processing Systems, 32, 2019.
  34. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on Pattern Analysis and Machine Intelligence, 30(11), 2008.
  35. Out-of-distribution detection with implicit outlier transformation. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=hdghx6wbGuD.
  36. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  37. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
  38. Wide Residual Networks. arXiv preprint arXiv:1605.07146, 2016.
  39. Universum prescription: Regularization using unlabeled data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets