Papers
Topics
Authors
Recent
Search
2000 character limit reached

Accuracy-Preserving Calibration via Statistical Modeling on Probability Simplex

Published 21 Feb 2024 in cs.LG and stat.ML | (2402.13765v2)

Abstract: Classification models based on deep neural networks (DNNs) must be calibrated to measure the reliability of predictions. Some recent calibration methods have employed a probabilistic model on the probability simplex. However, these calibration methods cannot preserve the accuracy of pre-trained models, even those with a high classification accuracy. We propose an accuracy-preserving calibration method using the Concrete distribution as the probabilistic model on the probability simplex. We theoretically prove that a DNN model trained on cross-entropy loss has optimality as the parameter of the Concrete distribution. We also propose an efficient method that synthetically generates samples for training probabilistic models on the probability simplex. We demonstrate that the proposed method can outperform previous methods in accuracy-preserving calibration tasks using benchmarks. The code is available at https://github.com/ToyotaCRDL/SimplexTS.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Multi-sample ζ𝜁\zetaitalic_ζ-mixup: Richer, more realistic synthetic samples from a p𝑝pitalic_p-series interpolant. arXiv preprint arXiv: 2204.03323.
  2. Adaptive temperature scaling for robust calibration of deep neural networks. arXiv preprint arXiv: 2208.00461.
  3. Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. In International Conference on Computational Statistics, pages 177–186. Physica-Verlag HD.
  4. A theoretical analysis of feature pooling in visual recognition. In International Conference on Machine Learning, page 111–118. Omnipress.
  5. Bulatov, Y. (2011). notMNIST dataset. Google (Books/OCR), Tech. Rep. https://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html, Accessed on February 1, 2024.
  6. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics, volume 15, pages 215–223. PMLR.
  7. Elements of Information Theory. Wiley-Interscience.
  8. Classifier calibration: A survey on how to assess and improve predicted class probabilities. arXiv preprint arXiv: 2112.10327.
  9. A survey of uncertainty in deep neural networks. arXiv preprint arXiv: 2107.03342.
  10. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics, volume 15, pages 315–323. PMLR.
  11. Gumbel, E. J. (1941). The return period of flood flows. The Annals of Mathematical Statistics, 12(2):163–190.
  12. On calibration of modern neural networks. In International Conference on Machine Learning, volume 70, pages 1321–1330. PMLR.
  13. Mixup as locally linear out-of-manifold regularization. AAAI Conference on Artificial Intelligence, 33(01):3714–3722.
  14. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 110:457–506.
  15. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition.
  16. Densely connected convolutional networks. In Conference on Computer Vision and Pattern Recognition.
  17. Inoue, H. (2018). Data augmentation by pairing samples for images classification. arXiv preprint arXiv: 1801.02929.
  18. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations.
  19. Joy, T. (2022). Adaptive temperature scaling [Source code]. https://github.com/thwjoy/adats, Accessed on July 21, 2023.
  20. Sample-dependent adaptive temperature scaling for improved calibration. AAAI Conference on Artificial Intelligence, 37(12):14919–14926.
  21. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980.
  22. Being bayesian, even just a bit, fixes overconfidence in ReLU networks. In International Conference on Machine Learning, volume 119, pages 5436–5446. PMLR.
  23. Learning multiple layers of features from tiny images. Technical report, University of Toronto.
  24. Calibrated and sharp uncertainties in deep learning via density estimation. In International Conference on Machine Learning, volume 162, pages 11683–11693. PMLR.
  25. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324.
  26. A general framework for ensemble distribution distillation. In International Workshop on Machine Learning for Signal Processing, pages 1–6.
  27. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations.
  28. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations.
  29. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
  30. Ensemble distribution distillation. In International Conference on Learning Representations.
  31. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning.
  32. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  33. Polak, E. (1997). Optimization : Algorithms and Consistent Approximations. Springer-Verlag.
  34. Powers, D. M. W. (2020). Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. arXiv preprint arXiv: 2010.16061.
  35. Density estimation in representation space to predict model uncertainty. In Engineering Dependable and Secure Machine Learning Systems, pages 84–96. Springer International Publishing.
  36. Scaling ensemble distribution distillation to many classes with proxy targets. In Advances in Neural Information Processing Systems, volume 34, pages 6023–6035. Curran Associates, Inc.
  37. Evidential deep learning to quantify classification uncertainty. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
  38. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556.
  39. A survey on deep transfer learning. In International Conference on Artificial Neural Networks, pages 270–279. Springer International Publishing.
  40. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  41. Parameterized temperature scaling for boosting the expressive power in post-hoc uncertainty calibration. In European Conference on Computer Vision, pages 555–569. Springer Nature Switzerland.
  42. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, volume 97, pages 6438–6447. PMLR.
  43. Non-parametric calibration for classification. In International Conference on Artificial Intelligence and Statistics, volume 108, pages 178–190. PMLR.
  44. Bayesian deep learning and a probabilistic perspective of generalization. In Advances in Neural Information Processing Systems, volume 33, pages 4697–4708. Curran Associates, Inc.
  45. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 1708.07747. The MIT License (MIT) Copyright © 2017 Zalando SE, https://tech.zalando.com.
  46. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv: 2110.11334.
  47. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.03365.
  48. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
  49. Mix-n-Match : Ensemble and compositional methods for uncertainty calibration in deep learning. In International Conference on Machine Learning, volume 119, pages 11117–11128. PMLR.
  50. Confidence regularized self-training. In IEEE/CVF International Conference on Computer Vision.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 2 likes about this paper.

Sign Up for Free