Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
157 tokens/sec
GPT-4o
43 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Consistency-Guided Temperature Scaling Using Style and Content Information for Out-of-Domain Calibration (2402.15019v1)

Published 22 Feb 2024 in cs.LG, cs.AI, and stat.ML

Abstract: Research interests in the robustness of deep neural networks against domain shifts have been rapidly increasing in recent years. Most existing works, however, focus on improving the accuracy of the model, not the calibration performance which is another important requirement for trustworthy AI systems. Temperature scaling (TS), an accuracy-preserving post-hoc calibration method, has been proven to be effective in in-domain settings, but not in out-of-domain (OOD) due to the difficulty in obtaining a validation set for the unseen domain beforehand. In this paper, we propose consistency-guided temperature scaling (CTS), a new temperature scaling strategy that can significantly enhance the OOD calibration performance by providing mutual supervision among data samples in the source domains. Motivated by our observation that over-confidence stemming from inconsistent sample predictions is the main obstacle to OOD calibration, we propose to guide the scaling process by taking consistencies into account in terms of two different aspects -- style and content -- which are the key components that can well-represent data samples in multi-domain settings. Experimental results demonstrate that our proposed strategy outperforms existing works, achieving superior OOD calibration performance on various datasets. This can be accomplished by employing only the source domains without compromising accuracy, making our scheme directly applicable to various trustworthy AI systems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, 1657–1664.
  2. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, 1050–1059. PMLR.
  3. Confidence calibration for domain generalization under covariate shift. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 8958–8967.
  4. In search of lost domain generalization. arXiv preprint arXiv:2007.01434.
  5. On calibration of modern neural networks. In International conference on machine learning, 1321–1330. PMLR.
  6. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  7. A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16081–16090.
  8. A baseline for detecting misclassified and out-of-distribution examples in neural networks. International Conference on Learning Representations.
  9. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, 1501–1510.
  10. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4893–4902.
  11. Improving model calibration with accuracy versus uncertainty optimization. Advances in Neural Information Processing Systems, 33: 18237–18248.
  12. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6): 84–90.
  13. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
  14. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, 5542–5550.
  15. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5400–5409.
  16. Uncertainty Modeling for Out-of-Distribution Generalization. In International Conference on Learning Representations.
  17. The devil is in the margin: Margin-based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 80–88.
  18. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  19. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE international conference on computer vision, 5715–5725.
  20. Domain generalization via invariant feature representation. In International conference on machine learning, 10–18. PMLR.
  21. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33: 15288–15299.
  22. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29.
  23. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, 625–632.
  24. Generalizing Across Domains via Cross-Gradient Training. In International Conference on Learning Representations.
  25. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32.
  26. Towards trustworthy predictions from deep neural networks with fast adversarial calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 9886–9896.
  27. Post-hoc uncertainty calibration for domain drift scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10124–10132.
  28. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5018–5027.
  29. Algorithmic learning in a random world, volume 29. Springer.
  30. Robust calibration with multi-domain temperature scaling. Advances in Neural Information Processing Systems, 35: 27510–27523.
  31. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, 609–616.
  32. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 694–699.
  33. Domain generalization in vision: A survey. arXiv preprint arXiv:2103.02503.
  34. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  35. Learning to generate novel domains for domain generalization. In European conference on computer vision, 561–578. Springer.
  36. Domain adaptive ensemble learning. IEEE Transactions on Image Processing, 30: 8008–8018.
  37. Domain Generalization with MixStyle. In International Conference on Learning Representations.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets