2000 character limit reached
Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift (2312.08033v1)
Published 13 Dec 2023 in cs.LG and cs.AI
Abstract: Knowing if a model will generalize to data 'in the wild' is crucial for safe deployment. To this end, we study model disagreement notions that consider the full predictive distribution - specifically disagreement based on Hellinger distance, Jensen-Shannon and Kullback-Leibler divergence. We find that divergence-based scores provide better test error estimates and detection rates on out-of-distribution data compared to their top-1 counterparts. Experiments involve standard vision and foundation models.
- Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35:19274–19289, 2022.
- Yaofo Chen. chenyaofo/pytorch-cifar-models, November 2021.
- Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13733–13742, 2021.
- Diversity with cooperation: Ensemble methods for few-shot classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3723–3731, 2019.
- Leveraging unlabeled data to predict out-of-distribution performance. arXiv preprint arXiv:2201.04234, 2022.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
- Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132, 2019.
- Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
- A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016.
- Assessing generalization of sgd via disagreement. arXiv preprint arXiv:2106.13799, 2021.
- A note on" assessing generalization of sgd via disagreement". arXiv preprint arXiv:2202.01851, 2022.
- Cifar-10 (canadian institute for advanced research).
- Cifar-100 (canadian institute for advanced research).
- Predicting out-of-distribution error with confidence optimal transport. arXiv preprint arXiv:2302.05018, 2023.
- Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018.
- Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650, 2020.
- Active learning for probability estimation using jensen-shannon divergence. In Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005. Proceedings 16, pages 268–279. Springer, 2005.
- Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning, pages 7721–7735. PMLR, 2021.
- Distributional generalization: A new kind of generalization. arXiv preprint arXiv:2009.08092, 2020.
- K Nigam. Employing em in pool-based active learning for text classification. machine learning. In Proceeding (s) of the Fifteenth International Conference (ICML’98), pages 350–358, 1998.
- Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
- (almost) provable error bounds under distribution shift via disagreement discrepancy. arXiv preprint arXiv:2306.00312, 2023.
- Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
- Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.