Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (2310.01755v2)

Published 3 Oct 2023 in cs.CV

Abstract: The task of out-of-distribution (OOD) detection is notoriously ill-defined. Earlier works focused on new-class detection, aiming to identify label-altering data distribution shifts, also known as "semantic shift." However, recent works argue for a focus on failure detection, expanding the OOD evaluation framework to account for label-preserving data distribution shifts, also known as "covariate shift." Intriguingly, under this new framework, complex OOD detectors that were previously considered state-of-the-art now perform similarly to, or even worse than the simple maximum softmax probability baseline. This raises the question: what are the latest OOD detectors actually detecting? Deciphering the behavior of OOD detection algorithms requires evaluation datasets that decouples semantic shift and covariate shift. To aid our investigations, we present ImageNet-OOD, a clean semantic shift dataset that minimizes the interference of covariate shift. Through comprehensive experiments, we show that OOD detectors are more sensitive to covariate shift than to semantic shift, and the benefits of recent OOD detection algorithms on semantic shift detection is minimal. Our dataset and analyses provide important insights for guiding the design of future OOD detectors.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
  2. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
  3. Pass: An imagenet replacement for self-supervised pretraining without humans. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  4. Unified out-of-distribution detection: A model-specific perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1453–1463, 2023.
  5. In or out? fixing imagenet out-of-distribution detection evaluation. In International Conference on Machine Learning, pp. 2471–2506. PMLR, 2023.
  6. Improved baselines with momentum contrastive learning, 2020.
  7. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  3606–3613, 2014.
  8. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  9. Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  10. Extremely simple activation shaping for out-of-distribution detection. arXiv preprint arXiv:2209.09858, 2022.
  11. Christiane Fellbaum. WordNet: An Electronic Lexical Database. Bradford Books, 1998.
  12. Exploring the limits of out-of-distribution detection. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
  13. A framework for benchmarking class-out-of-distribution detection and its application to imagenet. In The Eleventh International Conference on Learning Representations, 2023.
  14. Out-of-distribution detection is not all you need, 2023.
  15. Out-of-distribution detection in unsupervised continual learning, 2022.
  16. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.  1026–1034, 2015.
  17. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  770–778, 2016. doi: 10.1109/CVPR.2016.90.
  18. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
  19. A baseline for detecting misclassified and out-of-distribution examples in neural networks. Proceedings of International Conference on Learning Representations, 2017.
  20. The many faces of robustness: A critical analysis of out-of-distribution generalization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp.  8320–8329, 2020.
  21. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  15262–15271, 2021.
  22. Scaling out-of-distribution detection for real-world settings. In International Conference on Machine Learning, pp. 8759–8773. PMLR, 2022.
  23. The inaturalist species classification and detection dataset. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  8769–8778, 2018.
  24. Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10951–10960, 2020.
  25. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  4700–4708, 2017.
  26. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  27. On the importance of gradients for detecting distributional shifts in the wild. In Advances in Neural Information Processing Systems, 2021.
  28. A call to reflect on evaluation practices for failure detection in image classification. In The Eleventh International Conference on Learning Representations, 2023.
  29. What uncertainties do we need in bayesian deep learning for computer vision? In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  30. A theoretical study on solving continual learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
  31. Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
  32. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(7):1956–1981, 2020.
  33. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
  34. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
  35. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 2020.
  36. A unifying view on dataset shift in classification. Pattern recognition, 45(1):521–530, 2012.
  37. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
  38. Imagenet-21k pretraining for the masses. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  39. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015. doi: 10.1007/s11263-015-0816-y.
  40. On random weights and unsupervised feature learning. In International Conference on Machine Learning, 2011.
  41. React: Out-of-distribution detection with rectified activations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
  42. Out-of-distribution detection with deep nearest neighbors. arXiv preprint arXiv:2204.06507, 2022.
  43. YFCC100m. Communications of the ACM, 59(2):64–73, jan 2016. doi: 10.1145/2812802.
  44. Exploring covariate and concept shift for out-of-distribution detection. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
  45. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  9446–9454, 2018.
  46. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2022.
  47. Learning robust global representations by penalizing local predictive power. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  48. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  49. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  6857–6866, 2018.
  50. Augmenting softmax information for selective classification with out-of-distribution data. In Proceedings of the Asian Conference on Computer Vision, pp.  1995–2012, 2022.
  51. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.  3485–3492, 2010. doi: 10.1109/CVPR.2010.5539970.
  52. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  1492–1500, 2017.
  53. Regnet: Self-regulated network for image classification. IEEE Transactions on Neural Networks and Learning Systems, pp.  1–6, 2022. doi: 10.1109/TNNLS.2022.3158966.
  54. Semantically coherent out-of-distribution detection. In Proceedings of the IEEE International Conference on Computer Vision, 2021a.
  55. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334, 2021b.
  56. Full-spectrum out-of-distribution detection. International Journal of Computer Vision, 131(10):2607–2622, 2023.
  57. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pp.  547–558, 2020.
  58. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1–87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C.30.87.
  59. Openood v1. 5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306.09301, 2023.
  60. Decoupling maxlogit for out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3388–3397, 2023.
  61. Rethinking out-of-distribution detection from a human-centric perspective, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. William Yang (15 papers)
  2. Byron Zhang (4 papers)
  3. Olga Russakovsky (62 papers)
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now