Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the Learnability of Out-of-distribution Detection (2404.04865v1)

Published 7 Apr 2024 in cs.LG, cs.CV, and stat.ML

Abstract: Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution. To ease the above assumption, researchers have studied a more realistic setting: out-of-distribution (OOD) detection, where test data may come from classes that are unknown during training (i.e., OOD data). Due to the unavailability and diversity of OOD data, good generalization ability is crucial for effective OOD detection algorithms, and corresponding learning theory is still an open problem. To study the generalization of OOD detection, this paper investigates the probably approximately correct (PAC) learning theory of OOD detection that fits the commonly used evaluation metrics in the literature. First, we find a necessary condition for the learnability of OOD detection. Then, using this condition, we prove several impossibility theorems for the learnability of OOD detection under some scenarios. Although the impossibility theorems are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios. Based on this observation, we next give several necessary and sufficient conditions to characterize the learnability of OOD detection in some practical scenarios. Lastly, we offer theoretical support for representative OOD detection works based on our OOD theory.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. Uncertainty estimation using a single deep deterministic neural network. In ICML, 2020.
  2. P. L. Bartlett and W. Maass. Vapnik-chervonenkis dimension of neural nets. The handbook of brain theory and neural networks, 2003.
  3. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 2008.
  4. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20(63):1–17, 2019.
  5. A. Bendale and T. E. Boult. Towards open set deep networks. In CVPR, 2016.
  6. Y. Bulatov. Notmnist dataset. Google (Books/OCR), Tech. Rep.[Online]. Available: http://yaroslavvb. blogspot. it/2011/09/notmnist-dataset. html,2, 2011.
  7. Classification with rejection based on cost-sensitive classification. In ICML, 2021.
  8. Atom: Robustifying out-of-distribution detection using outlier mining. ECML, 2021a.
  9. Large-margin contrastive learning with distance polarization regularizer. In ICML, 2021b.
  10. Outlier-robust sparse estimation via non-convex optimization. In NeurIPS, 2021.
  11. C. K. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 1970.
  12. D. L. Cohn. Measure theory. Springer, 2013.
  13. Boosting with abstention. In NeurIPS, 2016a.
  14. Learning with rejection. In ALT, 2016b.
  15. Image anomaly detection with generative adversarial networks. In ECML, 2018.
  16. L. Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag., 2012.
  17. Reducing network agnostophobia. In NeurIPS, pages 9175–9186, 2018.
  18. I. Diakonikolas and D. M. Kane. Recent advances in algorithmic high-dimensional robust statistics. A shorter version appears as an Invited Book Chapter in Beyond the Worst-Case Analysis of Algorithms, 2020.
  19. Outlier-robust high-dimensional sparse estimation via iterative filtering. In NeurIPS, 2019.
  20. Outlier robust mean estimation with subgaussian rates via stability. In NeurIPS, 2020.
  21. Outlier-robust learning of ising models under dobrushin’s condition. In COLT, 2021.
  22. Outlier-robust sparse mean estimation for heavy-tailed distributions. In NeurIPS, 2022.
  23. What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation. In CVPR, 2020.
  24. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  25. Open set domain adaptation: Theoretical bound and algorithm. IEEE Transactions on Neural Networks and Learning Systems, 2020.
  26. Learning bounds for open-set learning. In ICML, 2021.
  27. Is out-of-distribution detection learnable? In NeurIPS, 2022a.
  28. Semi-supervised heterogeneous domain adaptation: Theory and algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022b.
  29. Exploring the limits of out-of-distribution detection. In NeurIPS, 2021a.
  30. Exploring the Limits of Out-of-Distribution Detection. In NeurIPS, 2021b.
  31. Optimal strategies for reject option classifiers. CoRR, abs/2101.12523, 2021.
  32. Beyond perturbations: Learning guarantees with arbitrary adversarial test examples. In NeurIPS, 2020.
  33. DROCC: deep robust one-class classification. In ICML, 2020.
  34. A kernel two-sample test. Journal of Machine Learning Research, 2012.
  35. D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
  36. Deep anomaly detection with outlier exposure. In ICLR, 2019.
  37. Generalized ODIN: detecting out-of-distribution image without learning from out-of-distribution data. In CVPR, 2020.
  38. Densely connected convolutional networks. In CVPR, 2017.
  39. On the Importance of Gradients for Detecting Distributional Shifts in the Wild. In NeurIPS, 2021.
  40. Binary classification from positive-confidence data. In NeurIPS, 2018.
  41. Positive-unlabeled learning with label distribution alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  42. A. T. Kalai and V. Kanade. Efficient learning with arbitrary covariate shift. In ALT, Proceedings of Machine Learning Research, 2021.
  43. M. Karpinski and A. Macintyre. Polynomial bounds for VC dimension of sigmoidal and general pfaffian neural networks. J. Comput. Syst. Sci., 54(1):169–176, 1997.
  44. D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NeurIPS, 2018.
  45. Positive-unlabeled learning with non-negative risk estimator. In NeurIPS, 2017.
  46. A. Krizhevsky and G. Hinton. Convolutional deep belief networks on cifar-10. Technical report, Citeseer, 2009.
  47. Cifar-10 and cifar-100 datasets. 2009.
  48. G. Kylberg. Kylberg texture dataset v. 1.0. 2011.
  49. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS, 2018.
  50. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
  51. Mood: Multi-level out-of-distribution detection. In CVPR, 2021.
  52. Open category detection with PAC guarantees. In ICML, 2018.
  53. Energy-based out-of-distribution detection. In NeurIPS, 2020.
  54. Progressive graph learning for open-set domain adaptation. In ICML, 2020.
  55. On the impact of spurious correlation for out-of-distribution detection. AAAI, 2022.
  56. Foundations of machine learning. MIT press, 2018.
  57. P. Morteza and Y. Li. Provable guarantees for understanding out-of-distribution detection. AAAI, 2022.
  58. Do deep generative models know what they don’t know? In ICLR, 2019.
  59. On the calibration of multiclass classification with rejection. In NeurIPS, 2019.
  60. Generative probabilistic novelty detection with adversarial autoencoders. In NeurIPS, 2018.
  61. A. Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143–195, 1999.
  62. Likelihood ratios for out-of-distribution detection. In NeurIPS, 2019.
  63. A simple fix to mahalanobis distance for improving near-ood detection. CoRR, abs/2106.09022, 2021.
  64. Robust statistics. John Wiley & Sons, 2009.
  65. Robust statistics: the approach based on influence functions. John Wiley & Sons, 2011.
  66. Deep one-class classification. In ICML, 2018.
  67. I. Safran and O. Shamir. Depth-width tradeoffs in approximating natural functions with neural networks. In ICML, 2017.
  68. A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. arXiv preprint arXiv:2110.14051, 2021.
  69. S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
  70. Learnability, stability and uniform convergence. J. Mach. Learn. Res., 11:2635–2670, 2010.
  71. React: Out-of-distribution detection with rectified activations. In NeurIPS, 2021.
  72. Out-of-distribution detection with deep nearest neighbors. In ICML, 2022.
  73. Out-of-distribution detection in classifiers via generation. In NeurIPS Workshop, 2019.
  74. Can multi-label classification networks know what they don’t know? In NeurIPS, 2021.
  75. Openauc: Towards auc-oriented open-set recognition. Advances in Neural Information Processing Systems, 35:25033–25045, 2022.
  76. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. In NeurIPS, 2020.
  77. Generalized out-of-distribution detection: A survey. CoRR, abs/2110.11334, 2021.
  78. Full-spectrum out-of-distribution detection. CoRR, 2022.
  79. Revisiting auc-oriented adversarial training with loss-agnostic perturbations. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–18, 2023.
  80. Out-of-distribution detection using union of 1-dimensional subspaces. In CVPR, 2021.
  81. Understanding failures in out-of-distribution detection with deep generative models. In ICML, 2021.
  82. Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell., 2018.
  83. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR, 2018.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets