Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning to Augment Distributions for Out-of-Distribution Detection (2311.01796v2)

Published 3 Nov 2023 in cs.LG

Abstract: Open-world classification systems should discern out-of-distribution (OOD) data whose labels deviate from those of in-distribution (ID) cases, motivating recent studies in OOD detection. Advanced works, despite their promising progress, may still fail in the open world, owing to the lack of knowledge about unseen OOD data in advance. Although one can access auxiliary OOD data (distinct from unseen ones) for model training, it remains to analyze how such auxiliary data will work in the open world. To this end, we delve into such a problem from a learning theory perspective, finding that the distribution discrepancy between the auxiliary and the unseen real OOD data is the key to affecting the open-world detection performance. Accordingly, we propose Distributional-Augmented OOD Learning (DAL), alleviating the OOD distribution discrepancy by crafting an OOD distribution set that contains all distributions in a Wasserstein ball centered on the auxiliary OOD distribution. We justify that the predictor trained over the worst OOD data in the ball can shrink the OOD distribution discrepancy, thus improving the open-world detection performance given only the auxiliary OOD data. We conduct extensive evaluations across representative OOD detection setups, demonstrating the superiority of our DAL over its advanced counterparts.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (80)
  1. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
  2. Is out-of-distribution detection learnable? In NeurIPS, 2022.
  3. Anomalous instance detection in deep learning: A survey. Technical report, Lawrence Livermore National Lab., 2020.
  4. Generalized out-of-distribution detection: a survey. arXiv preprint arXiv:2110.11334, 2021.
  5. A benchmark of medical out of distribution detection. arXiv preprint arXiv:2007.04250, 2020.
  6. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624, 2021.
  7. Deep anomaly detection with outlier exposure. In ICLR, 2019.
  8. POEM: Out-of-distribution detection with posterior sampling. In ICML, 2022.
  9. Out-of-distribution detection with implicit outlier transformation. In ICLR, 2023.
  10. Cédric Villani. Topics in Optimal Transportation. American Mathematical Society, 2021.
  11. Cédric Villani. Optimal Transport: Old and New. Springer, 2008.
  12. VOS: Learning what you don’t know by virtual outlier synthesis. In ICLR, 2022a.
  13. On certifying and improving generalization to unseen domains. arXiv preprint arXiv:2206.12364, 2022.
  14. ATOM: Robustifying out-of-distribution detection using outlier mining. In ECML, 2021.
  15. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS, 2018a.
  16. Energy-based out-of-distribution detection. In NeurIPS, 2020.
  17. Out-of-distribution detection with deep nearest neighbors. In ICML, 2022.
  18. Certifying some distributional robustness with principled adversarial training. In ICLR, 2018.
  19. Stochastic gradient methods for distributionally robust optimization with f-divergences. In NIPS, 2016.
  20. Modeling the second player in distributionally robust optimization. In ICLR, 2021.
  21. Distributionally robust optimization and generalization in kernel methods. In NeurIPS, 2019.
  22. Moderately distributional exploration for domain generalization. In ICML, 2023.
  23. José H. Blanchet and M. KarthyekRajhaaA. Quantifying distributional model risk via optimal transport. Risk Management eJournal, 2016.
  24. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009.
  25. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(11):1958–1970, 2008.
  26. Describing textures in the wild. In CVPR, 2014.
  27. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 2011.
  28. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452–1464, 2018.
  29. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
  30. Turkergaze: crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755, 2015.
  31. Wide residual networks. In BMVC, 2016.
  32. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017.
  33. Extremely simple activation shaping for out-of-distribution detection. In ICLR, 2023.
  34. ReAct: out-of-distribution detection with rectified activations. In NeurIPS, 2021.
  35. CSI: novelty detection via contrastive learning on distributionally shifted instances. In NeurIPS, 2020.
  36. ImageNet: a large-scale hierarchical image database. In CVPR, 2009.
  37. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11):2579–2605, 2008.
  38. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
  39. Can multi-label classification networks know what they don’t know? In NeurIPS, 2021a.
  40. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
  41. MOS: towards scaling out-of-distribution detection for large semantic space. In CVPR, 2021.
  42. Detecting out-of-distribution examples with gram matrices. In ICML, 2020.
  43. ViM: Out-of-distribution with virtual-logit matching. In CVPR, 2022a.
  44. MOOD: Multi-level out-of-distribution detection. In CVPR, 2021.
  45. Provable guarantees for understanding out-of-distribution detection. In AAAI, 2022.
  46. Source-free progressive graph learning for open-set domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):11240–11255, 2023.
  47. On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS, 2021.
  48. How useful are gradients for ood detection really? arXiv preprint arXiv:2205.10439, 2022.
  49. Unleashing mask: Explore the intrinsic out-of-distribution detection capability. In ICML, 2023a.
  50. SSD: A unified framework for self-supervised outlier detection. In ICLR, 2021.
  51. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. In ICML, 2022b.
  52. Out-of-distribution detection learning with unreliable out-of-distribution sources. In NeurIPS, 2023.
  53. SIREN: Shaping representations for detecting out-of-distribution objects. In NeurIPS, 2022b.
  54. How to exploit hyperspherical embeddings for out-of-distribution detection? In ICLR, 2023.
  55. Out-of-distribution detection using union of 1-dimensional subspaces. In CVPR, 2021.
  56. Mitigating neural network overconfidence with logit normalization. In ICML, 2022.
  57. Harnessing out-of-distribution examples via augmenting content and style. In ICLR, 2023a.
  58. Diversified outlier exposure for out-of-distribution detection via informative extrapolation. In NeurIPS, 2023b.
  59. Yi Li and Nuno Vasconcelos. Background data resampling for outlier-aware classification. In CVPR, 2020.
  60. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In CVPR, 2019.
  61. OOD-MAMl: Meta-learning for few-shot out-of-distribution detection and classification. In NeurIPS, 2020.
  62. Uncertainty estimation using a single deep deterministic neural network. In ICML, 2020.
  63. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In ICLR, 2018b.
  64. Out-of-distribution detection in classifiers via generation. In NeurIPS Workshop, 2019.
  65. Non-parametric outlier synthesis. In ICLR, 2023.
  66. Progressive graph learning for open-set domain adaptation. In ICML, 2020.
  67. Understanding failures in out-of-distribution detection with deep generative models. In ICML, 2021.
  68. Learning bounds for open-set learning. In ICML, 2021.
  69. Breaking down out-of-distribution detection: Many methods based on OOD training data estimate a combination of the same core quantities. In ICML, 2022.
  70. Roman Vershynin. High-Dimensional Probability. Springer, 2018.
  71. Robust wasserstein profile inference and applications to machine learning. Journal of Applied Probability, 56(3):830–857, 2019.
  72. Size-independent sample complexity of neural networks. In COLT, 2018.
  73. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
  74. Probabilistic margins for instance reweighting in adversarial training. In NeurIPS, 2021b.
  75. Robust generalization against photon-limited corruptions via worst-case sharpness minimization. In CVPR, 2023b.
  76. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):10045–10067, 2021.
  77. Adversarial robustness through the lens of causality. In ICLR, 2022.
  78. ImageNet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972, 2021.
  79. The iNaturalist species classification and detection dataset. In CVPR, 2018.
  80. Scaling out-of-distribution detection for real-world settings. In ICML, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Qizhou Wang (26 papers)
  2. Zhen Fang (58 papers)
  3. Yonggang Zhang (36 papers)
  4. Feng Liu (1212 papers)
  5. Yixuan Li (183 papers)
  6. Bo Han (282 papers)
Citations (27)
View on Semantic Scholar