Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Dream the Impossible: Outlier Imagination with Diffusion Models (2309.13415v1)

Published 23 Sep 2023 in cs.LG and cs.CV

Abstract: Utilizing auxiliary outlier datasets to regularize the machine learning model has demonstrated promise for out-of-distribution (OOD) detection and safe prediction. Due to the labor intensity in data collection and cleaning, automating outlier data generation has been a long-desired alternative. Despite the appeal, generating photo-realistic outliers in the high dimensional pixel space has been an open challenge for the field. To tackle the problem, this paper proposes a new framework DREAM-OOD, which enables imagining photo-realistic outliers by way of diffusion models, provided with only the in-distribution (ID) data and classes. Specifically, DREAM-OOD learns a text-conditioned latent space based on ID data, and then samples outliers in the low-likelihood region via the latent, which can be decoded into images by the diffusion model. Different from prior works, DREAM-OOD enables visualizing and understanding the imagined outliers, directly in the pixel space. We conduct comprehensive quantitative and qualitative studies to understand the efficacy of DREAM-OOD, and show that training with the samples generated by DREAM-OOD can benefit OOD detection performance. Code is publicly available at https://github.com/deeplearning-wisc/dream-ood.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (122)
  1. Spot the fake lungs: Generating synthetic medical images using neural diffusion models. arXiv preprint arXiv:2211.00902, 2022.
  2. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023.
  3. Leaving reality to imagination: Robust classification via generated datasets. arXiv preprint arXiv:2302.02503, 2023.
  4. Towards open set deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1563–1572, 2016.
  5. Discriminative out-of-distribution detection for semantic segmentation. arXiv preprint arXiv:1808.07703, 2018.
  6. In or out? fixing imagenet out-of-distribution detection evaluation. In International Conference on Machine Learning, pages 2471–2506, 2023.
  7. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
  8. A data augmentation perspective on diffusion models and retrieval. arXiv preprint arXiv:2304.10253, 2023.
  9. Diffusiondet: Diffusion model for object detection. arXiv preprint arXiv:2211.09788, 2022.
  10. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3606–3613, 2014.
  11. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  12. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020.
  13. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  14. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  15. Extremely simple activation shaping for out-of-distribution detection. In International Conference on Learning Representations, 2023.
  16. Siren: Shaping representations for detecting out-of-distribution objects. In Advances in Neural Information Processing Systems, 2022.
  17. Unknown-aware object detection: Learning what you don’t know from videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  18. Vos: Learning what you don’t know by virtual outlier synthesis. In Proceedings of the International Conference on Learning Representations, 2022.
  19. Is out-of-distribution detection learnable? In Advances in Neural Information Processing Systems, 2022.
  20. Diffusion denoised smoothing for certified and adversarial robust out-of-distribution detection. arXiv preprint arXiv:2303.14961, 2023.
  21. An image is worth one word: Personalizing text-to-image generation using textual inversion. In International Conference on Learning Representations, 2023.
  22. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning, pages 1050–1059, 2016.
  23. A framework for benchmarking class-out-of-distribution detection and its application to imagenet. In International Conference on Learning Representations, 2023.
  24. Dall-e for detection: Language-driven context image synthesis for object detection. arXiv preprint arXiv:2206.09592, 2022.
  25. Em-paste: Em-guided cut-paste with dall-e augmentation for image-level weakly supervised instance segmentation. arXiv preprint arXiv:2212.07629, 2022.
  26. Selectivenet: A deep neural network with an integrated reject option. In Proceedings of the International Conference on Machine Learning, pages 2151–2159, 2019.
  27. Improving robustness using generated data. Advances in Neural Information Processing Systems, 34:4218–4233, 2021.
  28. Denoising diffusion models for out-of-distribution detection. arXiv preprint arXiv:2211.07740, 2022.
  29. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  30. Is synthetic data from generative models ready for image recognition? In International Conference on Learning Representations, 2023.
  31. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 41–50, 2019.
  32. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8349, 2021.
  33. A baseline for detecting misclassified and out-of-distribution examples in neural networks. Proceedings of the International Conference on Learning Representations, 2017.
  34. Deep anomaly detection with outlier exposure. In Proceedings of the International Conference on Learning Representations, 2019.
  35. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representations, 2020.
  36. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262–15271, 2021.
  37. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
  38. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  39. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47–1, 2022.
  40. Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10951–10960, 2020.
  41. On the importance of gradients for detecting distributional shifts in the wild. In Advances in Neural Information Processing Systems, 2021.
  42. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8710–8719, 2021.
  43. Distilling model failures as directions in latent space. In International Conference on Learning Representations, 2023.
  44. Ood-maml: Meta-learning for few-shot out-of-distribution detection and classification. Advances in Neural Information Processing Systems, 33:3907–3916, 2020.
  45. Training OOD detectors in their natural habitats. In Proceedings of the International Conference on Machine Learning, pages 10848–10865, 2022.
  46. On outlier exposure with generative models. In NeurIPS ML Safety Workshop, 2022.
  47. Being bayesian, even just a bit, fixes overconfidence in relu networks. In International conference on machine learning, pages 5436–5446, 2020.
  48. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  49. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, pages 6402–6413, 2017.
  50. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In Proceedings of the International Conference on Learning Representations, 2018.
  51. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems, 31, 2018.
  52. Enhancing the reliability of out-of-distribution image detection in neural networks. In Proceedings of the International Conference on Learning Representations, 2018.
  53. Magicmix: Semantic mixing with diffusion models. arXiv preprint arXiv:2210.16056, 2022.
  54. Diffusion denoising process for perceptron bias in out-of-distribution detection. arXiv preprint arXiv:2211.11255, 2022.
  55. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.
  56. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33:21464–21475, 2020.
  57. Unsupervised out-of-distribution detection with diffusion inpainting. In Proceedings of the International Conference on Machine Learning, 2023.
  58. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 2022.
  59. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32:13153–13164, 2019.
  60. Predictive uncertainty estimation via prior networks. Advances in Neural Information Processing Systems, 31, 2018.
  61. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. In Advances in Neural Information Processing Systems, 2019.
  62. Directional statistics, volume 2. Wiley Online Library, 2000.
  63. Towards neural networks that provably know when they don’t know. In Proceedings of the International Conference on Learning Representations, 2020.
  64. POEM: out-of-distribution detection with posterior sampling. In Proceedings of the International Conference on Machine Learning, pages 15650–15665, 2022.
  65. How to exploit hyperspherical embeddings for out-of-distribution detection? In Proceedings of the International Conference on Learning Representations, 2023.
  66. Self-supervised learning for generalizable out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5216–5223, 2020.
  67. Reading digits in natural images with unsupervised feature learning. 2011.
  68. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 427–436, 2015.
  69. Imaginarynet: Learning object detectors without real images and annotations. In International Conference on Learning Representations, 2023.
  70. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. International Conference on Machine Learning, 2022.
  71. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162–8171, 2021.
  72. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
  73. Fast unsupervised brain anomaly detection and segmentation with diffusion models. arXiv preprint arXiv:2206.03461, 2022.
  74. Brain imaging generation with latent diffusion models. In MICCAI Workshop on Deep Generative Models, pages 117–126, 2022.
  75. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.
  76. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, pages 8748–8763, 2021.
  77. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  78. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389–5400, 2019.
  79. Out-of-distribution detection and selective generation for conditional language models. In International Conference on Learning Representations, 2023.
  80. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  81. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242, 2022.
  82. Improving dermatology classifiers across populations using images generated by large diffusion models. arXiv preprint arXiv:2211.13352, 2022.
  83. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022.
  84. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
  85. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  86. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  87. LAION-5b: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2022.
  88. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
  89. Generating high fidelity data from low-density regions using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11492–11501, 2022.
  90. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265, 2015.
  91. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
  92. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415–1428, 2021.
  93. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  94. React: Out-of-distribution detection with rectified activations. In Advances in Neural Information Processing Systems, volume 34, 2021.
  95. Dice: Leveraging sparsification for out-of-distribution detection. In Proceedings of European Conference on Computer Vision, 2022.
  96. Out-of-distribution detection with deep nearest neighbors. In Proceedings of the International Conference on Machine Learning, pages 20827–20840, 2022.
  97. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
  98. Non-parametric outlier synthesis. In Proceedings of the International Conference on Learning Representations, 2023.
  99. Effective data augmentation with diffusion models. arXiv preprint arXiv:2302.07944, 2023.
  100. Uncertainty estimation using a single deep deterministic neural network. In Proceedings of the International Conference on Machine Learning, pages 9690–9700, 2020.
  101. The inaturalist species classification and detection dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8769–8778, 2018.
  102. Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
  103. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4921–4930, 2022.
  104. Can multi-label classification networks know what they don’t know? Proceedings of the Advances in Neural Information Processing Systems, 2021.
  105. Larry Wasserman. Lecture notes of statistical methods for machine learning. 2019.
  106. Novel view synthesis with diffusion models. In International Conference on Learning Representations, 2023.
  107. Mitigating neural network overconfidence with logit normalization. In Proceedings of the International Conference on Machine Learning, pages 23631–23644, 2022.
  108. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
  109. Diffusion models for medical anomaly detection. arXiv preprint arXiv:2203.04306, 2022.
  110. Energy-based out-of-distribution detection for graph neural networks. In International Conference on Learning Representations, 2023.
  111. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 650–656, 2022.
  112. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3485–3492, 2010.
  113. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755, 2015.
  114. Semantically coherent out-of-distribution detection. In Proceedings of the International Conference on Computer Vision, pages 8281–8289, 2021.
  115. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334, 2021.
  116. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
  117. Not just pretty pictures: Text-to-image generators enable interpretable interventions for robust representations. arXiv preprint arXiv:2212.11237, 2022.
  118. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019.
  119. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 2022.
  120. Expanding small-scale datasets with guided imagination. arXiv preprint arXiv:2211.13976, 2022.
  121. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.
  122. Unleashing mask: Explore the intrinsic out-of-distribution detection capability. arXiv preprint arXiv:2306.03715, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Xuefeng Du (26 papers)
  2. Yiyou Sun (27 papers)
  3. Xiaojin Zhu (59 papers)
  4. Yixuan Li (183 papers)
Citations (32)
View on Semantic Scholar