Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Understanding Likelihood of Normalizing Flow and Image Complexity through the Lens of Out-of-Distribution Detection (2402.10477v1)

Published 16 Feb 2024 in cs.LG

Abstract: Out-of-distribution (OOD) detection is crucial to safety-critical machine learning applications and has been extensively studied. While recent studies have predominantly focused on classifier-based methods, research on deep generative model (DGM)-based methods have lagged relatively. This disparity may be attributed to a perplexing phenomenon: DGMs often assign higher likelihoods to unknown OOD inputs than to their known training data. This paper focuses on explaining the underlying mechanism of this phenomenon. We propose a hypothesis that less complex images concentrate in high-density regions in the latent space, resulting in a higher likelihood assignment in the Normalizing Flow (NF). We experimentally demonstrate its validity for five NF architectures, concluding that their likelihood is untrustworthy. Additionally, we show that this problem can be alleviated by treating image complexity as an independent variable. Finally, we provide evidence of the potential applicability of our hypothesis in another DGM, PixelCNN++.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 427–436, 2015.
  2. Vos: Learning what you don’t know by virtual outlier synthesis. In Proceedings of the International Conference on Learning Representations, 2022.
  3. Efficient out-of-distribution detection in digital pathology using multi-head convolutional neural networks. In Tal Arbel, Ismail Ben Ayed, Marleen de Bruijne, Maxime Descoteaux, Herve Lombaert, and Christopher Pal, editors, Proceedings of the Third Conference on Medical Imaging with Deep Learning, volume 121 of Proceedings of Machine Learning Research, pages 465–478. PMLR, 06–08 Jul 2020.
  4. OpenOOD: Benchmarking generalized out-of-distribution detection. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  5. Generalized out-of-distribution detection: A survey. CoRR, abs/2110.11334, 2021.
  6. Provable guarantees for understanding out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022.
  7. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7498–7512. Curran Associates, Inc., 2020.
  8. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8710–8719, June 2021.
  9. Do deep generative models know what they don’t know? In International Conference on Learning Representations, 2019.
  10. Waic, but why? generative ensembles for robust anomaly detection, 2018.
  11. Input complexity and out-of-distribution detection with likelihood-based generative models. In International Conference on Learning Representations, 2020.
  12. Detecting out-of-distribution inputs to deep generative models using typicality, 2020.
  13. A. Hyvärinen and E. Oja. Independent component analysis: Algorithms and applications. Neural Netw., 13(4–5):411–430, May 2000.
  14. Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948.
  15. Closing the dequantization gap: Pixelcnn as a single-layer flow. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3724–3734. Curran Associates, Inc., 2020.
  16. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  17. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21038–21049. Curran Associates, Inc., 2020.
  18. Why normalizing flows fail to detect out-of-distribution data. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20578–20589. Curran Associates, Inc., 2020.
  19. Further analysis of outlier detection with deep generative models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8982–8992. Curran Associates, Inc., 2020.
  20. Understanding failures in out-of-distribution detection with deep generative models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12427–12436. PMLR, 18–24 Jul 2021.
  21. A note on the evaluation of generative models. In International Conference on Learning Representations, Apr 2016.
  22. Modelling image complexity by independent component analysis, with application to content-based image retrieval. In Cesare Alippi, Marios Polycarpou, Christos Panayiotou, and Georgios Ellinas, editors, Artificial Neural Networks – ICANN 2009, pages 704–714, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg.
  23. Nice: Non-linear independent components estimation. In International Conference on Learning Representations, 2015.
  24. Pattern recognition and machine learning, volume 4. Springer, 2006.
  25. An information-theoretic framework for image complexity. In Proceedings of the First Eurographics Conference on Computational Aesthetics in Graphics, Visualization and Imaging, Computational Aesthetics’05, page 177–184, Goslar, DEU, 2005. Eurographics Association.
  26. Peter D. Grünwald and Paul M. B. Vitányi. Kolmogorov complexity and information theory. with an interpretation in terms of questions and answers. Journal of Logic, Language and Information, 12(4):497–529, 2003.
  27. Khalid Sayood. Introduction to data compression. Morgan Kaufmann, 2017.
  28. Exploiting intra-slice and inter-slice redundancy for learning-based lossless volumetric image compression. IEEE Transactions on Image Processing, 31:1697–1707, 2022.
  29. Likelihood-free out-of-distribution detection with invertible generative models. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2119–2125. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.
  30. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1530–1538, Lille, France, 07–09 Jul 2015. PMLR.
  31. Density estimation using real nvp. In International Conference on Learning Representations, 2017.
  32. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems 31, pages 10215–10224. Curran Associates, Inc., 2018.
  33. Invertible residual networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 573–582. PMLR, 09–15 Jun 2019.
  34. Residual flows for invertible generative modeling. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  35. Integer discrete flows and lossless compression. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  36. PixelCNN++: Improving the pixelCNN with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2017.
  37. Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108–116, 1995.
  38. Herbert Federer. Geometric measure theory. Springer, 2014.
  39. Learning multiple layers of features from tiny images. 2009.
  40. Understanding and mitigating exploding inverses in invertible neural networks. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1792–1800. PMLR, 13–15 Apr 2021.
  41. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  42. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  43. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.03365, 2015.
  44. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015.
  45. Yaroslav Bulatov. Notmnist dataset. Google (Books/OCR), Tech. Rep.[Online]. Available: http://yaroslavvb. blogspot. it/2011/09/notmnist-dataset. html, 2, 2011.
  46. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
  47. Mu Cai and Yixuan Li. Out-of-distribution detection via frequency-regularized generative models. In Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
  48. Elements of information theory. John Wiley & Sons, 2012.
  49. Carlos Fernandez-Granda. Optimization-based data analysis: Lecture notes 3: Randomness, Fall 2017.
  50. Compression with flows via local bits-back coding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  51. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
  52. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Genki Osada (4 papers)
  2. Tsubasa Takahashi (20 papers)
  3. Takashi Nishide (4 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets