Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CADet: Fully Self-Supervised Out-Of-Distribution Detection With Contrastive Learning (2210.01742v4)

Published 4 Oct 2022 in cs.LG and cs.CV

Abstract: Handling out-of-distribution (OOD) samples has become a major stake in the real-world deployment of machine learning systems. This work explores the use of self-supervised contrastive learning to the simultaneous detection of two types of OOD samples: unseen classes and adversarial perturbations. First, we pair self-supervised contrastive learning with the maximum mean discrepancy (MMD) two-sample test. This approach enables us to robustly test whether two independent sets of samples originate from the same distribution, and we demonstrate its effectiveness by discriminating between CIFAR-10 and CIFAR-10.1 with higher confidence than previous work. Motivated by this success, we introduce CADet (Contrastive Anomaly Detection), a novel method for OOD detection of single samples. CADet draws inspiration from MMD, but leverages the similarity between contrastive transformations of a same sample. CADet outperforms existing adversarial detection methods in identifying adversarially perturbed samples on ImageNet and achieves comparable performance to unseen label detection methods on two challenging benchmarks: ImageNet-O and iNaturalist. Significantly, CADet is fully self-supervised and requires neither labels for in-distribution samples nor access to OOD examples.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Adversarial example detection using latent neighborhood graph. In International Conference on Computer Vision, 2021.
  2. L. Bergman and Y. Hoshen. Classification-based anomaly detection for general data. arXiv preprint arXiv:2005.02359, 2020.
  3. N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In Symposium on Security and Privacy, 2017.
  4. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020a.
  5. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020b.
  6. X. Cheng and A. Cloninger. Classification logit two-sample testing by neural networks. arXiv preprint arXiv:1909.11298, 2019.
  7. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392, 2018.
  8. Fast two-sample testing with analytic representations of probability measures. arXiv preprint arXiv:1506.04725, 2015.
  9. Image anomaly detection with generative adversarial networks. In Joint european conference on machine learning and knowledge discovery in databases, 2018.
  10. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.
  11. Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, 2019.
  12. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410, 2017.
  13. Maximum mean discrepancy test is aware of adversarial attacks. In M. Meila and T. Z. 0001, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3564–3575. PMLR, 2021. URL http://proceedings.mlr.press/v139/gao21b.html.
  14. I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In Advances in Neural Information Processing Systems, 2018.
  15. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
  16. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03263, 2019.
  17. A fast, consistent kernel two-sample test. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems, 2009.
  18. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723–773, 2012.
  19. M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In International Conference on Artificial Intelligence and Statistics, 2010.
  20. Dimensionality reduction by learning an invariant mapping. In Computer Vision and Pattern Recognition, 2006.
  21. Momentum contrast for unsupervised visual representation learning. In Conference on Computer Vision and Pattern Recognition, 2020.
  22. D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016.
  23. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132, 2019a.
  24. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, 2019b.
  25. Natural adversarial examples. arXiv preprint arXiv:1907.07174, 2021.
  26. A new defense against adversarial images: Turning a weakness into a strength. In Advances in Neural Information Processing Systems, 2019.
  27. R. Huang and Y. Li. Mos: Towards scaling out-of-distribution detection for large semantic space. arXiv preprint arXiv:2105.01879, 2021.
  28. Interpretable distribution features with maximum testing power. arXiv preprint arXiv:1605.06796, 2016.
  29. Rodd: A self-supervised approach for robust out-of-distribution detection. arXiv preprint arXiv:2204.02553, 2022.
  30. Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In 2010 IEEE Intelligent Vehicles Symposium, pages 486–492, 2010. doi: 10.1109/IVS.2010.5548123.
  31. A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
  32. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, 2018.
  33. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690, 2017.
  34. Learning deep kernels for non-parametric two-sample tests. In International Conference on Machine Learning, 2020a.
  35. H. Liu and P. Abbeel. Hybrid discriminative-generative training via contrastive learning. arXiv preprint arXiv:2007.09070, 2020.
  36. Energy-based out-of-distribution detection. In Advances in Neural Information Processing Systems, 2020b.
  37. Energy-based out-of-distribution detection. arXiv preprint arXiv:2010.03759, 2020c.
  38. J. Lust and A. P. Condurache. Gran: an efficient gradient-norm based detector for adversarial and misclassified examples. arXiv preprint arXiv:2004.09179, 2020.
  39. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint arXiv:1801.02613, 2018a.
  40. Characterizing adversarial subspaces using local intrinsic dimensionality. CoRR, abs/1801.02613, 2018b. URL http://arxiv.org/abs/1801.02613.
  41. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  42. On detecting adversarial perturbations. arXiv preprint arXiv:1702.04267, 2017.
  43. Self-supervised learning for generalizable out-of-distribution detection. In AAAI Conference on Artificial Intelligence, 2020.
  44. Do deep generative models know what they don’t know? arXiv preprint arXiv:1810.09136, 2018.
  45. Detecting out-of-distribution inputs to deep generative models using a test for typicality. arXiv preprint arXiv:1906.02994, 2019.
  46. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
  47. N. Papernot and P. McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765, 2018.
  48. Ocgan: One-class novelty detection using gans with constrained latent representations. In Conference on Computer Vision and Pattern Recognition, 2019.
  49. Generative probabilistic novelty detection with adversarial autoencoders. In Advances in neural information processing systems, 2018.
  50. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019.
  51. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, 2019.
  52. Deep one-class classification. In International Conference on Machine Learning, 2018.
  53. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. arXiv preprint arXiv:1703.05921, 2017a.
  54. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference on information processing in medical imaging, 2017b.
  55. Support vector method for novelty detection. In Advances in neural information processing systems, 1999.
  56. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=v5gjXpmR8J.
  57. Input complexity and out-of-distribution detection with likelihood-based generative models. arXiv preprint arXiv:1909.11480, 2019.
  58. React: Out-of-distribution detection with rectified activations. arXiv preprint arXiv:2111.12797, 2021.
  59. Generative models and model criticism via optimized maximum mean discrepancy. arXiv preprint arXiv:1611.04488, 2016.
  60. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
  61. A. Uwimana1 and R. Senanayake. Out of distribution detection and adversarial attacks on deep neural networks for robust medical image analysis. arXiv preprint arXiv:2107.04882, 2021.
  62. Vim: Out-of-distribution with virtual-logit matching. In Conference on Computer Vision and Pattern Recognition, 2022.
  63. Learning deep kernels for exponential family densities. In International Conference on Machine Learning, 2019.
  64. Contrastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.05566, 2020.
  65. Unsupervised feature learning via non-parametric instance discrimination. In Conference on Computer Vision and Pattern Recognition, 2018.
  66. Feature squeezing: Detecting adversarial examples in deep neural networks. CoRR, abs/1704.01155, 2017. URL http://arxiv.org/abs/1704.01155.
  67. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017.
  68. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
  69. Deep structured energy based models for anomaly detection. In International Conference on Machine Learning, 2016.
  70. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations, 2018.
  71. F. Zuo and Q. Zeng. Exploiting the sensitivity of l2 adversarial examples to erase-and-restore. In Asia Conference on Computer and Communications Security, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Charles Guille-Escuret (10 papers)
  2. David Vazquez (73 papers)
  3. Ioannis Mitliagkas (61 papers)
  4. Joao Monteiro (25 papers)
  5. Pau Rodriguez (35 papers)
Citations (3)
View on Semantic Scholar