Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Out-of-Distribution Detection via Deep Multi-Comprehension Ensemble (2403.16260v2)

Published 24 Mar 2024 in cs.LG, cs.AI, cs.CV, and stat.ML

Abstract: Recent research underscores the pivotal role of the Out-of-Distribution (OOD) feature representation field scale in determining the efficacy of models in OOD detection. Consequently, the adoption of model ensembles has emerged as a prominent strategy to augment this feature representation field, capitalizing on anticipated model diversity. However, our introduction of novel qualitative and quantitative model ensemble evaluation methods, specifically Loss Basin/Barrier Visualization and the Self-Coupling Index, reveals a critical drawback in existing ensemble methods. We find that these methods incorporate weights that are affine-transformable, exhibiting limited variability and thus failing to achieve the desired diversity in feature representation. To address this limitation, we elevate the dimensions of traditional model ensembles, incorporating various factors such as different weight initializations, data holdout, etc., into distinct supervision tasks. This innovative approach, termed Multi-Comprehension (MC) Ensemble, leverages diverse training tasks to generate distinct comprehensions of the data and labels, thereby extending the feature representation field. Our experimental results demonstrate the superior performance of the MC Ensemble strategy in OOD detection compared to both the naive Deep Ensemble method and a standalone model of comparable size. This underscores the effectiveness of our proposed approach in enhancing the model's capability to detect instances outside its training distribution.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Deep ensembles work, but are they necessary? Advances in Neural Information Processing Systems, 35:33646–33660, 2022.
  2. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T.
  3. Mostly harmless econometrics: An empiricist’s companion. Princeton university press, 2009.
  4. Breiman, L. Bagging predictors. Machine learning, 24:123–140, 1996.
  5. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  9650–9660, 2021.
  6. Jigsaw clustering for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  11526–11535, 2021a.
  7. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PMLR, 2020.
  8. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  9640–9649, 2021b.
  9. Enhancing out-of-distribution detection in natural language understanding via implicit layer ensemble. arXiv preprint arXiv:2210.11034, 2022.
  10. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  3606–3613, 2014.
  11. Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
  12. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  13. Extremely simple activation shaping for out-of-distribution detection. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=ndYXTEL6cZz.
  14. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  15. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019.
  16. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259–3269. PMLR, 2020.
  17. Experiments with a new boosting algorithm. In icml, volume 96, pp.  148–156. Citeseer, 1996.
  18. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728, 2018.
  19. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  20. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9729–9738, 2020.
  21. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  16000–16009, 2022.
  22. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017.
  23. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019.
  24. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34:677–689, 2021.
  25. Protecting dnns from theft using an ensemble of diverse models. In International Conference on Learning Representations, 2021.
  26. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673, 2020.
  27. Bias plus variance decomposition for zero-one loss functions. In ICML, volume 96, pp.  275–83. Citeseer, 1996.
  28. Learning multiple layers of features from tiny images. 2009.
  29. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
  30. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
  31. kfolden: k-fold ensemble for out-of-distribution detection-fold ensemble for out-of-distribution detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.  3102–3115, 2021.
  32. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=H1VGkIxRZ.
  33. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33:21464–21475, 2020.
  34. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
  35. Ensemble distribution distillation. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BygSP6Vtvr.
  36. How to exploit hyperspherical embeddings for out-of-distribution detection? In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=aEFaE0W5pAd.
  37. Reading digits in natural images with unsupervised feature learning. 2011.
  38. Agree to disagree: Diversity through disagreement for better transferability. arXiv preprint arXiv:2202.04414, 2022.
  39. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  40. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems, 32, 2019.
  41. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
  42. Dice: Leveraging sparsification for out-of-distribution detection. In European Conference on Computer Vision, 2022.
  43. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144–157, 2021.
  44. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pp. 20827–20840. PMLR, 2022.
  45. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 33:11839–11852, 2020.
  46. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 33:15300–15311, 2020.
  47. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  8769–8778, 2018.
  48. Out-of-distribution detection using an ensemble of self supervised leave-out classifiers. In Proceedings of the European Conference on Computer Vision (ECCV), pp.  550–564, 2018.
  49. Out-of-distribution detection with implicit outlier transformation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=hdghx6wbGuD.
  50. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp.  3485–3492. IEEE, 2010.
  51. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755, 2015.
  52. Boosting out-of-distribution detection with multiple pre-trained models. arXiv preprint arXiv:2212.12720, 2022.
  53. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
  54. Block selection method for using feature norm in out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  15701–15711, 2023.
  55. Neural ensemble search for uncertainty estimation and dataset shift. Advances in Neural Information Processing Systems, 34:7898–7911, 2021.
  56. Out-of-distribution detection based on in-distribution data patterns memorization with modern hopfield energy. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=KkazG4lgKL.
  57. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets