Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

How to Overcome Curse-of-Dimensionality for Out-of-Distribution Detection? (2312.14452v1)

Published 22 Dec 2023 in cs.LG

Abstract: Machine learning models deployed in the wild can be challenged by out-of-distribution (OOD) data from unknown classes. Recent advances in OOD detection rely on distance measures to distinguish samples that are relatively far away from the in-distribution (ID) data. Despite the promise, distance-based methods can suffer from the curse-of-dimensionality problem, which limits the efficacy in high-dimensional feature space. To combat this problem, we propose a novel framework, Subspace Nearest Neighbor (SNN), for OOD detection. In training, our method regularizes the model and its feature representation by leveraging the most relevant subset of dimensions (i.e. subspace). Subspace learning yields highly distinguishable distance measures between ID and OOD data. We provide comprehensive experiments and ablations to validate the efficacy of SNN. Compared to the current best distance-based method, SNN reduces the average FPR95 by 15.96% on the CIFAR-100 benchmark.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. On the surprising behavior of distance metrics in high dimensional space. In International conference on database theory.
  2. Adaptive dropout for training deep neural networks. In Advances in neural information processing systems.
  3. Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection. In Proceedings of the International Conference on Machine Learning.
  4. Discriminative out-of-distribution detection for semantic segmentation. In arXiv preprint arXiv:1808.07703.
  5. When is “nearest neighbor” meaningful? In International conference on database theory.
  6. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  7. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  8. Improved regularization of convolutional neural networks with cutout. In arXiv preprint arXiv:1708.04552.
  9. Dream the Impossible: Outlier Imagination with Diffusion Models. In Advances in Neural Information Processing Systems.
  10. Unknown-Aware Object Detection: Learning What You Don’t Know from Videos in the Wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  11. VOS: Learning What You Don’t Know by Virtual Outlier Synthesis. In Proceedings of the International Conference on Learning Representations.
  12. Selectivenet: A deep neural network with an integrated reject option. In Proceedings of the International Conference on Machine Learning.
  13. Learning sparse networks using targeted dropout. In arXiv preprint arXiv:1905.13678.
  14. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  15. A Novel Data Augmentation Technique for Out-of-Distribution Sample Detection using Compounded Corruptions. In Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases.
  16. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  17. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the International Conference on Learning Representations.
  18. Deep anomaly detection with outlier exposure. In Proceedings of the International Conference on Learning Representations.
  19. What is the nearest neighbor in high dimensional spaces? In 26th Internat. Conference on Very Large Databases.
  20. Improving neural networks by preventing co-adaptation of feature detectors. In arXiv preprint arXiv:1207.0580.
  21. Ho, T. K. 1998. Nearest neighbors in random subspaces. In Joint IAPR international workshops on statistical techniques in pattern recognition (SPR) and structural and syntactic pattern recognition (SSPR).
  22. Can shared-neighbor distances defeat the curse of dimensionality? In International conference on scientific and statistical database management.
  23. Efficient algorithms for similarity search in axis-aligned subspaces. In International Conference on Similarity Search and Applications.
  24. Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  25. Densely connected convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  26. On the importance of gradients for detecting distributional shifts in the wild. In Advances in Neural Information Processing Systems.
  27. MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  28. Harnessing Out-Of-Distribution Examples via Augmenting Content and Style. In Proceedings of the International Conference on Learning Representations.
  29. Subspace nearest neighbor search-problem statement, approaches, and discussion. In International Conference on Similarity Search and Applications.
  30. Training OOD Detectors in their Natural Habitats. In Proceedings of the International Conference on Machine Learning.
  31. Outlier detection in axis-parallel subspaces of high dimensional data. In Pacific-asia conference on knowledge discovery and data mining.
  32. Clustering high-dimensional data: A survey on subspace clustering, pattern-based clustering, and correlation clustering. ACM Transactions on Knowledge Discovery from Data.
  33. Learning multiple layers of features from tiny images.
  34. Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. In Proceedings of the International Conference on Learning Representations.
  35. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems.
  36. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In Proceedings of the International Conference on Learning Representations.
  37. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision.
  38. Computational methods of feature selection.
  39. Energy-based Out-of-distribution Detection. In Advances in Neural Information Processing Systems.
  40. Mahalanobis, P. C. 1936. On the generalized distance in statistics. National Institute of Science of India.
  41. Towards neural networks that provably know when they don’t know. In Proceedings of the International Conference on Learning Representations.
  42. POEM: Out-of-Distribution Detection with Posterior Sampling. In Proceedings of the International Conference on Machine Learning.
  43. How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection? In Proceedings of the International Conference on Learning Representations.
  44. Provable Guarantees for Understanding Out-of-distribution Detection. In Proceedings of the AAAI Conference on Artificial Intelligence.
  45. Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems.
  46. When does label smoothing help? In Advances in neural information processing systems.
  47. Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence.
  48. Reading Digits in Natural Images with Unsupervised Feature Learning.
  49. Measuring Calibration in Deep Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.
  50. SSD: A Unified Framework for Self-Supervised Outlier Detection. In Proceedings of the International Conference on Learning Representations.
  51. RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection. In Advances in Neural Information Processing Systems.
  52. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research.
  53. ReAct: Out-of-distribution Detection With Rectified Activations. In Advances in Neural Information Processing Systems.
  54. DICE: Leveraging Sparsification for Out-of-Distribution Detection. In Proceedings of European Conference on Computer Vision.
  55. Out-of-Distribution Detection with Deep Nearest Neighbors. In Proceedings of the International Conference on Machine Learning.
  56. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in neural information processing systems.
  57. Learning to Augment Distributions for Out-of-Distribution Detection. In Advances in Neural Information Processing Systems.
  58. Out-of-distribution Detection with Implicit Outlier Transformation. In Proceedings of the International Conference on Learning Representations.
  59. Out-of-Distribution Detection via Conditional Kernel Independence Model. In Advances in Neural Information Processing Systems.
  60. Mitigating Neural Network Overconfidence with Logit Normalization. In Proceedings of the International Conference on Machine Learning.
  61. SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
  62. Leveraging sparse linear layers for debuggable deep networks. In Proceedings of the International Conference on Machine Learning.
  63. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. In arXiv preprint arXiv:1504.06755.
  64. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. In arXiv preprint arXiv:1506.03365.
  65. Analysis of knn density estimation. IEEE Transactions on Information Theory.
  66. Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources. In Advances in Neural Information Processing Systems.
  67. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now