Out-of-Distribution Detection with Deep Nearest Neighbors (2204.06507v3)

Abstract: Out-of-distribution (OOD) detection is a critical task for deploying machine learning models in the open world. Distance-based methods have demonstrated promise, where testing samples are detected as OOD if they are relatively far away from in-distribution (ID) data. However, prior methods impose a strong distributional assumption of the underlying feature space, which may not always hold. In this paper, we explore the efficacy of non-parametric nearest-neighbor distance for OOD detection, which has been largely overlooked in the literature. Unlike prior works, our method does not impose any distributional assumption, hence providing stronger flexibility and generality. We demonstrate the effectiveness of nearest-neighbor-based OOD detection on several benchmarks and establish superior performance. Under the same model trained on ImageNet-1k, our method substantially reduces the false positive rate (FPR@TPR95) by 24.77% compared to a strong baseline SSD+, which uses a parametric approach Mahalanobis distance in detection. Code is available: https://github.com/deeplearning-wisc/knn-ood.

• The paper introduces a non-parametric deep KNN approach for out-of-distribution detection that avoids Gaussian assumptions.
• The authors validate their method with extensive experiments on CIFAR and ImageNet, achieving a 24.77% reduction in false positive rates compared to state-of-the-art baselines.
• The study provides a theoretical framework showing that deep KNN can approximate the Bayes factor in binary classification, underscoring its robustness and versatility.

Out-of-Distribution Detection with Deep Nearest Neighbors

This paper, authored by Yiyou Sun et al., investigates a non-parametric approach to out-of-distribution (OOD) detection using deep nearest neighbors (KNN). The primary contribution lies in utilizing KNN for OOD detection without imposing distributional assumptions, offering an alternative to traditional parametric methods like Mahalanobis distance, which rely heavily on Gaussian distribution assumptions in feature space.

Key Contributions

1. Non-parametric Density Estimation: The paper introduces KNN as a more flexible method for estimating the likelihood of input data being OOD. This method eschews the restrictions of parametric approaches, widening applicability across varied datasets.
2. Empirical Validation: The authors demonstrate superior performance of the KNN-based approach against competitive baselines, including those employing Mahalanobis distance, across several OOD detection benchmarks. Notably, the paper spans various architectures (CNNs and ViTs) and training frameworks, including contrastive learning.
3. Theoretical Insights: A theoretical framework supports the empirical results, suggesting that KNN can approximate the optimal Bayes factor for binary classification tasks, effectively discriminating between ID and OOD inputs.
4. Robust Across Conditions: The method's robustness is evidenced by its ability to maintain strong OOD detection capabilities across different ID densities and large-scale datasets such as ImageNet, with minimal computational overhead.

Experimental Findings

The authors conducted extensive experiments on CIFAR and ImageNet datasets. In CIFAR benchmarks, KNN notably reduced the false positive rate by 24.77% compared to SSD+, a strong baseline utilizing Mahalanobis distance. When applied to ImageNet, KNN-based methods consistently outperformed parametric counterparts, achieving impressive results without sacrificing inference speed.

A noteworthy finding is KNN's compatibility with contrastive learning frameworks, where it leverages high-quality, compact feature embeddings to enhance detection capabilities. This compatibility underscores KNN's versatility across different training objectives.

Theoretical Justification

The paper provides a theoretical analysis positioning KNN within the broader landscape of density estimation techniques. The approach aligns with Bayes optimal classifiers in certain settings, reinforcing KNN's utility in universal OOD scenarios where data assumptions are minimal.

Implications and Future Directions

The non-parametric nature of KNN makes it attractive for real-world applications where distributional assumptions are impractical or invalid. Practically, its ease of integration and robustness across architectures extend its potential utility in deploying reliable ML models in diverse environments.

Theoretically, this work suggests pathways for further exploration into non-parametric OOD detection methods, potentially inspiring investigations into other distance-based measures or hybrid approaches integrating KNN with parametric insights.

Conclusion

This paper substantially contributes to the field of OOD detection by demonstrating the viability and effectiveness of a non-parametric nearest neighbor approach. The implications are broad, offering researchers and practitioners a flexible tool devoid of the traditional constraints imposed by parametric methods, thus paving the way for innovative applications and further theoretical advancements in OOD detection.