Testing for Outliers with Conformal p-values (2104.08279v3)

Abstract: This paper studies the construction of p-values for nonparametric outlier detection, taking a multiple-testing perspective. The goal is to test whether new independent samples belong to the same distribution as a reference data set or are outliers. We propose a solution based on conformal inference, a broadly applicable framework which yields p-values that are marginally valid but mutually dependent for different test points. We prove these p-values are positively dependent and enable exact false discovery rate control, although in a relatively weak marginal sense. We then introduce a new method to compute p-values that are both valid conditionally on the training data and independent of each other for different test points; this paves the way to stronger type-I error guarantees. Our results depart from classical conformal inference as we leverage concentration inequalities rather than combinatorial arguments to establish our finite-sample guarantees. Furthermore, our techniques also yield a uniform confidence bound for the false positive rate of any outlier detection algorithm, as a function of the threshold applied to its raw statistics. Finally, the relevance of our results is demonstrated by numerical experiments on real and simulated data.

• The paper introduces a method for testing outliers using conformal p-values, ensuring rigorous statistical validity both marginally and conditionally on training data.
• The research shows that standard multiple-testing procedures may fail due to p-value dependencies, but methods like Benjamini-Hochberg successfully control the false discovery rate.
• Numerical results show the method effectively controls the conditional false discovery rate and detects outliers in real data, with applications in fields like fraud detection.

Nonparametric Outlier Detection Using Conformal p-values

The paper "Testing for Outliers with Conformal p-values" presents a methodological framework for testing outliers using conformal p-values, addressing a significant issue inherent in nonparametric outlier detection: ensuring valid statistical inference in multiple hypothesis testing scenarios. The paper is rooted in conformal inference, known for its flexibility and robust application across different domains without making stringent assumptions about the underlying statistical models.

The authors propose a method for calculating conformal p-values that are rigorous both marginally—over test points—and conditionally on the training data. The essential advancement over traditional conformal methods lies in the conditional validity of these p-values, supporting robust type-I error guarantees, even in complex and high-dimensional data settings. The framework departs from classical combinatorial arguments by leveraging concentration inequalities, offering finite-sample guarantees which are essential in practical applications where data samples are limited.

The paper thoroughly explores the dependencies among conformal p-values, which are mutually dependent due to shared calibration data, challenging traditional multiple-testing procedures. Findings reveal that standard methods like Fisher's combination test can fail under the global null hypothesis due to these dependencies. However, methods like Benjamini-Hochberg (BH) are shown to maintain false discovery rate (FDR) control, beneficial in applications necessitating error rate adjustments across multiple comparisons.

Numerical results exhibit the efficacy of the proposed method in controlling conditional FDR while preserving substantial power in detecting true outliers, as evidenced by experiments on both simulated and real datasets. This is pivotal for applications such as fraud detection, medical diagnostics, and failure monitoring, where the cost and consequences of false identifications are substantial.

Despite the advancement, the approach naturally trades off power for increased conditional inference reliability. Simulation studies underscore that while conditional p-values yield higher confidence in the accuracy of individual practitioners' results, they are inherently more conservative than their marginal counterparts, particularly in large dataset contexts where power is equally pivotal.

In terms of theoretical implications, this work sets the stage for future research into out-of-distribution detection methodologies, especially in AI applications requiring rigorous guarantees about the generalizability and reliability of detection systems. With growing interest in such assurances in machine learning models, the introduction of stronger confidence bounds for false positive rates will be particularly impactful.

Looking forward, this paper opens avenues for the development of enhanced tools integrating conformal inference with modern machine learning frameworks, aiming to strike a harmonious balance between predictive accuracy and statistical validation across diverse application domains.

In conclusion, Bates et al.'s work is a commendable contribution to the field of statistical learning, providing a valuable toolkit for nonparametric outlier detection enriched with robust statistical guarantees, and laying a significant foundation for future explorations in predictive inference.