Evaluating Black-Box Classifiers via Stable Adaptive Two-Sample Inference

Abstract: We consider the problem of evaluating black-box multi-class classifiers. In the standard setup, we observe class labels $Y\in {0,1,\ldots,M-1}$ generated according to the conditional distribution $ Y|X \sim \text{ Multinom}\big(η(X)\big), $ where $X$ denotes the features and $η$ maps from the feature space to the $(M-1)$-dimensional simplex. A black-box classifier is an estimate $\hatη$ for which we make no assumptions about the training algorithm. Given holdout data, our goal is to evaluate the performance of the classifier $\hatη$. Recent work suggests treating this as a goodness-of-fit problem by testing the hypothesis $H_0: ρ((X,Y),(X',Y')) \le δ$, where $ρ$ is some metric between two distributions, and $(X',Y')\sim P_X\times \text{ Multinom}(\hatη(X))$. Combining ideas from algorithmic fairness, Neyman-Pearson lemma, and conformal p-values, we propose a new methodology for this testing problem. The key idea is to generate a second sample $(X',Y') \sim P_X \times \text{ Multinom}\big(\hatη(X)\big)$ allowing us to reduce the task to two-sample conditional distribution testing. Using part of the data, we train an auxiliary binary classifier called a distinguisher to attempt to distinguish between the two samples. The distinguisher's ability to differentiate samples, measured using a rank-sum statistic, is then used to assess the difference between $\hatη$ and $η$ . Using techniques from cross-validation central limit theorems, we derive an asymptotically rigorous test under suitable stability conditions of the distinguisher.

• The paper introduces a novel conditional goodness-of-fit test to evaluate black-box classifiers using adaptive two-sample inference.
• It combines conformal inference, Neyman-Pearson testing, and stability analysis to achieve valid Type I error control and robust power.
• Empirical validation on synthetic and real datasets demonstrates the method’s effectiveness in detecting subtle distributional mismatches and fairness issues.

Evaluating Black-Box Classifiers via Stable Adaptive Two-Sample Inference

Problem Formulation and Motivations

This paper introduces a rigorous framework for the evaluation of black-box multi-class classifiers, targeting scenarios where the internal mechanics of $\hat{\eta}$—a learned conditional probability function—are not transparent. Classical approaches to classifier evaluation often rely on fixed performance metrics (e.g., accuracy, AUC, cross-entropy), which provide limited diagnostic power in settings where distributional shifts, structured dependencies, or adversarial features may exist. The authors reconceptualize evaluation as a conditional goodness-of-fit (GoF) problem: does the classifier $\hat{\eta}$ generate predictions whose joint distribution with $X$ is indistinguishable from data generated by the ground truth $\eta$? The null hypothesis is framed as $H_0: \rho((X,Y), (X',Y')) \leq \delta$, using an explicit metric $\rho$. This sharpens evaluation to a two-sample conditional test, allowing for nonparametric, interpretable, and detection-oriented diagnostics.

Methodology: Stable Adaptive Two-Sample Inference

The central methodological contribution is a test procedure that synthesizes techniques from conformal inference, Neyman-Pearson two-sample testing, and algorithmic fairness auditing. The core steps are:

• Sample Generation: Construct two empirical samples: $(X, Y)$ from the observed holdout set sampled via the ground-truth distribution, and $(X', Y')$ where $X'$ is drawn from $P_X$ and $\hat{\eta}$0 is sampled according to the black-box classifier's predicted conditional, i.e., $\hat{\eta}$1.
• Distinguisher Training: An auxiliary binary classifier is trained to discriminate between $\hat{\eta}$2 and $\hat{\eta}$3. The classifier, termed the distinguisher, is learned on a subset of the joint samples.
• Statistic Construction: Utilizing a rank-sum (U-statistic) framework, the discriminative power of the distinguisher functions as the test statistic.
• Inference via Stability: Leveraging cross-validation CLT results, the paper provides an asymptotically valid test under stability assumptions on the distinguisher. Notably, the test adapts to the capacity of the distinguisher, which can incorporate deep nets or tree ensembles, so long as stability and regularity conditions are satisfied.

This adaptive, data-driven procedure subsumes simple tests (e.g., permutation-based tests on accuracy) as special cases and can operate with any classifier family for the distinguisher.

Theoretical Results

The authors deliver a thorough analysis of Type I and Type II errors in the adaptive two-sample setting. Central contributions include:

• Validity Via Cross-Validation CLT: Under stability conditions for the distinguisher’s learning procedure (including randomized cross-fitting), the test achieves correct Type I error control asymptotically.
• Power Analysis: The test’s power is linked to the distinguishability between $\hat{\eta}$4 and $\hat{\eta}$5, parametrized by the choice of metric $\hat{\eta}$6. In particular, when $\hat{\eta}$7 is close to $\hat{\eta}$8, the power diminishes, reflecting the difficulty of distinguishing two nearly identical conditional distributions.
• Robustness to Black-Box Structure: No structural restrictions (e.g., calibration or specific training regimes) are required on $\hat{\eta}$9, making the framework broadly applicable in real-world settings where classifiers are proprietary or opaque.

Empirical Validation

Simulation studies validate the theoretical claims and demonstrate practical performance. Salient findings include:

• Calibration and Power: On synthetic datasets, the test correctly controls the error rate under the null and exhibits substantial power when the black-box classifier deviates from $X$0 in both overall and class-conditional error.
• Comparison to Standard Metrics: The adaptive two-sample test can detect deficiencies that are invisible to standard single-number metrics, including subtle distributional mismatches and fairness violations.
• Real-World Datasets: Application to public datasets, including UCI benchmarks, demonstrates test sensitivity in the presence of label noise, synthetic covariate shifts, and class imbalance, affirming broad empirical utility.

Implications and Future Directions

This work provides a principled methodology for classifier evaluation, especially suitable for auditing deployed models in high-stakes or regulated contexts. Theoretically, it establishes a foundation for nonparametric conditional inference with black-box mechanisms in multidimensional output spaces. Practically, the framework can guide regulatory auditing, robustness evaluation, and post-deployment monitoring—domains where access to transparent model internals is restricted.

Future research directions could address:

• Extensions to Structured Outputs: Adapting the approach for sequence or graph-valued outputs, where conditional random fields or attention-based mechanisms are prevalent.
• Online Setting and Streaming Data: Investigation into extensions where data are acquired sequentially, and distribution drift is gradual.
• Distinguisher Design: Analyzing the optimality of distinguisher families, including model selection and the impact of over/underfitting on test power.
• Connections to Adversarial Machine Learning: Synthesis with adversarial robustness assessments, leveraging the two-sample framework to detect exploitable vulnerabilities.

Conclusion

The paper establishes a rigorous, adaptive, and stable test for evaluating black-box multi-class classifiers using a two-sample inference paradigm that is agnostic to classifier internals. Through both theoretical guarantees and empirical studies, it demonstrates that subtle conditional distributional mismatches can be reliably detected—facilitating robust model assessment in both research and regulatory applications. This methodology constitutes a significant step towards principled, nonparametric validation of black-box prediction mechanisms.