---
title: Two-Sample Neural Classifier Test
url: https://www.emergentmind.com/topics/two-sample-neural-classifier-test
type: topic
---

# Two-Sample Neural Classifier Test

A Two-Sample Neural Classifier Test is a two-sample procedure that recasts the hypothesis \(H_0:P=Q\) versus \(H_1:P\neq Q\) as a supervised prediction problem on pooled, artificially labeled data. Given samples \(X_1,\dots,X_n \sim P\) and \(Y_1,\dots,Y_m \sim Q\), one pools the observations, assigns a binary label indicating source membership, and trains a neural network or other probabilistic classifier to predict that label from the observation. Under \(H_0\), the pooled feature-label pair satisfies \(X \Indep Y\); under \(H_1\), the label is predictable from the feature, so \(X \cancel{\Indep} Y\). In this framework, held-out accuracy, logit or probability scores, witness mean discrepancies, likelihood-ratio-style statistics, and e-values become test statistics, while calibration may proceed through exact binomial laws, permutation, asymptotic approximations, or e-processes with optional stopping guarantees [1610.06545, 2210.13027].

## 1. Formal setup and decision-theoretic interpretation

The formal two-sample problem is to test
\[
H_0:P=Q \qquad \text{versus} \qquad H_1:P\neq Q.
\]
The classifier formulation pools the two samples and assigns labels \(L=1\) to points from \(P\) and \(L=0\) to points from \(Q\). The resulting regression function is
\[
\eta(x)=P(L=1\mid X=x)=\frac{\pi f(x)}{\pi f(x)+(1-\pi)g(x)},
\]
where \(\pi=P(L=1)\), \(f\) is the density of \(P\), and \(g\) is the density of \(Q\). This yields the odds-likelihood identity
\[
\frac{f(x)}{g(x)}=\frac{1-\pi}{\pi}\cdot \frac{\eta(x)}{1-\eta(x)}.
\]
Accordingly, a probabilistic classifier estimating \(\eta(x)\) is estimating a surrogate for the likelihood ratio. In the balanced case, the Bayes-optimal score is
\[
s^*(x)=\frac{p(x)}{p(x)+q(x)},
\qquad
\log\frac{p(x)}{q(x)}=\log\frac{s^*(x)}{1-s^*(x)}.
\]
These identities make classifier-based testing simultaneously a density-ratio, independence-testing, and discriminative inference procedure [1909.07836, 1610.06545].

The same perspective underlies witness-function formulations. If \(w:\mathcal X\to \mathbb R\) is a measurable witness, then the population discrepancy is
\[
\Delta(w)=\mathbb E_{X\sim P}[w(X)]-\mathbb E_{Y\sim Q}[w(Y)],
\]
with empirical counterpart
\[
T_{n,m}(w)=\frac1n\sum_{i=1}^n w(X_i)-\frac1m\sum_{j=1}^m w(Y_j).
\]
Under \(H_0\), \(\Delta(w)=0\). This places neural classifier tests in the broader family of mean-discrepancy and integral-probability-metric procedures, but with the witness learned from data rather than fixed a priori [2206.08843].

## 2. Classical classifier two-sample tests

The standard classifier two-sample test, often abbreviated C2ST, splits the pooled labeled dataset into training and test subsets, trains a classifier on the training portion, and computes a test statistic on the untouched test portion. The paper "Revisiting Classifier Two-Sample Tests" [1610.06545] focuses on hard accuracy, but the literature also uses soft probabilities, logits, cross-entropy, and AUC. For balanced classes and a held-out test set \(D_{\mathrm{te}}=\{(z_i,y_i)\}_{i=1}^N\), the canonical hard-accuracy statistic is
\[
T_{\mathrm{acc}}=\frac1N\sum_{i=1}^N \mathbf 1\{\mathbf I[s_i\ge \tau]=y_i\},
\]
with \(\tau=1/2\) under equal priors.

Under \(H_0\) and balanced classes, \(K:=NT_{\mathrm{acc}}\sim \mathrm{Binomial}(N,1/2)\), so the null is exact and simple. More generally, permutation calibration is standard: labels on the held-out set are shuffled, the statistic is recomputed, and the p-value is the fraction of permuted statistics exceeding the observed one. Kim, Ramdas, Singh, and Wasserman showed that this permutation-based approach is finite-sample valid for either half-permutation or full-permutation procedures, and that a Gaussian approximation based on held-out class-wise errors is asymptotically valid for any classifier satisfying their stability assumptions [1602.02210].

Before later variants are introduced, three classical facts are central. First, the test must preserve a strict split between training and evaluation data; otherwise Type I error control is compromised. Second, the method is consistent if the classifier approaches Bayes risk, because the held-out accuracy then exceeds chance by a fixed effect size. Third, accuracy is interpretable: for the Bayes classifier,
\[
A_*=\frac{1+\mathrm{TV}(P,Q)}{2},
\]
so accuracy above \(1/2\) directly quantifies total variation distance [1610.06545].

| Statistic | Form | Typical calibration |
|---|---|---|
| Hard accuracy | \(T_{\mathrm{acc}}\) on held-out labels | Exact binomial under balanced classes |
| Soft score | Mean predicted probability of the true label | Permutation |
| Logit mean difference | \(\left|\bar f(S_P^{te})-\bar f(S_Q^{te})\right|\) | Permutation |
| AUC | ROC-area from held-out scores | Permutation |

A recurrent limitation of this classical form is that a single split wastes data: only the test split contributes directly to the final statistic, while the training split is “used up” in learning. Standard p-values are also not anytime valid, so repeated looks at accumulating data inflate Type I error unless additional machinery is introduced [2210.13027].

## 3. E-values, sequential validity, and E-C2ST

The E-value Classifier Two-Sample Test (E-C2ST) replaces the one-shot p-value logic of classical C2ST with batchwise e-factors whose product forms an e-process. A conditional e-variable with respect to a null class \(H\) is a nonnegative measurable function \(E(x\mid z)\) such that
\[
\int E(x\mid z)\,P(dx\mid z)\le 1
\]
for all \(P\in H\) and all conditioning values \(z\). If data arrive sequentially and \(E_t:=E(X_t\mid X_1,\dots,X_{t-1})\), then
\[
E(\le t):=\prod_{\ell=1}^t E_\ell
\]
is a nonnegative supermartingale under \(H_0\). Ville’s inequality yields
\[
P_{H_0}\!\left(\exists t\ge 1:\,E(\le t)\ge \frac1\alpha\right)\le \alpha,
\]
so the rule “reject \(H_0\) when \(E(\le t)\ge 1/\alpha\)” is \(\alpha\)-safe under optional stopping [2210.13027].

E-C2ST specializes this construction to binary source labels. Under the independence formulation \(H_0:X\Indep Y\) versus \(H_1:X\cancel{\Indep}Y\), the null model is Bernoulli with unknown class prior and the alternative is Bernoulli with neural-network logit \(g_\theta(X)\):
\[
H_0: P(Y\mid X)=\mathrm{Bern}(q_\theta),\qquad
H_1: P(Y\mid X)=\mathrm{Bern}(\sigma(g_\theta(X))).
\]
After partitioning the data into batches \(I^{(1)},\dots,I^{(M)}\), batch \(m\) uses a classifier trained on previous batches and the current-batch null maximum-likelihood estimate \(\hat q^{(m)}=N_1^{(m)}/N^{(m)}\). The batch e-factor is the per-batch likelihood ratio
\[
E^{(m)}
=
\prod_{n\in I^{(m)}}
\left[\frac{\sigma(g_\theta(x_n))}{\hat q^{(m)}}\right]^{y_n}
\left[\frac{1-\sigma(g_\theta(x_n))}{1-\hat q^{(m)}}\right]^{1-y_n},
\]
and the cumulative evidence is \(E(\le m)=\prod_{\ell=1}^m E^{(\ell)}\). A bounded mixture alternative \(\tilde p_A=\lambda_m p_0+(1-\lambda_m)p_A\) ensures bounded log-e-values and retains e-variable validity via convexity. The training protocol rotates batches through training and validation roles, updates \(\lambda_{m+1}\) by maximizing previous-batch log-evidence, and stops as soon as \(E(\le m)\ge 1/\alpha\) [2210.13027].

This construction changes the role of sample splitting. Rather than a single train/test split, data are consumed prequentially, and every new batch can both improve the predictor and contribute multiplicative evidence. The paper reports that this multiple-batch strategy increases power while keeping Type I error well below the desired significance level, thereby addressing both data inefficiency and optional-stopping failure modes of standard C2ST [2210.13027].

## 4. Continuous witnesses, probability statistics, conformalization, and label efficiency

A common misconception is that classifier two-sample tests are fundamentally tests of held-out accuracy. Several later developments instead treat the classifier output as a continuous witness. In the AutoML two-sample test, one trains \(h:\mathcal X\to\mathbb R\) by minimizing weighted squared loss
\[
L_{P,Q,c}(h)=(1-c)\,\mathbb E_{X\sim P}(1-h(X))^2+c\,\mathbb E_{Y\sim Q}h(Y)^2,
\]
and then tests using the mean discrepancy of \(h\) or of the centered score \(f(x)=2h(x)-1\). Any minimizer \(h^*\) of this squared loss maximizes \(\mathrm{SNR}(h)\), and in the balanced case
\[
h^*(x)=\frac{p(x)}{p(x)+q(x)},\qquad
f^*(x)=\frac{p(x)-q(x)}{p(x)+q(x)}.
\]
Plugging \(f^*\) into the population discrepancy yields triangular discrimination. The same paper shows that, on a balanced held-out set, accuracy is just the mean discrepancy of a binary witness,
\[
\mathrm{Acc}=\frac12+\frac12 T_{n,m}(f),
\]
so replacing binary outputs with continuous scores reduces variance and improves power [2206.08843].

A closely related probability-based construction uses log-odds directly. If \(\hat\eta(x)\) estimates the class probability, then
\[
W_1=\frac1n\sum_{i=1}^n \log\frac{\hat\eta(X_i)}{1-\hat\eta(X_i)}-\log\frac{n}{m}
\]
approximates a likelihood-ratio statistic, while
\[
W_2=\frac1N\sum_{i=1}^N (\hat\eta(X_i)-\bar p)^2
\]
tests whether the classification probability is constant. Under uniform consistency of \(\hat\eta\), the \(W_1\)-based test is asymptotically most powerful in the sense stated in the paper [1909.07836].

Conformal variants shift the emphasis from probability calibration to score ranking. In the conformal C2ST for neural posterior validation, a classifier score \(s\) is converted into per-point conformal p-values by ranking each test score among calibration scores from \(p\). Under exchangeability, these p-values are exactly \(\mathrm{Unif}[0,1]\) under \(H_0\), so any one-sample uniformity test yields exact finite-sample Type-I control. The paper further proves that power degrades gently with score error: if the density-ratio estimate has \(L^2\) error \(\epsilon\), then the expected conformal p-values differ from the oracle ones by at most \(O(\epsilon^{2/3})\) [2507.17026].

Label-costly settings admit a further extension. The label-efficient framework begins with a uniformly labeled seed set, trains a probabilistic classifier \(Q(z\mid s)\), then uses bimodal querying to request labels for items with the largest \(Q(Z=1\mid s)\) and the largest \(Q(Z=0\mid s)\). The final test can be a batch permutation C2ST or a sequential likelihood-ratio-style statistic
\[
u_n=\prod_{i=1}^n \frac{\hat P(Z_i)}{Q_i(Z_i\mid S_i)},
\]
with anytime-valid guarantee \(P_0(\exists n\le N_q: U_n\le \alpha)\le \alpha\) [2501.03568].

## 5. Relations to kernels, IPMs, learned representations, and interpretable variants

Classifier tests are not isolated from kernel and IPM methodology; several papers make the equivalence explicit. In "Learning Deep Kernels for Non-Parametric Two-Sample Tests" [2002.09116], accuracy-based C2ST is shown to be exactly an MMD test with a sign kernel,
\[
k_f^{(S)}(x,y)=\frac14\,\mathbf 1\{f(x)>0\}\mathbf 1\{f(y)>0\},
\]
while logit-mean C2ST is MMD with the linear kernel
\[
k_f^{(L)}(x,y)=f(x)f(y).
\]
The same paper argues that deep-kernel MMD strictly generalizes C2ST by learning a spatially non-homogeneous kernel \(k_\omega\) and optimizing the power proxy \(\hat J_\lambda\) directly rather than cross-entropy.

An IPM formulation is developed for data supported on a low-dimensional manifold. There the test statistic is
\[
d_{\mathcal F}(P,Q)=\sup_{f\in\mathcal F}\{\mathbb E_P[f]-\mathbb E_Q[f]\},
\]
with \(\mathcal F\) either a Hölder class or a ReLU network class approximating it. The resulting neural-network IPM test attains type-II risk of order \(n^{-(s+\beta)/d}\), and the performance depends on intrinsic dimension \(d\) rather than ambient dimension \(D\) [2205.02043].

A distinct branch uses learned deep representations but abandons the classifier statistic itself. "Two-sample Testing Using Deep Learning" [1910.06239] trains a feature map \(\phi_N\) on auxiliary supervised or unsupervised tasks and then applies asymptotic location tests on hidden-layer means:
\[
S_{n,m}=\frac{nm}{n+m}\|\overline{\phi_N(X_n)}-\overline{\phi_N(Y_m)}\|^2
\]
and
\[
T_{n,m}=\frac{nm}{n+m} D_{n,m}^\top \hat\Sigma_{n,m}^{-1} D_{n,m}.
\]
These statistics are linear-time in sample size and asymptotically control the Type I error rate.

Interpretability has motivated further alternatives. The self-organizing-map two-sample test trains a SOM on pooled unlabeled data, projects both samples to the grid, and compares per-neuron hit histograms by
\[
\chi^2=\sum_{i=1}^M \frac{(K_1R_i-K_2S_i)^2}{R_i+S_i}\,\mathbf 1\{R_i+S_i>0\}.
\]
Because the map supports hit histograms, component planes, and U-matrices, it can reveal where and how \(P\) and \(Q\) differ rather than merely rejecting equality [2212.08960].

The same reduction-to-two-samples idea has also been used for evaluating black-box multiclass classifiers. One samples \((X',Y')\sim P_X\times \mathrm{Multinom}(\hat\eta(X))\), trains a distinguisher between \((X,Y)\) and \((X',Y')\), and evaluates separability by a rank-sum or AUC statistic under cross-fitting and stability conditions [2604.05470].

## 6. Empirical behavior, training dynamics, and limitations

Across the literature, a stable empirical pattern is that richer continuous statistics usually outperform hard accuracy, provided validity is preserved. For E-C2ST, the reported results are explicit: on Blob, the method reaches maximum power with fewer samples while maintaining type I error strictly below \(\alpha=0.05\); on KDEF, it achieves \(\mathrm{power}=1\) fastest and keeps type I error lower than baselines; and on Corrupted MNIST, it exhibits superior power across corruption levels while maintaining type I error below \(\alpha\) in the \(p=0\) case [2210.13027]. AutoML witness tests likewise show that continuous witness statistics outperform binary-output variants; on distribution-shift benchmarks, AutoML (bin) consistently underperforms, while continuous-witness AutoML tests outperform MMDAgg and MMD-D in most regimes except the very smallest \(n\) [2206.08843].

Deep-kernel and deep-representation results reinforce the same conclusion in different form. Deep kernels generally outperform C2ST variants on Blob, HDGM, Higgs, MNIST, and CIFAR-10.1, especially when differences are subtle, local, or highly structured [2002.09116]. Deep representation tests on audio, images, and MRI report decreases in type-II error rate of up to 35 percentage points relative to kernel methods and classifier two-sample tests [1910.06239].

Training dynamics have also become an object of theory. The NTK analysis of neural network C2ST derives a theoretical minimum training time needed to detect a deviation-level and a theoretical maximum training time before the NTK test detects that deviation-level. In the resulting small-time regime, the times needed to detect the same deviation-level in the null and alternative scenarios are well-separated, which justifies early-stopping strategies rather than training to convergence [2407.04806].

The principal limitations remain consistent across papers. Standard one-shot C2ST wastes data through a single split and is not anytime valid. All classifier-based procedures are sensitive to representation quality, overfitting, and leakage between training and evaluation. Batchwise class imbalance must be handled through the null class prior rather than by fixing \(q=0.5\) without justification. Very small batches can destabilize prequential training. Permutation calibration, cross-fitting, or conformalization usually improves validity, but at additional computational cost. Finally, in simple, well-specified low-dimensional settings, classical parametric tests or carefully tuned kernel tests may remain preferable; neural classifier tests are primarily designed for complex, high-dimensional regimes in which representation learning or adaptive scoring is a substantive advantage [2210.13027, 1602.02210, 2206.08843].

In contemporary usage, the term therefore denotes not a single test but a family of related procedures: hard-accuracy C2ST, logit and probability tests, continuous witness tests, e-value and sequential tests, conformal score-rank tests, and kernel or IPM formulations that reinterpret the classifier as a learned witness. Their unifying principle is the same: two-sample inference is reduced to source discrimination, and the statistical problem becomes one of turning predictive signal into valid evidence for \(P\neq Q\).

Source: https://www.emergentmind.com/topics/two-sample-neural-classifier-test