---
title: Likelihood-Free Hypothesis Testing (LFHT)
url: https://www.emergentmind.com/topics/likelihood-free-hypothesis-testing-lfht
type: topic
---

# Likelihood-Free Hypothesis Testing (LFHT)

Searching arXiv for relevant papers on likelihood-free hypothesis testing and related methods.
Likelihood-Free Hypothesis Testing (LFHT) denotes hypothesis-testing procedures in settings where the likelihood is unavailable or intractable, but forward simulation from the model is possible. Across the literature, LFHT replaces explicit likelihood evaluation by test statistics learned or estimated from simulated data, including classifier-derived likelihood-ratio surrogates, odds-ratio estimators, divergence estimators, and kernel discrepancies. In the formulation developed from likelihood-free inference by ratio estimation, the central object is a density ratio between distributions of summary statistics, estimated by logistic regression and then repurposed as a log-likelihood-ratio test statistic [1611.10242]. In a more explicitly frequentist formulation, LFHT is the construction of level-\(\alpha\) tests and confidence sets in simulator-based models by replacing the classical likelihood ratio test with an estimated odds-ratio statistic and calibrating rejection thresholds or p-values from simulation [2002.10399]. Related lines of work show that LFHT can also be implemented by divergence estimation [2405.06397], kernel-based testing [2308.09043], optimal-statistic learning in the presence of nuisance parameters [2203.13079], and classification-based discrepancy measures [1407.4981].

## 1. Conceptual formulation

LFHT addresses binary or composite hypothesis testing when one can sample from model-implied distributions but cannot evaluate \(p(x \mid \theta)\) or the corresponding likelihood in closed form. In the frequentist setting, the objective is to construct tests that control size, attain high power, and yield confidence sets with nominal coverage without requiring explicit likelihood evaluations [2002.10399].

A common setup uses summary statistics \(s(x)\) computed from raw data \(x\). In the ratio-estimation formulation, the marginal distribution of summaries is
\[
p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,
\]
with \(p(\theta)\) a prior and \(p(s \mid \theta)\) induced by the simulator \(p(x \mid \theta)\) via \(s(x)\). The key target is the marginal ratio
\[
r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},
\]
which yields posterior inference through
\[
p(\theta \mid x) \propto p(\theta) r_\theta(s(x)).
\]
This same density-ratio machinery can be redirected toward hypothesis testing by replacing the marginal denominator \(p(s)\) with a competing hypothesis distribution \(p(s \mid H_0)\) or \(p(s \mid H_1)\) [1611.10242].

In the explicit LFHT formulation for two hypotheses, the test statistic is the estimated log-likelihood ratio on summaries,
\[
T(x) = \log \frac{p(s(x) \mid H_1)}{p(s(x) \mid H_0)}
      \approx f_{H_1,H_0}(s(x)) - \log(\pi_1/\pi_0),
\]
where \(f_{H_1,H_0}\) is the logit of a trained classifier discriminating simulated summaries from \(H_1\) and \(H_0\), and \(\pi_1,\pi_0\) are class priors [1611.10242].

The classical baseline remains the likelihood ratio test. For composite hypotheses,
\[
H_0: \theta \in \Theta_0 \quad \text{versus} \quad H_1: \theta \in \Theta_1=\Theta\setminus\Theta_0,
\]
the log-likelihood ratio statistic is
\[
\Lambda(D;\Theta_0) = \log \frac{\sup_{\theta\in\Theta_0} L(D;\theta)}{\sup_{\theta\in\Theta} L(D;\theta)}.
\]
LFHT seeks to reconstruct this object, or an equivalent decision rule, without direct access to \(L(D;\theta)\) [2002.10399].

## 2. Ratio estimation and classifier-based likelihood ratios

A central insight of LFHT is that likelihood ratios can be recovered from classification. In the LFIRE construction, one defines a binary classification problem in which samples from \(p(s \mid \theta)\) are labeled \(y=1\) and samples from the marginal \(p(s)\) are labeled \(y=0\). The Bayes-optimal logit satisfies
\[
f^*(s) = \log \frac{p(s \mid \theta)}{p(s)} + \log \frac{\pi_1}{\pi_0}.
\]
Thus an estimate \(f(s)\) implies
\[
r_\theta(s) \approx \exp\{f(s)-\log(\pi_1/\pi_0)\}.
\]
The paper emphasizes logistic regression with a linear logit \(f(s)=\beta_0+\beta^\top s\), fitted by minimizing a penalized logistic loss with an \(\ell_1\) penalty for automatic summary-statistic selection [1611.10242].

For LFHT, the same derivation applies directly to hypothesis discrimination. If one trains a classifier between \(p(s \mid H_1)\) and \(p(s \mid H_0)\), the Bayes-optimal logit is
\[
f^*(s)=\log \frac{p(s \mid H_1)}{p(s \mid H_0)}+\log\frac{\pi_1}{\pi_0},
\]
so the classifier learns the log-likelihood ratio up to a known constant [1611.10242].

ACORE makes the same connection in a parametrized form. It introduces the odds
\[
O(x;\theta)=\frac{P(Y=1\mid \theta,x)}{P(Y=0\mid \theta,x)},
\]
where \(Y\) indicates whether \(x\) came from \(F_\theta\) or from a reference distribution \(G\). By Bayes’ rule,
\[
O(x;\theta)=\frac{f_\theta(x)p}{g(x)(1-p)}.
\]
Taking odds ratios cancels the reference:
\[
OR(x;\theta_0,\theta_1)=\frac{O(x;\theta_0)}{O(x;\theta_1)}=\frac{f_{\theta_0}(x)}{f_{\theta_1}(x)}.
\]
This exactly equals the likelihood ratio at \(x\), and allows a single parametrized classifier \(P(Y=1\mid \theta,x)\) to amortize likelihood-ratio estimation across \(\theta\) [2002.10399].

Balanced two-class sampling yields the canonical classifier identity
\[
\eta^*(x;\theta,\theta')=\frac{p_\theta(x)}{p_\theta(x)+p_{\theta'}(x)},\qquad
\operatorname{logit}(\eta^*(x;\theta,\theta'))=\log\frac{p_\theta(x)}{p_{\theta'}(x)},
\]
which links optimal classification and likelihood ratios directly [2002.10399]. The same identity appears in classification-based likelihood-free inference more broadly: with equal priors,
\[
p(y=1 \mid x)=\frac{p_\theta(x)}{p_\theta(x)+q(x)},\qquad
\logit(p(y=1\mid x))=\log p_\theta(x)-\log q(x),
\]
and
\[
r_\theta(x)=\frac{p_\theta(x)}{q(x)}=\frac{p(y=1\mid x)}{1-p(y=1\mid x)}.
\]
This provides a direct route from classifier outputs to likelihood-ratio-type test statistics [1407.4981].

## 3. Statistical constructions and calibration

LFHT procedures differ mainly in the form of the test statistic and in how the rejection threshold is calibrated. ACORE defines, for a simple null \(H_{0,\theta}\),
\[
T(D,\theta):=\inf_{\theta'\in\Theta}\sum_{i=1}^n \log[OR(x_i^{obs};\theta,\theta')],
\]
and for a composite null,
\[
\tau(D;\Theta_0):=\sup_{\theta_0\in\Theta_0}\inf_{\theta_1\in\Theta}\sum_{i=1}^n \log[OR(x_i^{obs};\theta_0,\theta_1)].
\]
Under perfect odds estimation, ACORE proves Fisher consistency:
\[
\tau(D;\Theta_0)=\Lambda(D;\Theta_0),
\]
so the learned statistic equals the classical likelihood ratio test statistic [2002.10399].

The rejection threshold for the simple null is the conditional \(\alpha\)-quantile
\[
c_\alpha(\theta)=F^{-1}_{T\mid \theta}(\alpha\mid \theta), \qquad
F_{T\mid \theta}(t)=P_\theta(T(X,\theta)\le t).
\]
For a composite null, one uses
\[
C=\inf_{\theta\in\Theta_0} c_\alpha(\theta),
\]
which ensures size \(\le \alpha\) [2002.10399]. ACORE estimates \(c_\alpha(\theta)\) by quantile regression of the test statistic on \(\theta\), then constructs confidence regions by inversion:
\[
C_\alpha(x)=\{\theta\in\Theta:T(x,\theta)\ge c_\alpha(\theta)\}.
\]
Under quantile-regression consistency and finite \(|\Theta|\), the learned cutoffs converge in probability to valid level-\(\alpha\) thresholds, irrespective of odds-estimation quality [2002.10399].

LF2I generalizes this amortized calibration principle. It treats the test statistic distribution, critical value \(C_\theta\), p-value, and coverage as conditional functionals that vary smoothly in \(\theta\). Given a learned statistic \(\lambda(D;\theta)\), it fits \(\theta\mapsto \widehat C_\theta\) by quantile regression, yielding tests of the form: reject \(H_{0,\theta_0}\) if \(\lambda(D_{obs};\theta_0)<\widehat C_{\theta_0}\). Confidence sets follow by inversion,
\[
\widehat R_\alpha(D_{obs})=\{\theta\in\Theta:\lambda(D_{obs};\theta)\ge \widehat C_\theta\},
\]
or equivalently through p-values [2107.03920]. This modular view permits ACORE and BFF statistics to be embedded in a single frequentist construction with empirical coverage diagnostics [2107.03920].

An alternative route uses direct Monte Carlo calibration of the null statistic. In the LFIRE-derived LFHT construction, one simulates \(x^{(b)}\sim p(x\mid H_0)\), computes \(T(x^{(b)})\), sets \(\tau_\alpha\) as the \((1-\alpha)\)-quantile, and rejects \(H_0\) when \(T(x_{obs})\ge \tau_\alpha\) [1611.10242]. This is conceptually simple and separates statistic learning from size control.

## 4. Major methodological families

LFHT is not a single algorithm but a family of procedures united by simulation access and lack of explicit likelihoods. The major families represented in the literature differ in what they estimate.

| Family | Core object | Representative paper |
|---|---|---|
| Ratio / odds estimation | Log-likelihood ratio via classifier logits or odds ratios | [1611.10242], [2002.10399] |
| Classification discrepancy | Accuracy, AUC, or log-loss as discrepancy statistic | [1407.4981], [2306.11085] |
| Divergence estimation | \(f\)-divergence or IPM estimated from samples | [2405.06397], [2205.10890] |
| Kernel testing | MMD-based test statistics for LFHT or mixed LFHT | [2308.09043] |
| Learned optimal statistic | Neural statistic optimized for average power | [2203.13079] |

Classifier-based discrepancy methods treat the distinguishability of observed and simulated samples as the test statistic itself. With equal priors, Bayes accuracy satisfies
\[
e^*=\frac{1-\operatorname{TV}(P,Q)}{2},\qquad
a^*=\frac{1+\operatorname{TV}(P,Q)}{2},
\]
so under \(H_0\), no classifier can exceed chance, while under \(H_1\), higher classification performance indicates greater discrepancy [1407.4981]. This supports tests based on cross-validated accuracy, ROC-AUC, or negative log-loss, calibrated by permutation or simulation-based nulls [1407.4981].

The CAT framework sharpens this view by showing that classifier/classification-accuracy testing can achieve near-minimax or minimax sample complexity for goodness-of-fit, two-sample testing, and LFHT in several nonparametric classes [2306.11085]. Its test statistic is the empirical difference
\[
T_S(A,B)=\frac1a\sum_{i=1}^{a}\mathbf{1}\{A_i\in S\}-\frac1b\sum_{j=1}^{b}\mathbf{1}\{B_j\in S\},
\]
where \(S\) is a separating set learned from training data [2306.11085]. The paper’s contribution is not merely classifier use, but the proof that appropriately designed classifiers and thresholds recover minimax sample-complexity behavior.

Divergence-based LFHT replaces the likelihood ratio by a discrepancy \(D(P,Q)\) satisfying \(D(P,Q)\ge0\) and \(D(P,Q)=0\) iff \(P=Q\). The 2024 divergence-based framework emphasizes \(f\)-divergences such as KL, JS, TV, Hellinger, and Rényi, as well as IPMs such as MMD and Wasserstein, all estimated from samples by variational objectives optimized over neural critics [2405.06397]. In this view, a large estimated divergence \(\widehat D\) becomes the test statistic, and calibration is performed by permutation or bootstrap [2405.06397].

For categorical simulator outputs, Jensen–Shannon divergence yields a particularly direct asymptotic theory. If \(\hat p\) is the empirical distribution and \(p_\theta\) the model-implied category probabilities, then under \(H_0:p=p_\theta\),
\[
2n\,\pi(1-\pi)\,\mathrm{JS}_\pi(\hat p\Vert p)\xrightarrow{d}\chi^2(K-1).
\]
In the symmetric case,
\[
8n\,\mathrm{JS}(\hat p\Vert p)\xrightarrow{d}\chi^2(K-1).
\]
This yields asymptotically calibrated LFHT without repeated null resampling, and can be adapted to simulator-only access via large synthetic samples and bias correction [2205.10890].

Kernel-based LFHT uses maximum mean discrepancy. In the balanced two-class setting, one observes simulated \(X_1,\dots,X_n\sim P\), \(Y_1,\dots,Y_n\sim Q\), and unlabeled \(Z_1,\dots,Z_m\sim R\), with the classical LFHT problem
\[
H_0:R=P \qquad \text{versus} \qquad H_1:R=Q.
\]
The proposed MMD statistic is
\[
T(X,Y,Z)=\frac{1}{nm}\sum_{i=1}^n\sum_{j=1}^m [k(Z_j,Y_i)-k(Z_j,X_i)],
\]
with threshold
\[
\gamma(X,Y,\pi)=\pi\, \mathrm{MMD}_u^2(\widehat P_X,\widehat P_Y)+T(X,Y,X),
\]
and decision rule
\[
\Psi_\pi=1\{T(X,Y,Z)\ge \gamma(X,Y,\pi)\}.
\]
For classic LFHT, \(\pi=1/2\), and the test reduces to choosing the closer class in MMD [2308.09043].

## 5. Theory, optimality, and sample complexity

Several strands of theory justify LFHT. In the LFIRE setting, the minimizer of the logistic regression loss converges to
\[
h^*(s)=\log r_\theta(s)=\log p(s\mid\theta)-\log p(s),
\]
and more generally to the log density ratio between any two distributions represented as the classifier classes [1611.10242]. This supports asymptotically valid likelihood-ratio approximation for testing as simulation budgets grow.

ACORE strengthens this connection with explicit frequentist guarantees. If \(\widehat P(Y=1\mid \theta,x)=P(Y=1\mid \theta,x)\) for all \(\theta,x\), then its test statistic equals the exact log-likelihood ratio statistic [2002.10399]. A theorem on cutoff consistency shows that, under quantile-regression consistency and finite \(|\Theta|\), the learned critical value converges in probability to a valid level-\(\alpha\) threshold, so type-I error is controlled asymptotically regardless of odds-estimation error [2002.10399]. A separate power theorem states that when classifier posteriors converge and cutoffs converge appropriately, the rejection probabilities of the ACORE test converge to those of the likelihood ratio test [2002.10399].

The learned-optimal-statistic approach addresses nuisance parameters from a different angle. For nested hypotheses \(H_0:\mu=\mu_0,\nu\) free, it trains a neural statistic \(s_\phi(x;\mu_0)\) by classification between null samples and mixtures of alternatives lying on surfaces of equal Fisher distance from \(\mu_0\). Under standard regularity conditions, minimizing average cross-entropy over these surfaces yields a statistic that is a monotone transform of the profile likelihood ratio, hence asymptotically equivalent to the classical optimal test statistic [2203.13079]. This establishes a route to profile-likelihood-ratio-type LFHT without explicit profiling of the nuisance parameter in the likelihood.

Another theoretical axis concerns minimax sample complexity. Gerber and Polyanskiy formulate LFHT as
\[
H_0:P_Z=P_X \qquad \text{versus} \qquad H_1:P_Z=P_Y,
\]
where \(P_X\) and \(P_Y\) are accessed only through \(n\) simulated samples each, the test batch \(Z\) has size \(m\), and \(TV(P_X,P_Y)\ge \epsilon\) [2211.01126]. For regular nonparametric classes, they show the feasible region
\[
\mathcal R_{LF}\asymp\{(n,m): m\ge 1/\epsilon^2,\ n\ge n_{\sf GoF},\ nm\ge n_{\sf GoF}^2\},
\]
where \(n_{\sf GoF}(\epsilon,\mathcal P)\) is the minimax goodness-of-fit sample complexity [2211.01126]. The key implication is that LFHT can avoid full density estimation when \(m\gg 1/\epsilon^2\), but cannot circumvent estimation difficulty when \(m\) is minimal.

CAT closes the high-probability version of this picture in several classes. For bounded discrete, Hölder-smooth, and Gaussian-sequence families, it proves minimax high-probability LFHT sample complexity of the same form, with \(m\), \(n\), and \(nm\) scaling according to the underlying goodness-of-fit complexity [2306.11085]. For unrestricted discrete distributions, the rates are near-minimax within polylogarithmic factors [2306.11085].

The Gaussian-sequence analysis refines these tradeoffs geometrically. For quadratically convex \(\Gamma\), the LFHT feasibility region is characterized, up to constants, by
\[
m \gtrsim \epsilon^{-2},\qquad
n \gtrsim \frac{\sqrt{D(\Gamma,c\epsilon)}}{\epsilon^2},\qquad
mn \gtrsim \frac{D(\Gamma,c\epsilon)}{\epsilon^4},
\]
where \(D(\Gamma,\epsilon)\) is the Kolmogorov dimension [2507.16734]. For \(1\le p\le2\), new simulator–observation tradeoffs arise via an effective dimension \(d(\Gamma,n,\epsilon)\), showing that the simple ellipsoidal rule \(mn\asymp n_{\sf GoF}^2\) does not extend verbatim to non–quadratically-convex classes [2507.16734]. This suggests that LFHT sample complexity is sensitive not only to separation \(\epsilon\) but to the geometry of the hypothesis class.

Kernel-based LFHT adds another theoretical layer: in mixed LFHT, where the unlabeled batch may be a mixture \(R_\alpha=\alpha Q+(1-\alpha)P\), the MMD-based test achieves total error \(\le \alpha\) under explicit upper bounds of the form
\[
\min\{m,n\}\ge c \frac{C\|\lambda\|_\infty \log(1/\alpha)}{(\epsilon\delta/(1+R))^2},\qquad
\min\{n,\sqrt{nm}\}\ge c \frac{C\|\lambda\|_2 \log(1/\alpha)}{\epsilon^2\delta},
\]
with matching lower bounds up to constants and logs [2308.09043]. The asymmetric \(\min\{n,\sqrt{nm}\}\) dependence formalizes the simulation–experimentation tradeoff.

## 6. Applications, empirical illustrations, and limitations

Empirical studies of LFHT span canonical low-dimensional examples and challenging nonlinear simulators. LFIRE demonstrates Gaussian mean estimation, ARCH(1), Lorenz ’96 with stochastic parametrization, and a cell spreading model with 291 summaries. In these examples, penalized logistic regression recovers relevant summary terms, improves posterior approximation relative to synthetic likelihood, and remains robust to irrelevant summaries [1611.10242]. The LFHT reinterpretation of this machinery naturally extends to model discrimination and parameter-level tests in these same settings [1611.10242].

ACORE evaluates toy Poisson and Gaussian mixture models, along with a high-energy physics signal-detection problem. In the toy examples, smaller cross-entropy loss corresponds to higher power and smaller confidence sets, while all constructed confidence sets achieve nominal 90% coverage by diagnostics [2002.10399]. In the HEP application, selected deep classifiers and deep quantile regression produce confidence regions closely matching exact likelihood-ratio sets, and diagnostics identify undercoverage regions for suboptimal calibrators [2002.10399].

LF2I broadens the empirical scope to Gaussian mixtures, Poisson counting with nuisance parameters, and \(51{,}200\)-dimensional muon-energy images. It shows that Wilks-type asymptotics can under-cover in finite samples, while amortized quantile-regressed cutoffs recover nominal coverage without per-\(\theta\) Monte Carlo [2107.03920]. In the Poisson-nuisance example, h-ACORE is conservative and h-BFF yields tighter intervals with localized undercoverage, illustrating the value of explicit coverage diagnostics [2107.03920].

The exchangeable-neural-network framework in population genetics presents a summary-statistic-free version of LFHT. A classifier \(D_\theta(X)\approx P(H_1\mid X)\) yields the approximate likelihood ratio
\[
\Lambda_\theta(X)=\frac{D_\theta(X)}{1-D_\theta(X)}\cdot \frac{\pi_0}{\pi_1},\qquad
T_\theta(X)=\log \Lambda_\theta(X),
\]
with thresholds calibrated by null simulations [1802.06153]. Applied to recombination hotspot testing, the method substantially outperforms LDhot under realistic recombination maps, while respecting permutation invariance across haplotypes through the network architecture [1802.06153]. This suggests that LFHT can be carried out directly on structured raw data when suitable inductive biases are available.

Divergence-based LFHT shows that high-dimensional alternatives invisible to low-order marginals can still be detected through learned variational critics. In a 9-dimensional parity-asymmetry example, KL divergence estimated by a neural critic yields a lower bound of approximately \(0.077 \pm 0.003\), providing strong evidence of a difference between the simulated and reference distributions despite matching one- and two-dimensional marginals [2405.06397].

Several limitations recur across methods. Summary-statistic approaches remain vulnerable to misspecified or uninformative summaries; if \(s(x)\) cannot discriminate \(H_0\) from \(H_1\), power collapses [1611.10242]. Classifier-based methods can lose power when the classifier family is inadequate, miscalibrated, or overfit, although correct threshold calibration can still preserve validity [2002.10399]. Variational divergence estimators can be optimistically biased if evaluated on training data, and KL-type objectives may be numerically unstable because of exponential terms [2405.06397]. Kernel methods depend strongly on kernel choice and bandwidth, and theoretical guarantees for learned kernels remain incomplete [2308.09043]. In nuisance-parameter settings, profiling tends to be conservative while marginalization can under-cover in specific regions [2107.03920].

A frequent misconception is that LFHT is merely approximate Bayesian computation applied to testing. The literature points instead to a broader landscape. ABC uses acceptance thresholds and discrepancy measures tied to summary statistics, whereas LFHT often targets the likelihood-ratio principle directly through odds-ratio estimation, divergence duality, or explicit minimax constructions [2002.10399]. Another misconception is that classifier-based testing is inherently heuristic. The Neyman–Pearson linkage, the Fisher-consistency results for ACORE, the asymptotic profile-likelihood equivalence in nuisance settings, and the minimax sample-complexity theorems for CAT all show that classification-based LFHT can be theoretically aligned with optimal testing, provided the classifier is embedded in a calibrated testing procedure [2002.10399] [2203.13079] [2306.11085].

Taken together, these developments position LFHT as a simulator-native reformulation of hypothesis testing. Its core principle is stable across implementations: when explicit likelihoods are unavailable, one can still learn or estimate a decision statistic from simulations, calibrate it under the null, and recover valid frequentist or Neyman–Pearson-style inference. The differences among LFIRE, ACORE, LF2I, divergence-based methods, kernel tests, and classification-accuracy tests concern not the objective itself, but the representation of the discriminating signal and the mechanism used to control error.

Source: https://www.emergentmind.com/topics/likelihood-free-hypothesis-testing-lfht