Likelihood-Free Hypothesis Testing (LFHT)

• Likelihood-Free Hypothesis Testing (LFHT) is a suite of simulation-based techniques for testing hypotheses when direct likelihood evaluations are intractable.
• LFHT methods leverage classifier-derived ratios, divergence estimators, and kernel discrepancies to approximate likelihood ratios and control error rates.
• These approaches are applied in diverse fields such as physics and population genetics, offering calibrated tests with near-optimal power in complex settings.

Searching arXiv for relevant papers on likelihood-free hypothesis testing and related methods. Likelihood-Free Hypothesis Testing (LFHT) denotes hypothesis-testing procedures in settings where the likelihood is unavailable or intractable, but forward simulation from the model is possible. Across the literature, LFHT replaces explicit likelihood evaluation by test statistics learned or estimated from simulated data, including classifier-derived likelihood-ratio surrogates, odds-ratio estimators, divergence estimators, and kernel discrepancies. In the formulation developed from likelihood-free inference by ratio estimation, the central object is a density ratio between distributions of summary statistics, estimated by logistic regression and then repurposed as a log-likelihood-ratio test statistic (Thomas et al., 2016). In a more explicitly frequentist formulation, LFHT is the construction of level-$\alpha$ tests and confidence sets in simulator-based models by replacing the classical likelihood ratio test with an estimated odds-ratio statistic and calibrating rejection thresholds or p-values from simulation (Dalmasso et al., 2020). Related lines of work show that LFHT can also be implemented by divergence estimation (Wilkinson et al., 2024), kernel-based testing (Gerber et al., 2023), optimal-statistic learning in the presence of nuisance parameters (Heinrich, 2022), and classification-based discrepancy measures (Gutmann et al., 2014).

1. Conceptual formulation

LFHT addresses binary or composite hypothesis testing when one can sample from model-implied distributions but cannot evaluate $p(x \mid \theta)$ or the corresponding likelihood in closed form. In the frequentist setting, the objective is to construct tests that control size, attain high power, and yield confidence sets with nominal coverage without requiring explicit likelihood evaluations (Dalmasso et al., 2020).

A common setup uses summary statistics $s(x)$ computed from raw data $x$. In the ratio-estimation formulation, the marginal distribution of summaries is

$p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$

with $p(\theta)$ a prior and $p(s \mid \theta)$ induced by the simulator $p(x \mid \theta)$ via $s(x)$. The key target is the marginal ratio

$r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$

which yields posterior inference through

$p(x \mid \theta)$0

This same density-ratio machinery can be redirected toward hypothesis testing by replacing the marginal denominator $p(x \mid \theta)$1 with a competing hypothesis distribution $p(x \mid \theta)$2 or $p(x \mid \theta)$3 (Thomas et al., 2016).

In the explicit LFHT formulation for two hypotheses, the test statistic is the estimated log-likelihood ratio on summaries,

$p(x \mid \theta)$4

where $p(x \mid \theta)$5 is the logit of a trained classifier discriminating simulated summaries from $p(x \mid \theta)$6 and $p(x \mid \theta)$7, and $p(x \mid \theta)$8 are class priors (Thomas et al., 2016).

The classical baseline remains the likelihood ratio test. For composite hypotheses,

$p(x \mid \theta)$9

the log-likelihood ratio statistic is

$s(x)$0

LFHT seeks to reconstruct this object, or an equivalent decision rule, without direct access to $s(x)$1 (Dalmasso et al., 2020).

2. Ratio estimation and classifier-based likelihood ratios

A central insight of LFHT is that likelihood ratios can be recovered from classification. In the LFIRE construction, one defines a binary classification problem in which samples from $s(x)$2 are labeled $s(x)$3 and samples from the marginal $s(x)$4 are labeled $s(x)$5. The Bayes-optimal logit satisfies

$s(x)$6

Thus an estimate $s(x)$7 implies

$s(x)$8

The paper emphasizes logistic regression with a linear logit $s(x)$9, fitted by minimizing a penalized logistic loss with an $x$0 penalty for automatic summary-statistic selection (Thomas et al., 2016).

For LFHT, the same derivation applies directly to hypothesis discrimination. If one trains a classifier between $x$1 and $x$2, the Bayes-optimal logit is

$x$3

so the classifier learns the log-likelihood ratio up to a known constant (Thomas et al., 2016).

ACORE makes the same connection in a parametrized form. It introduces the odds

$x$4

where $x$5 indicates whether $x$6 came from $x$7 or from a reference distribution $x$8. By Bayes’ rule,

$x$9

Taking odds ratios cancels the reference: $p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$0 This exactly equals the likelihood ratio at $p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$1, and allows a single parametrized classifier $p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$2 to amortize likelihood-ratio estimation across $p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$3 (Dalmasso et al., 2020).

Balanced two-class sampling yields the canonical classifier identity

$p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$4

which links optimal classification and likelihood ratios directly (Dalmasso et al., 2020). The same identity appears in classification-based likelihood-free inference more broadly: with equal priors,

$p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$5

and

$p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$6

This provides a direct route from classifier outputs to likelihood-ratio-type test statistics (Gutmann et al., 2014).

3. Statistical constructions and calibration

LFHT procedures differ mainly in the form of the test statistic and in how the rejection threshold is calibrated. ACORE defines, for a simple null $p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$7,

$p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$8

and for a composite null,

$p(s) = \int p(s \mid \theta) p(\theta)\, d\theta,$9

Under perfect odds estimation, ACORE proves Fisher consistency: $p(\theta)$0 so the learned statistic equals the classical likelihood ratio test statistic (Dalmasso et al., 2020).

The rejection threshold for the simple null is the conditional $p(\theta)$1-quantile

$p(\theta)$2

For a composite null, one uses

$p(\theta)$3

which ensures size $p(\theta)$4 (Dalmasso et al., 2020). ACORE estimates $p(\theta)$5 by quantile regression of the test statistic on $p(\theta)$6, then constructs confidence regions by inversion: $p(\theta)$7 Under quantile-regression consistency and finite $p(\theta)$8, the learned cutoffs converge in probability to valid level-$p(\theta)$9 thresholds, irrespective of odds-estimation quality (Dalmasso et al., 2020).

LF2I generalizes this amortized calibration principle. It treats the test statistic distribution, critical value $p(s \mid \theta)$0, p-value, and coverage as conditional functionals that vary smoothly in $p(s \mid \theta)$1. Given a learned statistic $p(s \mid \theta)$2, it fits $p(s \mid \theta)$3 by quantile regression, yielding tests of the form: reject $p(s \mid \theta)$4 if $p(s \mid \theta)$5. Confidence sets follow by inversion,

$p(s \mid \theta)$6

or equivalently through p-values (Dalmasso et al., 2021). This modular view permits ACORE and BFF statistics to be embedded in a single frequentist construction with empirical coverage diagnostics (Dalmasso et al., 2021).

An alternative route uses direct Monte Carlo calibration of the null statistic. In the LFIRE-derived LFHT construction, one simulates $p(s \mid \theta)$7, computes $p(s \mid \theta)$8, sets $p(s \mid \theta)$9 as the $p(x \mid \theta)$0-quantile, and rejects $p(x \mid \theta)$1 when $p(x \mid \theta)$2 (Thomas et al., 2016). This is conceptually simple and separates statistic learning from size control.

4. Major methodological families

LFHT is not a single algorithm but a family of procedures united by simulation access and lack of explicit likelihoods. The major families represented in the literature differ in what they estimate.

Family	Core object	Representative paper
Ratio / odds estimation	Log-likelihood ratio via classifier logits or odds ratios	(Thomas et al., 2016, Dalmasso et al., 2020)
Classification discrepancy	Accuracy, AUC, or log-loss as discrepancy statistic	(Gutmann et al., 2014, Gerber et al., 2023)
Divergence estimation	$p(x \mid \theta)$3-divergence or IPM estimated from samples	(Wilkinson et al., 2024, Corander et al., 2022)
Kernel testing	MMD-based test statistics for LFHT or mixed LFHT	(Gerber et al., 2023)
Learned optimal statistic	Neural statistic optimized for average power	(Heinrich, 2022)

Classifier-based discrepancy methods treat the distinguishability of observed and simulated samples as the test statistic itself. With equal priors, Bayes accuracy satisfies

$p(x \mid \theta)$4

so under $p(x \mid \theta)$5, no classifier can exceed chance, while under $p(x \mid \theta)$6, higher classification performance indicates greater discrepancy (Gutmann et al., 2014). This supports tests based on cross-validated accuracy, ROC-AUC, or negative log-loss, calibrated by permutation or simulation-based nulls (Gutmann et al., 2014).

The CAT framework sharpens this view by showing that classifier/classification-accuracy testing can achieve near-minimax or minimax sample complexity for goodness-of-fit, two-sample testing, and LFHT in several nonparametric classes (Gerber et al., 2023). Its test statistic is the empirical difference

$p(x \mid \theta)$7

where $p(x \mid \theta)$8 is a separating set learned from training data (Gerber et al., 2023). The paper’s contribution is not merely classifier use, but the proof that appropriately designed classifiers and thresholds recover minimax sample-complexity behavior.

Divergence-based LFHT replaces the likelihood ratio by a discrepancy $p(x \mid \theta)$9 satisfying $s(x)$0 and $s(x)$1 iff $s(x)$2. The 2024 divergence-based framework emphasizes $s(x)$3-divergences such as KL, JS, TV, Hellinger, and Rényi, as well as IPMs such as MMD and Wasserstein, all estimated from samples by variational objectives optimized over neural critics (Wilkinson et al., 2024). In this view, a large estimated divergence $s(x)$4 becomes the test statistic, and calibration is performed by permutation or bootstrap (Wilkinson et al., 2024).

For categorical simulator outputs, Jensen–Shannon divergence yields a particularly direct asymptotic theory. If $s(x)$5 is the empirical distribution and $s(x)$6 the model-implied category probabilities, then under $s(x)$7,

$s(x)$8

In the symmetric case,

$s(x)$9

This yields asymptotically calibrated LFHT without repeated null resampling, and can be adapted to simulator-only access via large synthetic samples and bias correction (Corander et al., 2022).

Kernel-based LFHT uses maximum mean discrepancy. In the balanced two-class setting, one observes simulated $r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$0, $r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$1, and unlabeled $r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$2, with the classical LFHT problem

$r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$3

The proposed MMD statistic is

$r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$4

with threshold

$r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$5

and decision rule

$r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$6

For classic LFHT, $r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$7, and the test reduces to choosing the closer class in MMD (Gerber et al., 2023).

5. Theory, optimality, and sample complexity

Several strands of theory justify LFHT. In the LFIRE setting, the minimizer of the logistic regression loss converges to

$r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$8

and more generally to the log density ratio between any two distributions represented as the classifier classes (Thomas et al., 2016). This supports asymptotically valid likelihood-ratio approximation for testing as simulation budgets grow.

ACORE strengthens this connection with explicit frequentist guarantees. If $r_\theta(s) = \frac{p(s \mid \theta)}{p(s)},$9 for all $p(x \mid \theta)$00, then its test statistic equals the exact log-likelihood ratio statistic (Dalmasso et al., 2020). A theorem on cutoff consistency shows that, under quantile-regression consistency and finite $p(x \mid \theta)$01, the learned critical value converges in probability to a valid level-$p(x \mid \theta)$02 threshold, so type-I error is controlled asymptotically regardless of odds-estimation error (Dalmasso et al., 2020). A separate power theorem states that when classifier posteriors converge and cutoffs converge appropriately, the rejection probabilities of the ACORE test converge to those of the likelihood ratio test (Dalmasso et al., 2020).

The learned-optimal-statistic approach addresses nuisance parameters from a different angle. For nested hypotheses $p(x \mid \theta)$03 free, it trains a neural statistic $p(x \mid \theta)$04 by classification between null samples and mixtures of alternatives lying on surfaces of equal Fisher distance from $p(x \mid \theta)$05. Under standard regularity conditions, minimizing average cross-entropy over these surfaces yields a statistic that is a monotone transform of the profile likelihood ratio, hence asymptotically equivalent to the classical optimal test statistic (Heinrich, 2022). This establishes a route to profile-likelihood-ratio-type LFHT without explicit profiling of the nuisance parameter in the likelihood.

Another theoretical axis concerns minimax sample complexity. Gerber and Polyanskiy formulate LFHT as

$p(x \mid \theta)$06

where $p(x \mid \theta)$07 and $p(x \mid \theta)$08 are accessed only through $p(x \mid \theta)$09 simulated samples each, the test batch $p(x \mid \theta)$10 has size $p(x \mid \theta)$11, and $p(x \mid \theta)$12 (Gerber et al., 2022). For regular nonparametric classes, they show the feasible region

$p(x \mid \theta)$13

where $p(x \mid \theta)$14 is the minimax goodness-of-fit sample complexity (Gerber et al., 2022). The key implication is that LFHT can avoid full density estimation when $p(x \mid \theta)$15, but cannot circumvent estimation difficulty when $p(x \mid \theta)$16 is minimal.

CAT closes the high-probability version of this picture in several classes. For bounded discrete, Hölder-smooth, and Gaussian-sequence families, it proves minimax high-probability LFHT sample complexity of the same form, with $p(x \mid \theta)$17, $p(x \mid \theta)$18, and $p(x \mid \theta)$19 scaling according to the underlying goodness-of-fit complexity (Gerber et al., 2023). For unrestricted discrete distributions, the rates are near-minimax within polylogarithmic factors (Gerber et al., 2023).

The Gaussian-sequence analysis refines these tradeoffs geometrically. For quadratically convex $p(x \mid \theta)$20, the LFHT feasibility region is characterized, up to constants, by

$p(x \mid \theta)$21

where $p(x \mid \theta)$22 is the Kolmogorov dimension (Jia et al., 22 Jul 2025). For $p(x \mid \theta)$23, new simulator–observation tradeoffs arise via an effective dimension $p(x \mid \theta)$24, showing that the simple ellipsoidal rule $p(x \mid \theta)$25 does not extend verbatim to non–quadratically-convex classes (Jia et al., 22 Jul 2025). This suggests that LFHT sample complexity is sensitive not only to separation $p(x \mid \theta)$26 but to the geometry of the hypothesis class.

Kernel-based LFHT adds another theoretical layer: in mixed LFHT, where the unlabeled batch may be a mixture $p(x \mid \theta)$27, the MMD-based test achieves total error $p(x \mid \theta)$28 under explicit upper bounds of the form

$p(x \mid \theta)$29

with matching lower bounds up to constants and logs (Gerber et al., 2023). The asymmetric $p(x \mid \theta)$30 dependence formalizes the simulation–experimentation tradeoff.

6. Applications, empirical illustrations, and limitations

Empirical studies of LFHT span canonical low-dimensional examples and challenging nonlinear simulators. LFIRE demonstrates Gaussian mean estimation, ARCH(1), Lorenz ’96 with stochastic parametrization, and a cell spreading model with 291 summaries. In these examples, penalized logistic regression recovers relevant summary terms, improves posterior approximation relative to synthetic likelihood, and remains robust to irrelevant summaries (Thomas et al., 2016). The LFHT reinterpretation of this machinery naturally extends to model discrimination and parameter-level tests in these same settings (Thomas et al., 2016).

ACORE evaluates toy Poisson and Gaussian mixture models, along with a high-energy physics signal-detection problem. In the toy examples, smaller cross-entropy loss corresponds to higher power and smaller confidence sets, while all constructed confidence sets achieve nominal 90% coverage by diagnostics (Dalmasso et al., 2020). In the HEP application, selected deep classifiers and deep quantile regression produce confidence regions closely matching exact likelihood-ratio sets, and diagnostics identify undercoverage regions for suboptimal calibrators (Dalmasso et al., 2020).

LF2I broadens the empirical scope to Gaussian mixtures, Poisson counting with nuisance parameters, and $p(x \mid \theta)$31-dimensional muon-energy images. It shows that Wilks-type asymptotics can under-cover in finite samples, while amortized quantile-regressed cutoffs recover nominal coverage without per-$p(x \mid \theta)$32 Monte Carlo (Dalmasso et al., 2021). In the Poisson-nuisance example, h-ACORE is conservative and h-BFF yields tighter intervals with localized undercoverage, illustrating the value of explicit coverage diagnostics (Dalmasso et al., 2021).

The exchangeable-neural-network framework in population genetics presents a summary-statistic-free version of LFHT. A classifier $p(x \mid \theta)$33 yields the approximate likelihood ratio

$p(x \mid \theta)$34

with thresholds calibrated by null simulations (Chan et al., 2018). Applied to recombination hotspot testing, the method substantially outperforms LDhot under realistic recombination maps, while respecting permutation invariance across haplotypes through the network architecture (Chan et al., 2018). This suggests that LFHT can be carried out directly on structured raw data when suitable inductive biases are available.

Divergence-based LFHT shows that high-dimensional alternatives invisible to low-order marginals can still be detected through learned variational critics. In a 9-dimensional parity-asymmetry example, KL divergence estimated by a neural critic yields a lower bound of approximately $p(x \mid \theta)$35, providing strong evidence of a difference between the simulated and reference distributions despite matching one- and two-dimensional marginals (Wilkinson et al., 2024).

Several limitations recur across methods. Summary-statistic approaches remain vulnerable to misspecified or uninformative summaries; if $p(x \mid \theta)$36 cannot discriminate $p(x \mid \theta)$37 from $p(x \mid \theta)$38, power collapses (Thomas et al., 2016). Classifier-based methods can lose power when the classifier family is inadequate, miscalibrated, or overfit, although correct threshold calibration can still preserve validity (Dalmasso et al., 2020). Variational divergence estimators can be optimistically biased if evaluated on training data, and KL-type objectives may be numerically unstable because of exponential terms (Wilkinson et al., 2024). Kernel methods depend strongly on kernel choice and bandwidth, and theoretical guarantees for learned kernels remain incomplete (Gerber et al., 2023). In nuisance-parameter settings, profiling tends to be conservative while marginalization can under-cover in specific regions (Dalmasso et al., 2021).

A frequent misconception is that LFHT is merely approximate Bayesian computation applied to testing. The literature points instead to a broader landscape. ABC uses acceptance thresholds and discrepancy measures tied to summary statistics, whereas LFHT often targets the likelihood-ratio principle directly through odds-ratio estimation, divergence duality, or explicit minimax constructions (Dalmasso et al., 2020). Another misconception is that classifier-based testing is inherently heuristic. The Neyman–Pearson linkage, the Fisher-consistency results for ACORE, the asymptotic profile-likelihood equivalence in nuisance settings, and the minimax sample-complexity theorems for CAT all show that classification-based LFHT can be theoretically aligned with optimal testing, provided the classifier is embedded in a calibrated testing procedure (Dalmasso et al., 2020, Heinrich, 2022, Gerber et al., 2023).

Taken together, these developments position LFHT as a simulator-native reformulation of hypothesis testing. Its core principle is stable across implementations: when explicit likelihoods are unavailable, one can still learn or estimate a decision statistic from simulations, calibrate it under the null, and recover valid frequentist or Neyman–Pearson-style inference. The differences among LFIRE, ACORE, LF2I, divergence-based methods, kernel tests, and classification-accuracy tests concern not the objective itself, but the representation of the discriminating signal and the mechanism used to control error.