Post-Hoc Hypothesis Testing

• Post-hoc hypothesis testing is the practice of selecting significance thresholds, rejection sets, or variable pathways after data inspection, challenging the classical fixed-alpha framework.
• Modern approaches employ simultaneous guarantees and e-value-based risk control to maintain valid Type I error rates despite data-driven adaptations.
• Recent advances incorporate decision-theoretic and multiple testing frameworks to deliver robust, adaptable inference with precise error control mechanisms.

Post-hoc hypothesis testing denotes inferential settings in which some component that is classically fixed before analysis—most prominently the significance level $\alpha$, but also a rejection set, a variable set, a pathway collection, or a local region of interest—is chosen after inspection of the data. In classical Neyman–Pearson testing this is ordinarily invalid, because size control relies on choosing $\alpha$ independently of the data. Contemporary work therefore treats post-hoc testing in two main ways: by deriving simultaneous guarantees that hold for all admissible post-data choices, or by replacing fixed-$\alpha$ tail-probability control with e-value-based risk control that remains valid under data-dependent choices of $\alpha$ (Hemerik et al., 2024, Goeman et al., 2012, Grünwald, 2022).

1. Meanings and problem formulations

In the cited literature, the expression post-hoc hypothesis testing refers to several distinct but related inferential problems. One concerns choosing $\alpha$ after observing the data, which violates the standard assumption that $\alpha$ is fixed independently of the realized $p$-value. A second concerns choosing the rejected subset $R$ or $S$ after seeing the data, while still wanting a valid statement about the number of false rejections. A third, older usage refers to pairwise or localized follow-up procedures applied after rejection of a global null. These uses are not identical, and much of the modern literature is concerned with separating them conceptually while restoring formal guarantees in each setting (Blanchard et al., 2017, Benavoli et al., 2015, Fryer et al., 2023).

Formulation	Object chosen after seeing data	Guarantee
Data-dependent $\alpha$	Significance level	Type-I risk control via e-values or post-hoc $\alpha$0-values
User-agnostic multiple testing	Rejection set $\alpha$1 or $\alpha$2	Simultaneous bound on false rejections or false positives
Classical post-hoc comparison	Pairwise contrasts after omnibus rejection	Family-wise control under a model-specific procedure

The first formulation is central to "Choosing alpha post hoc: the danger of multiple standard significance thresholds" (Hemerik et al., 2024). The second is central to the “reversed roles” approach of Goeman and Solari, the JER framework of Blanchard et al., the forest-structured spatial bounds of Durand et al., and exact closed testing for Globaltest pathway analysis (Goeman et al., 2012, Blanchard et al., 2017, Durand et al., 2018, Xu et al., 2020). The third includes both critiques of invalid follow-up procedures and construction of valid ones, as in mean-ranks comparisons after Friedman’s test and Tukey-style post-hoc comparison of Sharpe ratios (Benavoli et al., 2015, Pav, 2019).

2. Data-dependent significance levels and size inflation

In classical Neyman–Pearson testing, one specifies before seeing the data a significance level $\alpha$3. A test $\alpha$4 rejects the null $\alpha$5 exactly when the data $\alpha$6 fall into a pre-defined rejection region of size $\alpha$7, so that under $\alpha$8,

$\alpha$9

The same pre-specification requirement underlies the nominal coverage of confidence intervals obtained by inverting such tests. If $\alpha$0 is allowed to depend on the observed data, written $\alpha$1, the guarantee no longer has its usual interpretation (Hemerik et al., 2024).

The 2024 analysis of multiple standard thresholds formalizes this distortion by considering a set $\alpha$2 of possible significance levels and defining, for each $\alpha$3, the conditional discrepancy

$\alpha$4

and the relative discrepancy

$\alpha$5

If $\alpha$6, then conditional on $\alpha$7 the test rejects too often. The overall expected discrepancy ratio is

$\alpha$8

This quantity can exceed $\alpha$9 even when each fixed-$\alpha$0 test is valid (Hemerik et al., 2024).

Two examples make the point explicit. With two candidate thresholds $\alpha$1, suppose the rule is to set $\alpha$2 if $\alpha$3 and $\alpha$4 if $\alpha$5. Then conditional on $\alpha$6 one must have $\alpha$7, hence

$\alpha$8

so $\alpha$9. By contrast, conditional on $\alpha$0 the procedure is conservative, yet the overall expected ratio is

$\alpha$1

For $\alpha$2 and $\alpha$3, this gives $\alpha$4, so on average one rejects $\alpha$5 more often than intended. In a second example, if one sets $\alpha$6 over infinitely many candidate thresholds, then the expected ratio is infinite, and the test is “disastrously invalid” (Hemerik et al., 2024).

This analysis is directly relevant to proposals to replace a single field-wide threshold by multiple “accepted” thresholds such as $\alpha$7 and $\alpha$8. If different journals or venues accept different thresholds, researchers with $\alpha$9 may “shop” for venues accepting $\alpha$0, whereas those with $\alpha$1 may prefer venues demanding $\alpha$2. The underlying problem is not merely sociological: the post-hoc choice of $\alpha$3 violates the independence assumption, inflates Type I error, and breaks the nominal interpretation of both tests and confidence intervals (Hemerik et al., 2024).

3. E-values, post-hoc $\alpha$4-values, and decision-theoretic admissibility

The modern solution to post-hoc $\alpha$5 choice is based on e-values. An e-value is a nonnegative statistic $\alpha$6 such that under every null distribution,

$\alpha$7

By Markov’s inequality, the rule “reject if $\alpha$8” has Type I error at most $\alpha$9 for every fixed $p$0. More importantly, e-values support a generalized Neyman–Pearson formulation in which the downstream decision task, or loss function, may itself be chosen after observation of the data. In that setting, sufficiently rich decision problems have only e-value-based admissible rules (Grünwald, 2022).

This perspective yields a precise characterization of post-hoc $p$1-values. A statistic $p$2 is a post-hoc $p$3-value if and only if

$p$4

that is, if and only if $p$5 is an e-value. Unlike regular $p$6-values, such post-hoc $p$7-values allow one to “reject at level $p$8” while retaining the relevant expectation guarantee. They also combine multiplicatively under independence: if $p$9 and $R$0 are independent post-hoc $R$1-values, then $R$2 is again a post-hoc $R$3-value because

$R$4

This equivalence also clarifies the role of e-values as the reciprocal evidence measure underlying post-hoc testing (Koning, 2023).

The decision-theoretic formulation goes further. For a loss budget $R$5, a rule $R$6 is Type I-risk safe if

$R$7

The generalized Neyman–Pearson theorem states that if $R$8 is admissible, then there exists an e-variable $R$9 such that

$S$0

Conversely, maximally compatible rules based on sharp e-variables are admissible. In this sense, e-value-based rules form a complete class for post-hoc decision tasks (Grünwald, 2022).

The 2025 theory of $S$1-admissibility sharpens this by modeling an adversary that maps the data to a significance level. If $S$2 contains all constant mappings and $S$3 is $S$4-admissible, then

$S$5

almost surely for every $S$6. In the binary setting this reduces to

$S$7

When $S$8 contains only constant adversaries and the loss span collapses to a singleton, this recovers the classical Neyman–Pearson likelihood-ratio test; when $S$9 allows arbitrary data-dependent adversaries, admissible tests are exactly the canonical tests arising from sharp e-variables (Chugg et al., 1 Aug 2025).

4. Simultaneous post-hoc inference in multiple testing

A second major strand does not try to salvage ordinary fixed-$\alpha$0 tests under data-driven threshold choice; instead, it returns a statement that is valid simultaneously for every post-hoc chosen rejection set. In “Multiple Testing for Exploratory Research,” the conventional roles are reversed: the analyst chooses any candidate rejection set $\alpha$1 after seeing the data, and the procedure returns an integer $\alpha$2 such that, with confidence at least $\alpha$3,

$\alpha$4

where $\alpha$5 is the number of false rejections in $\alpha$6. Equivalently, if $\alpha$7 is the number of correct discoveries, then

$\alpha$8

simultaneously for all $\alpha$9. The machinery is classical closed testing, but the crucial insight is that non-consonant rejections, often treated as a nuisance in ordinary FWER theory, are informative because they shrink the upper bound on the number of false rejections (Goeman et al., 2012).

Blanchard et al. generalized this idea in large-scale multiple testing via the joint-family-wise-error rate (JER). For a sequence of thresholds $\alpha$00, JER is

$\alpha$01

If JER is controlled at level $\alpha$02, then

$\alpha$03

is a simultaneous post-hoc upper bound on the number of false positives in any user-chosen set $\alpha$04, and $\alpha$05 is a lower-confidence bound on the number of true discoveries. The framework accommodates both known dependence and permutation-based calibration, and step-down calibration adapts to the unknown quantity of signal (Blanchard et al., 2017).

Durand et al. specialized this perspective to spatially structured hypotheses. For a reference family $\alpha$06 satisfying a joint error-rate control and a forest-structure condition, they construct a data-dependent upper bound $\alpha$07 such that

$\alpha$08

The coverage is simultaneous over all $\alpha$09 subsets $\alpha$10, so the subset may be chosen arbitrarily, even after repeated inspection of the data. The forest structure yields an explicit interpolation formula and a low-complexity dynamic-programming algorithm; the implementation is available in the R package sansSouci (Durand et al., 2018).

Exact post-hoc multiple testing has also been developed for metabolomics pathway analysis. In pathway testing with Globaltest, the family $\alpha$11 is closed under unions, and closed testing therefore controls FWER simultaneously over all possible feature sets. Xu et al. derive a shortcut, based on convex-hull envelopes for the minimum test statistic and majorization envelopes for the maximum critical value, that makes exact closed testing computationally feasible at metabolomics scale. The result is that one may choose the pathway database after seeing the data without jeopardizing error control; the implementation is provided in the R package ctgt (Xu et al., 2020).

5. Pairwise post-hoc procedures after omnibus tests

In a narrower and older usage, post-hoc tests are follow-up comparisons performed after a global null has been rejected. This usage is common in algorithm comparison, medicine, psychology, finance, and other applied fields. The modern literature emphasizes that such procedures are not automatically valid merely because they are labeled “post-hoc”: the validity depends on whether the pairwise decision for a given pair depends only on that pair, and whether the multiplicity correction matches the inferential target (Benavoli et al., 2015, Pav, 2019).

Benavoli et al. show that the mean-ranks post-hoc test used after Friedman’s test is inconsistent because the outcome of the comparison between algorithms $\alpha$12 and $\alpha$13 depends on the performance of the other algorithms included in the original experiment. In their example with $\alpha$14 data sets, algorithms $\alpha$15 and $\alpha$16 are tied in every direct comparison, so when considered alone they have $\alpha$17 and $\alpha$18. After adding three auxiliary algorithms $\alpha$19, the same pair yields $\alpha$20, $\alpha$21, $\alpha$22, and

$\alpha$23

so $\alpha$24 versus $\alpha$25 is declared significant at $\alpha$26 with Bonferroni correction for $\alpha$27. By choosing a different triple of auxiliary algorithms, one can reverse the conclusion again. The recommended alternatives are tests whose outcome depends only on the paired differences, such as the sign test or the Wilcoxon signed-rank test, combined with multiplicity correction (Benavoli et al., 2015).

A contrasting example is Pav’s post hoc test on the Sharpe ratio. After rejection of the global null

$\alpha$28

under a Gaussian equi-correlation model for contemporaneous returns, the pairwise difference in sample Sharpe ratios can be compared using a Tukey-style studentized-range cutoff. The honest significant difference is

$\alpha$29

where $\alpha$30 with $\alpha$31 and $\alpha$32 or $\alpha$33. The family-wise guarantee follows from the range distribution, and simulation results indicate that the $\alpha$34 variant achieves nearly correct $\alpha$35 over a grid of $\alpha$36 and $\alpha$37, whereas the $\alpha$38 rule can be anti-conservative when $\alpha$39 is large and $\alpha$40 is small (Pav, 2019).

These examples illustrate a persistent misconception. Rejection of a global null does not by itself justify arbitrary follow-up comparisons. Valid post-hoc pairwise inference requires either a pairwise test whose sampling law depends only on the two items being compared, or a joint reference distribution that genuinely controls family-wise error for the entire family of contrasts. The mean-ranks critique and the Sharpe-ratio construction occupy opposite sides of this distinction (Benavoli et al., 2015, Pav, 2019).

6. Confidence sets, asymptotic theory, and emerging extensions

Because ordinary confidence intervals are obtained by inverting fixed-level tests, post-hoc choice of $\alpha$41 invalidates their nominal coverage. The same point appears in both the classical critique of choosing $\alpha$42 after seeing the data and the decision-theoretic e-value literature: to ensure advertised $\alpha$43 coverage, $\alpha$44 must be fixed in advance, or one must use e-confidence sets, e-posteriors, or other constructions that remain valid under data-dependent $\alpha$45 (Hemerik et al., 2024, Grünwald, 2022).

A 2026 extension develops post-hoc large-sample statistical inference through asymptotic e-values. For the mean, one example is the IWR e-variable

$\alpha$46

which is an asymptotic e-value under only finite-second-moment or domain-of-attraction conditions. Inversion yields asymptotic post-hoc confidence intervals and asymptotic post-hoc $\alpha$47-values. The framework also treats mixture constructions over $\alpha$48, and extends to post-hoc confidence sequences via asymptotic e-processes (Chugg et al., 9 Mar 2026).

The same period has seen post-hoc validity imported into other inferential paradigms. “Doublethink” shows that Bayesian model-averaged hypothesis testing is a closed testing procedure that controls the frequentist familywise error rate in the strong sense. It computes simultaneous posterior odds and asymptotic $\alpha$49-values, and permits post-hoc variable selection because the relevant intersection nulls are already tested; the paper also emphasizes finite-sample inflation and the need to mitigate highly correlated covariates, for example by testing groups of correlated variables (Fryer et al., 2023). In equivalence testing, post-hoc margin selection is handled by reporting a data-dependent bound $\alpha$50 and, more generally, a uniformly valid curve $\alpha$51 obtained by inverting a non-decreasing family of e-values (Koobs et al., 17 Mar 2026).

A further development is local post-hoc detection rather than global rejection. For dependence testing, Ćmiel and Gibas replace a single critical threshold by critical surfaces $\alpha$52 and $\alpha$53 for the quantile dependence function, so that one can identify where dependence occurs while preserving the overall significance level of the test. By Bonferroni, with local level $\alpha$54 on a $\alpha$55 grid and $\alpha$56, the global probability of any false local detection is at most $\alpha$57 (Ćmiel et al., 23 Dec 2025).

Taken together, these developments suggest that post-hoc hypothesis testing is no longer a single technical warning about “peeking” at $\alpha$58-values. It has become a general theory of inference under analyst adaptivity, with at least two non-equivalent forms of validity: simultaneous setwise guarantees in multiple testing, and e-value-based control of post-hoc risk under data-dependent $\alpha$59. The main unresolved point is not whether unrestricted post-hoc choice is harmless—it is not for ordinary $\alpha$60-value testing—but which replacement guarantee is scientifically appropriate for a given inferential task (Hemerik et al., 2024, Grünwald, 2022, Chugg et al., 9 Mar 2026).