Compound p-values: Methods & Applications

• Compound p-values are p-like inferential quantities that integrate information from multiple tests or studies to produce a unified significance measure.
• They employ a variety of combination rules—such as Fisher’s, Stouffer’s, and Tippett’s methods—to optimize sensitivity under different evidence profiles and dependence structures.
• These approaches extend conventional p-values to composite null hypotheses and multiple-testing frameworks, improving replicability and error control in research.

Compound p-values are p-like inferential quantities formed by combining information beyond a single simple test statistic. In the literature, the term is used in several related but nonidentical senses: as p-values that depend on all available data rather than only the data for one hypothesis; as combined p-values obtained by merging several study-level or test-level p-values; and, in a recent multiple-testing formulation, as p-values that satisfy superuniformity only on average across the true nulls rather than coordinatewise. Related work also treats p-values for composite null models and partial-conjunction or replicability hypotheses as part of the same conceptual family (Habiger et al., 2011, Ignatiadis et al., 2024, Barber et al., 29 Jul 2025, Vexler, 2020).

1. Terminological scope

The phrase has no single universally accepted definition. Some papers use it for cross-hypothesis borrowing in multiple testing, some for evidence combination across studies, and some for average-validity relaxations of ordinary p-values. A useful way to organize the literature is by the inferential object being preserved: individual null validity, combined-study evidence, or average null validity.

Usage	Core condition or construction	Representative source
Cross-hypothesis borrowing	$P_k=P_k(X)$ depends on all data but remains a bona fide p-value	(Habiger et al., 2011, Ignatiadis et al., 2024)
Combined evidence	Merge $p_1,\dots,p_K$ into one valid p-value	(Vovk et al., 2012, Heard et al., 2017)
Average-validity multiple testing	$\sum_{i\in H} P(p_i\le t)\le mt$	(Barber et al., 29 Jul 2025)
Composite-null validity	$P(p\le a)\le a$ uniformly over nuisance values	(Vexler, 2020)

In the cross-hypothesis sense, compound p-values are ordinary valid p-values that use information from other hypotheses. Habiger-style compound p-values are explicitly described as bona fide p-values that depend on the full data $X$, rather than only on the $k$-th component $X_k$; the same section of the later e-value literature suggests that “non-separable p-values” is a more precise label for this usage (Ignatiadis et al., 2024). In the average-validity sense, by contrast, the defining condition is not per-null superuniformity but the weaker ensemble property

$\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$

which generalizes ordinary p-values and is the formal definition of compound p-values in that framework (Barber et al., 29 Jul 2025).

A broader interpretation includes p-values for composite nulls with nuisance parameters. In that literature, the relevant question is how to define a valid p-value for $H_0:X\sim f_0(x;\theta)$ when $\theta$ is unknown, for example via

$p_1,\dots,p_K$0

with validity understood as $p_1,\dots,p_K$1 for all $p_1,\dots,p_K$2 (Vexler, 2020).

2. Combination rules and merged p-values

A major branch of the subject treats compound p-values as single omnibus p-values obtained by combining several input p-values. Under independence, classical combination rules have distinct power profiles. Fisher’s statistic

$p_1,\dots,p_K$3

is sensitive to many moderately small p-values; Stouffer’s inverse-normal sum

$p_1,\dots,p_K$4

is natural when evidence aggregates linearly on a $p_1,\dots,p_K$5-scale; Tippett’s method uses $p_1,\dots,p_K$6 and is strongest for sparse alternatives; Edgington’s method uses the sum $p_1,\dots,p_K$7 and is less dominated by a single extreme observation. The central comparative result is that no combiner is uniformly best: different rules are optimal against different alternative structures (Heard et al., 2017).

When dependence is unknown, valid merging requires explicit calibration. A general arbitrary-dependence theory based on generalized means shows that

$p_1,\dots,p_K$8

are valid merged p-values built from the arithmetic, geometric, and harmonic means, respectively. The factor $p_1,\dots,p_K$9 for the arithmetic mean is exact and cannot be reduced in general; $\sum_{i\in H} P(p_i\le t)\le mt$0 is asymptotically precise for the geometric mean; and $\sum_{i\in H} P(p_i\le t)\le mt$1 is the correct asymptotic scale for the harmonic mean, while $\sum_{i\in H} P(p_i\le t)\le mt$2 is a finite-sample safe rule (Vovk et al., 2012).

The same combination logic appears in partial-conjunction and replicability testing. If $\sum_{i\in H} P(p_i\le t)\le mt$3 are ordered p-values and the target null is that at most $\sum_{i\in H} P(p_i\le t)\le mt$4 component nulls are false, then a valid partial-conjunction p-value is obtained by applying any valid global-null combiner $\sum_{i\in H} P(p_i\le t)\le mt$5 to the largest $\sum_{i\in H} P(p_i\le t)\le mt$6 p-values: $\sum_{i\in H} P(p_i\le t)\le mt$7 In that setting, simulations found that Stouffer works well when null p-values are uniform and signal is low, Fisher works better when null p-values are conservative, and the minimum method works well when evidence is concentrated in a few non-null components (Hoang et al., 2021).

In meta-analysis based on combined p-value functions, the choice of combiner affects not only hypothesis testing but also point and interval estimation. Among the methods systematically compared for one-sided p-value functions $\sum_{i\in H} P(p_i\le t)\le mt$8, only Edgington’s sum-of-p-values method was orientation-invariant, because replacing $\sum_{i\in H} P(p_i\le t)\le mt$9 by $P(p\le a)\le a$0 leaves the Irwin–Hall symmetry intact. The same study emphasized that Edgington’s method can yield confidence intervals that are not constrained to be symmetric around the point estimate (Held et al., 2024).

3. Cross-hypothesis borrowing in multiple testing

In large-scale testing, compound p-values often mean p-values that borrow strength across hypotheses while preserving the null properties required by standard multiple-testing procedures. The canonical construction uses a multiple decision process $P(p\le a)\le a$1 and defines

$P(p\le a)\le a$2

A p-value statistic is called simple if the $P(p\le a)\le a$3-th p-value depends only on the $P(p\le a)\le a$4-th row or test, and compound if it depends on the full data matrix. The key technical device is sample splitting: training data $P(p\le a)\le a$5 are used to estimate shared structure across tests, and test-specific data $P(p\le a)\le a$6 are used to calibrate the conditional size. If

$P(p\le a)\le a$7

under the null, then the resulting p-values remain null-uniform; with suitable independence assumptions on the null components, they also remain null-independent (Habiger et al., 2011).

The location-shift example in that framework illustrates the mechanism. Shared training data estimate a directional weight $P(p\le a)\le a$8, and the compound p-value becomes

$P(p\le a)\le a$9

This construction uses the full dataset through $X$0 but still satisfies the conditional null calibration needed for standard FDR procedures (Habiger et al., 2011).

A more recent empirical-partially-Bayes formulation uses the entire collection of nuisance estimates to construct per-hypothesis p-values. In the Gaussian means model with unknown variances, the oracle partially Bayes p-value is

$X$1

Replacing the unknown variance prior $X$2 by a nonparametric MLE $X$3 yields $X$4, which is compound because the p-value for hypothesis $X$5 depends on all sample variances through $X$6. The paper proves nearly parametric approximation rates for these p-values and asymptotic BH/FDR control both in the hierarchical model and in a fixed-variance compound setting (Ignatiadis et al., 2023).

4. Replicability and additive compound p-values

A particularly simple and influential two-study construction uses the sum of p-values as both a combination method and a replicability criterion. With one-sided p-values $X$7 from the original study and $X$8 from the replication study, define

$X$9

Under the intersection null and independence, $k$0 has an Irwin–Hall distribution with parameter $k$1, and the valid combined p-value is

$k$2

To match the overall type-I error of the two-trials rule at one-sided level $k$3, replication success is declared when

$k$4

which in the practically relevant branch is equivalent to

$k$5

At $k$6, this gives the explicit budget $k$7, allowing a strong replication to rescue a borderline original while preserving the same overall false-positive probability $k$8 under the intersection null (Held et al., 2024).

The same work develops a weighted version

$k$9

intended to downweight the original study when it is viewed as more vulnerable to questionable research practice, publication bias, or effect inflation. For $X_k$0 and $X_k$1, the practically relevant success condition becomes

$X_k$2

so at $X_k$3,

$X_k$4

This admits a less convincing original result than the unweighted rule but requires a stricter replication; indeed, success is impossible if $X_k$5. The conditional replication thresholds are

$X_k$6

for the unweighted rule and

$X_k$7

for the $X_k$8-weighted rule. The unweighted version can reduce replication sample size when the original study is already very convincing, with reported maximum reductions of $X_k$9 or $\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$0 for conditional power targets $\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$1 and $\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$2, and $\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$3 or $\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$4 under predictive-power planning; the weighted version always requires a larger replication sample than the two-trials rule (Held et al., 2024).

The same paper positions additive compound p-values against three alternatives. Relative to the two-trials rule, the sum criterion is less dichotomous because it uses a linear budget rather than fixed marginal cutoffs. Relative to Fisher’s method and fixed-effect meta-analysis, it enforces the idea that both studies must contribute evidence, rather than allowing one spectacular p-value to overwhelm a weak or even contradictory replication. In the empirical analyses of four major replication projects, the additive method was mostly similar to the two-trials rule but was modestly more permissive in cases where the original p-value was just above $\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$5 and the replication p-value was very small (Held et al., 2024).

5. Average-validity, p*-values, and calibration to e-values

A newer multiple-testing meaning of compound p-values relaxes individual validity to average validity across the true nulls. If $\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$6 is the true-null index set, then $\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$7 are compound p-values when

$\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$8

Under independence, BH applied to compound p-values no longer has exact nominal control, but it still satisfies

$\sum_{i\in H} P(p_i\le t)\le mt,\qquad t\in[0,1],$9

This cannot be improved to $H_0:X\sim f_0(x;\theta)$0 in general: there are independent compound p-values for which $H_0:X\sim f_0(x;\theta)$1. Under the global null, the upper bound improves to

$H_0:X\sim f_0(x;\theta)$2

with a corresponding lower bound $H_0:X\sim f_0(x;\theta)$3. Under PRDS-type positive dependence, however, BH can suffer inflation of order $H_0:X\sim f_0(x;\theta)$4, and the paper gives a lower bound of $H_0:X\sim f_0(x;\theta)$5 (Barber et al., 29 Jul 2025).

Closely related is the notion of average significance controlling p-values, defined by

$H_0:X\sim f_0(x;\theta)$6

That framework presents these p-values as the p-value analogue of compound e-values. It also makes explicit that Habiger-style compound p-values and average-significance-valid p-values are not the same object. In that terminology, p-BH need not control FDR even if the p-values are jointly independent and average significance controlling, whereas p-BY controls FDR under arbitrary dependence, and p-BH controls under weak-dependence asymptotics (Ignatiadis et al., 2024).

The p*-value framework further broadens the landscape by introducing an intermediate object between p-values and e-values. A p*-variable is characterized by the stochastic order $H_0:X\sim f_0(x;\theta)$7, where $H_0:X\sim f_0(x;\theta)$8. Every p-variable is automatically a p*-variable; every mid p-value is a p*-value; and a p*-variable is exactly a convex combination of p-variables, with any p*-variable representable as the arithmetic average of three p-variables. This makes p*-values a natural closure class for averaging and coarsening operations. In particular, if $H_0:X\sim f_0(x;\theta)$9 are p*-variables, then $\theta$0 is again a p*-variable, and the universal calibrator

$\theta$1

turns a combined p*-value into an ordinary p-value. The same theory provides a dependence-robust geometric merger

$\theta$2

and explicit bridges between p*-values and e-values (Wang, 2020).

6. Composite-null, discrete, and dependence-sensitive regimes

Compound-p-value methodology becomes more delicate when nulls are composite, test statistics are discrete, or dependence is substantial. For composite null models with nuisance parameters, one classical route is to take the worst-case p-value

$\theta$3

or the Berger–Boos/Silvapulle refinement

$\theta$4

These are valid in the sense that $\theta$5 for all $\theta$6, but they may be strongly conservative. In fact, in one-observation Kolmogorov–Smirnov examples with an unknown nuisance parameter, the paper shows $\theta$7, and the confidence-set version is also noninformative (Vexler, 2020).

A sharper existence theory for powerful exact composite p-values is available for convex-polytopal null and alternative classes. For finite $\theta$8 and $\theta$9, exact nontrivial p-values exist if and only if

$p_1,\dots,p_K$00

whereas ordinary conservative nontrivial p-values exist if and only if

$p_1,\dots,p_K$01

A notable implication is that exact powerful p-values may fail to exist in the full data filtration but may appear after coarsening the data or batching observations (Zhang et al., 2023).

Discrete inputs create a different problem: the standard continuous null references for Fisher, Pearson, George, Stouffer, and Edgington no longer apply. One recent framework addresses this by replacing each transformed discrete p-value with a Wasserstein-optimal adjusted value and approximating the null distribution of the sum by a continuous surrogate with matching first two moments. For Edgington’s statistic, the optimal adjusted component is the midpoint

$p_1,\dots,p_K$02

which coincides with the mid-p-value, and the paper emphasizes the variance ratio $p_1,\dots,p_K$03 as a practical selector of which combination statistic is likely to be best calibrated for a given discrete null distribution (Contador et al., 4 Aug 2025).

A complementary discrete-data literature studies randomized p-values for composite nulls. In one-parameter exponential-family settings, the least-favorable-configuration p-value $p_1,\dots,p_K$04, a single-stage randomized version $p_1,\dots,p_K$05, a first-stage exact randomized tail probability $p_1,\dots,p_K$06, and a two-stage randomized version $p_1,\dots,p_K$07 are all valid. The two-stage procedure is designed to address both discreteness and the extra conservativeness induced by interior null values, and in the binomial examples it is closest to $p_1,\dots,p_K$08 under the null. The same paper reports that powers based on $p_1,\dots,p_K$09 and $p_1,\dots,p_K$10 increase monotonically with sample size, unlike the non-randomized least-favorable p-value (Ochieng et al., 2022).

Dependence can also alter the meaning of compound p-values. Heavy-tailed combination tests such as the Cauchy combination and harmonic-mean-style rules are asymptotically valid for fixed $p_1,\dots,p_K$11 and $p_1,\dots,p_K$12 under pairwise Gaussian dependence without perfect correlations, but in that regime they become asymptotically equivalent to weighted Bonferroni whenever the transformed variables are pairwise quasi-asymptotically independent. Under $p_1,\dots,p_K$13-like tail dependence, however, simulations suggest that the same heavy-tailed combinations can remain well calibrated and can substantially outperform Bonferroni (Gui et al., 2023).

The literature therefore treats compound p-values less as a single formula than as a family of constructions for preserving p-value-like calibration while borrowing information, combining evidence, or relaxing validity in controlled ways. The unifying theme is that the inferential burden is shifted from a single-test null law to a richer structure: across studies, across hypotheses, across nuisance values, or across nulls on average.