Replicability Failure Probability Explained

• Replicability failure probability is the chance that a repeated experiment does not reproduce the same substantive outcome, applicable across algorithms, transfer learning, and classical statistics.
• It is quantified through diverse methods—exact output agreement, performance divergence beyond a tolerance, or failing replication criteria—depending on the testing framework.
• Practical insights show that achieving lower failure rates often requires larger sample sizes, while heterogeneity, adaptive sensitivity, and environmental shifts can increase the risk of replication failure.

Replicability failure probability is the probability that a nominal repetition of a procedure, fed fresh data from the same underlying mechanism, does not reproduce the same substantive outcome. In algorithmic work, the outcome is often an exact output such as a test decision or policy; in transfer learning it may be a performance gap exceeding a tolerance; and in classical replication statistics it may be failure of a future study to attain the same sign, significance threshold, or replication-success criterion. The quantity is therefore not tied to a single formalism, but to a family of agreement, divergence, and decision-risk notions indexed by what counts as a successful replication (Diakonikolas et al., 3 Jul 2025, Singh et al., 6 Aug 2025, Costello et al., 2022).

1. Formal meanings across research traditions

In the algorithmic replicability framework, the canonical definition uses two independent samples from the same distribution together with shared internal randomness. For distribution testing, a randomized algorithm $A:\mathcal X^n\mapsto \mathcal Y$ is $\rho$-replicable if

$$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$

equivalently,

$\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$

Here $\rho$ is the replicability failure probability: the probability of disagreement across two independent resamplings from the same target distribution. The same exact-output notion is used in discounted tabular reinforcement learning for policy estimation, where failure means that two executions with the same internal randomness output different policies (Diakonikolas et al., 3 Jul 2025, Karbasi et al., 2023).

In transfer learning, the failure event is not exact model disagreement but excessive performance divergence. A procedure is $\rho$-replicable with tolerance $\epsilon$ if, for two independent target samples $T,T'$,

$$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$

Thus $\rho$ measures the chance that two independently trained models differ in true target performance by more than the tolerated amount (Singh et al., 6 Aug 2025).

In classical replication statistics, the same phrase often refers to the complement of a replication-success probability. Under a distributional-null model, for example, the paper on distributional null hypothesis tests defines

$\rho$0

where $\rho$1 is the probability that a future replication is significant in the same direction as the original result. Other statistical papers define failure as failure to exceed a substantively meaningful cutoff in an exact replication, failure to satisfy a sceptical-$\rho$2-value or sum-of-$\rho$3-values success rule, or false declaration of replication success under null configurations (Costello et al., 2022, Segal, 2018, Micheloud et al., 2022).

Setting	Failure event	Formal parameterization
Distribution testing	Two runs disagree on the test output	$\rho$4
Transfer learning	Two runs differ in target risk by more than $\rho$5	$\rho$6
Replication statistics	Future study fails the chosen success criterion	Often $\rho$7

These definitions are not interchangeable. Shared-randomness algorithmic replicability requires exact or tolerance-based agreement under repeated sampling from one data-generating process, whereas statistical replication frameworks may incorporate between-study heterogeneity, predictive conditioning on an original result, or Type-I error control for replication claims. A plausible implication is that “replicability failure probability” is best understood as a task-dependent functional rather than a universally standardized scalar.

2. Canonical disagreement-based formulations in algorithms and learning

A major structural development in hypothesis testing is that any replicable tester can be reduced, without worsening accuracy or sample complexity, to a canonical random-threshold form. In that form, the algorithm computes a deterministic statistic $\rho$8, samples $\rho$9, and accepts iff $$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$0. The disagreement probability between two executions on fixed datasets $$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$1 is then exactly

$$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$2

The same work shows that the tester can be taken to be invariant to sample order, and for symmetric discrete testing problems also invariant to relabeling of domain elements. This makes replicability failure analyzable as an expected absolute gap between deterministic scores on independent samples (Aamand et al., 3 Jul 2025).

That canonicalization has two consequences. First, it separates accuracy from stability: a tester must be stable on all distributions, not only on the completeness and soundness classes. Second, it aligns lower bounds with concentration questions about acceptance statistics rather than with ad hoc properties of a particular randomized implementation. This suggests that replicability failure is often governed by how much a statistic fluctuates under independent resampling, with randomization serving primarily as a thresholding device.

In learning theory, a related but distinct notion is global stability, which does not condition on shared randomness. A learner is $$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$3-globally stable if for every distribution $$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$4 there exists a predictor $$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$5 such that

$$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$6

equivalently,

$$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$7

The paper on replicability and stability proves that global stability is equivalent to list replicability and that for every concept class $$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$8,

$$\Pr_{r,\,T,T'}\!\left(A(T;r)=A(T';r)\right)\ge 1-\rho,$$9

Hence the exact-agreement failure probability in this setting is

$\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$0

The same paper shows that, unlike shared-randomness replicability, global stability generally cannot be boosted to probability arbitrarily close to $\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$1, and that except for trivial cases, highly replicable algorithms in the Impagliazzo et al. sense must be randomized (Chase et al., 2023).

3. Quantitative tradeoffs in testing and reinforcement learning

Replicable distribution testing makes the cost of lowering failure probability explicit. For uniformity testing over $\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$2, the sample complexity of $\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$3-replicable testing is

$\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$4

For closeness testing over $\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$5, the lower and upper bounds match up to polylogarithmic factors: $\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$6 and

$\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$7

For independence testing over $\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$8 with $\Pr_{r,T,T'}\big[A(T;r)\neq A(T';r)\big]\le \rho.$9, the upper bound is

$\rho$0

The authors emphasize a heuristic bridge to high-confidence testing: $\rho$1 They also remark that if a tester is $\rho$2-replicable and correct with probability $\rho$3, then its correctness can effectively be boosted to at least $\rho$4 in the replicability framework (Diakonikolas et al., 3 Jul 2025).

The lower-bound method for these results combines Poissonization, a meta-lemma that reduces randomized lower bounds to deterministic ones via a “good string,” and a random-walk concentration argument on Poissonized hard instances. For uniformity hard instances, the sample random walk has mixing time

$\rho$5

and if $\rho$6 is $\rho$7-replicable with respect to the hard family, then

$\rho$8

An analogous acceptance-probability concentration statement is proved for closeness. The near-matching upper and lower bounds show that, for uniformity and closeness, $\rho$9 cannot be pushed arbitrarily low without a corresponding increase in sample size (Diakonikolas et al., 3 Jul 2025).

A closely related structural line gives state-of-the-art upper and lower bounds for several testing problems. For coin testing, expected and worst-case sample complexities scale as

$\rho$0

with success probability at least

$\rho$1

For Gaussian mean testing, the paper gives a polynomial-time $\rho$2-replicable tester using

$\rho$3

samples, and proves a lower bound of

$\rho$4

These results reinforce the same conclusion: low disagreement probability is attainable, but its dependence on $\rho$5 is intrinsic rather than a proof artifact (Aamand et al., 3 Jul 2025).

In reinforcement learning, strict replicability is defined by exact policy equality under shared randomness: $\rho$6 For discounted tabular MDPs with a generative model, strict $\rho$7-replicable policy estimation has sample and time complexity

$\rho$8

while deterministic $\rho$9-replicable policy estimation requires

$\epsilon$0

samples. The same paper distinguishes the accuracy failure parameter $\epsilon$1 from the replicability failure parameter $\epsilon$2, studies the weaker notion of $\epsilon$3-TV indistinguishability with complexity

$\epsilon$4

and introduces approximate replicability with divergence-based failure probability $\epsilon$5 and complexity

$\epsilon$6

Only the strict notion guarantees exact equality of outputs; the latter two replace equality by distributional or divergence-based closeness (Karbasi et al., 2023).

4. Replication probabilities in statistical inference

A major statistical interpretation of replicability failure probability arises from predictive replication under between-experiment heterogeneity. In the distributional-null framework, the true effect varies across experiments as

$\epsilon$7

so an experiment-level sample mean obeys

$\epsilon$8

Writing $\epsilon$9, the predictive replication probability for a result with statistic $T,T'$0 is

$T,T'$1

and the failure probability is its complement: $T,T'$2 The paper validates this framework on Many Labs 1, reporting a correlation of about $T,T'$3 between predicted and observed replication rates, with slope about $T,T'$4, and argues that point-null NHST underestimates variability because it ignores between-experiment variation (Costello et al., 2022).

A more direct exact-replication perspective is given by exceedance probability. If a replication estimate is normally distributed, the probability that it exceeds a cutoff $T,T'$5 is

$T,T'$6

The paper derives frequentist confidence intervals for this probability and interprets low exceedance probability as high replication-failure risk relative to the chosen cutoff. Its examples show that statistical significance can coexist with substantial instability: in one simulated normal-mean example with one-sided $T,T'$7, the probability that a replication with $T,T'$8 exceeds $T,T'$9 could still be as low as about $$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$0 with $$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$1 confidence; in an Open Science Collaboration example with one-sided $$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$2, the probability that a future replication estimate is positive could be as low as $$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$3, implying up to a $$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$4 chance of sign reversal from sampling variability alone (Segal, 2018).

The distinction between same-sign and same-significance replication is especially sharp. One paper states that when the original study has one-sided $$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$5 and the replication has the same sample size and variance, the probability that the replication again achieves one-sided $$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$6 is about $$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$7, so the corresponding failure probability is about $$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$8. For the weaker same-sign criterion, the success probability is about $$\Pr_{T, T' \sim \mathcal{D}_T^n} \left[ |R_{D_T}(h_T) - R_{D_T}(h_{T'})| > \epsilon \right] \leq \rho.$$9, with failure about $\rho$0. In the infinite-replication-sample limit, the same-sign replication probability becomes $\rho$1 for an original one-sided $\rho$2 (Llewelyn, 15 Dec 2025).

The variability of replication $\rho$3-values is itself a source of failure-probability inflation. The paper on $\rho$4-value variability argues that classical $\rho$5-intervals rely on unrealistic implicit assumptions about extremely diffuse effect-size distributions. Under selection, their coverage can collapse: for $\rho$6 and $\rho$7, reported coverage drops to $\rho$8, and for the minimum $\rho$9-value among $\rho$00 tests with the same $\rho$01, coverage drops to $\rho$02. The proposed Mixture Bayes intervals are presented as directly interpretable probabilistic bounds for replication $\rho$03-values and as more resistant to winner’s-curse effects (Vsevolozhskaya et al., 2016).

5. Error-controlled replication claims and decision rules

In some statistical frameworks, replicability failure probability is not the chance that two studies disagree, but the probability of wrongly declaring replication success. The calibrated sceptical $\rho$04-value is constructed precisely for this purpose. Under independent normal effect estimates,

$\rho$05

replication success at level $\rho$06 is defined through the joint criterion

$\rho$07

together with $\rho$08 and $\rho$09. The recalibrated quantity $\rho$10 is defined so that

$\rho$11

Thus the false replication-success probability under the global null is exactly $\rho$12. The same paper distinguishes overall, partial, and conditional Type-I error, and states that the conditional Type-I error rate is always bounded by $\rho$13 (Micheloud et al., 2022).

The sum-of-$\rho$14-values approach provides another error-controlled replication rule. In its unweighted form, replication success is declared when

$\rho$15

which at $\rho$16 becomes $\rho$17. In its weighted form with $\rho$18, success requires

$\rho$19

The paper calibrates both versions to preserve the same overall Type-I error as the two-trials rule: $\rho$20 It also reports conditional false-success thresholds given the original result, such as $\rho$21 in the unweighted case. This framework treats replicability failure probability as a decision-theoretic error rate associated with a replication-success rule rather than as a predictive disagreement probability (Held et al., 2024).

Selection-adjusted large-scale replication assessment introduces yet another interpretation. The paper on statistical methods for replicability assessment defines failure through three distinct objects: false directional claims, exact effect mismatch, and effect decline. For false directional claims, the directional null is

$\rho$22

and the directional false discovery proportion $\rho$23 acts as a failure-rate estimand among published significant results. In the Reproducibility Project: Psychology data, the paper reports an estimate of $\rho$24 false directional claims with a $\rho$25 upper confidence bound of $\rho$26; using replication $\rho$27-values as a check gives $\rho$28 with upper bound $\rho$29. Under stricter thresholds, the estimates fall to $\rho$30 at $\rho$31, $\rho$32 at $\rho$33, and $\rho$34 at $\rho$35. For exact replication, only $\rho$36 studies reject the exact-replication null after selection adjustment, compared with $\rho$37 without adjustment. For effect decline, the estimated fraction declining is $\rho$38, with $\rho$39 confidence interval $\rho$40 (Hung et al., 2019).

These decision-rule formulations shift attention from predictive repeatability to inferential error control. They do not ask whether two studies literally agree under repeated sampling; they ask whether a replication-success claim can be made while controlling the probability of false success under specified null hypotheses. A plausible implication is that “replicability failure probability” can denote either a predictive instability quantity or an error-rate quantity, depending on whether the object of interest is repeated outcome agreement or the validity of a replication claim.

6. Heterogeneity, adaptivity, and limits of a single failure-probability summary

Adaptive training procedures make failure probability sensitive to how strongly the method reacts to small data perturbations. In transfer learning with adaptive data selection, selection sensitivity is defined by

$\rho$41

The paper proves a stability lemma

$\rho$42

and the replicability bound

$\rho$43

This yields the sample-size requirement

$\rho$44

and, in the appendix,

$\rho$45

On MultiNLI, direct fine-tuning at batch size $\rho$46 gives failure rates of $\rho$47 for uniform sampling, $\rho$48 for importance weighting, $\rho$49 for confidence sampling, $\rho$50 for curriculum learning, $\rho$51 for uncertainty-aware curriculum, and $\rho$52 for gradient-based selection; two-stage fine-tuning with source-domain pretraining reduces these to $\rho$53, $\rho$54, $\rho$55, $\rho$56, $\rho$57, and $\rho$58, respectively (Singh et al., 6 Aug 2025).

Environmental heterogeneity can produce the same phenomenon in classical experiments. In the mixed-model account of replication across environments, the environmental effect ratio is

$\rho$59

Under the follow-up model,

$\rho$60

has a noncentral $\rho$61-distribution, and replicability power is

$\rho$62

For large samples,

$\rho$63

while the large-sample limit is

$\rho$64

The wrong-direction significance probability approaches

$\rho$65

Thus large sample size does not necessarily drive failure probability to zero when environment-by-treatment interaction remains substantial (Higgins et al., 2019).

At the level of replication programs, non-exactness complicates the very meaning of a single failure probability. In the benchmark model for binary replication verdicts,

$\rho$66

with $\rho$67 Beta-distributed, the mean replicability-rate estimator has variance

$\rho$68

so that

$\rho$69

as $\rho$70. The effective sample size is

$\rho$71

In the operational model, where each experiment has its own latent rate $\rho$72, the heterogeneity parameter $\rho$73 disappears entirely when there is only one binary verdict per experiment, $\rho$74. The paper concludes that standard replication data with one verdict per experiment do not identify between-experiment heterogeneity and cannot reliably distinguish high- from low-replicability regimes (Devezer et al., 29 Apr 2026).

An applied engineering study reaches a related practical conclusion without a formal failure-probability calculation. In inter-laboratory replication of a finite-element model for human femur failure load, the reproduced model was highly correlated with the original on the Leuven dataset ($\rho$75 for reproduced versus original predictions), but performance degraded on the Lyon dataset to $\rho$76 overall and $\rho$77 for lesion cases, with changed bias and larger dispersion. The authors therefore infer that agreement is much higher on the same dataset than across changed data sources and preprocessing pipelines, even though no explicit probability of replicability failure is reported (Gardegaront et al., 2024).

Across these literatures, a consistent pattern emerges. Replicability failure probability decreases with more data when the data-generating regime is stable, but it rises with adaptive sensitivity, between-experiment heterogeneity, environmental change, and problem formulations that demand exact output equality rather than coarse directional agreement. The main controversy is not whether such probabilities matter, but whether a single scalar can summarize them outside well-specified models. The recent literature is therefore divided between settings where $\rho$78 is a precise algorithmic disagreement probability and settings where heterogeneity makes any universal “replicability rate” unstable or non-identifiable.