The Difference Between "Replicable" and "Not replicable" is not Itself Scientifically Replicable

Abstract: Replication studies estimate the replicability rate of scientific results by aggregating binary verdicts of experiments. Exact replications are rarely attainable, so most replication sequences are non-exact. Experiments differ in ways that matter and do not share a single data-generating process. We formalize two statistical interpretations of non-exactness. In a shared latent rate (benchmark) model, experiments are exchangeable and depend on a common random replicability rate. In a conditionally independent rates (operational) model, each experiment has its own replicability rate drawn from a population distribution. Under the benchmark model, even small variability among replicability rates induces an irreducible variance floor on the estimated mean replicability rate that no amount of replication can eliminate. Under the operational model, the degree of non-exactness is not identifiable from standard replication data, because one binary verdict per experiment carries no information about between-experiment heterogeneity. Researchers cannot tell which precision regime they are in or whether high- and low-replicability sequences can be distinguished in principle. The usual data structure cannot support reliable demarcation between "replicable" and "not replicable" results and systematically understates uncertainty, making high- and low-replicability sequences appear discriminable when they are not. We show how common sources of heterogeneity amplify these problems and demonstrate practical consequences in a reanalysis of Many Labs 4. Aggregating replicability rates across heterogeneous literatures produces averages that conflate incommensurable regimes and lack a stable interpretation. Replicability rate is not a reliable demarcation criterion. The replication crisis, if there is one, cannot be established by the methods used to declare it.

• The paper shows that binary replicability assessments are fundamentally limited by an irreducible variance floor induced by latent heterogeneity (ρ).
• It employs analytical and simulation methods to quantify the loss in effective sample size and reveals severe overlap in posterior probabilities for different replicability rates.
• The study advocates for methodological reforms, including richer data collection and hierarchical modeling, to better account for non-exact replications.

Demarcation Failures in Replicability Rate Estimation

Statistical Foundations and Model Formalization

The paper provides a rigorous statistical treatment of replication in science by elucidating two core models addressing non-exact replications: the benchmark (shared latent rate) and operational (conditionally independent rates) models. The benchmark model assumes outcome exchangeability with a common latent replicability rate $\phi$, governed by intraclass correlation $\rho$, inducing a Beta-Binomial likelihood. This generates an irreducible variance floor for estimators of the mean replicability rate $\hat{\mu}$, quantifiable as $\mu(1-\mu)\rho$, which persists regardless of accumulated replication count. The operational model instead posits that each experiment has its own replicability rate, leading to independent Bernoulli or Beta-Binomial outcomes, and critically, rendering $\rho$ non-identifiable from the typical data structure: one binary verdict per experiment.

This distinction is not merely theoretical but exposes structural inferential limits in standard replication practices. While the benchmark model can diagnose the impact of heterogeneity (quantified by $\rho$) on estimation precision, the operational model demonstrates that, given standard design protocols, experimenters are fundamentally incapable of learning about heterogeneity from binary outcomes alone.

Irreducible Variance and Sample Size Effects

The paper quantifies the effect of non-exactness through analytical and simulation-based evaluations. Under exact replication ($\rho=0$), variance of $\hat{\mu}$ decreases linearly with replication count $m$, supporting consistent discrimination between high- and low-replicability regimes.

However, even modest non-exactness ($\rho>0$) results in an irreducible variance floor (Figure 1).

Figure 1: Theoretical sensitivity of the estimated mean replicability rate $\rho$0 to intraclass correlation $\rho$1, as governed by the benchmark model—demonstrating the emergence and persistence of a variance floor.

Boxes and figures further clarify that effective sample size $\rho$2 collapses rapidly with increasing $\rho$3. For large-scale projects (SCORE or RPP), the inferential utility is determined by $\rho$4 rather than the nominal replication count; at $\rho$5, a sequence of 274 replications is equivalent to 10 independent ones (Figure 2).

Figure 2: Relationship between nominal and effective replication count shows severe informational loss under even mild non-exactness in large-scale replication projects.

Failure of Binary Demarcation and Posterior Analysis

The core claim is formalized in likelihood and posterior analyses, which demonstrate that—even with favorable prior assumptions and large replication counts—the marginal posteriors for different true replicability rates ($\rho$6) substantially overlap. The pairwise posterior probability mass overlap for a sequence of $\rho$7 replications reveals that even extreme pairs (e.g., $\rho$8 vs. $\rho$9), under typical priors, cannot be reliably discriminated (Figure 3).

Figure 3: Pairwise marginal posterior overlaps for $\hat{\mu}$0 imply severe limits on discriminability between high- and low-replicability regimes.

Conditional posteriors for $\hat{\mu}$1 flatten and overlap as $\hat{\mu}$2 increases, meaning that under plausible non-exactness, even sequences with very high or low replicability rates cannot be reliably distinguished (Figure 4).

Figure 4: Conditional posterior distributions for several candidate $\hat{\mu}$3 under varying $\hat{\mu}$4 values demonstrate loss of discriminability.

Sampling distributions of $\hat{\mu}$5 further reinforce this point; moderate or high $\hat{\mu}$6 results in HDIs for high- and low-replicability sequences overlapping for practically feasible $\hat{\mu}$7, extinguishing any valid binary demarcation (Figure 5).

Figure 5: Sampling distribution of $\hat{\mu}$8 for the ML4 sequence shows that distinction between regimes is impossible at realistic replication counts and observed heterogeneity.

Empirical and Practical Consequences

Reanalysis of the Many Labs 4 dataset demonstrates severe practical ramifications. Even under the most favorable assumptions (minimal $\hat{\mu}$9 estimated from Hedges’ $\mu(1-\mu)\rho$0), the distribution of $\mu(1-\mu)\rho$1 remains so wide—due to high non-exactness—that binary demarcation is infeasible. Protocol standardization (AA vs. IH) only minimally reduces $\mu(1-\mu)\rho$2, and credible intervals for $\mu(1-\mu)\rho$3 span nearly the full scientifically relevant range.

The aggregation of replication rates across heterogeneous results and literatures, as commonly practiced in meta-science, further conflates incommensurable experimental regimes; mixture means are uninterpretable and their variance is dominated by heterogeneity unaccounted for by binary verdict data.

Comparisons between observed replicability and nominal power, or crisis declarations predicated on aggregate replicability rates, are shown to be statistically meaningless absent specification of heterogeneity and base rates. Even the supposed reference experiment may inject noise that distorts the entire sequence.

Theoretical and Practical Implications

The analysis establishes that binary verdict-based replication studies are structurally incapable of demarcating reliable from unreliable results under realistic non-exactness conditions. The mean replicability rate, as estimated from the canonical data structure, cannot serve as a disciplinary criterion. The methods currently used to diagnose or declare a "replication crisis" are inadequate, as the signal for demarcation is fundamentally absent.

Practical implications include the necessity to adopt richer data structures (e.g., continuous measures, hierarchical designs) to partially recover information about heterogeneity, and abandoning binary verdict-based aggregation as a demarcation tool. Future developments in meta-scientific methodology should focus on quantifying and modeling sources of non-exactness, incorporating continuous outcomes, and designing protocols to maximize identifiability of heterogeneity parameters.

Theoretically, the limitations shown here for binary replicability rates generalize to any practice where structural heterogeneity and aggregation over regimes are ignored; extensions to hierarchical Bayesian modeling and random-effects frameworks are warranted.

Conclusion

This paper rigorously proves that standard replication practices employing binary verdict aggregation are structurally incapable of reliably demarcating "replicable" from "not replicable" results under non-exactness. The irreducible variance floor and non-identifiability of heterogeneity preclude valid discriminability, rendering aggregate replicability rates scientifically uninterpretable. Methodological reform must prioritize rich data collection, formal heterogeneity modeling, and contextualized inference over simplistic binary aggregation.