Determine meaningful effect sizes and sources of variance for rigorous NHST in cell biology

Determine biologically meaningful effect sizes and characterize sources of experimental variance for cell biology experiments sufficiently to enable the design and execution of rigorous null hypothesis significance testing (NHST) with pre-specified quantitative null hypotheses and effect-size thresholds.

Background

The paper argues that routine testing of a nil null (no-effect) hypothesis has led to misleading interpretations in cell biology, especially as sample sizes grow large and subtle effects become statistically detectable. It proposes focusing on effect sizes and using confidence intervals as default reporting.

Within this framework, the author notes that in many typical cell biology experiments, researchers lack sufficient prior knowledge of what constitutes a biologically meaningful effect size and of the major sources of variance. This lack of knowledge prevents the formulation of rigorous, quantitative null hypotheses for NHST and highlights a need to establish effect-size criteria and variance characterization to support hypothesis-driven testing when appropriate.

References

Now let's imagine that we understand the limits of testing for no effect, but we also don't know enough about the meaning of potential effect sizes or about sources of variance to design a rigorous NHST. This state of uncertainty about meaningful effect sizes is the norm for cell biology experiments.

— Alternatives to the statistical mass confusion of testing for no-effect  (2407.07114 - Morgan, 2024) in Main Text, section "The solution starts and ends with considering effect sizes," paragraph beginning "Now let's imagine that we understand the limits of testing for no effect..." (following Figure 1)