Significance and Replication in simple counting experiments: Distributional Null Hypothesis Testing (2010.09209v1)

Abstract: Null Hypothesis Significance Testing (NHST) has long been of central importance to psychology as a science, guiding theory development and underlying the application of evidence-based intervention and decision-making. Recent years, however, have seen growing awareness of serious problems with NHST as it is typically used; this awareness has led to proposals to limit the use of NHST techniques, to abandon these techniques and move to alternative statistical approaches, or even to ban the use of NHST entirely. These proposals are premature, because the observed problems with NHST arise as a consequence of an historically contingent, essentially unmotivated, and fundamentally incorrect, choice: that of NHST testing against point-form null hypotheses. Using simple counting experiments we give a detailed presentation of an alternative, more general approach: that of testing against distributional nulls. We show that this distributional approach is well-motivated mathematically, practically and experimentally, and that the use of distributional nulls addresses various problems with the standard point-form NHST approach, avoiding issues to do with sample size and allowing a coherent estimation of the probability of replication of a given experimental result. Rather than abandoning NHST, we should use the NHST approach in its most general form, with distributional rather than point-form null hypotheses.