Principled selection of the p-value significance threshold α

Determine a principled criterion for selecting the significance threshold α that distinguishes significant from non-significant p-values in classical null hypothesis significance testing, thereby demarcating unlikely versus likely null hypotheses.

Background

The paper reviews the classical logic of significance testing and highlights a fundamental ambiguity: while p-values quantify the extremeness of observed data under a null hypothesis, there is no agreed-upon rule for where to draw the boundary between “unlikely” and “likely” hypotheses based on a fixed threshold α.

This ambiguity is central to widespread misinterpretations, as noted in the ASA statement and related literature. The authors’ proposed fuzzy-set perspective aims to avoid the arbitrary choice of α by interpreting the p-value as a membership function over parameter values, but the underlying question of establishing a principled α selection criterion remains explicitly stated as unresolved.

References

Although these arguments seem plausible, it is not known where to draw the line between unlikely and likely hypotheses, that is to say, it is not clear how to select the threshold a between significant and non-significant p-values, easily generating misinterpretations (Greenland, 2016).

— The p-value from a fuzzy point of view  (2508.07474 - Quatto, 10 Aug 2025) in Section 1 (Introduction)

In addition, when correcting for multiple comparisons, it is important to note that for the number of samples we generated per null distribution, we are unable to get precise p-values that are below $0.001$.

— Efficient Convex Optimization Methods for State-Space Heartbeat Dynamics Models with Gamma Generalized Linear Models  (2610.00884 - Liu et al., 1 Oct 2026) in Section 6, Discussion, paragraph beginning “Beyond model fitting”