A five-decision testing procedure to infer on unidimensional parameter

Abstract: A statistical test can be seen as a procedure to produce a decision based on observed data, where some decisions consist of rejecting a hypothesis (yielding a significant result) and some do not, and where one controls the probability to make a wrong rejection at some pre-specified significance level. Whereas traditional hypothesis testing involves only two possible decisions (to reject or not a null hypothesis), Kaiser's directional two-sided test as well as the more recently introduced Jones and Tukey's testing procedure involve three possible decisions to infer on unidimensional parameter. The latter procedure assumes that a point null hypothesis is impossible (e.g. that two treatments cannot have exactly the same effect), allowing a gain of statistical power. There are however situations where a point hypothesis is indeed plausible, for example when considering hypotheses derived from Einstein's theories. In this article, we introduce a five-decision rule testing procedure, which combines the advantages of the testing procedures of Kaiser (no assumption on a point hypothesis being impossible) and of Jones and Tukey (higher power), allowing for a non-negligible (typically 20%) reduction of the sample size needed to reach a given statistical power to get a significant result, compared to the traditional approach.