Confidence intervals with maximal average power

Abstract: We propose a frequentist testing procedure that maintains a defined coverage and is optimal in the sense that it gives maximal power to detect deviations from a null hypothesis when the alternative to the null hypothesis is sampled from a pre-specified distribution (the prior distribution). Selecting a prior distribution allows to tune the decision rule. This leads to an increased power, if the true data generating distribution happens to be compatible with the prior. It comes at the cost of losing power, if the data generating distribution or the observed data are incompatible with the prior. We illustrate the proposed approach for a binomial experiment, which is sufficiently simple such that the decision sets can be illustrated in figures, which should facilitate an intuitive understanding. The potential beyond the simple example will be discussed: the approach is generic in that the test is defined based on the likelihood function and the prior only. It is comparatively simple to implement and efficient to execute, since it does not rely on Minimax optimization. Conceptually it is interesting to note that for constructing the testing procedure the Bayesian posterior probability distribution is used.