Valid p-Values and Expectations of p-Values Revisited (2001.05126v1)

Abstract: A storm of favorable or critical publications regarding p-values-based procedures has been observed in both the theoretical and applied literature. We focus on valid definitions of p-values in the scenarios when composite null models are in effect. A valid p-value (VpV) statistic can be used to make a prefixed level-decision. In this context, Kolmogorov Smirnov goodness-of-fit tests and the normal two sample problem are considered. In particular, we examine an issue regarding the goodness-of-fit testability based on a single observation. This article exemplifies constructions of new test procedures, advocating practical reasons to implement VpV-based mechanisms. The VpV framework induces an extension of the conventional expected p-value (EPV) tool for measuring the performance of a test. Associating the EPV concept with the receiver operating characteristic (ROC) curve methodology, a well-established biostatistical approach, we propose a Youden index based optimality principle to derive critical values of decision making procedures. In these terms, the significance level alpha=0.05 can be suggested, in many situations. In light of an ROC curve analysis, we introduce partial EPVs to characterize properties of tests including their unbiasedness. We also provide the intrinsic relationship between the Bayes Factor (BF) test statistic and the BF of test statistics. Keywords: AUC; Bayes Factor; Expected p-value; Kolmogorov Smirnov tests; Likelihood ratio; Nuisance parameters; P-value; ROC curve; Pooled data; Single observation; Type I error rate; Youden index