On Determining the Distribution of a Goodness-of-Fit Test Statistic

Abstract: We consider the problem of goodness-of-fit testing for a model that has at least one unknown parameter that cannot be eliminated by transformation. Examples of such problems can be as simple as testing whether a sample consists of independent Gamma observations, or whether a sample consists of independent Generalised Pareto observations given a threshold. Over time the approach to determining the distribution of a test statistic for such a problem has moved towards on-the-fly calculation post observing a sample. Modern approaches include the parametric bootstrap and posterior predictive checks. We argue that these approaches are merely approximations to integrating over the posterior predictive distribution that flows naturally from a given model. Further, we attempt to demonstrate that shortcomings which may be present in the parametric bootstrap, especially in small samples, can be reduced through the use of objective Bayes techniques, in order to more reliably produce a test with the correct size.