Limitations of Randomization Tests in Finite Samples (2512.07099v1)

Abstract: Randomization tests yield exact finite-sample Type 1 error control when the null satisfies the randomization hypothesis. However, achieving these guarantees in practice often requires stronger conditions than the null hypothesis of primary interest. For instance, sign-change tests for mean zero require symmetry and fail to control finite-sample error for non-symmetric mean-zero distributions. We investigate whether such limitations stem from specific test choices or reflect a fundamental inability to construct valid randomization tests for certain hypotheses. We develop a framework providing a simple necessary and sufficient condition for when null hypotheses admit randomization tests. Applying this framework to one-sample tests, we provide characterizations of which nulls satisfy this condition for both finite and continuous supports. In doing so, we prove that certain null hypotheses -- including mean zero -- do not admit randomization tests. We further show that nulls that admit randomization tests based on linear group actions correspond only to subsets of symmetric or normal distributions. Overall, our findings affirm that practitioners are not inadvertently incurring additional Type 1 error when using existing tests and further motivate focusing on the asymptotic validity of randomization tests.