On admissibility in post-hoc hypothesis testing

Abstract: The validity of classical hypothesis testing requires the significance level $\alpha$ be fixed before any statistical analysis takes place. This is a stringent requirement. For instance, it prohibits updating $\alpha$ during (or after) an experiment due to changing concern about the cost of false positives, or to reflect unexpectedly strong evidence against the null. Perhaps most disturbingly, witnessing a p-value $p\ll\alpha$ vs $p\leq \alpha$ has no (statistical) relevance for any downstream decision-making. Following recent work of Gr\"unwald (2024), we develop a theory of post-hoc hypothesis testing, enabling $\alpha$ to be chosen after seeing and analyzing the data. To study "good" post-hoc tests we introduce $\Gamma$-admissibility, where $\Gamma$ is a set of adversaries which map the data to a significance level. A test is $\Gamma$-admissible if, roughly speaking, there is no other test which performs at least as well and sometimes better across all adversaries in $\Gamma$. For point nulls and alternatives, we prove general properties of any $\Gamma$-admissible test for any $\Gamma$ and show that they must be based on e-values. We also classify the set of admissible tests for various specific $\Gamma$.