Safe Testing (1906.07801v5)

Abstract: We develop the theory of hypothesis testing based on the e-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e-values are safe, i.e. they preserve Type-I error guarantees, under such optional continuation. We define growth-rate optimality (GRO) as an analogue of power in an optional continuation context, and we show how to construct GRO e-variables for general testing problems with composite null and alternative, emphasizing models with nuisance parameters. GRO e-values take the form of Bayes factors with special priors. We illustrate the theory using several classic examples including a one-sample safe t-test and the 2 x 2 contingency table. Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, e-values may provide a methodology acceptable to adherents of all three schools.

• The paper introduces GRO $$-values, a novel method enabling sequential hypothesis testing with safe Type-I error control.
• It details the construction of GRO $$-variables using approaches like the right Haar prior to manage nuisance parameters.
• The method facilitates meta-analysis by allowing optional continuation, robustly synthesizing evidence across multiple studies.

Safe Testing: An Expert Analysis

The paper "Safe Testing" by Peter Grünwald, Rianne de Heide, and Wouter M. Koolen introduces a novel approach to hypothesis testing through the use of $$-values, which allows for combining results from multiple studies in a statistically rigorous manner under the condition known as "optional continuation." This approach aims to preserve Type-I error guarantees while enabling the sequential updating of evidence, a feature that traditional$$-value methodologies struggle to maintain due to their sensitivity to the stopping rule applied.

The authors propose the concept of $-variables, nonnegative random variables that satisfy an expectation constraint under the null hypothesis \(H_0\). The key relationship \({\bf E}_P[E] \leq 1\) for all \(P \in H_0\) ensures that these variables provide a conservative measure of evidence against the null. The paper demonstrates the construction of these$-variables and compares them to the classical $-values in terms of their practical applicability and advantages. ### Key Contributions and Findings 1. **Theoretical Foundations of$-Values*: - $-values are presented as alternatives to classical$-values, with advantages in scenarios where the decision to conduct additional studies is contingent on previous outcomes. - The authors define Growth-Rate Optimality (GRO) for $$-variables, analogous to power in traditional testing but situating it within an optional continuation framework. 2. **Construction of GRO$$-Variables*: - GRO $$-values are illustrated through examples such as the one-sample safe$$t$-test and$2 \times 2$$contingency tables, showing their equivalence to Bayes factors when appropriate priors are chosen. - Importantly, in models with nuisance parameters, GRO$$-values are constructed using special priors, such as the right Haar prior in the $t$-test example, thus providing a new methodology for dealing with complexities in testing.

1. Handling Composite Hypotheses and Nuisance Parameters:
   • The paper extends the GRO concept to general testing scenarios with composite null and alternative hypotheses, particularly emphasizing models with nuisance parameters.
   • Strategies for integrating prior knowledge and dealing with worst-case scenarios are developed to ensure that the testing remains robust and valid under various conditions.
2. **Implications of $-Values in Meta-Analysis**: - The use of$-values is particularly promising in meta-analyses, where the aim is to synthesize findings across multiple studies without violating statistical assumptions.
   • The enhanced interpretability and intrinsic safety of $-tests under optional continuation make them suitable for accumulating evidence across studies. ### Practical and Theoretical Implications The adoption of$-values presents a significant advancement in statistical methodologies, particularly in research contexts that require sequential testing and interim analysis. By redefining the evidence measure in hypothesis testing, $$-values offer a more flexible and robust approach to evidence accumulation, thereby facilitating a more realistic adaptation of paper designs to natural scientific workflows. The theoretical implications are also notable: the unification of frequentist and Bayesian perspectives in the context of GRO$$-variables provides a bridge that allows adherents of both schools to utilize a common methodology, thereby reducing contention between disparate statistical methodologies. This approach could potentially shift the paradigm in statistical hypothesis testing, focusing on a blend of evidence-based and error control properties.

Future Developments in Artificial Intelligence

Looking forward, the concepts and methodologies introduced in this paper could significantly impact developments in artificial intelligence, particularly in areas involving adaptive learning and decision-making processes. As AI systems increasingly rely on ongoing learning and decision-making based on sequential data, the robustness of $$-values in adaptive contexts can lead to more reliable and interpretable AI models. Moreover, integration with Bayesian models and machine learning algorithms could extend the applicability of$$-values in various AI-driven research fields.

In conclusion, the paper by Grünwald et al. offers substantial contributions to statistical testing, combining theoretical rigor with practical adaptability that appeals to both frequentist and Bayesian paradigms. This work presents a pathway towards safer and more flexible statistical methodologies that are imperative for modern scientific inquiry.