2000 character limit reached
The Online Closure Principle (2211.11400v3)
Published 21 Nov 2022 in stat.ME
Abstract: The closure principle is fundamental in multiple testing and has been used to derive many efficient procedures with familywise error rate control. However, it is often unsuitable for modern research, which involves flexible multiple testing settings where not all hypotheses are known at the beginning of the evaluation. In this paper, we focus on online multiple testing where a possibly infinite sequence of hypotheses is tested over time. At each step, it must be decided on the current hypothesis without having any information about the hypotheses that have not been tested yet. Our main contribution is a general and stringent mathematical definition of online multiple testing and a new online closure principle which ensures that the resulting closed procedure can be applied in the online setting. We prove that any familywise error rate controlling online procedure can be derived by this online closure principle and provide admissibility results. In addition, we demonstrate how short-cuts of these online closed procedures can be obtained under a suitable consonance property.
- Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 57(1):289–300.
- Optimal testing of multiple hypotheses with common effect direction. Biometrika, 96(2):399–410.
- A graphical approach to sequentially rejective multiple test procedures. Statistics in Medicine, 28(4):586–604.
- Gatekeeping strategies for clinical trials that do not require all primary effects to be significant. Statistics in Medicine, 22(15):2387–2400.
- Online multiple testing with super-uniformity reward. arXiv preprint arXiv:2110.01255.
- Approval policies for modifications to machine learning-based software as a medical device: A study of bio-creep. Biometrics, 77(1):31–44.
- Sequential algorithmic modification with test data reuse. In Uncertainty in Artificial Intelligence, pages 674–684. PMLR.
- α𝛼\alphaitalic_α-investing: A procedure for sequential control of expected false discoveries. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(2):429–444.
- Gabriel, K. R. (1969). Simultaneous test procedures–some theory of multiple comparisons. The Annals of Mathematical Statistics, 40(1):224–250.
- Exceedance control of the false discovery proportion. Journal of the American Statistical Association, 101(476):1408–1417.
- Only closed testing procedures are admissible for controlling false discovery proportions. The Annals of Statistics, 49(2):1218–1238.
- Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing. Biometrika, 106(4):841–856.
- Multiple testing for exploratory research. Statistical Science, 26(4):584–597.
- Closed procedures are better and often admit a shortcut. Journal of Statistical Planning and Inference, 76(1-2):79–91.
- Powerful short-cuts for multiple testing procedures with special reference to gatekeeping strategies. Statistics in Medicine, 26(22):4063–4073.
- Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8):e124.
- Online rules for control of false discovery rate and false discovery exceedance. The Annals of statistics, 46(2):526–554.
- Prevalence of sexual dimorphism in mammalian phenotypic traits. Nature Communications, 8(1):1–12.
- Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1168–1176.
- Testing statistical hypotheses, volume 3. Springer.
- On closed testing procedures with special reference to ordered analysis of variance. Biometrika, 63(3):655–660.
- Multiple comparisons in drug clinical trials and preclinical assays: a-priori ordered hypotheses. In Joachim, V., editor, Biometrie in der Chemisch-Pharmazeutischen Industrie, pages 3–18. Fischer Verlag, Stuttgart.
- The international mouse phenotyping consortium (impc): A functional catalogue of the mammalian genome that informs conservation. Conservation Genetics, 19(4):995–1005.
- Online control of the false discovery rate with decaying memory. Advances in Neural Information Processing Systems, 30.
- Saffron: An adaptive algorithm for online control of the false discovery rate. In International Conference on Machine Learning, pages 4286–4294. PMLR.
- Online error control for platform trials. arXiv preprint arXiv:2202.03838.
- Online multiple hypothesis testing for reproducible research. arXiv preprint arXiv:2208.11418.
- Graphical approaches for the control of generalized error rates. Statistics in Medicine, 39(23):3135–3155.
- onlinefdr: An r package to control the false discovery rate for growing data repositories. Bioinformatics, 35(20):4196–4199.
- Consonance and the closure method in multiple testing. The International Journal of Biostatistics, 7(1):0000102202155746791300.
- Exact and approximate stepdown methods for multiple hypothesis testing. Journal of the American Statistical Association, 100(469):94–108.
- Experiences of the data monitoring committee for the RECOVERY trial, a large-scale adaptive platform randomised trial of treatments for patients hospitalised with COVID-19. Trials, 23(1):881.
- Plots of p-values to evaluate many tests simultaneously. Biometrika, 69(3):493–502.
- Vollständigkeitssätze für multiple testprobleme. In Bauer, P., Hommel, G., and Sonnemann, E., editors, Multiple Hypothesenprüfung/Multiple Hypotheses Testing, pages 121–135. Springer, Berlin.
- Addis: An adaptive discarding algorithm for online fdr control with conservative nulls. Advances in Neural Information Processing Systems, 32.
- Online control of the familywise error rate. Statistical Methods in Medical Research, 30(4):976–993.
- Online control of the false discovery rate in group-sequential platform trials. Statistical Methods in Medical Research, 0(0):09622802221129051. PMID: 36189481.
- Multiple testing when many p-values are uniformly conservative, with application to testing qualitative interaction in educational interventions. Journal of the American Statistical Association, 114(527):1291–1304.
- The power of batching in multiple hypothesis testing. In International Conference on Artificial Intelligence and Statistics, pages 3806–3815. PMLR.
- Asynchronous online testing of multiple hypotheses. J. Mach. Learn. Res., 22:33–1.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
