Papers
Topics
Authors
Recent
2000 character limit reached

Flexible control of the median of the false discovery proportion (2208.11570v4)

Published 24 Aug 2022 in stat.ME

Abstract: We introduce a multiple testing procedure that controls the median of the proportion of false discoveries (FDP) in a flexible way. The procedure only requires a vector of p-values as input and is comparable to the Benjamini-Hochberg method, which controls the mean of the FDP. Our method allows freely choosing one or several values of alpha after seeing the data -- unlike Benjamini-Hochberg, which can be very liberal when alpha is chosen post hoc. We prove these claims and illustrate them with simulations. Our procedure is inspired by a popular estimator of the total number of true hypotheses. We adapt this estimator to provide simultaneously median unbiased estimators of the FDP, valid for finite samples. This simultaneity allows for the claimed flexibility. Our approach does not assume independence. The time complexity of our method is linear in the number of hypotheses, after sorting the p-values.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Permutation-based true discovery proportions for fMRI cluster analysis. Statistics in Medicine. Online First version, 2023.
  2. Controlling the false discovery rate via knockoffs. The Annals of Statistics, 43(5):2055–2085, 2015.
  3. Empirical bayes control of the false discovery exceedance. arXiv preprint arXiv:2111.03885, 2021.
  4. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the royal statistical society. Series B (Methodological), pages 289–300, 1995.
  5. The control of the false discovery rate in multiple testing under dependency. Annals of statistics, pages 1165–1188, 2001.
  6. Global and multiple test procedures using ordered p-values—a review. Statistical Papers, 45(1):1–14, 2004.
  7. RNA-Seq of tumor-educated platelets enables blood-based pan-cancer, multiclass, and molecular pathway cancer diagnostics. Cancer cell, 28(5):666–676, 2015.
  8. Notip: Non-parametric true discovery proportion control for brain imaging. NeuroImage, 260(119492), 2022.
  9. Post hoc confidence bounds on false positives using reference families. Annals of Statistics, 48(3):1281–1303, 2020.
  10. New procedures controlling the false discovery proportion via Romano-Wolf’s heuristic. The Annals of Statistics, 43(3):1141–1177, 2015.
  11. Dickhaus, T. Simultaneous statistical inference: with applications in the life sciences. Springer Science & Business Media, 2014.
  12. Variability and stability of the false discovery proportion. Electronic Journal of Statistics, 13(1):882–910, 2019.
  13. Controlling the false discovery exceedance for heterogeneous tests. Electronic Journal of Statistics, 14(2):4244–4272, 2020.
  14. Efron, B. Correlation and large-scale simultaneous significance testing. Journal of the American Statistical Association, 102(477):93–103, 2007.
  15. Farcomeni, A. A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Statistical methods in medical research, 17(4):347–388, 2008.
  16. A stochastic process approach to false discovery control. Annals of Statistics, pages 1035–1061, 2004.
  17. Exceedance control of the false discovery proportion. Journal of the American Statistical Association, 101(476):1408–1417, 2006.
  18. Multiple testing for exploratory research. Statistical Science, 26(4):584–597, 2011.
  19. Multiple hypothesis testing in genomics. Statistics in medicine, 33(11):1946–1978, 2014.
  20. Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing. Biometrika, 106(4):841–856, 2019.
  21. Only closed testing procedures are admissible for controlling false discovery proportions. The Annals of Statistics, 49(2):1218–1238, 2021.
  22. Grünwald, P. Beyond Neyman-Pearson. arXiv preprint arXiv:2205.00901, 2022.
  23. A generalized Sidak-Holm procedure and control of generalized error rates under independence. Statistical applications in genetics and molecular biology, 6(1), 2007.
  24. Further results on controlling the false discovery proportion. The Annals of Statistics, 42(3):1070–1101, 2014.
  25. An evaluation of alternative multiple testing methods for finance applications. The Review of Asset Pricing Studies, 10(2):199–248, 2020.
  26. False discovery proportion estimation by permutations: confidence for significance analysis of microarrays. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 80(1):137–155, 2018.
  27. Permutation-based simultaneous confidence bounds for the false discovery proportion. Biometrika, 106(3):635–649, 2019.
  28. On the usage of randomized p-values in the Schweder–Spjøtvoll estimator. Annals of the Institute of Statistical Mathematics, 74(2):289–319, 2022.
  29. More powerful procedures for multiple significance testing. Statistics in medicine, 9(7):811–818, 1990.
  30. Hubbard, R. Alphabet soup: Blurring the distinctions between p’s and alpha’s in psychological research. Theory & Psychology, 14(3):295–327, 2004.
  31. Towards “simultaneous selective inference”: post hoc bounds on the false discovery proportion. arXiv:1803.06790v3, 2018.
  32. Simultaneous high-probability bounds on the false discovery proportion in structured, regression and online settings. The Annals of Statistics, 48(6):3465–3487, 2020.
  33. Estimating the proportion of true null hypotheses, with application to DNA microarray data. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(4):555–572, 2005.
  34. Generalizations of the familywise error rate. The Annals of Statistics, 33(3):1138–1154, 2005.
  35. AdaPT. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 80(4):649–679, 2018.
  36. A general interactive framework for false discovery rate control under structural constraints. Biometrika, 108(2):253–267, 2021.
  37. Accumulation tests for FDR control in ordered hypothesis testing. Journal of the American Statistical Association, 112(518):837–849, 2017.
  38. Adaptive and dynamic adaptive procedures for false discovery rate control and estimation. Journal of the Royal Statistical Society Series B: Statistical Methodology, 74(1):163–182, 2012.
  39. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2. Genome biology, 15(12):1–21, 2014.
  40. Competition-based control of the false discovery proportion. arXiv preprint arXiv:2011.11939, 2020.
  41. On closed testing procedures with special reference to ordered analysis of variance. Biometrika, 63(3):655–660, 1976.
  42. Meinshausen, N. False discovery control for multiple tests of association under general dependence. Scandinavian Journal of Statistics, 33(2):227–237, 2006.
  43. Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. The Annals of Statistics, 34(1):373–393, 2006.
  44. Exceedance control of the false discovery proportion via high precision inversion method of Berk Jones statistics. (Submitted), 2022.
  45. Neuvial, P. Asymptotic properties of false discovery rate controlling procedures under independence. Electronic journal of statistics, 2:1065–1110, 2008.
  46. R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. URL https://www.R-project.org/.
  47. Controlling the false discovery rate via knockoffs: is the+ 1 needed? arXiv preprint arXiv:2204.13248, 2022.
  48. Estimating prevalence from the results of a screening test. American journal of epidemiology, 107(1):71–76, 1978.
  49. Stepup procedures for control of generalizations of the familywise error rate. The Annals of Statistics, 34(4):1850–1873, 2006.
  50. Control of generalized error rates in multiple testing. The Annals of Statistics, 35(4):1378–1408, 2007.
  51. Formalized data snooping based on generalized error rates. Econometric Theory, 24(2):404–447, 2008.
  52. Roquain, E. Type I error rate control for testing many hypotheses: a survey with proofs. hal-00547965v2, 2011.
  53. Rosenblatt, J. D. Prevalence estimation. In Handbook of Multiple Comparisons, pages 183–210. Chapman and Hall/CRC, 2021.
  54. The effect of correlation in false discovery rate estimation. Biometrika, 98(1):199–214, 2011.
  55. Plots of p-values to evaluate many tests simultaneously. Biometrika, 69(3):493–502, 1982.
  56. Minimally adaptive BH: A tiny but uniform improvement of the procedure of Benjamini and Hochberg. Biometrical Journal, 59(4):776–780, 2017.
  57. Storey, J. D. A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(3):479–498, 2002.
  58. Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Statistical applications in genetics and molecular biology, 3(1):15, 2004.
  59. Permutation-based true discovery guarantee by sum tests. Journal of the Royal Statistical Society. Series B (Statistical Methodology). Online First version, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free