Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Boosting e-BH via conditional calibration (2404.17562v1)

Published 26 Apr 2024 in stat.ME, math.ST, and stat.TH

Abstract: The e-BH procedure is an e-value-based multiple testing procedure that provably controls the false discovery rate (FDR) under any dependence structure between the e-values. Despite this appealing theoretical FDR control guarantee, the e-BH procedure often suffers from low power in practice. In this paper, we propose a general framework that boosts the power of e-BH without sacrificing its FDR control under arbitrary dependence. This is achieved by the technique of conditional calibration, where we take as input the e-values and calibrate them to be a set of "boosted e-values" that are guaranteed to be no less -- and are often more -- powerful than the original ones. Our general framework is explicitly instantiated in three classes of multiple testing problems: (1) testing under parametric models, (2) conditional independence testing under the model-X setting, and (3) model-free conformalized selection. Extensive numerical experiments show that our proposed method significantly improves the power of e-BH while continuing to control the FDR. We also demonstrate the effectiveness of our method through an application to an observational study dataset for identifying individuals whose counterfactuals satisfy certain properties.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. A survey of anomaly detection techniques in financial domain. Future Generation Computer Systems, 55:278–288.
  2. Controlling the false discovery rate via knockoffs. The Annals of Statistics, 43(5):2055 – 2085.
  3. Derandomized novelty detection with fdr control via conformal e-values. Advances in Neural Information Processing Systems, 36.
  4. Testing for outliers with conformal p-values. The Annals of Statistics, 51(1):149–178.
  5. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289–300.
  6. The control of the false discovery rate in multiple testing under dependency. Annals of statistics, pages 1165–1188.
  7. Panning for gold:‘model-x’knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3):551–577.
  8. Assessing treatment effect variation in observational studies: Results from a data challenge. Observational Studies, 5(2):21–35.
  9. Novelty detection-based approach for alzheimer’s disease and mild cognitive impairment diagnosis from eeg. Medical & Biological Engineering & Computing, 59:2287–2296.
  10. Multiple testing under negative dependence. arXiv preprint arXiv:2212.09706.
  11. Farcomeni, A. (2007). Some results on the control of the false discovery rate under dependence. Scandinavian Journal of Statistics, 34(2):275–297.
  12. On the benjamini–hochberg method.
  13. Conditional calibration for false discovery rate control under dependence. The Annals of Statistics, 50(6):3091 – 3118.
  14. Gao, Z. (2023). Adaptive storey’s null proportion estimator. arXiv preprint arXiv:2310.06357.
  15. Simultaneous hypothesis testing using internal negative controls with an application to proteomics. arXiv preprint arXiv:2303.01552.
  16. A stochastic process approach to false discovery control.
  17. Safe testing. Journal of the Royal Statistical Society, Series B (Methodology), with discussion.
  18. Grünwald, P. D. (2023). The e-posterior. Philosophical Transactions of the Royal Society A, 381(2247):20220146.
  19. Time-uniform chernoff bounds via nonnegative supermartingales. Probability Surveys.
  20. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2):1055 – 1080.
  21. A two-sample conditional distribution test using conformal prediction and weighted rank sum. Journal of the American Statistical Association, pages 1–19.
  22. E-values as unnormalized weights in multiple testing. Biometrika, page asad057.
  23. Causal inference in statistics, social, and biomedical sciences. Cambridge university press.
  24. Model-free selective inference under covariate shift via weighted conformal p-values. arXiv preprint arXiv:2307.09291.
  25. The numeraire e-variable. arXiv preprint arXiv:2402.18810.
  26. E-values, multiple testing and beyond. arXiv preprint arXiv:2312.02905.
  27. Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers. arXiv preprint arXiv:2208.11111.
  28. Improving knockoffs with conditional calibration. arXiv preprint arXiv:2208.09542.
  29. Semi-supervised multiple testing. Electronic Journal of Statistics, 16(2):4926–4981.
  30. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830.
  31. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4):576–601.
  32. Randomized and exchangeable improvements of markov’s, chebyshev’s and chernoff’s inequalities. arXiv preprint arXiv:2304.02611.
  33. Derandomised knockoffs: leveraging e-values for false discovery rate control. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(1):122–154.
  34. Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688.
  35. Sarkar, S. K. (2023). On controlling the false discovery rate in multiple testing of the means of correlated normals against two-sided alternatives. arXiv preprint arXiv:2304.05261.
  36. Shafer, G. (2021). Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society Series A: Statistics in Society, 184(2):407–431.
  37. Test Martingales, Bayes Factors and p-Values. Statistical Science, 26(1):84 – 101.
  38. In situ cell-type-specific cell-surface proteomic profiling in mice. Neuron, 110(23):3882–3896.
  39. Powerful knockoffs via minimizing reconstructability. The Annals of Statistics, 50(1):252–276.
  40. Storey, J. D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society Series B: Statistical Methodology, 64(3):479–498.
  41. Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society Series B: Statistical Methodology, 66(1):187–205.
  42. Novelty detection for the identification of masses in mammograms.
  43. Conformal prediction under covariate shift. Advances in neural information processing systems, 32.
  44. Nonparametric e-tests of symmetry. New England Journal of Statistics in Data Science.
  45. Algorithmic learning in a random world, volume 29. Springer.
  46. E-values: Calibration, combination and applications. The Annals of Statistics, 49(3):1736–1754.
  47. Confidence and discoveries with e-values. Statistical Science, 38(2):329–354.
  48. E-backtesting. arXiv preprint arXiv:2209.00991.
  49. False discovery rate control with e-values. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):822–852.
  50. Universal inference. Proceedings of the National Academy of Sciences, 117(29):16880–16890.
  51. Time-uniform central limit theory and asymptotic confidence sequences. arXiv preprint arXiv:2103.06476.
  52. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(1):1–27.
  53. Anytime-valid off-policy inference for contextual bandits. ACM/JMS Journal of Data Science.
  54. A power and prediction analysis for knockoffs with lasso statistics. arXiv preprint arXiv:1712.06465.
  55. More powerful multiple testing under dependence via randomization. arXiv preprint arXiv:2305.11126.
  56. Online multiple testing with e-values. arXiv preprint arXiv:2311.06412.
  57. Post-selection inference for e-value based confidence intervals. arXiv preprint arXiv:2203.12572.
  58. On the existence of powerful p-values and e-values for composite hypotheses. ArXiv preprint. arXiv:2305.16539.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now