Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
143 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On Large-Scale Multiple Testing Over Networks: An Asymptotic Approach (2211.16059v4)

Published 29 Nov 2022 in stat.ME, cs.LG, cs.SY, eess.SP, and eess.SY

Abstract: This work concerns developing communication- and computation-efficient methods for large-scale multiple testing over networks, which is of interest to many practical applications. We take an asymptotic approach and propose two methods, proportion-matching and greedy aggregation, tailored to distributed settings. The proportion-matching method achieves the global BH performance yet only requires a one-shot communication of the (estimated) proportion of true null hypotheses as well as the number of p-values at each node. By focusing on the asymptotic optimal power, we go beyond the BH procedure by providing an explicit characterization of the asymptotic optimal solution. This leads to the greedy aggregation method that effectively approximates the optimal rejection regions at each node, while computation efficiency comes from the greedy-type approach naturally. Moreover, for both methods, we provide the rate of convergence for both the FDR and power. Extensive numerical results over a variety of challenging settings are provided to support our theoretical findings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Y. Benjamini and Y. Hochberg, “Controlling the false discovery rate: a practical and powerful approach to multiple testing,” Journal of the royal statistical society. Series B (Methodological), pp. 289–300, 1995.
  2. Y. Benjamini and D. Yekutieli, “The control of the false discovery rate in multiple testing under dependency,” Annals of statistics, pp. 1165–1188, 2001.
  3. J. D. Storey, “A direct approach to false discovery rates,” Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 64, no. 3, pp. 479–498, 2002.
  4. S. K. Sarkar, “Some results on false discovery rate in stepwise multiple testing procedures,” The Annals of Statistics, vol. 30, no. 1, pp. 239–257, 2002.
  5. C. Genovese and L. Wasserman, “Operating characteristics and extensions of the false discovery rate procedure,” Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 64, no. 3, pp. 499–517, 2002.
  6. G. Blanchard and E. Roquain, “Two simple sufficient conditions for FDR control,” Electronic journal of Statistics, vol. 2, pp. 963–992, 2008.
  7. A. K. Ramdas, R. F. Barber, M. J. Wainwright, and M. I. Jordan, “A unified treatment of multiple testing with prior knowledge using the p-filter,” 2019.
  8. C. R. Genovese, N. A. Lazar, and T. Nichols, “Thresholding of statistical maps in functional neuroimaging using the false discovery rate,” Neuroimage, vol. 15, no. 4, pp. 870–878, 2002.
  9. F. Abramovich, Y. Benjamini, D. L. Donoho, and I. M. Johnstone, “Adapting to unknown sparsity by controlling the false discovery rate,” The Annals of Statistics, vol. 34, no. 2, pp. 584–653, 2006.
  10. M. Gölz, A. M. Zoubir, and V. Koivunen, “Multiple hypothesis testing framework for spatial signals,” IEEE Transactions on Signal and Information Processing over Networks, vol. 8, pp. 771–787, 2022.
  11. R. R. Tenney and N. R. Sandell, “Detection with distributed sensors,” IEEE Transactions on Aerospace and Electronic systems, no. 4, pp. 501–510, 1981.
  12. J. N. Tsitsiklis, “Problems in decentralized decision making and computation.” DTIC Document, Tech. Rep., 1984.
  13. R. Viswanathan and P. K. Varshney, “Distributed detection with multiple sensors Part I. Fundamentals,” Proceedings of the IEEE, vol. 85, no. 1, pp. 54–63, 1997.
  14. R. S. Blum, S. A. Kassam, and H. V. Poor, “Distributed detection with multiple sensors II. Advanced topics,” Proceedings of the IEEE, vol. 85, no. 1, pp. 64–79, 1997.
  15. E. B. Ermis and V. Saligrama, “Adaptive statistical sampling methods for decentralized estimation and detection of localized phenomena,” in Fourth International Symposium on Information Processing in Sensor Networks, 2005.   IEEE, 2005, pp. 143–150.
  16. P. Ray, P. K. Varshney, and R. Niu, “A novel framework for the network-wide distributed detection problem,” in 10th International Conference on Information Fusion.   IEEE, 2007, pp. 1–8.
  17. E. B. Ermis and V. Saligrama, “Distributed detection in sensor networks with limited range multimodal sensors,” IEEE Transactions on Signal Processing, vol. 58, no. 2, pp. 843–858, 2009.
  18. P. Ray and P. K. Varshney, “False discovery rate based sensor decision rules for the network-wide distributed detection problem,” IEEE Transactions on Aerospace and Electronic Systems, vol. 47, no. 3, pp. 1785–1799, 2011.
  19. A. Ramdas, J. Chen, M. Wainwright, and M. Jordan, “QuTE: Decentralized multiple testing on sensor networks with false discovery rate control,” in IEEE 56th Annual Conference on Decision and Control, 2017, pp. 6415–6421.
  20. Y. Xiang, “Distributed false discovery rate control with quantization,” in 2019 IEEE International Symposium on Information Theory.   IEEE, 2019, pp. 246–249.
  21. A. Ramdas, J. Chen, M. Wainwright, and M. Jordan, “QuTE: Decentralized multiple testing on sensor networks with false discovery rate control,” arXiv preprint arXiv:2210.04334, 2022.
  22. M. Pournaderi and Y. Xiang, “Sample-and-forward: Communication-efficient control of the false discovery rate in networks,” arXiv preprint arXiv:2210.02555, 2022.
  23. Z. Chi, “On the performance of FDR control: constraints and a partial solution,” The Annals of Statistics, vol. 35, no. 4, pp. 1409–1431, 2007.
  24. E. Arias-Castro, S. Chen, and A. Ying, “A scan procedure for multiple testing: Beyond threshold-type procedures,” Journal of Statistical Planning and Inference, vol. 210, pp. 42–52, 2021.
  25. A. C. Djedouboum, A. A. Abba Ari, A. M. Gueroui, A. Mohamadou, and Z. Aliouat, “Big data collection in large-scale wireless sensor networks,” Sensors, vol. 18, no. 12, p. 4474, 2018.
  26. A. A. A. Ari, B. O. Yenke, N. Labraoui, I. Damakoa, and A. Gueroui, “A power efficient cluster-based routing algorithm for wireless sensor networks: Honeybees swarm intelligence based approach,” Journal of Network and Computer Applications, vol. 69, pp. 77–97, 2016.
  27. F. Bajaber and I. Awan, “An efficient cluster-based communication protocol for wireless sensor networks,” Telecommunication Systems, vol. 55, pp. 387–401, 2014.
  28. A. Fascista, “Toward integrated large-scale environmental monitoring using wsn/uav/crowdsensing: a review of applications, signal processing, and future perspectives,” Sensors, vol. 22, no. 5, p. 1824, 2022.
  29. A. Khalifeh, K. A. Darabkh, A. M. Khasawneh, I. Alqaisieh, M. Salameh, A. AlAbdala, S. Alrubaye, A. Alassaf, S. Al-HajAli, R. Al-Wardat et al., “Wireless sensor networks for smart cities: Network design, implementation and performance evaluation,” Electronics, vol. 10, no. 2, p. 218, 2021.
  30. S. Misra, M. Reisslein, and G. Xue, “A survey of multimedia streaming in wireless sensor networks,” IEEE communications surveys & tutorials, vol. 10, no. 4, pp. 18–39, 2008.
  31. J. D. Rosenblatt, L. Finos, W. D. Weeda, A. Solari, and J. J. Goeman, “All-resolutions inference for brain imaging,” Neuroimage, vol. 181, pp. 786–796, 2018.
  32. M. Pournaderi and Y. Xiang, “Communication-efficient distributed multiple testing for large-scale inference,” in IEEE International Symposium on Information Theory, 2022, pp. 1477–1482.
  33. C. Bonferroni, “Teoria statistica delle classi e calcolo delle probabilita,” Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze, vol. 8, pp. 3–62, 1936.
  34. B. Efron, R. Tibshirani, J. D. Storey, and V. Tusher, “Empirical bayes analysis of a microarray experiment,” Journal of the American statistical association, vol. 96, no. 456, pp. 1151–1160, 2001.
  35. C. Genovese and L. Wasserman, “A stochastic process approach to false discovery control,” The Annals of Statistics, vol. 32, no. 3, pp. 1035–1061, 2004.
  36. A. Tarski, “A lattice-theoretical fixpoint theorem and its applications.” 1955.
  37. Y. Hochberg and Y. Benjamini, “More powerful procedures for multiple significance testing,” Statistics in Medicine, vol. 9, no. 7, pp. 811–818, 1990.
  38. N. W. Hengartner and P. B. Stark, “Finite-sample confidence envelopes for shape-restricted densities,” The Annals of Statistics, pp. 525–550, 1995.
  39. J. W. Swanepoel, “The limiting behavior of a modified maximal symmetric 2⁢s2𝑠2s2 italic_s-spacing with applications,” The Annals of Statistics, vol. 27, no. 1, pp. 24–35, 1999.
  40. Y. Benjamini and Y. Hochberg, “On the adaptive control of the false discovery rate in multiple testing with independent statistics,” Journal of Educational and Behavioral Statistics, vol. 25, no. 1, pp. 60–83, 2000.
  41. R. R. Bahadur, “A note on quantiles in large samples,” The Annals of Mathematical Statistics, vol. 37, no. 3, pp. 577–580, 1966.
  42. T. Tao, “Analysis II, texts and readings in mathematics,” 2015.
  43. T. T. Cai and W. Sun, “Simultaneous testing of grouped hypotheses: Finding needles in multiple haystacks,” Journal of the American Statistical Association, vol. 104, no. 488, pp. 1467–1481, 2009.
  44. D. Storey, “The positive false discovery rate: A bayesian interpretation and the q-value,” Annals of Statistics, vol. 31, pp. 2013–2035, 2003.
  45. O. L. Davies et al., “Statistical methods in research and production.” Statistical methods in research and production., 1947.
  46. D. Freedman and P. Diaconis, “On the histogram as a density estimator: L 2 theory,” Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, vol. 57, no. 4, pp. 453–476, 1981.
  47. N. V. Smirnov, “Approximate laws of distribution of random variables from empirical data,” Uspekhi Matematicheskikh Nauk, no. 10, pp. 179–206, 1944.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now