2000 character limit reached
Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization (2407.17358v1)
Published 24 Jul 2024 in cs.IT, cs.LG, and math.IT
Abstract: The increasing adoption of AI in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem.
- K. B. Letaief, W. Chen, Y. Shi, J. Zhang, and Y.-J. A. Zhang, “The roadmap to 6g: Ai empowered wireless networks,” IEEE communications magazine, vol. 57, no. 8, pp. 84–90, 2019.
- J. Wu, X.-Y. Chen, H. Zhang, L.-D. Xiong, H. Lei, and S.-H. Deng, “Hyperparameter optimization for machine learning models based on Bayesian optimization,” Journal of Electronic Science and Technology, vol. 17, no. 1, pp. 26–40, 2019.
- M. Feurer and F. Hutter, “Hyperparameter optimization,” Automated machine learning: Methods, systems, challenges, pp. 3–33, 2019.
- J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization.,” Journal of machine learning research, vol. 13, no. 2, 2012.
- L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter optimization,” Journal of Machine Learning Research, vol. 18, no. 185, pp. 1–52, 2018.
- B. Bischl, M. Binder, M. Lang, T. Pielok, J. Richter, S. Coors, J. Thomas, T. Ullmann, M. Becker, A.-L. Boulesteix, et al., “Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 13, no. 2, p. e1484, 2023.
- A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, and L. Lei, “Learn then test: Calibrating predictive algorithms to achieve risk control,” arXiv preprint arXiv:2110.01052, 2021.
- J. Rice, Mathematical Statistics and Data Analysis. Advanced series, Cengage Learning, 2007.
- C. Lu, A. N. Angelopoulos, and S. Pomerantz, “Improving trustworthiness of ai disease severity rating in medical imaging with ordinal conformal prediction sets,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 545–554, Springer, 2022.
- T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Tran, Y. Tay, and D. Metzler, “Confident adaptive language modeling,” Advances in Neural Information Processing Systems, vol. 35, pp. 17456–17472, 2022.
- C. Ganewattha, Z. Khan, J. J. Lehtomäki, and M. Matinmikko-Blue, “Real-Time Quantile-Based Estimation of Resource Utilization on an FPGA Platform Using HLS,” IEEE Access, vol. 8, pp. 43301–43313, 2020.
- Z. V. Valkova-Jarvis and V. R. Stoynov, “Novel throughput quantile averaging methods for the proportional fair algorithm in indoor mobile networks,” in 2017 Advances in Wireless and Optical Communications (RTUWO), pp. 198–202, IEEE, 2017.
- L. Cheng, J. Niu, Y. Gu, T. He, and Q. Zhang, “Energy-efficient statistical delay guarantee for duty-cycled wireless sensor networks,” in 2015 12th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pp. 46–54, IEEE, 2015.
- A. N. Angelopoulos, S. Bates, et al., “Conformal prediction: A gentle introduction,” Foundations and Trends® in Machine Learning, vol. 16, no. 4, pp. 494–591, 2023.
- S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. Jordan, “Distribution-free, risk-controlling prediction sets,” Journal of the ACM (JACM), vol. 68, no. 6, pp. 1–34, 2021.
- J. C. Snell, T. P. Zollo, Z. Deng, T. Pitassi, and R. Zemel, “Quantile risk control: A flexible framework for bounding the probability of high-loss predictions,” arXiv preprint arXiv:2212.13629, 2022.
- Z. Deng, T. Zollo, J. Snell, T. Pitassi, and R. Zemel, “Distribution-free statistical dispersion control for societal applications,” Advances in Neural Information Processing Systems, vol. 36, 2024.
- P. M. de Sant Ana and N. Marchenko, “Radio Access Scheduling using CMA-ES for Optimized QoS in Wireless Networks,” in 2020 IEEE Globecom Workshops (GC Wkshps, pp. 1–6, IEEE, 2020.
- P. Bauer, “Multiple testing in clinical trials,” Statistics in medicine, vol. 10, no. 6, pp. 871–890, 1991.
- S. R. Howard and A. Ramdas, “Sequential estimation of quantiles with applications to A/B testing and best-arm identification,” Bernoulli, vol. 28, no. 3, pp. 1704–1728, 2022.
- A. Valcarce, “Wireless Suite: A collection of problems in wireless telecommunications,” https://github.com/nokia/wireless-suite, 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.