Python Wrapper for Simulating Multi-Fidelity Optimization on HPO Benchmarks without Any Wait
Abstract: Hyperparameter (HP) optimization of deep learning (DL) is essential for high performance. As DL often requires several hours to days for its training, HP optimization (HPO) of DL is often prohibitively expensive. This boosted the emergence of tabular or surrogate benchmarks, which enable querying the (predictive) performance of DL with a specific HP configuration in a fraction. However, since the actual runtime of a DL training is significantly different from its query response time, simulators of an asynchronous HPO, e.g. multi-fidelity optimization, must wait for the actual runtime at each iteration in a na\"ive implementation; otherwise, the evaluation order during simulation does not match with the real experiment. To ease this issue, we developed a Python wrapper and describe its usage. This wrapper forces each worker to wait so that we yield exactly the same evaluation order as in the real experiment with only $10{-2}$ seconds of waiting instead of waiting several hours. Our implementation is available at https://github.com/nabenabe0928/mfhpo-simulator/.
- Optuna: A next-generation hyperparameter optimization framework. In International Conference on Knowledge Discovery & Data Mining.
- HPO-B: A large-scale reproducible benchmark for black-box HPO based on OpenML. arXiv:2106.06257.
- DEHB: Evolutionary HyperBand for scalable, robust and efficient hyperparameter optimization. arXiv:2105.09821.
- JAHS-Bench-201: A foundation for research on joint architecture and hyperparameter search. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track.
- Algorithms for hyper-parameter optimization. Advances in Neural Information Processing Systems.
- Bayesian optimization in AlphaGo. arXiv:1812.06855.
- NAS-Bench-201: Extending the scope of reproducible neural architecture search. arXiv:2001.00326.
- Efficient benchmarking of hyperparameter optimizers via surrogates. In AAAI Conference on Artificial Intelligence.
- HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. arXiv:2109.06716.
- BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning.
- Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence.
- Non-stochastic best arm identification and hyperparameter optimization. In International Conference on Artificial Intelligence and Statistics.
- Multi-fidelity Bayesian optimisation with continuous approximations. In International Conference on Machine Learning.
- Tuning hyperparameters without grad students: Scalable and robust Bayesian optimisation with Dragonfly. Journal of Machine Learning Research, 21.
- Tabular benchmarks for joint architecture and hyperparameter optimization. arXiv:1905.04970.
- HyperBand: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18.
- Hyper-Tune: towards efficient hyper-parameter tuning at scale. arXiv:2201.06834.
- Tune: A research platform for distributed model selection and training. arXiv:1807.05118.
- SMAC3: A versatile Bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23.
- NAS-Bench-Suite: NAS evaluation is (now) surprisingly easy. arXiv:2201.13396.
- TrivialAugment: Tuning-free yet state-of-the-art data augmentation. In International Conference on Computer Vision.
- YAHPO Gym – an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. In International Conference on Automated Machine Learning.
- Syne Tune: A library for large scale hyperparameter tuning and reproducible research. In International Conference on Automated Machine Learning.
- Watanabe, S. (2023). Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance. arXiv:2304.11127.
- Auto-PyTorch: Multi-fidelity metalearning for efficient and robust AutoDL. Transactions on Pattern Analysis and Machine Intelligence.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
