Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Provably Efficient Bayesian Optimization with Unknown Gaussian Process Hyperparameter Estimation (2306.06844v3)

Published 12 Jun 2023 in stat.ML and cs.LG

Abstract: Gaussian process (GP) based Bayesian optimization (BO) is a powerful method for optimizing black-box functions efficiently. The practical performance and theoretical guarantees of this approach depend on having the correct GP hyperparameter values, which are usually unknown in advance and need to be estimated from the observed data. However, in practice, these estimations could be incorrect due to biased data sampling strategies used in BO. This can lead to degraded performance and break the sub-linear global convergence guarantee of BO. To address this issue, we propose a new BO method that can sub-linearly converge to the objective function's global optimum even when the true GP hyperparameters are unknown in advance and need to be estimated from the observed data. Our method uses a multi-armed bandit technique (EXP3) to add random data points to the BO process, and employs a novel training loss function for the GP hyperparameter estimation process that ensures consistent estimation. We further provide theoretical analysis of our proposed method. Finally, we demonstrate empirically that our method outperforms existing approaches on various synthetic and real-world problems.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Gambling in a rigged casino: The adversarial multi-armed bandit problem. Electron. Colloquium Comput. Complex., TR00-068, 2000.
  2. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235–256, 2002a.
  3. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48–77, 2002b.
  4. Francois Bachoc. Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification. Computational Statistics and Data Analysis, 66:55–69, 2013.
  5. Thomas Back. Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford university press, 1996.
  6. Efficient hyperparameter optimization by using Bayesian optimization for drug-target interaction prediction. In IEEE 7th International Conference on Computational Advances in Bio and Medical Sciences, pages 1–6, 2017.
  7. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems 24, pages 2546–2554, 2011.
  8. No-regret Bayesian optimization with unknown hyperparameters. The Journal Machine Learning Research, 20:50:1–50:24, 2019.
  9. Misspecified Gaussian process bandit optimization. In Advances in Neural Information Processing Systems 34, pages 3004–3015, 2021.
  10. Time-varying Gaussian process bandit optimization. In Artificial Intelligence and Statistics, pages 314–323, 2016a.
  11. Truncated variance reduction: A unified approach to Bayesian optimization and level-set estimation. In Advances in Neural Information Processing Systems, pages 1507–1515, 2016b.
  12. The bernstein-von mises theorem for markov processes. The Annals of Mathematical Statistics, 42(4):1241–1253, 1971.
  13. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599, 2010.
  14. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
  15. Adam D Bull. Convergence rates of efficient global optimization algorithms. The Journal of Machine Learning Research, 12:2879–2904, 2011.
  16. Lenient regret and good-action identification in Gaussian process bandits. In International Conference on Machine Learning, 2021.
  17. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SigKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. ACM, 2016.
  18. Multi-information source bayesian optimization of culture media for cellular agriculture. Biotechnology and Bioengineering, 119(9):2447–2458, 2022.
  19. Hebo: pushing the limits of sample-efficient hyper-parameter optimisation. Journal of Artificial Intelligence Research, 74:1269–1349, 2022.
  20. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  21. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing Systems, pages 5496–5507, 2019.
  22. On statistical bias in active learning: How and when to fix it. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
  23. Peter I Frazier. A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811, 2018.
  24. Algorithmic assurance: An active approach to algorithmic testing using Bayesian optimisation. In Advances in Neural Information Processing Systems (NeurIPS), pages 5465–5473, 2018.
  25. Nikolaus Hansen. The cma evolution strategy: a comparing review. Towards a new evolutionary computation, pages 75–102, 2006.
  26. Portfolio allocation for Bayesian optimization. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence (UAI), pages 327–336, 2011.
  27. Gryffin: An algorithm for bayesian optimization of categorical variables informed by expert knowledge. Applied Physics Reviews, (8):031406, 2021.
  28. Tuning materials-binding peptide sequences toward gold- and silver-binding selectivity with bayesian optimization. ACS Nano, 15(11):18260–18269, 2021.
  29. Pibo: Augmenting acquisition functions with user beliefs for bayesian optimization. International Conference on Learning Representation, 2022.
  30. Self-correcting Bayesian optimization through Bayesian active learning. abs/2304.11005, 2023. doi: 10.48550/arXiv.2304.11005.
  31. Bayesian optimization with tree-structured dependencies. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1655–1664, 2017.
  32. Donald R Jones. A taxonomy of global optimization methods based on response surfaces. Journal of global optimization, 21(4):345–383, 2001.
  33. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455–492, 1998.
  34. Neural architecture search with Bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems, pages 2016–2025, 2018.
  35. Lassobench: A high-dimensional hyperparameter optimization benchmark suite for lasso. International Conference on Automated Machine Learning, 2022.
  36. Uniform error bounds for Gaussian process regression with application to safe control. In Advances in Neural Information Processing Systems, pages 657–667, 2019.
  37. Accelerating experimental design by incorporating experimenter hunches. In 2018 IEEE International Conference on Data Mining (ICDM), pages 257–266. IEEE Computer Society, 2018.
  38. Batch Bayesian optimization via multi-objective acquisition ensemble for automated analog circuit design. In Proceedings of the 35th International Conference on Machine Learning, 2018.
  39. Automating Bayesian optimization with Bayesian optimization. In Advances in Neural Information Processing Systems (NeurIPS), pages 5988–5997, 2018.
  40. Bayesian optimisation for intelligent environmental monitoring. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
  41. Radford M. Neal. MCMC Using Hamiltonian Dynamics, volume 2. CRC Press, 2011.
  42. Regret for expected improvement over the best-observed value and stopping condition. In Proceedings of The 9th Asian Conference on Machine Learning (ACML), pages 279–294, 2017.
  43. Gaussian processes for global optimization. In The 3rd International Conference on Learning and Intelligent Optimization (LION3), pages 1–15. Springer-Verlag, 2009.
  44. Provably efficient online hyperparameter optimization with population-based bandits. Advances in Neural Information Processing Systems, 33, 2020.
  45. Automated reinforcement learning (autorl): A survey and open problems. Journal of Artificial Intelligence Research, 74:517–568, 2022.
  46. Michael JD Powell. On search directions for minimization algorithms. Mathematical programming, 4(1):193–201, 1973.
  47. Gaussian processes for machine learning. 2006.
  48. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. 2021.
  49. J. Sayyad Shirabad and T.J. Menzies. The PROMISE Repository of Software Engineering Databases. School of Information Technology and Engineering, University of Ottawa, Canada, 2005. URL http://promise.site.uottawa.ca/SERepository.
  50. Jonathan Scarlett. Tight regret bounds for Bayesian optimization in one dimension. In International Conference on Machine Learning, pages 4500–4508. PMLR, 2018.
  51. Information consistency of nonparametric Gaussian process methods. IEEE Transactions on Information Theory, 54(5):2376–2382, 2008.
  52. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2016.
  53. Jan Paul Siebert. Vehicle recognition using rule based methods. 1987.
  54. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, pages 2951–2959, 2012.
  55. Scalable Bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on Machine Learning, pages 2171–2180, 2015.
  56. Bayesian optimization with robust Bayesian neural networks. In Advances in Neural Information Processing Systems, pages 4134–4142, 2016.
  57. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on Machine Learning, pages 1015–1022, 2010.
  58. Combo: an efficient Bayesian optimization library for materials science. Materials discovery, 4:18–21, 2016.
  59. A. W. van der Vaart. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998. doi: 10.1017/CBO9780511802256.
  60. A.J. van Soest and L.J.R. Casius. The merits of a parallel genetic algorithm in solving hard optimization problems. European Journal of Industrial Engineering, 125:141–146, 2003. ISSN 1751-5254.
  61. Theoretical analysis of Bayesian optimisation with unknown Gaussian process hyper-parameters. arXiv preprint arXiv:1406.7758, 2014.
  62. Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10293–10301, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Huong Ha (19 papers)
  2. Vu Nguyen (45 papers)
  3. Hongyu Zhang (147 papers)
  4. Anton van den Hengel (188 papers)
  5. Hung Tran-The (10 papers)
  6. Xiuzhen Zhang (35 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets