Papers
Topics
Authors
Recent
2000 character limit reached

A General Framework for User-Guided Bayesian Optimization (2311.14645v2)

Published 24 Nov 2023 in cs.LG and stat.ML

Abstract: The optimization of expensive-to-evaluate black-box functions is prevalent in various scientific disciplines. Bayesian optimization is an automatic, general and sample-efficient method to solve these problems with minimal knowledge of the underlying function dynamics. However, the ability of Bayesian optimization to incorporate prior knowledge or beliefs about the function at hand in order to accelerate the optimization is limited, which reduces its appeal for knowledgeable practitioners with tight budgets. To allow domain experts to customize the optimization routine, we propose ColaBO, the first Bayesian-principled framework for incorporating prior beliefs beyond the typical kernel structure, such as the likely location of the optimizer or the optimal value. The generality of ColaBO makes it applicable across different Monte Carlo acquisition functions and types of user beliefs. We empirically demonstrate ColaBO's ability to substantially accelerate optimization when the prior information is accurate, and to retain approximately default performance when it is misleading.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. Unexpected improvements to expected improvement for bayesian optimization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=1vyAG6j9PE.
  2. Bayesian optimization of function networks. Advances in neural information processing systems, 34:14463–14475, 2021.
  3. Botorch: A framework for efficient monte-carlo bayesian optimization. In Advances in Neural Information Processing Systems, 2020. URL http://arxiv.org/abs/1910.06403.
  4. Algorithms for hyper-parameter optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger (eds.), Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NeurIPS’11), pp.  2546–2554, 2011a.
  5. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 24. Curran Associates, Inc., 2011b.
  6. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv:1012.2599v1 [cs.LG], 2010.
  7. Adam D. Bull. Convergence rates of efficient global optimization algorithms. 12:2879–2904, 2011.
  8. Bayesian gait optimization for bipedal locomotion. In P. Pardalos and M. Resende (eds.), Proceedings of the Eighth International Conference on Learning and Intelligent Optimization (LION’14), 2014.
  9. Hpvm2fpga: Enabling true hardware-agnostic fpga programming. In Proceedings of the 33rd IEEE International Conference on Application-specific Systems, Architectures, and Processors, 2022.
  10. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing Systems, pp.  5496–5507, 2019. URL http://papers.nips.cc/paper/8788-scalable-global-optimization-via-local-bayesian-optimization.pdf.
  11. Initializing bayesian hyperparameter optimization via meta-learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp.  1128–1135, 2015.
  12. Practical transfer learning for bayesian optimization. ArXiv abs/1802.02219, 2018.
  13. Practical transfer learning for Bayesian optimization. arXiv preprint 1802.02219, 2022.
  14. The knowledge-gradient policy for correlated normal beliefs. INFORMS journal on Computing, 21(4):599–613, 2009.
  15. R. Garnett. Bayesian Optimization. Cambridge University Press, 2022. Available for free at https://bayesoptbook.com/.
  16. Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chemical Science, 2020.
  17. P. Hennig and C. J. Schuler. Entropy search for information-efficient global optimization. Journal of Machine Learning Research, 13(1):1809–1837, June 2012. ISSN 1532-4435.
  18. Predictive entropy search for efficient global optimization of black-box functions. In Advances in Neural Information Processing Systems, 2014. URL https://proceedings.neurips.cc/paper/2014/file/069d3bb002acd8d7dd095917f9efe4cb-Paper.pdf.
  19. Predictive entropy search for bayesian optimization with unknown constraints. In International conference on machine learning, pp.  1699–1707. PMLR, 2015.
  20. Bayesian optimization augmented with actively elicited expert knowledge, 2022.
  21. Sequential model-based optimization for general algorithm configuration. In C. Coello (ed.), Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION’11), volume 6683, pp.  507–523, 2011.
  22. Joint entropy search for maximally-informed bayesian optimization. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022a.
  23. PiBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization. In International Conference on Learning Representations, 2022b.
  24. Self-correcting bayesian optimization through bayesian active learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=dX9MjUtP1A.
  25. Objective bound conditional gaussian process for bayesian optimization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.  4819–4828. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/jeong21a.html.
  26. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13:455–492, 12 1998. doi: 10.1023/A:1008306431147.
  27. Mining geostatistics, Jan 1976.
  28. Gaussian Process Bandit Optimisation with Multi-fidelity Evaluations. In D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett (eds.), Proceedings of the 30th International Conference on Advances in Neural Information Processing Systems (NeurIPS’16), pp.  992–1000, 2016.
  29. Parallelised Bayesian optimisation via Thompson sampling. In A. Storkey and F Perez-Cruz (eds.), Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS), volume 84, pp.  133–142. Proceedings of Machine Learning Research, 2018.
  30. Auto-encoding variational bayes, 2014. URL https://arxiv.org/abs/1312.6114.
  31. Human-ai collaborative bayesian optimisation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.  16233–16245. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/6751611b394a3464cea53eed91cf163c-Paper-Conference.pdf.
  32. H. J. Kushner. A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise. Journal of Basic Engineering, 86(1):97–106, 03 1964. ISSN 0021-9223. doi: 10.1115/1.3653121. URL https://doi.org/10.1115/1.3653121.
  33. Constrained Bayesian optimization with noisy experiments. Bayesian Analysis, 2018.
  34. Sampling from gaussian process posteriors using stochastic gradient descent. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Sf9goJtTCE.
  35. Smac3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23(54):1–9, 2022. URL http://jmlr.org/papers/v23/21-0888.html.
  36. Priorband: Practical hyperparameter optimization in the age of deep learning. arXiv preprint 2306.12370, 2023.
  37. B. Matérn. Spatial variation. Meddelanden fran Statens Skogsforskningsinstitut, 1960.
  38. Learning skill-based industrial robot tasks with user priors. IEEE 18th International Conference on Automation Science and Engineering, 2022. URL https://arxiv.org/abs/2208.01605.
  39. The application of Bayesian methods for seeking the extremum. Towards Global Optimization, 2(117-129):2, 1978.
  40. Gibbon: General-purpose information-based bayesian optimisation. Journal of Machine Learning Research, 22(235):1–49, 2021. URL http://jmlr.org/papers/v22/21-0120.html.
  41. Transformers can do bayesian inference. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=KSugKcbNf9.
  42. PFNs4BO: In-context learning for Bayesian optimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.  25444–25470. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/muller23a.html.
  43. Efficient high dimensional bayesian optimization with additivity and quadrature fourier features. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/4e5046fc8d6a97d18a5f54beaed54dea-Paper.pdf.
  44. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pp.  347–358. IEEE, 2019.
  45. Bayesian algorithm execution: Estimating computable properties of black-box functions using mutual information. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.  8005–8015. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/neiswanger21a.html.
  46. Vu Nguyen and Michael A. Osborne. Knowing the what but not the where in Bayesian optimization. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.  7317–7326. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/nguyen20d.html.
  47. BOCK : Bayesian optimization with cylindrical kernels. In International Conference on Machine Learning, pp.  3865–3874, 2018.
  48. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In Advances in Neural Information Processing Systems, 2019.
  49. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper_files/paper/2007/file/013a006f03dbc5392effeb8f18fda755-Paper.pdf.
  50. C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
  51. Stochastic backpropagation and approximate inference in deep generative models. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp.  1278–1286, Bejing, China, 22–24 Jun 2014. PMLR. URL https://proceedings.mlr.press/v32/rezende14.html.
  52. Pacoh: Bayes-optimal meta-learning with pac-guarantees. In Proceedings of the 38th International Conference on Machine Learning, pp.  9116–9126, 2021a.
  53. Meta-learning reliable priors in the function space. In Advances in Neural Information Processing Systems, volume 34, 2021b.
  54. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=j9Rv7qdXjd.
  55. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2016.
  56. L. Smith. A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820, 2018.
  57. Practical Bayesian optimization of machine learning algorithms. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger (eds.), Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NeurIPS’12), pp.  2960–2968, 2012.
  58. Bayesian optimization with a prior for the optimum. In Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III, volume 12977 of Lecture Notes in Computer Science, pp.  265–296. Springer, 2021.
  59. Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory, 58(5):3250–3265, May 2012. ISSN 1557-9654. doi: 10.1109/tit.2011.2182033. URL http://dx.doi.org/10.1109/TIT.2011.2182033.
  60. Multi-task Bayesian optimization. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger (eds.), Proceedings of the 27th International Conference on Advances in Neural Information Processing Systems (NeurIPS’13), pp.  2004–2012, 2013.
  61. Multi-fidelity Bayesian optimization with max-value entropy search and its parallelization. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.  9334–9345. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/takeno20a.html.
  62. Sequential and parallel constrained max-value entropy search via information lower bound. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.  20960–20986. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/takeno22a.html.
  63. W. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285–294, 1933.
  64. Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In David van Dyk and Max Welling (eds.), Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp.  567–574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16–18 Apr 2009. PMLR. URL https://proceedings.mlr.press/v5/titsias09a.html.
  65. Joint entropy search for multi-objective bayesian optimization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=ZChgD8OoGds.
  66. Atmseer: Increasing transparency and controllability in automated machine learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, pp.  1–12. Association for Computing Machinery, 2019.
  67. Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In International Conference on Machine Learning (ICML), 2017.
  68. Batched large-scale bayesian optimization in high-dimensional spaces. In Amos Storkey and Fernando Perez-Cruz (eds.), Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pp.  745–754. PMLR, 09–11 Apr 2018. URL https://proceedings.mlr.press/v84/wang18c.html.
  69. Pre-trained Gaussian processes for Bayesian optimization. arXiv preprint arXiv:2109.08215, 2023.
  70. BANANAS: Bayesian optimization with neural architectures for neural architecture search. In Q. Yang, K. Leyton-Brown, and Mausam (eds.), Proceedings of the Thirty-Fifth Conference on Artificial Intelligence (AAAI’21), pp.  10293–10301. Association for the Advancement of Artificial Intelligence, AAAI Press, 2021.
  71. Maximizing acquisition functions for bayesian optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/file/498f2c21688f6451d9f5fd09d53edda7-Paper.pdf.
  72. The reparameterization trick for acquisition functions, 2017. URL https://arxiv.org/abs/1712.00424.
  73. Efficiently sampling functions from gaussian process posteriors. In International Conference on Machine Learning, 2020. URL https://arxiv.org/abs/2002.09309.
  74. Hyperparameter search space pruning - A new component for sequential model-based hyperparameter optimization. In A. Appice, P. Rodrigues, V. Costa, J. Gama, A. Jorge, and C. Soares (eds.), Machine Learning and Knowledge Discovery in Databases (ECML/PKDD’15), volume 9285, pp.  104–119, 2015.
  75. Few-shot bayesian optimization with deep kernel surrogates. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=bJxgv5C3sYc.
  76. Auto-pytorch tabular: Multi-fidelity metalearning for efficient and robust autodl. ArXiv, abs/2006.13799, 2020. URL https://api.semanticscholar.org/CorpusID:220041844.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Video Overview

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 9 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free