Information-Theoretic Safe Bayesian Optimization (2402.15347v2)
Abstract: We consider a sequential decision making task, where the goal is to optimize an unknown function without evaluating parameters that violate an a~priori unknown (safety) constraint. A common approach is to place a Gaussian process prior on the unknown functions and allow evaluations only in regions that are safe with high probability. Most current methods rely on a discretization of the domain and cannot be directly extended to the continuous case. Moreover, the way in which they exploit regularity assumptions about the constraint introduces an additional critical hyperparameter. In this paper, we propose an information-theoretic safe exploration criterion that directly exploits the GP posterior to identify the most informative safe parameters to evaluate. The combination of this exploration criterion with a well known Bayesian optimization acquisition function yields a novel safe Bayesian optimization selection criterion. Our approach is naturally applicable to continuous domains and does not require additional explicit hyperparameters. We theoretically analyze the method and show that we do not violate the safety constraint with high probability and that we learn about the value of the safe optimum up to arbitrary precision. Empirical evaluations demonstrate improved data-efficiency and scalability.
- BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
- Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning Journal, Special Issue on Robust Machine Learning, 2016a.
- Safe controller optimization for quadrotors with gaussian processes. In IEEE International Conference on Robotics and Automation (ICRA), 2016b.
- Information-Theoretic Safe Exploration with Gaussian Processes. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
- Openai gym. arXiv preprint arXiv:1606.01540, 2016.
- Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning, 5(1):1–112, 2012.
- On the dynamics of the furuta pendulum. Hindawi Publishing Corporation Journal of Control Science and Engineering, 2011.
- Safe explorative bayesian optimization – towards personalized treatments in plasma medicine. IEEE Conference on Decision and Control (CDC), 2023.
- On kernelized multi-armed bandits. In International Conference on Machine Learning (ICML), 2017.
- Gaussian process optimization with mutual information. In International Conference on International Conference on Machine Learning (ICML), 2014.
- Constrained Bayesian optimization with particle swarms for safe adaptive controller tuning. In Proc. of the IFAC (International Federation of Automatic Control), 2017.
- Challenges of real-world reinforcement learning. In International Conference on International Conference on Machine Learning (ICML), 2019.
- Practical and rigorous uncertainty bounds for gaussian process regression. In AAAI Conference on Artificial Intelligence, 2021.
- Noisy-input entropy search for efficient robust bayesian optimization. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
- Roman Garnett. Bayesian Optimization. Cambridge University Press, 2022. in preparation.
- Bayesian optimization with unknown constraints. In Conference on Uncertainty in Artificial Intelligence (UAI), 2014.
- Active learning for level set estimation. In International Joint Conference on Artificial Intelligence (IJCAI), 2013.
- Safe exploration for reinforcement learning. In European Symposium on Artificial Neural Networks (ESANN), 2008.
- Entropy search for information-efficient global optimization. Journal of Machine Learning Research, 13:1809–1837, 2012.
- Predictive entropy search for efficient global optimization of black-box functions. In International Conference on Neural Information Processing Systems (NIPS), 2014.
- Joint entropy search for maximally-informed bayesian optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
- Adaptive and safe bayesian optimization in high dimensions via one-dimensional subspaces. In International Conference for Machine Learning (ICML), 2019.
- Safe active learning for multi-output gaussian processes. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
- Safe exploration in markov decision processes. In International Coference on International Conference on Machine Learning (ICML), 2012.
- Constrained bayesian optimization with max-value entropy search. In NeurIPS 2019 Workshop on Metalearning, 2019.
- Blue river controls: A toolkit for reinforcement learning control systems on hardware. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
- Near-optimal multi-agent learning for safe coverage control. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
- Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. MIT Press, Cambridge, MA, USA, 2006.
- Safe active learning and safe bayesian optimization for tuning a pi-controller. IFAC-PapersOnLine, 50(1):5967–5972, 2017.
- Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT Press, Cambridge, MA, USA, 2002.
- Safe exploration for active learning with gaussian processes. In European Conference on Machine Learning and Knowledge Discovery in Databases (ECMLPKDD), 2015.
- Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE, 104:148–175, 2016.
- Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning (ICML), 2010.
- Safe exploration for optimization with Gaussian processes. In International Conference on Machine Learning (ICML), 2015.
- Stagewise safe Bayesian optimization with Gaussian processes. In International Conference on Machine Learning (ICML), 2018.
- Safe exploration in finite markov decision processes with gaussian processes. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
- Safe exploration for interactive machine learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
- Coverage Control, pages 11–67. Springer London, 2012.
- Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In International Conference on Machine Learning (ICML), 2017.