Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Reinforced In-Context Black-Box Optimization (2402.17423v3)

Published 27 Feb 2024 in cs.LG, cs.AI, and cs.NE

Abstract: Black-Box Optimization (BBO) has found successful applications in many fields of science and engineering. Recently, there has been a growing interest in meta-learning particular components of BBO algorithms to speed up optimization and get rid of tedious hand-crafted heuristics. As an extension, learning the entire algorithm from data requires the least labor from experts and can provide the most flexibility. In this paper, we propose RIBBO, a method to reinforce-learn a BBO algorithm from offline data in an end-to-end fashion. RIBBO employs expressive sequence models to learn the optimization histories produced by multiple behavior algorithms and tasks, leveraging the in-context learning ability of large models to extract task information and make decisions accordingly. Central to our method is to augment the optimization histories with \textit{regret-to-go} tokens, which are designed to represent the performance of an algorithm based on cumulative regret over the future part of the histories. The integration of regret-to-go tokens enables RIBBO to automatically generate sequences of query points that satisfy the user-desired regret, which is verified by its universally good empirical performance on diverse problems, including BBO benchmark functions, hyper-parameter optimization and robot control problems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Two decades of blackbox optimization applications. EURO Journal on Computational Optimization, 9:100011, 2021.
  2. HPO-B: A large-scale reproducible benchmark for black-box HPO based on OpenML. arXiv preprint arXiv:2106.06257, 2021.
  3. Thinking inside the box: A tutorial on grey-box Bayesian optimization. In Proceedings of the 51st Winter Simulation Conference (WSC’21), pp.  1–15, Phoenix, AZ, 2021.
  4. Derivative-Free and Blackbox Optimization. Springer, 2017.
  5. Back, T. Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford University Press, 1996.
  6. Transfer learning for Bayesian optimization: A survey. arXiv preprint arXiv:2302.05927, 2023.
  7. A framework for behavioural cloning. Machine Intelligence, 15:103–129, 1995.
  8. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. In Advances in Neural Information Processing Systems 33 (NeurIPS’20), pp.  10113–10124, Virtual, 2020.
  9. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 (NeurIPS’20), pp.  1877–1901, Virtual, 2020.
  10. Bayesian optimization for learning gaits under uncertainty - An experimental comparison on a dynamic bipedal walker. Annals of Mathematics and Artificial Intelligence, 76(1-2):5–23, 2016.
  11. A survey on policy search algorithms for learning robot controllers in a handful of trials. IEEE Transactions on Robotics, 36(2):328–347, 2019.
  12. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems 34 (NeurIPS’21), pp.  15084–15097, Virtual, 2021.
  13. Learning to learn without gradient descent by gradient descent. In Proceedings of the 34th International Conference on Machine Learning (ICML’17), pp.  748–756, Sydney, Australia, 2017.
  14. Towards learning universal hyperparameter optimizers with transformers. In Advances in Neural Information Processing Systems 35 (NeurIPS’22), pp.  32053–32068, New Orleans, LA, 2022.
  15. COCO: The large scale black-box optimization benchmarking (BBOB-largescale) test suite. arXiv preprint arXiv:1903.06396, 2019.
  16. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing Systems 32 (NeurIPS’19), pp.  5497–5508, Vancouver, Canada, 2019.
  17. Initializing Bayesian hyperparameter optimization via meta-learning. In Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI’15), pp.  1128–1135, Austin, TX, 2015.
  18. OpenML-Python: An extensible Python API for OpenML. The Journal of Machine Learning Research, 22(1):4573–4577, 2021.
  19. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML’17), pp.  1126–1135, Sydney, Australia, 2017.
  20. Frazier, P. I. A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811, 2018.
  21. Bayesian Optimization for Materials Design. Springer, 2016.
  22. Conditional neural processes. In Proceedings of the 35th International Conference on Machine Learning (ICML’18), pp.  1704–1713, Stockholm, Sweden, 2018.
  23. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science, 4(2):268 – 276, 2018.
  24. Bayesian optimization for adaptive experimental design: A review. IEEE Access, 8:13937–13948, 2020.
  25. How do transformers learn in-context beyond simple functions? A case study on learning with representations. arXiv preprint arXiv:2310.10616, 2023.
  26. Hansen, N. The CMA evolution strategy: A tutorial. arXiv preprint arXiv:1604.00772, 2016.
  27. TabPFN: A transformer that solves small tabular classification problems in a second. In Proceedings of the 11th International Conference on Learning Representations (ICLR’23), Kigali, Rwanda, 2023.
  28. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5149–5169, 2021.
  29. Reinforced few-shot acquisition function learning for Bayesian optimization. In Advances in Neural Information Processing Systems 34 (NeurIPS’21), pp.  7718–7731, Virtual, 2021.
  30. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4):455–492, 1998.
  31. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  32. Transformers in vision: A survey. ACM Computing Surveys, 54(10s):1–41, 2022.
  33. Discovering attention-based genetic algorithms via meta-black-box optimization. In Proceedings of the 25th Conference on Genetic and Evolutionary Computation (GECCO’23), pp.  929–937, Lisbon, Portugal, 2023a.
  34. Discovering evolution strategies via meta-black-box optimization. In Proceedings of the 11th International Conference on Learning Representations (ICLR’23), Kigali, Rwanda, 2023b.
  35. In-context reinforcement learning with algorithm distillation. In Proceedings of the 11th International Conference on Learning Representations (ICLR’23), Kigali, Rwanda, 2023.
  36. Transfer learning based search space design for hyperparameter tuning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD’22), pp.  967–977, Washington, DC, 2022.
  37. Transformers as algorithms: Generalization and stability in in-context learning. In Proceedings of the 40th International Conference on Machine Learning (ICML’23), pp.  19565–19594, Honolulu, HI, 2023.
  38. End-to-end meta-Bayesian optimisation with transformer neural processes. In Advances in Neural Information Processing Systems 36 (NeurIPS’23), New Orleans, LA, 2023.
  39. Transformers can do Bayesian inference. In Proceedings of the 10th International Conference on Learning Representations (ICLR’22), Virtual, 2022.
  40. PFNs4BO: In-context learning for Bayesian optimization. In Proceedings of the 40th International Conference on Machine Learning (ICML’23), pp.  25444–25470, Honolulu, HI, 2023.
  41. The knowledge-gradient algorithm for sequencing experiments in drug discovery. INFORMS Journal on Computing, 23(3):346–363, 2011.
  42. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In Proceedings of the 39th International Conference on Machine Learning (ICML’22), pp.  16569–16594, Baltimore, MD, 2022.
  43. ExPT: Synthetic pretraining for few-shot experimental design. arXiv preprint arXiv:2310.19961, 2023.
  44. Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning. In Advances in Neural Information Processing Systems 32 (NeurIPS’19), pp.  12751–12761, Vancouver, Canada, 2019.
  45. Scalable hyperparameter transfer learning. In Advances in Neural Information Processing Systems 31 (NeurIPS’18), pp.  6846–6856, Montreal, Canada, 2018.
  46. Warm starting Bayesian optimization. In Proceedings of the 46th Winter Simulation Conference (WSC’16), pp.  770–781, Washington, DC, 2016.
  47. Improving language understanding by generative pre-training. OpenAI Blog, 2018.
  48. Gaussian Processes for Machine Learning. The MIT Press, 2006.
  49. Regularized evolution for image classifier architecture search. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI’19), pp.  4780–4789, Honolulu, HI, 2019.
  50. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2016.
  51. Open Source Vizier: Distributed infrastructure and API for reliable and flexible black-box optimization. In Proceedings of the 1st International Conference on Automated Machine Learning (AutoML Conference’22), pp.  1–17, Baltimore, MD, 2022.
  52. Black-box optimization for automated discovery. Accounts of Chemical Research, 54(6):1334–1346, 2021.
  53. Meta-learning for black-box optimization. In Proceedings of European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD’19), pp.  366–381, Würzburg, Germany, 2019.
  54. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NIPS’17), pp.  5998–6008, Long Beach, Canada, 2017.
  55. A perspective view and survey of meta-learning. Artificial Intelligence Review, 18(2):77–95, 2002.
  56. Meta-learning acquisition functions for transfer learning in Bayesian optimization. In Proceedings of the 8th International Conference on Learning Representations (ICLR’20), Addis Ababa, Ethiopia, 2020.
  57. Batched large-scale Bayesian optimization in high-dimensional spaces. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS’18), pp.  745–754, Playa Blanca, Spain, 2018.
  58. Pre-trained Gaussian processes for Bayesian optimization. arXiv preprint arXiv:2109.08215, 2021.
  59. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125, 2022.
  60. Maximizing acquisition functions for Bayesian optimization. In Advances in Neural Information Processing Systems 31 (NeurIPS’18), pp.  9906–9917, Montréal, Canada, 2018.
  61. Few-shot Bayesian optimization with deep kernel surrogates. In Proceedings of the 9th International Conference on Learning Representations (ICLR’21), Virtual, 2021.
  62. Transformers: State-of-the-art natural language processing. In Proceedings of the 25th Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP’20), pp.  38–45, Virtual, 2020.
  63. Eagle strategy using Lévy walk and firefly algorithms for stochastic optimization. In Proceedings of the 4th Nature Inspired Cooperative Strategies for Optimization (NICSO’10), pp.  101–111, Granada, Spain, 2010.
  64. Towards omni-generalizable neural methods for vehicle routing problems. In Proceedings of the 40th International Conference on Machine Learning (ICML’23), pp.  42769–42789, Honolulu, HI, 2023.
  65. Evolutionary Learning: Advances in Theories and Algorithms. Springer, 2019.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now