Papers
Topics
Authors
Recent
2000 character limit reached

Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spaces (2307.00618v2)

Published 2 Jul 2023 in cs.LG

Abstract: Impactful applications such as materials discovery, hardware design, neural architecture search, or portfolio optimization require optimizing high-dimensional black-box functions with mixed and combinatorial input spaces. While Bayesian optimization has recently made significant progress in solving such problems, an in-depth analysis reveals that the current state-of-the-art methods are not reliable. Their performances degrade substantially when the unknown optima of the function do not have a certain structure. To fill the need for a reliable algorithm for combinatorial and mixed spaces, this paper proposes Bounce that relies on a novel map of various variable types into nested embeddings of increasing dimensionality. Comprehensive experiments show that Bounce reliably achieves and often even improves upon state-of-the-art performance on a variety of high-dimensional problems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. Webpage of the Fourth Max-SAT Evaluation. http://www.maxsat.udl.cat/09/index.php?disp=submitted-benchmarks. Last access: 2023-05-02.
  2. MaxSAT Evaluation 2018 : Solver and Benchmark Descriptions. 2018. URL http://hdl.handle.net/10138/237139. Publisher: Department of Computer Science, University of Helsinki.
  3. AHMAXSAT: Description and evaluation of a branch and bound Max-SAT solver. Journal on Satisfiability, Boolean Modeling and Computation, 9(1):89–128, 2014.
  4. MaxSAT Evaluation 2018: New developments and detailed results. Journal on Satisfiability, Boolean Modeling and Computation, 11(1):99–131, 2019.
  5. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.
  6. Bayesian optimization of combinatorial structures. In International Conference on Machine Learning, pages 462–471. PMLR, 2018.
  7. Online Black-Box Algorithm Portfolios for Continuous Optimization. In Parallel Problem Solving from Nature – PPSN XIII, pages 40–49, Cham, 2014. Springer International Publishing.
  8. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 13(2), 2012.
  9. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 24, 2011.
  10. A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization, 2(2):1–26, 2022.
  11. A warped kernel improving robustness in Bayesian optimization via random embeddings. In International Conference on Learning and Intelligent Optimization (LION), pages 281–286. Springer, 2015.
  12. On the choice of the low-dimensional domain for global optimization via random embeddings. Journal of global optimization, 76(1):69–90, 2020.
  13. Closed-loop automatic gradient design for liquid chromatography using Bayesian optimization. Analytica Chimica Acta, 1242, 2023.
  14. Efficient global optimization for high-dimensional constrained problems by using the Kriging models combined with the partial least squares method. Engineering Optimization, 50(12):2038–2053, 2018.
  15. A mobile robotic chemist. Nature, 583(7815):237–241, 2020.
  16. Bayesian Gait Optimization for Bipedal Locomotion. In Learning and Intelligent Optimization, pages 274–290. Springer International Publishing, 2014.
  17. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 76(1):5–23, 2016.
  18. Multi-Information Source Bayesian Optimization of Culture Media for Cellular Agriculture. Biotechnology and Bioengineering, 2022.
  19. Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.
  20. Bayesian optimization over permutation spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6515–6523, 2022.
  21. Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddings. In International Conference on Artificial Intelligence and Statistics, pages 7021–7039. PMLR, 2023.
  22. HPVM2FPGA: Enabling True Hardware-Agnostic FPGA Programming. In Proceedings of the 33rd IEEE International Conference on Application-specific Systems, Architectures, and Processors, 2022.
  23. High-dimensional Bayesian optimization with sparse axis-aligned subspaces. In Cassio de Campos and Marloes H. Maathuis, editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 493–503. PMLR, 27–30 Jul 2021.
  24. Scalable Constrained Bayesian Optimization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 730–738. PMLR, 13–15 Apr 2021.
  25. Scalable Global Optimization via Local Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pages 5496–5507, 2019.
  26. Advances in protein structure prediction and de novo protein design: A review. Chemical Engineering Science, 61(3):966–988, 2006.
  27. Peter I Frazier. A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811, 2018.
  28. Bayesian Optimization for Materials Design, pages 45–75. Springer International Publishing, Cham, 2016. ISBN 978-3-319-23871-5.
  29. GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
  30. Multi-information source constrained Bayesian optimization. Structural and Multidisciplinary Optimization, 59:977–991, 2019.
  31. A fusion-based multi-information source optimization approach using knowledge gradient policies. In 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, page 1159, 2018.
  32. Algorithms for the maximum satisfiability problem. Computing, 44(4):279–303, 1990.
  33. Phoenics: a Bayesian optimizer for chemistry. ACS central science, 4(9):1134–1145, 2018.
  34. Gryffin: An algorithm for Bayesian optimization of categorical variables informed by expert knowledge. Applied Physics Reviews, 8(3):031406, 2021.
  35. High-dimensional Bayesian Optimization with Group Testing. arXiv preprint arXiv:2310.03515, 2023a.
  36. BaCO: A Fast and Portable Bayesian Compiler Optimization Framework. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2023b.
  37. Efficient search of compositional space for hybrid organic–inorganic perovskites via Bayesian optimization. npj Computational Materials, 4(1):1–7, 2018.
  38. Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1470–1479. PMLR, 06–11 Aug 2017.
  39. Contamination control in food supply chain. In Proceedings of the 2010 Winter Simulation Conference, pages 2678–2681. IEEE, 2010.
  40. Tuning Materials-Binding Peptide Sequences toward Gold-and Silver-Binding Selectivity with Bayesian Optimization. ACS nano, 15(11):18260–18269, 2021.
  41. Sequential Model-Based Optimization for General Algorithm Configuration. In Carlos A. Coello Coello, editor, Learning and Intelligent Optimization, pages 507–523, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-25566-3.
  42. Deterministic job-shop scheduling: Past, present and future. European journal of operational research, 113(2):390–434, 1999.
  43. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455, 1998.
  44. Combinatorial Bayesian optimization with random mapping functions to convex polytopes. In Uncertainty in Artificial Intelligence, pages 1001–1011. PMLR, 2022.
  45. Advances in Bayesian optimization with applications in aerospace engineering. In 2018 AIAA Non-Deterministic Approaches Conference, page 1656, 2018.
  46. Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1546–1558, 2020.
  47. Global optimization with parametric function approximation. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 22113–22136, 2023.
  48. Automatic Gait Optimization With Gaussian Process Regression. In IJCAI, volume 7, pages 944–949, 2007.
  49. Active subspaces for shape optimization. In 10th AIAA multidisciplinary design optimization conference, page 1171, 2014.
  50. Skill-based Multi-objective Reinforcement Learning of Industrial Robot Tasks with Planning and Knowledge Integration. CoRR, abs/2203.10033, 2022.
  51. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 347–358, 2019.
  52. A framework for Bayesian Optimization in Embedded Subspaces. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research (PMLR), pages 4752–4761, 09–15 Jun 2019.
  53. The Knowledge-Gradient Algorithm for Sequencing Experiments in Drug Discovery. INFORMS Journal on Computing, 23(3):346–363, 2011.
  54. Dirichlet and related distributions: Theory, methods and applications. 2011.
  55. A survey of combinatorial testing. ACM Computing Surveys (CSUR), 43(2):1–29, 2011.
  56. Combinatorial Bayesian Optimization using the Graph Cartesian Product. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
  57. Closed-loop, multiobjective optimization of analytical instrumentation: gas chromatography/time-of-flight mass spectrometry of the metabolomes of human serum and of yeast fermentations. Analytical Chemistry, 77(1):290–303, 2005.
  58. Low autocorrelation binary sequences. Journal of Physics A: Mathematical and Theoretical, 49(16):165001, 2016.
  59. Daniel Packwood. Bayesian Optimization for Materials Science. Springer, 2017.
  60. Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.
  61. An experimental evaluation of fast approximation algorithms for the maximum satisfiability problem. Journal of Experimental Algorithmics (JEA), 22:1–18, 2017.
  62. Multi-information source optimization. Advances in neural information processing systems, 30, 2017.
  63. Warren B Powell. A unified framework for stochastic optimization. European Journal of Operational Research, 275(3):795–821, 2019.
  64. Bayesian optimization using domain knowledge on the ATRIAS biped. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1771–1778, 2018.
  65. Combining radial basis function surrogates and dynamic coordinate search in high-dimensional expensive black-box optimization. Engineering Optimization, 45(5):529–555, 2013.
  66. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278–1286. PMLR, 2014.
  67. Bayesian optimisation over multiple continuous and categorical inputs. In International Conference on Machine Learning, pages 8276–8285. PMLR, 2020.
  68. Machine learning meets continuous flow chemistry: Automated optimization towards the Pareto front of multiple objectives. Chemical Engineering Journal, 352:277–282, 2018.
  69. LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso. In First Conference on Automated Machine Learning (Main Track), 2022.
  70. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2015.
  71. Bayesian reaction optimization as a tool for chemical synthesis. Nature, 590(7844):89–96, 2021.
  72. A tutorial on support vector regression. Statistics and computing, 14:199–222, 2004.
  73. Francisco J. Solis and Roger J-B. Wets. Minimization by Random Search Techniques. Mathematics of Operations Research, 6(1):19–30, 1981.
  74. Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimization. Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.
  75. DeepFreak: Learning crystallography diffraction patterns with automated machine learning. arXiv preprint arXiv:1904.11834, 2019.
  76. Autonomous drug design with multi-armed bandits. In 2022 IEEE International Conference on Big Data (Big Data), pages 5584–5592. IEEE, 2022.
  77. Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces. Advances in Neural Information Processing Systems, 35:37401–37415, 2022.
  78. COMBO: An efficient Bayesian optimization library for materials science. Materials Discovery, 4:18–21, 2016.
  79. Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10663–10674. PMLR, 18–24 Jul 2021.
  80. Parallel Bayesian global optimization of expensive functions. Operations Research, 68(6):1850–1865, 2020a.
  81. Ke Wang and Alexander W Dowling. Bayesian optimization for chemical products and functional materials. Current Opinion in Chemical Engineering, 36:100728, 2022.
  82. Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search. Advances in Neural Information Processing Systems (NeurIPS), 33:19511–19522, 2020b.
  83. Recent Advances in Bayesian Optimization. ACM Comput. Surv., 2023.
  84. Bayesian Optimization in a Billion Dimensions via Random Embeddings. Journal of Artificial Intelligence Research (JAIR), 55:361–387, 2016.
  85. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
  86. The reparameterization trick for acquisition functions. NeurIPS Workshop on Bayesian Optimization, 2017.
  87. David P Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science, 10(1–2):1–157, 2014.
  88. Are Random Decompositions all we need in High Dimensional Bayesian Optimisation? In International Conference on Machine Learning, pages 43347–43368. PMLR, 2023.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up