Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization (2306.09803v3)

Abstract: This paper introduces a modular framework for Mixed-variable and Combinatorial Bayesian Optimization (MCBO) to address the lack of systematic benchmarking and standardized evaluation in the field. Current MCBO papers often introduce non-diverse or non-standard benchmarks to evaluate their methods, impeding the proper assessment of different MCBO primitives and their combinations. Additionally, papers introducing a solution for a single MCBO primitive often omit benchmarking against baselines that utilize the same methods for the remaining primitives. This omission is primarily due to the significant implementation overhead involved, resulting in a lack of controlled assessments and an inability to showcase the merits of a contribution effectively. To overcome these challenges, our proposed framework enables an effortless combination of Bayesian Optimization components, and provides a diverse set of synthetic and real-world benchmarking tasks. Leveraging this flexibility, we implement 47 novel MCBO algorithms and benchmark them against seven existing MCBO solvers and five standard black-box optimization algorithms on ten tasks, conducting over 4000 experiments. Our findings reveal a superior combination of MCBO primitives outperforming existing approaches and illustrate the significance of model fit and the use of a trust region. We make our MCBO library available under the MIT license at \url{https://github.com/huawei-noah/HEBO/tree/master/MCBO}.

• The paper presents a comprehensive framework and standardized benchmarking suite for Combinatorial and Mixed-variable Bayesian Optimization (MCBO) to address the lack of standardized evaluation.
• The proposed modular framework integrates specialized surrogate models, acquisition functions, and discrete-space optimizers adapted for mixed and combinatorial variables.
• Benchmarking 47 algorithms across 10 tasks showed that trust regions improve performance, surrogate model choice is critical, and Genetic Algorithms excel as acquisition optimizers.

A Comprehensive Framework and Benchmarking for Mixed-variable and Combinatorial Bayesian Optimization

The paper presents a framework designed to address key challenges in Mixed-variable and Combinatorial Bayesian Optimization (MCBO). Despite the potential of MCBO in tackling complex optimization problems across various domains, the absence of standardized benchmarking and evaluation practices has hindered progress. The authors propose a modular framework that facilitates the combination of MCBO components and provides a diverse set of benchmarking tasks.

Framework Overview

The MCBO framework offers a structured approach to building and evaluating various Bayesian Optimization (BO) strategies. It integrates components such as probabilistic surrogate models, acquisition functions, and optimizers, adapting these for mixed-variable and combinatorial spaces.

Core Components

1. Surrogate Models: The framework supports Gaussian Processes (GPs), known for their efficacy in modeling black-box functions due to the combination of flexibility and uncertainty quantification. Key kernels for combinatorial inputs, such as the Overlap and Transformed Overlap kernels, are utilized.
2. Acquisition Functions: These are employed to balance exploration and exploitation. The framework includes Expected Improvement (EI), Probability of Improvement (PI), Lower Confidence Bound (LCB), and Thompson Sampling (TS).
3. Optimizers: Addressing discrete spaces, the framework includes methods like Hill Climbing (HC), Genetic Algorithms (GA), and Simulated Annealing (SA), with modifications to support trust regions for efficient local optimization.

Benchmarking and Results

The authors implement 47 novel MCBO algorithms and benchmark them against existing solutions and standard black-box optimization methods across ten tasks. The benchmarking involves over 4000 experiments, providing extensive data on algorithm performance.

Key Findings

• Trust Region Effectiveness: Incorporating trust regions generally enhances optimization performance, allowing more effective exploitation of local structures.
• Surrogate Model Importance: The choice of surrogate model significantly impacts performance. For combinatorial tasks, models utilizing kernels like SSK (string subsequence kernel) perform well.
• Acquisition Optimization: Genetic Algorithms (GA) prove to be effective acquisition optimizers, providing a strong exploration-exploitation balance in high-dimensional spaces.

Implications and Future Work

The development of a standardized and flexible framework for MCBO has profound implications. It provides researchers with a tool to evaluate new algorithms systematically and explore novel combinations of optimization primitives. The findings underscore the importance of model fit and adaptive acquisition optimization strategies in Bayesian frameworks.

The paper suggests future developments in adaptive surrogate modeling and advanced kernel methods tailored to specific problem domains. The presented framework lays the groundwork for ongoing improvements in MCBO methodologies and their application to complex real-world problems.

Conclusion

This paper offers significant contributions to MCBO by providing a comprehensive framework and standardizing evaluation processes. The framework's implementation allows for streamlined experimentation and evaluation, encouraging further research and innovation in Bayesian Optimization strategies.