Bayesian Optimisation over Multiple Continuous and Categorical Inputs (1906.08878v2)

Abstract: Efficient optimisation of black-box problems that comprise both continuous and categorical inputs is important, yet poses significant challenges. We propose a new approach, Continuous and Categorical Bayesian Optimisation (CoCaBO), which combines the strengths of multi-armed bandits and Bayesian optimisation to select values for both categorical and continuous inputs. We model this mixed-type space using a Gaussian Process kernel, designed to allow sharing of information across multiple categorical variables, each with multiple possible values; this allows CoCaBO to leverage all available data efficiently. We extend our method to the batch setting and propose an efficient selection procedure that dynamically balances exploration and exploitation whilst encouraging batch diversity. We demonstrate empirically that our method outperforms existing approaches on both synthetic and real-world optimisation tasks with continuous and categorical inputs.

• The paper presents CoCaBO, which integrates multi-armed bandits with Gaussian Process optimisation to efficiently address mixed continuous and categorical input challenges.
• It introduces a tailored GP kernel that captures intra- and inter-category interactions without relying on one-hot encoding.
• Experimental results show superior performance over traditional methods on both synthetic benchmarks and real-world machine learning tasks.

Bayesian Optimisation over Multiple Continuous and Categorical Inputs

The paper "Bayesian Optimisation over Multiple Continuous and Categorical Inputs" introduces Continuous and Categorical Bayesian Optimisation (CoCaBO), an innovative approach designed to efficiently address the complexities of black-box optimisation problems featuring mixed continuous and categorical inputs. While traditional methods such as one-hot encoding and category-specific models have limitations concerning the dimensionality of the search space and sample efficiency, CoCaBO leverages a unified framework combining multi-armed bandits (MABs) and Gaussian Process (GP) based Bayesian optimisation.

Overview

The research illustrates the challenges inherent in optimising mixed input spaces, typically found within machine learning algorithms such as neural networks and gradient boosting decision trees. Categorical variables disrupt the assumption of differentiability that underpins many Bayesian optimisation techniques, necessitating novel solutions beyond simplistic transformations like one-hot encoding. This transformation significantly increases dimensionality, rendering optimisation algorithms less effective due to the resultant high-dimensional, sparse, and non-smooth search spaces.

CoCaBO innovatively circumvents these issues by utilizing a joint reward strategy within a multi-agent MAB system to select categorical inputs. This coordination across agents, combined with GP-based optimisation of continuous spaces, allows CoCaBO to efficiently identify high-performing configurations in mixed-type input scenarios. Within CoCaBO, categorical inputs are selected using the EXP3 bandit algorithm, accommodating the non-stationary nature of rewards observed as optimisation progresses. This integration of MAB for categorical selection and GP for continuous optimisation ensures computational efficiency, bypassing the dimensional pitfalls of one-hot encoding.

Additionally, the paper introduces a tailored GP kernel combining additive and multiplicative terms to capture both intra-category and inter-category information, without reverting to one-hot encoding. This hybrid kernel approach provides CoCaBO with enhanced expressiveness, allowing complex interactions between inputs to be modeled effectively.

Experimental Evaluation

Empirically, CoCaBO is validated against established methods such as SMAC, TPE, and one-hot encoded GP Bayesian optimisation, across a suite of synthetic benchmarks and real-world machine learning tasks. These tasks span diverse domains, from optimizing support vector machine parameters for the Boston housing dataset to neural architecture search within the NAS-Bench-101 framework. CoCaBO consistently demonstrates superior performance, particularly in scenarios with large categorical input spaces and multiple categorical variables. The experimental results underline the efficacy of the CoCaBO kernel and its ability to handle sparse data and mixed-type inputs, reaffirming the advantages of efficient category sharing strategies and dynamic acquisition functions.

Implications and Future Directions

CoCaBO offers significant implications for practical optimization problems commonly encountered in machine learning and AI application domains. By reducing computational overhead and improving sample efficiency, CoCaBO facilitates faster, more reliable tuning of algorithms with complex hyperparameter spaces. Its ability to handle parallel evaluations further enhances its real-world applicability, making it an attractive choice for distributed and cloud-based environments.

Future avenues may explore enhancing CoCaBO's parallel selection strategies, leveraging advanced probabilistic models, and integrating further with scalable computing platforms. Moreover, expanding on theoretical guarantees concerning convergence rates and regret bounds under varying conditions would bolster its foundational robustness and foster broader adoption.

In conclusion, the introduction of CoCaBO marks a substantive contribution to the field of Bayesian optimisation, providing professionals with a tool adept at handling the intricacies of mixed-variable spaces with remarkable precision and efficiency.