A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (2305.20028v2)

Abstract: Bayesian optimization is a highly efficient approach to optimizing objective functions which are expensive to query. These objectives are typically represented by Gaussian process (GP) surrogate models which are easy to optimize and support exact inference. While standard GP surrogates have been well-established in Bayesian optimization, Bayesian neural networks (BNNs) have recently become practical function approximators, with many benefits over standard GPs such as the ability to naturally handle non-stationarity and learn representations for high-dimensional data. In this paper, we study BNNs as alternatives to standard GP surrogates for optimization. We consider a variety of approximate inference procedures for finite-width BNNs, including high-quality Hamiltonian Monte Carlo, low-cost stochastic MCMC, and heuristics such as deep ensembles. We also consider infinite-width BNNs, linearized Laplace approximations, and partially stochastic models such as deep kernel learning. We evaluate this collection of surrogate models on diverse problems with varying dimensionality, number of objectives, non-stationarity, and discrete and continuous inputs. We find: (i) the ranking of methods is highly problem dependent, suggesting the need for tailored inductive biases; (ii) HMC is the most successful approximate inference procedure for fully stochastic BNNs; (iii) full stochasticity may be unnecessary as deep kernel learning is relatively competitive; (iv) deep ensembles perform relatively poorly; (v) infinite-width BNNs are particularly promising, especially in high dimensions.

• The paper shows that surrogate model performance in Bayesian optimization is highly problem-specific, advocating for adaptive models tailored to unique inductive biases.
• It demonstrates that Hamiltonian Monte Carlo offers high-quality inference for fully stochastic BNNs despite its computational intensity.
• The research highlights that infinite-width BNNs and deep kernel learning are promising alternatives to traditional Gaussian processes in high-dimensional, non-stationary settings.

Bayesian Neural Network Surrogates for Bayesian Optimization

The paper "A Study of Bayesian Neural Network Surrogates for Bayesian Optimization" explores the potential of Bayesian neural networks (BNNs) as surrogate models in the context of Bayesian Optimization (BO). Bayesian Optimization is a sophisticated method for optimizing costly-to-evaluate objective functions and relies heavily on the choice of surrogate models to efficiently explore the search space. Conventionally, Gaussian processes (GPs) are used due to their analytical tractability and strong priors, but they come with limitations like stationarity assumptions and challenges in high-dimensional spaces. This paper investigates whether BNNs can offer a viable alternative with certain advantages over GPs.

Methodological Overview

The research examines various BNN architectures, both finite-width and infinite-width, and compares different approximate inference techniques, including:

• Hamiltonian Monte Carlo (HMC): Known for its high-quality samples.
• Stochastic Gradient Hamiltonian Monte Carlo (SGHMC): A lower-cost alternative to HMC.
• Deep Ensembles: A heuristic method simulating BNN properties through ensemble approaches of deterministic networks.
• Infinite-width BNNs: Leveraging the neural network Gaussian process (NNGP) equivalence.
• Deep Kernel Learning (DKL): Combines the strengths of kernel methods with deep learning representations.

The paper conducts comprehensive experiments on both synthetic and real-world datasets, covering various input dimensionalities, objective count, and data characteristics such as stationarity and input types (discrete/continuous).

Key Findings

1. Problem Dependency: The efficacy of a surrogate model is highly problem-specific, indicating that adaptive approaches tuned to specific inductive biases of problems may outperform generic models.
2. HMC Performance: HMC emerges as a reliable inference method for fully-stochastic BNNs but is resource-intensive. Despite providing competitive results, it doesn't universally surpass simpler models such as GPs.
3. Infinite-width BNNs: These models show significant promise in high-dimensional scenarios, potentially due to their ability to model non-stationarity and utilize robust priors derived from deep architectures.
4. Deep Kernel Learning: This approach competes closely with fully-stochastic BNNs, suggesting that full stochastic models might not be strictly necessary for all applications.
5. Model Variability: No surrogate model consistently dominates across all problem types, emphasizing the need for model selection tailored to the specific characteristics of the optimization task.

Practical and Theoretical Implications

The research contributes to both theoretical and practical domains of machine learning and optimization. Theoretically, it highlights the nuanced trade-offs between different modeling paradigms in handling the specificity and variance of the objective functions being optimized. Practically, the paper presents compelling evidence for integrating non-GP surrogate models in BO frameworks, particularly in settings deviating from the assumptions underlying GPs, such as non-stationarity and high-dimensional inputs.

Future Directions

Future work could focus on automating the model selection and architecture tuning process to accommodate diverse problem characteristics dynamically. Additionally, further exploration of hybrid models that can seamlessly balance the strength of both BNNs and GPs could usher in a new era of robust, efficient Bayesian optimization frameworks.

The research underscores the nascent potential of BNN models in optimization and calls for more diverse benchmark problems to faithfully evaluate the performance landscape as the toolset expands beyond established norms.