Practical Bayesian Optimization for Model Fitting with Bayesian Adaptive Direct Search

Abstract: Computational models in fields such as computational neuroscience are often evaluated via stochastic simulation or numerical approximation. Fitting these models implies a difficult optimization problem over complex, possibly noisy parameter landscapes. Bayesian optimization (BO) has been successfully applied to solving expensive black-box problems in engineering and machine learning. Here we explore whether BO can be applied as a general tool for model fitting. First, we present a novel hybrid BO algorithm, Bayesian adaptive direct search (BADS), that achieves competitive performance with an affordable computational overhead for the running time of typical models. We then perform an extensive benchmark of BADS vs. many common and state-of-the-art nonconvex, derivative-free optimizers, on a set of model-fitting problems with real data and models from six studies in behavioral, cognitive, and computational neuroscience. With default settings, BADS consistently finds comparable or better solutions than other methods, including `vanilla' BO, showing great promise for advanced BO techniques, and BADS in particular, as a general model-fitting tool.

• The paper introduces Bayesian Adaptive Direct Search (BADS), a hybrid method that optimizes complex models in noisy, nonconvex parameter landscapes.
• BADS combines a mesh-adaptive direct search with Gaussian process-based Bayesian optimization to balance exploration and exploitation.
• Empirical benchmarks show that BADS outperforms traditional optimization techniques in efficiency and accuracy for challenging model fitting tasks.

Practical Bayesian Optimization for Model Fitting with Bayesian Adaptive Direct Search

The paper "Practical Bayesian Optimization for Model Fitting with Bayesian Adaptive Direct Search" presents a significant contribution to the field of computational model fitting. The research addresses the challenge of fitting complex computational models, a task that frequently arises in disciplines such as computational neuroscience, where models are often evaluated through computationally intensive simulations or approximation methods. The intrinsic difficulty stems from the need to navigate noisy, nonconvex parameter landscapes, a task for which gradient-based optimization is typically infeasible.

This study introduces Bayesian Adaptive Direct Search (BADS), a novel hybrid Bayesian optimization algorithm designed to optimize expensive black-box functions effectively. BADS particularly targets the fitting of complex models where traditional optimization algorithms struggle due to noise and nonconvexity in the parameter landscape. The method builds on existing Bayesian optimization frameworks, enhancing them with adaptive direct search techniques to achieve substantial improvements in efficiency and accuracy with relatively low computational overhead.

Key Contributions and Methodology

1. Development of BADS: BADS integrates a mesh-adaptive direct search (MADS) framework with a Bayesian optimization scheme using Gaussian processes (GPs). This hybrid approach allows BADS to adaptively refine the search space using local GPs, thereby balancing exploration and exploitation effectively.
2. Benchmarking and Comparisons: The research extensively benchmarks BADS against a range of existing optimization algorithms, including other Bayesian optimization techniques and various derivative-free optimizers. The experiments cover deterministic and noisy model-fitting problems derived from real-world data in behavioral, cognitive, and computational neuroscience. The results consistently show that BADS either matches or exceeds the performance of alternative methods, particularly when tasked with finding global optima within complex, noisy landscapes.
3. Algorithmic innovations: The paper highlights critical innovations in BADS, such as the hedged strategy for adaptive search directions and the integration of GP-based local surrogate models. These enhancements allow the algorithm to maintain robustness and efficiency, even as model evaluation costs vary between moderately expensive to very costly regimes.
4. Practical Implementations: Importantly, the authors provide BADS as a MATLAB package, designed for ease of use with a familiar interface for practitioners. The package requires minimal tuning, making it accessible for users without in-depth technical expertise in Bayesian methods.

Implications and Future Directions

The development of BADS expands the toolkit available for optimizing complex models, particularly in fields where direct evaluation is computationally prohibitive. The method's ability to efficiently navigate rough parameter spaces with bounded computational resources forecasts a broad range of applications beyond neuroscience, potentially impacting fields like engineering optimization and automated machine learning.

Furthermore, the paper suggests several avenues for future exploration. For instance, it opens the possibility of incorporating alternative statistical models within the BADS framework, such as random forests or neural surrogates, which might provide advantages in specific contexts or for certain types of noise models. There is also a prospect of integrating multi-start optimization strategies for enhanced global search capability.

In summary, the paper delivers a robust and practical methodological advancement for Bayesian optimization in model fitting, demonstrated with strong empirical evidence. BADS offers a compelling blend of theoretical innovation and practical usability, setting the stage for broader adoption and future enhancements across diverse scientific and engineering domains.