Constrained Bayesian Optimization with Noisy Experiments (1706.07094v2)

Abstract: Randomized experiments are the gold standard for evaluating the effects of changes to real-world systems. Data in these tests may be difficult to collect and outcomes may have high variance, resulting in potentially large measurement error. Bayesian optimization is a promising technique for efficiently optimizing multiple continuous parameters, but existing approaches degrade in performance when the noise level is high, limiting its applicability to many randomized experiments. We derive an expression for expected improvement under greedy batch optimization with noisy observations and noisy constraints, and develop a quasi-Monte Carlo approximation that allows it to be efficiently optimized. Simulations with synthetic functions show that optimization performance on noisy, constrained problems outperforms existing methods. We further demonstrate the effectiveness of the method with two real-world experiments conducted at Facebook: optimizing a ranking system, and optimizing server compiler flags.

• The paper introduces a novel Noisy Expected Improvement acquisition function that directly addresses noisy, constrained experimental settings.
• It employs quasi-Monte Carlo integration to robustly handle high noise levels and supports batch optimization in practical applications.
• Empirical evaluations on synthetic functions and real-world cases show significant performance gains over traditional optimization methods.

Essay on "Constrained Bayesian Optimization with Noisy Experiments"

The research paper titled "Constrained Bayesian Optimization with Noisy Experiments" explores the development and application of a Bayesian optimization method designed to efficiently optimize objective functions in the presence of noisy observations and constraints. Authored by Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy, the paper explores the intricacies of optimizing complex, continuous parameters often involved in randomized experiments. These experiments are common in various fields such as internet services, economics, and medicine, where traditional optimization methods struggle due to the inherent noise and variance in the data.

Core Contributions

The central contribution of this paper is the derivation of a new acquisition function, named Noisy Expected Improvement (NEI), specifically tailored for scenarios with both noisy objective measurements and constraints. This approach mitigates the limitations found in traditional methods like Expected Improvement (EI), which do not handle noise directly and often rely on heuristics that can fail in high noise settings. The NEI formulation integrates over the posterior distributions of the observed values using a quasi-Monte Carlo (QMC) approach, allowing it to maintain robustness in the presence of substantial measurement errors.

The authors address several critical challenges in Bayesian optimization:

• Handling High Noise Levels: The NEI method eschews simplistic heuristics and directly accounts for noise, enabling it to identify better solutions despite the high variance of the outcomes.
• Incorporating Constraints: Unlike other approaches that often treat constraints using problematic approximations or ignore them, this method accounts for infeasibility within its utility framework, thus supporting effective optimization in realistic settings where trade-offs are necessary.
• Batch Optimization: Recognizing the feasibility of running multiple experiments in parallel, the paper extends the NEI methodology to efficiently accommodate asynchronous and batch evaluations, crucial for practical applications where time constraints exist.

Results and Implications

Empirical results, both from synthetic functions and real-world applications, highlight the effectiveness of the NEI method. Experiments demonstrate that NEI outperforms existing techniques, including EI with heuristics and the predictive entropy search. The research includes rigorous synthetic benchmarks such as the constrained Hartmann6 and Branin functions, showing that NEI consistently attains superior optimization outcomes as compared to conventional methods.

Practically, the paper showcases substantial real-world implementations. These include optimizing a ranking system within the operational infrastructure at Facebook and improving compiler flags for server performance. These case studies not only underscore the capability of NEI to deliver genuinely improved solutions but also its scalability and applicability in industrial contexts.

Comparison With Other Methods

The paper situates NEI against other acquisition functions like knowledge gradient and Thompson sampling. Notably, most existing methods either fail to address noise adequately or involve complex entropic approximations that hinder practicality. The straightforward yet theoretically grounded nature of NEI enables its easy integration and application without excessively resorting to external approximations or estimates.

Future Prospects

The paper positions NEI as an advancement in the field of Bayesian optimization, particularly for noisy and constrained settings. However, the authors acknowledge that NEI focuses on myopic utility functions and does not prioritize replication of observations, which could be valuable in certain conditions. Future research might explore adaptive mechanisms to incorporate replication value or expand the utility model to include infeasible sampling for greater exploration of the optimization landscape.

In summary, "Constrained Bayesian Optimization with Noisy Experiments" offers a robust framework and a novel algorithmic approach for optimizing high-noise systems with constraints. It extends the reach of Bayesian optimization techniques into new application domains, fostering improved decision-making in settings characterized by complexity and uncertainty.