Batch Bayesian Optimization

• Batch Bayesian Optimization is a method that selects multiple query points concurrently using surrogate models to efficiently balance accuracy and experiment speed.
• It employs techniques such as fixed-size batching, dynamic batch adaptation, and local penalization to manage dependencies and optimize parallel evaluations.
• Practical applications in hyperparameter tuning, experimental sciences, and materials design illustrate its potential to significantly reduce the overall optimization time.

Batch Bayesian Optimization (batch BO) refers to the class of Bayesian optimization algorithms designed to select multiple query points to evaluate in parallel during each iteration, as opposed to the fully sequential paradigm where only one point is chosen per iteration. This approach is crucial in experimental scenarios or computational environments where parallel resources are available and the bottleneck is in experiment turnaround rather than model computation. Batch Bayesian optimization methods address the trade-off between the statistical efficiency of sequential sampling and the practical gains in wall-clock time achieved by concurrent evaluations.

1. Foundations and Motivations

The canonical Bayesian optimization setup sequentially selects a point $x_t$ at each round $t$ by maximizing an acquisition function (such as Expected Improvement, EI) which is informed by a surrogate model—commonly a Gaussian Process (GP). Each new query is chosen conditional on all previous observations, leveraging the updated posterior to maximize expected progress toward the optimum while minimizing redundant samples. However, in many practical settings, the cost of waiting for single queries to complete renders sequential selection inefficient; if multiple experiments or computations can be executed simultaneously, batching queries significantly accelerates the overall optimization process.

Batch BO schemes have been motivated by the need to leverage modern parallel computational infrastructure or experimental designs involving arrays of assays or runs. The main challenge, contrasting with the sequential case, stems from the mutual dependence of batch elements: the decision to select point $x_i$ generally depends on, but cannot condition on, the unknown outcomes of $x_1, \ldots, x_{i-1}$ within the same batch.

2. Methodological Approaches

Batch Bayesian optimization algorithms can be categorized based on their batch selection mechanisms and their strategies for managing dependencies between batch points:

• Fixed-size Batch Selection: Algorithms such as "Constant Liar" or "Kriging Believer" create batches by iteratively selecting candidates via greedy maximization of an acquisition function, simulating outcomes at pending points (e.g., hallucinating them as the predicted mean or a constant value). These points are treated as if their outcomes are known, allowing the surrogate to be updated in silico before each new batch element selection.
• Dynamic Batch Size Schemes: The "Dynamic Batch Bayesian Optimization" algorithm (Azimi et al., 2011) introduces a mechanism whereby the batch size $p_t$ at iteration $t$ is determined adaptively. The algorithm simulates the effect of each candidate point via a “fantasized” outcome to determine if subsequent candidate selections would be nearly independent of previous ones—quantified by upper-bounding the expected change in the posterior mean due to unknown outcomes. Points are added to the batch if the anticipated mutual dependence (measured by quantities such as $E[|\Delta^*(\mu_z)|]$) is below a threshold $\epsilon$, allowing the batch to grow as large as independence conditions permit.
• Local Penalization: The "Local Penalization" approach (González et al., 2015) avoids intractable joint acquisition maximization by defining exclusion zones (balls) of radius $r_j = (M - f(x_j))/L$ around previously selected batch points, based on an estimated Lipschitz constant $L$ for the unknown function. A local penalizer function $\phi(x; x_j)$ diminishes the acquisition value in the vicinity of $x_j$, thereby enforcing diversity among batch points in a principled manner.
• Optimal Batch Formulation: The parallel knowledge gradient method (qKG) (Wu et al., 2016) mathematically formulates batch selection as a one-step Bayes-optimal experiment design, maximizing the expected decrement in the minimum posterior mean across the domain given the new batch. This approach optimizes over the batch jointly rather than greedily and employs stochastic gradient-based optimization (e.g., via infinitesimal perturbation analysis) leveraging Monte Carlo techniques for intractable expectations.
• Hybrid and Adaptive Batch Strategies: Algorithms such as "Hybrid Batch Bayesian Optimization" (Azimi et al., 2012) dynamically choose between sequential and batch modes by simulating pseudo-outcomes at selected points and bounding the propagation of simulation error. Batch expansion is halted when the supposed simulation deviation would meaningfully change the quantity or location of subsequent candidates.
• Batch Size Discovery Methods: "Budgeted Batch Bayesian Optimization" (B3O) (Nguyen et al., 2017) employs Infinite Gaussian Mixture Models (IGMMs) and batch generalized slice sampling to estimate the number of “modes” or significant peaks in the current acquisition function, setting the batch size accordingly rather than fixing it a priori.
• Unsupervised and Optimization-based Approaches: Techniques such as K-means Batch Bayesian Optimization (KMBBO) (Groves et al., 2018) use clustering algorithms (e.g., K-means) to identify multiple peaks in the acquisition function and thus build batches that are representative and diverse, addressing the challenge of efficiently sampling high-dimensional or multimodal landscapes.

3. Batch Size Adaptation and Independence Criteria

Dynamic determination of batch size is essential for balancing statistical efficiency and computational throughput. In (Azimi et al., 2011), the independence criterion is formalized as an upper bound:

$E[|\Delta^*(\mu_z)|] \leq \epsilon$

This bound is computed by simulating outcomes at the current batch front-runner (via assigning a high value $M$ or a surrogate based on the current best observation) and propagating its effect through the GP posterior. Candidates whose mean and acquisition value do not change substantially are deemed “independent” and can safely be co-sampled.

This mathematically explicit criterion eliminates undue redundancy and ensures that batch selections remain near-optimal relative to purely sequential policies. The practical implication is that the batch size at each iteration varies depending on how rapidly the acquisition function decorrelates across the domain, which in turn is a function of the underlying kernel and data distribution.

4. Acquisition Function Adaptations

Most batch BO algorithms utilize standard acquisition functions (EI, UCB, KG), but these require adaptation to the batch setting:

• Simulated or Fantasized Outcomes: Candidates are evaluated with the acquisition function after “simulating” outcomes at other batch points, either by setting them to the posterior mean, maximum observed value, or a designated surrogate. This allows continued greedy maximization while tentatively accounting for the as-yet-unknown batch experiment results.
• Upper Bounding and Surrogacy: The dynamic batch algorithm (Azimi et al., 2011) computes an upper bound on the change of EI for subsequent candidates, ensuring that only points insensitive to previous pretend outcomes are batched. The EI function in particular maintains monotonicity in the mean prediction when variance is held fixed, supporting the use of deterministic simulation surrogates.
• Penalization and Exploration Control: Local penalization (González et al., 2015) modifies the acquisition function by multiplying it with penalizer terms derived from a GP-driven estimate of the function’s Lipschitz constant, directly enforcing non-redundancy in selected points.

5. Empirical Performance and Trade-offs

Extensive benchmarks (Azimi et al., 2011, Azimi et al., 2012) on both synthetic and real-world objectives (e.g., multi-dimensional Hartmann, Rosenbrock, Shekel functions; fuel cell and hydrogen experiments) demonstrate that dynamic or hybrid batch strategies typically achieve simple regret and cumulative regret comparable to, or only marginally exceeding, that of sequential methods. However, they can yield substantial acceleration in wall-clock time, running up to 18% (dynamic batch) or 78% (hybrid scheme) of all evaluations in parallel.

A table comparing typical features is as follows:

Method	Batch Size Adaptivity	Optimality vs. Sequential	Empirical Speedup
Fixed Batch (Standard)	No	Lower	Moderate
Dynamic Batch (Azimi et al., 2011)	Yes	Near-identical	6–18%
Hybrid Batch (Azimi et al., 2012)	Yes	Near-identical	up to 78%
Local Penalization (González et al., 2015)	No	Comparable/matched	Moderate/$n$-dependent

Key empirical findings include:

• Performance in terms of simple regret is generally preserved, with losses relative to the sequential oracle bounded and sometimes negligible.
• The realized batch size varies dynamically, sometimes being limited to a single point if high inter-dependence is detected.
• Strategies that avoid co-evaluating highly correlated points (via independence tests, local penalization, or IGMM) are able to avoid the performance degradation common to naïve fixed-size batch methods.

6. Practical Implementations and Real-world Implications

Batch BO is particularly relevant in domains such as hyperparameter tuning for machine learning, experimental science (e.g., genomics, chemistry), materials design, and robotics, where the bottleneck is often in conducting the experiments rather than in model computation. Batch strategies facilitate the use of parallel hardware and can substantially reduce total time-to-discovery or optimality, provided that the batch selection algorithm avoids superfluous or highly correlated samples.

Hybrid and dynamic batch methods (Azimi et al., 2012, Azimi et al., 2011) are advantageous in early-stage optimization when uncertainty is large and each evaluation greatly updates the surrogate; in this regime, sequential selection is naturally optimal. As the model becomes more confident and the posterior mean stabilizes, these methods smoothly transition to forming larger, more diverse batches, maximizing parallel efficiency.

Parameter tuning (such as the independence threshold $\epsilon$ in dynamic or hybrid batch strategies) governs the exploitation–exploration balance and requires consideration based on the application’s noise level, budget, and domain of interest.

7. Limitations and Future Directions

Several inherent challenges and open research directions are highlighted by the literature:

• Model Assumptions: Most existing adaptive batch strategies rely on GPs as surrogates. Extension to non-Gaussian surrogates, ensembles, or deep models remains limited and may lack the theoretical guarantees leveraged in GP-centered algorithms.
• High-dimensional Scaling: Scalability in very high-dimensional settings is not well characterized. While KMBBO with compressed sensing (Groves et al., 2018) and local penalization heuristics (González et al., 2015) offer some solutions, efficient diversification and uncertainty quantification for batch proposals in high dimensions is a persistent challenge.
• Exploration Parameter Sensitivity: The performance of batch methods can be highly sensitive to user-specified parameters like the independence threshold $\epsilon$, maximum batch size $n_b$, or the estimated Lipschitz constant $L$ in local penalization.
• Alternative Acquisition Functions: Most dynamic and hybrid batch methods target EI, with less work extending these frameworks to information-theoretic criteria, multi-objective settings, or alternative policies.
• Simulation and Surrogacy Effects: The accuracy of simulated or surrogate outcomes, especially in the early (uncertain) regime, can impact batch diversity and downstream model fidelity. Inaccurate surrogate outcomes may inadvertently restrict batch size (forcing near-sequential operation) or degrade optimization performance.

Advancing batch BO toward flexible, model-agnostic schemes with provable guarantees, improved robustness to surrogate inaccuracies, and scalability to large or structured parameter spaces remains a central focus for ongoing research.