Local Enumeration Bayesian Optimization

• LE-BO is a framework that applies Bayesian optimization to a local-search defined function, preserving the global optima of the original objective.
• It transforms the optimization landscape by mapping starting points to their local minima, thereby implicitly enumerating basins of attraction.
• The method demonstrates efficient performance on multimodal benchmarks and logistic regression tasks, though it depends on reliable local search and faces high-dimensional challenges.

Searching arXiv for the specified paper to ground the article in the published record. arxiv.search({"query":"id:(Gao et al., 2019)","max_results":5,"sort_by":"relevance"}) arxiv.search returned 1 result:

• (Gao et al., 2019) — "Bayesian optimization with local search" Local Enumeration Bayesian Optimization (LE-BO) is a perspective on global optimization in which Bayesian optimization is applied not directly to the original objective $f$, but to a transformed function induced by local search. In the formulation introduced as “Bayesian Optimization with Local Search (BOwLS),” each queried starting point is passed to a local optimizer, and the returned local minimum value is treated as the observation for the Bayesian model. Under the paper’s stated conditions, this local-search-defined function attains the same global optima as the original objective, so minimizing the transformed function is equivalent to identifying the basin of attraction containing a global minimizer (Gao et al., 2019).

1. Problem setting and nomenclature

LE-BO addresses the bound-constrained global optimization problem

$\min_{x \in \Omega} f(x),$

where $\Omega \subset \mathbb{R}^n$ is compact. The formulation explicitly targets multimodal objectives that admit multiple local minima and assumes that local optimization can be performed efficiently. No specific smoothness or Lipschitz assumptions are imposed on $f$. The method is intended for black-box objectives, with the additional possibility that gradients may be available and exploited by the local solver (Gao et al., 2019).

The underlying paper names the method “Bayesian Optimization with Local Search (BOwLS)” and states that it can be understood as constructing a new function by conducting local searches of the original objective function, where the new function attains the same global optima as the original one. Within this interpretation, Local Enumeration Bayesian Optimization is the viewpoint that Bayesian optimization drives the enumeration of local basins through its choice of starting points, while the surrogate models the basin-induced objective $F_{\mathcal{L}}$.

The term “enumeration” refers to the fact that each starting point is mapped to the value of the local minimum reached from its basin of attraction. This does not enumerate local minima by explicit combinatorial listing; rather, it enumerates them implicitly through repeated local searches initiated at Bayesian-optimized starting locations. A plausible implication is that LE-BO is best viewed as a basin-selection strategy rather than as direct global modeling of the raw objective.

2. Local-search-defined objective and preservation of global minima

The central object in LE-BO is a local search operator $\mathcal{L}$. For a starting point $x$, the local optimizer is applied to $f$ to obtain a local minimum in the basin of attraction containing $x$: $x^* = \mathcal{L}(f(\cdot), x).$

The formulation requires that for any initial point $\min_{x \in \Omega} f(x),$0, the local solver $\min_{x \in \Omega} f(x),$1 returns a unique local minimum $\min_{x \in \Omega} f(x),$2. This induces a partition of $\min_{x \in \Omega} f(x),$3 into basins of attraction with a deterministic mapping $\min_{x \in \Omega} f(x),$4. Using this mapping, the transformed function is defined by

$\min_{x \in \Omega} f(x),$5

If $\min_{x \in \Omega} f(x),$6 has a finite number of local minima, then $\min_{x \in \Omega} f(x),$7 is discrete-valued, taking values from the set $\min_{x \in \Omega} f(x),$8, where $\min_{x \in \Omega} f(x),$9 are the local minimizers. In this sense, all points in the same basin collapse to the same function value. This is the mechanism by which LE-BO replaces direct exploration of the original landscape with exploration over basin labels encoded by local minimum values (Gao et al., 2019).

Under the stated conditions that $\Omega \subset \mathbb{R}^n$0 deterministically maps every $\Omega \subset \mathbb{R}^n$1 to a unique local minimum and that $\Omega \subset \mathbb{R}^n$2 has a finite set of local minima $\Omega \subset \mathbb{R}^n$3, the transformed function preserves global minima: $\Omega \subset \mathbb{R}^n$4 The paper’s argument is straightforward. Since $\Omega \subset \mathbb{R}^n$5 always equals $\Omega \subset \mathbb{R}^n$6 for some local minimizer $\Omega \subset \mathbb{R}^n$7, minimizing $\Omega \subset \mathbb{R}^n$8 amounts to choosing the basin whose associated local minimum value is smallest. If $\Omega \subset \mathbb{R}^n$9 is a global minimizer of $f$0, then any point in its basin satisfies $f$1, while no value of $f$2 can be smaller than the minimum over the set of local minima. Hence $f$3, and the argmin set of $f$4 corresponds to the basins of global minima.

The paper also states caveats. If $f$5 is stochastic or non-unique, or if $f$6 has flat regions with non-isolated minima, additional care is needed. These caveats are fundamental because the equivalence between the original and transformed objectives is the core rationale for LE-BO.

3. Bayesian optimization on the basin-induced function

In LE-BO, Bayesian optimization is applied to $f$7, not to $f$8. The surrogate $f$9 is therefore a probabilistic model of the local-search-defined function. The next starting point is selected by optimizing an acquisition function: $F_{\mathcal{L}}$0

The probabilistic model is Gaussian process (GP) regression. Given data

$F_{\mathcal{L}}$1

a GP prior with mean $F_{\mathcal{L}}$2 and covariance kernel $F_{\mathcal{L}}$3, and i.i.d. observation noise of variance $F_{\mathcal{L}}$4, the posterior at a test point $F_{\mathcal{L}}$5 is Gaussian with mean and variance

$F_{\mathcal{L}}$6

$F_{\mathcal{L}}$7

where $F_{\mathcal{L}}$8, $F_{\mathcal{L}}$9, $\mathcal{L}$0, and $\mathcal{L}$1. The paper does not fix a specific kernel; squared exponential, Matérn, and other standard choices are stated to be compatible (Gao et al., 2019).

The observations used to train the GP are transformed outputs rather than raw function values: $\mathcal{L}$2 This is the defining modeling choice of LE-BO. The surrogate is not asked to interpolate the original multimodal landscape directly; it instead models the mapping from start points to attained local minimum values.

The paper mentions standard acquisition functions without fixing one. For minimization, the canonical forms given are

$\mathcal{L}$3

and

$\mathcal{L}$4

where $\mathcal{L}$5, $\mathcal{L}$6 and $\mathcal{L}$7 are the standard normal CDF and PDF, and $\mathcal{L}$8 controls the exploration–exploitation trade-off. The paper also mentions Probability of Improvement.

Because $\mathcal{L}$9 is induced by basins, maximizing the acquisition over $x$0 implicitly enumerates basins. Regions with high posterior uncertainty $x$1 correspond to unexplored or poorly characterized basin assignments, while regions with low posterior mean $x$2 correspond to basins likely to contain low local minimum values. This operationally links local optimization with the usual Bayesian optimization exploration–exploitation mechanism.

4. Algorithmic structure and implementation

The BOwLS procedure directly instantiates LE-BO through BO-driven selection of starting points, local-search-based enumeration, and GP modeling of the transformed function. The algorithmic steps stated in the paper are:

1. Initialize $x$3.
2. For $x$4 in the initial design: a. Select $x$5 in $x$6 such as random or space-filling. b. Run local search: $x$7. c. Set $x$8; augment $x$9.
3. Fit GP model $f$0 to $f$1.
4. For $f$2 until stopping: a. Select $f$3. b. Run local search: $f$4. c. Set $f$5; augment $f$6. d. Update GP to $f$7.
5. Return $f$8 as the estimated global minimum of $f$9 (Gao et al., 2019).

The initial phase uses $x$0 starting points chosen randomly or by space-filling design. The paper notes that larger $x$1 can stabilize the initial GP. It also states that one local search is performed per BO iteration in the sequential setting, while multiple starting points per iteration can be accommodated through batch acquisitions.

For the local optimizer $x$2, the experiments use conjugate gradient from SciPy. The paper lists alternatives including L-BFGS, trust-region, and Nelder–Mead, depending on gradient availability and problem characteristics. Tolerance and maximum iterations are to be set so as to ensure reliable convergence to a local minimum, with determinism preferred because the theoretical equivalence between $x$3 and $x$4 relies on uniqueness.

The paper also discusses several practical choices. Hyperparameters of the GP kernel can be trained by maximum likelihood or MAP. Observation noise $x$5 can be used to model stochasticity in $x$6 when local search is noisy and set small when local search is deterministic. Bound constraints are handled by projecting acquisition optimization into $x$7, and local search respects the same bounds. The practical guidance given includes choosing $x$8 in the range $x$9–$x^* = \mathcal{L}(f(\cdot), x).$0 based on dimension and budget, using EI in noiseless settings, and using UCB with an exploration schedule in noisy or discrete-valued $x^* = \mathcal{L}(f(\cdot), x).$1 cases.

5. Theoretical properties and computational profile

The principal theoretical property of LE-BO is preservation of the global optimum under deterministic local search and finitely many local minima. This establishes the legitimacy of replacing the original objective with the local-search-defined function. Beyond this equivalence, however, the paper does not provide formal convergence rates or guarantees for BOwLS. It notes that standard asymptotic consistency results for Bayesian optimization depend on assumptions on $x^* = \mathcal{L}(f(\cdot), x).$2, the kernel, and acquisition optimization, and do not directly cover the discrete-valued, basin-induced function $x^* = \mathcal{L}(f(\cdot), x).$3 (Gao et al., 2019).

This absence of direct BO theory for $x^* = \mathcal{L}(f(\cdot), x).$4 is significant because the transformed function may be piecewise-constant with sharp transitions at basin boundaries. Such structure is atypical for standard GP surrogates. A plausible implication is that LE-BO derives much of its practical behavior from the surrogate’s ability to rank or separate basins rather than from accurate interpolation of a smooth latent surface.

The paper gives a per-iteration computational decomposition. GP updating costs $x^* = \mathcal{L}(f(\cdot), x).$5 in the number of data points $x^* = \mathcal{L}(f(\cdot), x).$6 because of kernel matrix inversion. It suggests sparse GP approximations or incremental Cholesky updates as mitigations. Acquisition optimization requires global maximization over $x^* = \mathcal{L}(f(\cdot), x).$7; this is described as typically a small overhead compared to local search in expensive problems, but nontrivial in high dimensions. The cost of local search is problem-dependent and often dominant when gradients are available.

Scalability with dimension is identified as a limitation inherited from Bayesian optimization. The paper notes BO’s known difficulty in high dimensions and recommends dimensionality reduction methods such as random embeddings and active subspaces to extend BO, and therefore LE-BO, to higher-dimensional settings. It also mentions additive kernels and trust-region BO variants as possible extensions for stabilizing GP fits in high dimension.

Several failure modes are stated explicitly. LE-BO may miss the global optimum if local search fails to converge reliably, returns non-unique minima, or if the global minimum lies in a negligibly small basin that BO never explores. If local search outputs are noisy, the GP likelihood becomes more complicated and noisy BO settings may be required.

6. Empirical behavior, comparisons, and application to logistic regression

The experimental evaluation uses six standard test functions from the Global Optimization benchmark suite: Price 2D, Branin, Cosine-mixture 4D, Trid 6D, Hartmann 6D, and Ackley 2D/4D. Local search is conjugate gradient from SciPy. The cost metric is the combined number of function and gradient evaluations, and the stopping criterion on the test functions is a total budget capped at $x^* = \mathcal{L}(f(\cdot), x).$8 combined evaluations. Because of algorithmic randomness, each experiment is repeated $x^* = \mathcal{L}(f(\cdot), x).$9 times. The methods compared are MLSL (Multi-Level Single Linkage), EMS (Efficient Multi-Start), and BOwLS (Gao et al., 2019).

For the first five test functions, excluding Ackley, all three methods reach the global optimum within the budget, and BOwLS requires the least computational cost in four of the five cases, all except Price. On Ackley in 2D and 4D, some trials fail to reach the global optimum within the budget, but BOwLS outperforms EMS and MLSL in both dimensions, with a more pronounced advantage in 4D. The paper interprets this as indicating robustness on highly multimodal functions.

The paper also includes a logistic regression case study on the Pima Indians Diabetes dataset with $\min_{x \in \Omega} f(x),$00 predictors and $\min_{x \in \Omega} f(x),$01 instances, split into $\min_{x \in \Omega} f(x),$02 train and $\min_{x \in \Omega} f(x),$03 test examples. The objective is minimization of the logistic loss over parameters $\min_{x \in \Omega} f(x),$04, with

$\min_{x \in \Omega} f(x),$05

$\min_{x \in \Omega} f(x),$06

and

$\min_{x \in \Omega} f(x),$07

This experiment is repeated $\min_{x \in \Omega} f(x),$08 times. BOwLS achieves the best performance in terms of minimal loss versus combined evaluations, while EMS performs better than MLSL. The paper further reports that prediction accuracy improves across BO iterations, measured by the number of local-search runs, illustrating that better local minima, identified with the global optimum in the paper’s wording, yield superior predictive performance.

These results position LE-BO as a hybrid between global model-based search and classical multi-start local optimization. The factual empirical claim is not that BOwLS eliminates the difficulty of multimodality, but that under the stated budgets it often reduces computational cost relative to MLSL and EMS on the tested problems.

7. Limitations, misconceptions, and extensions

A common misconception would be to treat LE-BO as ordinary Bayesian optimization applied to the original objective. The method differs fundamentally in its target of inference: the GP is fit to transformed outputs $\min_{x \in \Omega} f(x),$09, not to raw values $\min_{x \in \Omega} f(x),$10. Another misconception would be to interpret the method as guaranteeing convergence because BO is used. The paper explicitly does not provide formal convergence rates or guarantees for BOwLS, and standard BO guarantees do not directly apply to the basin-induced transformed function (Gao et al., 2019).

The method’s limitations are clearly identified. High-dimensional scalability remains difficult because GP modeling and acquisition optimization degrade with dimension. Dependence on reliable local search is structural: if $\min_{x \in \Omega} f(x),$11 fails or returns inconsistent minima, then $\min_{x \in \Omega} f(x),$12 becomes noisy and the uniqueness assumption breaks. The transformed objective may be discrete-valued or piecewise-constant with sharp basin boundaries, which complicates GP modeling.

The extension directions listed in the paper are correspondingly pragmatic. For high-dimensional BO, it suggests random embeddings, active subspaces, and additive kernels. For noisy or stochastic local search, it suggests modeling observation noise with $\min_{x \in \Omega} f(x),$13, using noise-aware acquisitions such as noisy EI, or replicating local searches to reduce variance. It also mentions batch or parallel LE-BO, in which multiple starts are selected per iteration, integration with SGD-based local solvers for large-scale learning objectives, and trust-region BO variants that restrict acquisition optimization to trust regions.

Taken together, these points characterize LE-BO as a basin-oriented reformulation of multimodal global optimization. Its defining idea is to let local search collapse the original landscape into a set of local-minimum values and then let Bayesian optimization allocate search effort over starting points whose basins are uncertain or promising. This suggests a division of labor in which local search handles within-basin descent and Bayesian optimization handles across-basin allocation.