BOIDS: High-dimensional Bayesian Optimization via Incumbent-guided Direction Lines and Subspace Embeddings (2412.12918v1)

Abstract: When it comes to expensive black-box optimization problems, Bayesian Optimization (BO) is a well-known and powerful solution. Many real-world applications involve a large number of dimensions, hence scaling BO to high dimension is of much interest. However, state-of-the-art high-dimensional BO methods still suffer from the curse of dimensionality, highlighting the need for further improvements. In this work, we introduce BOIDS, a novel high-dimensional BO algorithm that guides optimization by a sequence of one-dimensional direction lines using a novel tailored line-based optimization procedure. To improve the efficiency, we also propose an adaptive selection technique to identify most optimal lines for each round of line-based optimization. Additionally, we incorporate a subspace embedding technique for better scaling to high-dimensional spaces. We further provide theoretical analysis of our proposed method to analyze its convergence property. Our extensive experimental results show that BOIDS outperforms state-of-the-art baselines on various synthetic and real-world benchmark problems.

• The paper introduces BOIDS, a novel Bayesian Optimization method for high-dimensional spaces combining incumbent-guided direction lines, adaptive line selection via multi-armed bandit, and subspace embedding.
• BOIDS leverages incumbent information from global and personal bests to guide the search directions, unlike methods using purely random directions.
• Experiments on synthetic and real-world problems show that BOIDS outperforms state-of-the-art methods for high-dimensional optimization, supported by theoretical convergence analysis.

The paper introduces a novel high-dimensional Bayesian Optimization (BO) algorithm called High-dimensional \underline{Bayesian \underline{O}ptimization via \underline{I}ncumbent-guided \underline{D}irection Lines and \underline{S}ubspace Embeddings} (BOIDS). BOIDS addresses the challenges of optimizing expensive black-box functions in high-dimensional spaces by integrating incumbent-guided direction lines, adaptive line selection using a multi-armed bandit technique, and subspace embedding.

Key components and contributions:

• Incumbent-Guided Direction Lines: BOIDS uses incumbents, specifically global and personal incumbents inspired by Particle Swarm Optimization (PSO), to guide the search process. The incumbent-guided direction $\mathbf{v}^{(t+1)}_i$ for the $i$-th data point $\mathbf{x}^{(t)}_i$ is defined as:

  $\mathbf{v}^{(t+1)}_i = w\bar{\mathbf{x}^{(t)}_i + \mathbf{r}_1 c_1 \bar{\mathbf{p}_i^{(t)} + \mathbf{r}_2 c_2 \bar{\mathbf{g}^{(t)}$

  where:

  • $w$, $c_1$, and $c_2$ are coefficients controlling exploration and exploitation.
  • $\mathbf{r}_1$ and $\mathbf{r}_2$ are uniformly random vectors sampled from $\mathcal{U}([0,1]^d)$.
  • $\bar{\mathbf{x}^{(t)}_i}$ is the displacement vector of the $i$-th point $\mathbf{x}^{(t)}_i$ relative to its last update.
  • $\bar{\mathbf{p}_i^{(t)}}$ is the personal best direction of the $i$-th point $\mathbf{x}^{(t)}_i$.
  • $\bar{\mathbf{g}^{(t)}}$ is the global best direction.

  This approach contrasts with LineBO, which uses uniformly random directions, by exploiting information from the incumbents to guide the search towards promising regions.

• Adaptive Line Selection: The algorithm uses a multi-armed bandit (MAB) approach with Thompson Sampling (TS) to select the most optimal line from a set of $m$ incumbent-guided lines. The reward $r_i$ of line $\mathcal{L}_i$ is defined as the maximum TS values among all data points on $\mathcal{L}_i$, i.e., $$r_i=\max_{\mathbf{x}\in\mathcal{L}_i}{g(\mathbf{x})}$$, where $g$ is a random realization from the Gaussian Process (GP) posterior. The line $\mathcal{L}_{i^*}$ is selected by maximizing the following equation:

  $i^* = \argmax_{i=1,\dots,m}{\max_{\mathbf{x}\in \mathcal{L}_i}{g(\mathbf{x})} \quad \text{where} \quad g \sim \mathcal{GP}(\mathcal{D})$

• Incumbent-Guided Line-Based Optimization: BOIDS optimizes an acquisition function $\alpha(.)$ (e.g., EI, TS) in the entire search space $\mathcal{X}$ while imposing Euclidean distance constraints to the personal incumbent of $\hat{\mathbf{x}_{i^*}^{(t)}$ and the current global incumbents: $L_{\mathbf{p}(\mathbf{x}) = \Vert\mathbf{x}- \mathbf{p}_{i^*}^{(t)}\Vert$ and $L_{\mathbf{g}(\mathbf{x}) = \Vert\mathbf{x} - \mathbf{g}^{(t)}\Vert$. This is formulated as a multi-objective (MO) optimization problem:

  $\mathcal{P} = \argmax_{\mathbf{x}\in\mathcal{X}{ \big( f_\alpha(\mathbf{x}), f_{\mathbf{p}(\mathbf{x}), f_{\mathbf{g}(\mathbf{x}) \big)}$

  where $f_\alpha(.) = \alpha(.)$, $f_\mathbf{p}(.)=-L_\mathbf{p}(.)$ and $f_\mathbf{g}(.)=-L_\mathbf{g}(.)$. The solution with the best acquisition objective on the Pareto front is selected as the next data point.

• Subspace Embedding: To further improve scalability, BOIDS incorporates a linear random subspace embedding strategy, specifically the adaptive expanding subspace embedding known as BAxUS. This reduces the dimensionality of the optimization problem by projecting data points from a low-dimensional subspace $\mathcal{A}$ to the original input space $\mathcal{X}$ via a projection matrix $\mathbf{S}$.
• Theoretical Analysis: The paper provides theoretical analysis of BOIDS, including local and global convergence properties. It derives a sub-linear simple regret bound for the incumbent-guided line-based optimization, showing convergence to a local optimum. The simple regret $r_t$ is bounded as follows:

  $$\mathbb{E}[r_T] \le \mathcal{O}((d \log{T}/ T)^{1/2-\kappa} )$$

  where $d$ is the input dimension and $\kappa$ depends on the smoothness of the kernel. The global convergence property is analyzed using the probability of the BAxUS embedding to contain the global optimum.

• Experimental Results: The proposed method is evaluated against state-of-the-art baselines, including standard BO, LineBO, BAxUS, HESBO, ALEBO, SAASBO, TuRBO, CMA-BO, RDUCB, CMA-ES and PSO, on a range of synthetic and real-world benchmark problems with dimensions ranging from 100 to 500. The results demonstrate that BOIDS outperforms these baselines on most problems. Ablation studies are conducted to assess the contribution of each component of BOIDS. Removing the incumbent-guided line component and the tailored line-based optimization significantly degrades performance.

In summary, BOIDS presents a method for high-dimensional black-box optimization that combines incumbent-guided search directions with multi-objective acquisition optimization and subspace embedding. The theoretical analysis and empirical results support the effectiveness of the proposed approach.