Vanilla Bayesian Optimization Performs Great in High Dimensions

Abstract: High-dimensional problems have long been considered the Achilles' heel of Bayesian optimization algorithms. Spurred by the curse of dimensionality, a large collection of algorithms aim to make it more performant in this setting, commonly by imposing various simplifying assumptions on the objective. In this paper, we identify the degeneracies that make vanilla Bayesian optimization poorly suited to high-dimensional tasks, and further show how existing algorithms address these degeneracies through the lens of lowering the model complexity. Moreover, we propose an enhancement to the prior assumptions that are typical to vanilla Bayesian optimization algorithms, which reduces the complexity to manageable levels without imposing structural restrictions on the objective. Our modification - a simple scaling of the Gaussian process lengthscale prior with the dimensionality - reveals that standard Bayesian optimization works drastically better than previously thought in high dimensions, clearly outperforming existing state-of-the-art algorithms on multiple commonly considered real-world high-dimensional tasks.

• The paper reveals that adjusting Gaussian process lengthscale priors effectively mitigates high-dimensional complexity in optimization tasks.
• The methodology consistently outperforms specialized high-dimensional methods across benchmarks including synthetic functions, SVM tuning, and MuJoCo simulations.
• The study challenges traditional BO assumptions by offering a simpler, more adaptable approach that reduces model complexity without restrictive objective function assumptions.

Insights on "Vanilla Bayesian Optimization Performs Great in High Dimensions"

The research paper "Vanilla Bayesian Optimization Performs Great in High Dimensions" by Carl Hvarfner, Erik O. Hellsten, and Luigi Nardi presents a significant investigation into the performance of Bayesian Optimization (BO) in high-dimensional spaces. Traditionally, high-dimensional optimization problems have been deemed a challenging domain for Bayesian optimization, largely due to the curse of dimensionality. The study addresses these challenges by questioning the complexity assumptions conventionally held about objective functions in BO and proposing a methodological refinement that enhances the performance of vanilla BO in high-dimensional scenarios.

Core Contributions

The paper's central thesis is that the perceived shortcomings of vanilla BO in high-dimensional settings are predominantly driven by overly complex assumptions concerning the objective function. To this end, the authors introduce a new perspective by reevaluating the complexity assumptions traditionally held in the domain. This is complemented by a methodological contribution: the scaling of Gaussian process lengthscale priors proportional to dimensionality, which modulates the model complexity effectively without imposing restrictive assumptions on the objective's structure.

Key contributions include:

1. Complexity Reduction Clarification: The paper demystifies the relationship between dimensionality and model complexity, underscoring how common high-dimensional BO strategies alleviate this complexity through various structural assumptions.
2. Proposition of a Refined BO Approach: The researchers propose a modification to the vanilla BO algorithm that simplifies the lengthscale prior of the GP kernel, achieving effective high-dimensional performance.
3. Empirical Validation: Robust validation across a spectrum of dimensionalities has demonstrated that vanilla BO, when adjusted as suggested, significantly surpasses state-of-the-art HDBO methods over an extensive range of real-world problems.

Theoretical and Experimental Findings

The authors explore the relationship between Bayesian optimization's model complexity and the curse of dimensionality, providing a theoretical backdrop that supports their empirical findings. They introduce the concept of Maximal Information Gain (MIG) as a measure of assumed complexity and analyze the pitfalls of high-complexity assumptions in traditional BO approaches. Their proof shows that uninformed EI (Expected Improvement) does not exhibit exploratory behavior typically perceived at the boundaries, challenging earlier studies like those of Swersky (2017).

Empirically, the modified vanilla BO consistently outperforms specialized high-dimensional methods across various benchmark tasks, including standard synthetic functions and high-dimensional real-world tasks such as MOPTA08, SVM-based hyperparameter tuning, and MuJoCo simulations. These results are remarkable since they illustrate the potential for simplified models to perform efficiently even under high-dimensional pressures.

Implications and Future Directions

The implications of this research are manifold. Practically, this work offers a more straightforward and computationally inexpensive alternative to complex HDBO algorithms, which often rely on specific assumptions that may not always align with the problem's nature. By conjecturing that the impediment of dimensionality can be alleviated through suitable prior adjustments, the authors open the door to simpler and potentially more adaptable approaches in BO applications.

Theoretically, this paper contributes to a refined understanding of the interplay between model complexity and dimensionality in BO. It encourages further exploration into Bayesian models that balance complexity optimally without stringent assumptions on the function structure. Future research could explore latent space models or deep kernel learning frameworks that incorporate this complexity consideration more intuitively.

The study challenges conventional practices in BO, urging the community to reassess established notions of complexity and dimensionality, ultimately advancing the discourse towards more effective optimization techniques that stand robust in the face of high-dimensional challenges.