Rethinking Trust Region Bayesian Optimization in High Dimensions

Abstract: Trust Region Bayesian Optimization (TuRBO) is an effective strategy for alleviating the curse of dimensionality in high-dimensional black-box optimization. However, inappropriate lengthscale design can cause the local Gaussian process (GP) model within the trust region to degenerate, leading to suboptimal performance in high dimensions. In this work, we show that TuRBO's local GP may remain either excessively complex or overly simple as the dimension $D$ and trust region side length $L$ vary. To address this issue, we propose a straightforward variant, AdaScale-TuRBO, which scales the GP lengthscale with both the problem dimension and trust region size, thereby preserving kernel geometry and maintaining consistent prior complexity. Empirically, we show that AdaScale-TuRBO can robustly outperform standard TuRBO and other popular high-dimensional BO methods on synthetic benchmarks and real-world trajectory planning tasks.

• The paper introduces a scaling law (ℓ ∝ L√D) that calibrates GP complexity for effective high-dimensional trust region optimization.
• It develops AdaScale-TuRBO, employing a TR- and dimension-aware LogNormal prior with MAP estimation to stabilize surrogate modeling.
• Empirical evidence demonstrates significant improvements in sample efficiency and performance on both synthetic benchmarks and real-world tasks.

Trust Region Calibration in High-Dimensional Bayesian Optimization

Motivation and Failure Modes of Local Gaussian Processes

Trust Region Bayesian Optimization (TuRBO) constitutes a paradigm for high-dimensional black-box optimization, leveraging local surrogate modeling by constraining optimization to adaptive trust regions. The canonical TuRBO workflow involves fitting a local Gaussian process (GP) model inside each trust region and optimizing an acquisition function to propose new evaluation points. This spatial restriction mitigates some global high-dimensional pathologies but does not automatically guarantee a well-calibrated complexity assumption in the local surrogate model.

The authors identify that, in high-dimensional settings, improper GP lengthscale selection is a dominant failure mode that persists even when restricting to local trust regions. Despite using adaptive regions to limit the search space, the canonical practice—e.g., maintaining a fixed or globally scaled lengthscale—results in a local GP that is either overly complex (near-independence) or overly simple (excessively smooth), depending on the scaling with both the ambient dimension $D$ and trust region size $L$. The authors diagnose this pathology using the Maximum Information Gain (MIG), which measures the information a GP prior can extract from $N$ observations. Specifically, when pairwise distances in the trust region still scale as $\Theta(L\sqrt{D})$, lengthscales that do not compensate for this geometric scaling yield a local kernel matrix near the identity, resulting in near-independence and poor generalization (Figure 1).

Figure 1: MIG scaling as a function of number of observations $N$, for various $D$ and trust region sizes $L$. When $L$ is large or $D$ is high and lengthscale is not properly scaled, MIG approaches the independent-kernel regime.

Theoretical Analysis and Regularization Approach

The core contribution is a proven lengthscale scaling law that ensures the invariance of the GP prior complexity under scaling of both the domain and dimensionality. By formal analysis, for a trust region $\mathcal{X}_L = [0, L]^D$, typical pairwise distances between points scale as $L$0, so to preserve kernel geometry (and thus maintain consistent function class complexity across instantiations), the lengthscale should scale as $L$1. The authors formalize this via:

$L$2

and prove that this scaling ensures the MIG—and thus the "learning capacity" of the local GP—is invariant to both $L$3 and $L$4 when performing domain or region scaling. This property is crucial for avoiding both underfitting (over-smoothed priors) and overfitting (independent kernels) as dimensions and trust region sizes vary.

To implement this prescription in practice, the authors propose a LogNormal prior for the lengthscales, parameterized to have its mode at $L$5, and recommend maximum a posteriori (MAP) hyperparameter estimation instead of the brittle maximum likelihood estimation commonly used. This addresses the over-parameterization and regularization deficiencies of MLE in high dimensions.

AdaScale-TuRBO: Algorithmic Realization

AdaScale-TuRBO is the resulting trust region Bayesian optimization variant. It augments TuRBO by integrating:

• TR- and dimension-aware LogNormal prior over lengthscales
• MAP fitting of GP hyperparameters at each iteration to stabilize training and enforce complexity constraints

AdaScale-TuRBO thus achieves a robust inductive bias adaptation as optimization focuses on progressively smaller local regions (with dynamically reduced $L$6) in high dimensions.

Empirical Performance and Ablations

Extensive empirical validation is performed on both synthetic high-dimensional benchmarks (Schwefel, Rastrigin, Michalewicz) for $L$7, and on a high-dimensional rover trajectory planning task ($L$8). In all cases, AdaScale-TuRBO demonstrates strict improvement over canonical TuRBO, as well as over vanilla BO with dimension-scaled priors and recent high-dimensional approaches. Critically, D-scaled TuRBO—enforcing lengthscales scaled only with $L$9, but not with $N$0—performs similarly or worse than canonical TuRBO, highlighting the necessity of joint trust region- and dimension-aware scaling.

Figure 2: Best observed function value vs. evaluation budget for synthetic benchmarks ($N$1). AdaScale-TuRBO dominates all baselines including TuRBO, D-scaled TuRBO, and Linear BO.

Figure 3: Best observed value for a real-world 60D rover trajectory planning task. AdaScale-TuRBO achieves the highest quality solutions throughout.

Results on GP-prior generated functions show that AdaScale-TuRBO achieves superior sample efficiency across a spectrum of data-generating smoothness levels, with the largest gains observed for functions with shorter lengthscales (greater local variability).

Figure 4: Objective value vs. evaluations on GP-sampled functions for varied true lengthscales. AdaScale-TuRBO maintains robust performance even as the true target functions become less smooth.

Ablations over the initial trust region side length $N$2 (Figure 5) confirm the robustness of the method: gains over TuRBO are not sensitive to this hyperparameter. Violin plots of final performance distributions across 10 replicates on all tasks (Figures 6–8) consistently show that AdaScale-TuRBO obtains lower median and tighter interquartile ranges for best objective values, underscoring its stability and effectiveness.

Practical and Theoretical Implications

This work provides a rigorous prescription for complexity calibration in local Bayesian optimization. Theoretically, it pinpoints the precise reason why canonical trust region methods do not automatically resolve the curse of dimensionality: geometric scaling of pairwise distances requires a matching scaling in kernel bandwidth to retain meaningful priors. The explicit $N$3 scaling delivers consistency across all considered regimes.

From a practical standpoint, the AdaScale-TuRBO prescription enables robust, self-tuning surrogates that maintain sample efficiency as problem difficulty increases. This approach is complementary to structure-exploiting BO methods (e.g., additive or low-rank assumptions) and could be combined orthogonally with those strategies to further improve high-dimensional search.

Future Outlook

Potential extensions include integrating structure learning (e.g., automatic relevance determination, sparsity-inducing priors) with AdaScale-TuRBO's trust region calibration, leveraging the described scaling laws for other surrogate classes (kernel regression, Bayesian neural networks), and further systematic study of lengthscale adaptation in more general black-box and multi-fidelity settings. The stabilization of surrogate complexity in local regions provides foundational tools for scalable BO in applied domains such as hyperparameter tuning, materials discovery, and trajectory design.

Conclusion

This work establishes the necessity of joint dimension- and trust-region-aware lengthscale scaling for effective surrogate modeling in high-dimensional Bayesian optimization. AdaScale-TuRBO, by regularizing the local GP prior via principled scaling and MAP fitting, systematically eliminates the major failure mode of existing trust region-based BO methods and achieves state-of-the-art performance across challenging benchmarks (2604.22967). The analytical framework and resulting methodology represent a substantial advancement for scalable black-box optimization.