Bayesian Optimization and Hyperband (BOHB)

• BOHB is a multi-fidelity hyperparameter optimization algorithm that combines Bayesian methods with Hyperband’s aggressive early-stopping to allocate resources efficiently.
• It utilizes a TPE-based surrogate model and successive halving to focus evaluations on promising configurations, achieving rapid convergence and lower regret.
• Recent extensions like FlexHB enhance BOHB with finer fidelity modeling and adaptive ranking strategies, further improving scalability and performance in ML tuning tasks.

Bayesian Optimization and Hyperband (BOHB) is a multi-fidelity hyperparameter optimization algorithm that integrates Bayesian optimization (BO) with the aggressive early-stopping and budget allocation strategies of Hyperband (HB). BOHB and its derivatives have become the standard in scalable hyperparameter optimization for machine learning and deep learning, efficiently balancing exploration and exploitation under strict resource constraints. The algorithm uses model-based guidance to focus evaluations on promising configurations while leveraging multi-fidelity scheduling to avoid excessive expenditure on suboptimal candidates. Recent advancements, such as FlexHB, further enhance BOHB’s flexibility and efficiency.

1. Hyperband and Bayesian Optimization: Foundational Principles

Hyperband is a multi-armed bandit algorithm for allocating a finite computational budget among a large set of hyperparameter configurations. It operates by running multiple “brackets” of Successive Halving (SH), where each bracket varies the number of initial configurations and the minimum resource allocated per configuration. At each round in SH, only the top-performing $1/\eta$ fraction of configurations is promoted to the next rung, where they receive an increased budget, until a single configuration remains. This allows rapid discarding of poor configurations using minimal resources and focuses subsequent resources on more promising candidates (Wang et al., 2018, Falkner et al., 2018).

Bayesian Optimization (BO) is a model-based approach for optimizing black-box, expensive-to-evaluate functions. BO uses a surrogate model (commonly a Gaussian process, random forest, or the Tree-structured Parzen Estimator—TPE) to approximate the objective and selects the next configuration by maximizing an acquisition function such as expected improvement (EI) or upper confidence bound (UCB). BO is highly data-efficient but traditionally applies only to single-fidelity, full-budget scenarios, leading to high resource consumption per iteration in deep learning settings (Wang et al., 2018, Falkner et al., 2018).

2. The BOHB Algorithm: Core Components and Pseudocode

BOHB combines the strengths of Hyperband’s early-stopping and multi-fidelity allocation with BO’s model-based guidance (Falkner et al., 2018, Lindauer et al., 2019). It operates as follows:

• Surrogate Model: BOHB uses the TPE approach, employing two kernel density estimators (KDEs) $\ell(x|b)$ and $g(x|b)$ that model the densities over the “good” and “bad” fraction of observed configurations at each fidelity $b$. Candidates are proposed by maximizing the ratio $\ell(x|b)/g(x|b)$, which approximates expected improvement.
• Sampling Mix: Each proposal step samples a fixed fraction $\rho$ of candidates uniformly (e.g., $\rho=0.2$) to guarantee exploration and theoretical convergence.
• Integration with Hyperband: BOHB uses the Hyperband bracket and scheduling structure. For each bracket, initial configurations are drawn with BO-guided sampling, then SH is run as in standard HB.

A simplified outline of BOHB’s main loop:

$g(x|b)$6

This coupled loop yields rapid anytime performance (from SH) and strong final regret (from BO guidance), with low overhead due to efficient sampling and KDE model fitting (Falkner et al., 2018, Lindauer et al., 2019).

3. Surrogate, Acquisition, and Multi-Fidelity Aspects

The surrogate in BOHB is a nonparametric model—most commonly a multivariate KDE, although Gaussian processes and random forests are also supported. Given observation history at fidelity $b$, BOHB splits data into the best $q$-quantile and the remainder, fitting KDEs $\ell(x)$ and $\ell(x|b)$0 as follows (Falkner et al., 2018, Lindauer et al., 2019):

• $\ell(x|b)$1: KDE over the $\ell(x|b)$2-best observed configurations at $\ell(x|b)$3
• $\ell(x|b)$4: KDE over all others
• Ratio $\ell(x|b)$5 is maximized among a candidate pool from $\ell(x|b)$6 (with bandwidth inflation for exploration)

BOHB supports multiple fidelities by maintaining KDEs at each budget and proposing at the highest observed fidelity with sufficient data, leveraging the correlation between low-fidelity and high-fidelity losses. The multi-fidelity aspect allows BOHB to avoid excessive evaluations at the highest resource setting.

4. Empirical Performance, Benchmarks, and Ablations

BOHB consistently outperforms random search, vanilla Bayesian optimization, and Hyperband across a variety of benchmarks—feed-forward NNs on OpenML datasets, CNNs (e.g., ResNet) on CIFAR-10, Bayesian neural networks, and deep RL problems. Empirical findings include:

• BOHB matches Hyperband’s initial rapid progress but achieves substantially lower final regret, e.g., up to 100× faster to HB’s final quality on NN surrogates (Falkner et al., 2018, Lindauer et al., 2019).
• In high-dimensional or large search spaces, BOHB reliably finds superior configurations, while Hyperband often stagnates at random-search levels.
• BOHB achieves competitive test error with neural architecture search methods at a fraction of the compute (Falkner et al., 2018).
• Practical experiments demonstrate that BOHB and its descendants are robust, parallelizable, and scalable.

Empirical guidelines emphasize choosing the budget axis such that lower fidelities correlate with final performance and retaining modest exploration by setting $\ell(x|b)$7 (Falkner et al., 2018, Lindauer et al., 2019).

5. Extensions and FlexHB: Algorithmic Innovations

Recent work has extended and improved upon BOHB. FlexHB introduces several innovations (Zhang et al., 2024):

• Fine-Grained Fidelity (FGF): Instead of evaluating at exponentially spaced budgets only, FGF injects more measurements at intermediate resource amounts, resulting in denser, more informative data for surrogates at all fidelities.
• Ranking-Loss Based Product-of-Experts: Surrogates at each fidelity are combined via a weighted product-of-experts. Weights $\ell(x|b)$8 for each fidelity’s surrogate $\ell(x|b)$9 are assigned using a ranking-loss that measures the consistency of $g(x|b)$0 with rankings at the highest fidelity. This increases the utility of intermediate fidelities.
• GloSH (Successive Halving with Global Ranking): Instead of ranking only current candidates per SH stage, GloSH merges current and previously eliminated candidates, globally ranks the union, and allows previously killed configurations to re-enter promotion with a tunable probability $g(x|b)$1 per stage.
• FlexBand: Hyperband brackets are self-adaptively shifted based on the observed agreement (Kendall’s $g(x|b)$2) between rankings at adjacent fidelities; highly redundant brackets are replaced by more explorative ones if adjacent fidelities exhibit high rank correlation, further tuning the exploration/exploitation balance.
• Anytime Performance and Efficiency: FlexHB achieves up to $g(x|b)$3 speedup over vanilla BOHB (and up to $g(x|b)$4 over other MFES-HB variants) on neural network and XGBoost HPO tasks, maintaining or improving final error/perplexity (Zhang et al., 2024).

Variant	Surrogate Model	Promotion Rule	Bracket Adaptation
BOHB	KDE/TPE	Vanilla SH	Static HB
FlexHB	Weighted PoE	GloSH	FlexBand (self-adapt)

Advances in FlexHB demonstrate that both finer fidelity modeling and more global, flexible configuration ranking and allocation can yield dramatic gains in multi-fidelity optimization efficiency without compromising solution quality.

6. Related Approaches and Problem Settings

BOHB and its extensions have been adopted in diverse applications, including black-box prompt selection for LLMs (Schneider et al., 2024). HbBoPs, for instance, extends the BOHB design by integrating a structural-aware deep kernel Gaussian process surrogate and a Hyperband-based scheduler, optimizing both sample- and query-efficiency for prompt selection. This structurally modular BO surrogate (using separately encoded instruction/exemplar pairs) achieves state-of-the-art results in LLM prompt optimization, outperforming both classical and neural bandit baselines in test error and query cost (Schneider et al., 2024).

Other practical toolkits, such as BOAH, offer scalable, multi-fidelity BOHB implementations with robust default settings and analytical capabilities (e.g., fANOVA, CAVE), easing adoption in model tuning scenarios with arbitrary fidelity/resource axes (Lindauer et al., 2019).

7. Discussion, Limitations, and Future Directions

BOHB and descendants are limited by the structure of the surrogate (KDE/TPE can struggle for $g(x|b)$5 or with highly conditional spaces) and rely on the practitioner to define the fidelity/budget axis for meaningful low-fidelity evaluations. While theoretical convergence to the optimal full-fidelity solution is retained (as a random fraction of candidates is always explored), no strong regret bounds are proven for multi-fidelity BO in general (Falkner et al., 2018, Wang et al., 2018, Zhang et al., 2024). Extensions toward asynchronous, distributed, and conditional search spaces, as well as automated budget and bracket parameterization (as explored in FlexBand), are ongoing research directions. Empirical comparison of alternative surrogates (GP, random forest, neural surrogates) with advanced multi-fidelity architectures remains a key area for future study (Wang et al., 2018, Zhang et al., 2024).

For practical deployment, BOHB and its improved versions such as FlexHB present a robust, extensible foundation for automated hyperparameter and architecture tuning across a range of computationally demanding ML tasks, consistently outperforming random search, vanilla BO, and classic HB in both convergence time and final performance (Falkner et al., 2018, Lindauer et al., 2019, Zhang et al., 2024, Schneider et al., 2024, Wang et al., 2018).