HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation (2012.03826v6)

Abstract: In this work we rigorously analyse assumptions inherent to black-box optimisation hyper-parameter tuning tasks. Our results on the Bayesmark benchmark indicate that heteroscedasticity and non-stationarity pose significant challenges for black-box optimisers. Based on these findings, we propose a Heteroscedastic and Evolutionary Bayesian Optimisation solver (HEBO). HEBO performs non-linear input and output warping, admits exact marginal log-likelihood optimisation and is robust to the values of learned parameters. We demonstrate HEBO's empirical efficacy on the NeurIPS 2020 Black-Box Optimisation challenge, where HEBO placed first. Upon further analysis, we observe that HEBO significantly outperforms existing black-box optimisers on 108 machine learning hyperparameter tuning tasks comprising the Bayesmark benchmark. Our findings indicate that the majority of hyper-parameter tuning tasks exhibit heteroscedasticity and non-stationarity, multi-objective acquisition ensembles with Pareto front solutions improve queried configurations, and robust acquisition maximisers afford empirical advantages relative to their non-robust counterparts. We hope these findings may serve as guiding principles for practitioners of Bayesian optimisation. All code is made available at https://github.com/huawei-noah/HEBO.

• The paper introduces HEBO, an algorithm that advances hyperparameter tuning by mitigating heteroscedasticity and non-stationarity using Box-Cox and Yeo-Johnson transformations.
• It employs a multi-objective acquisition strategy with a Pareto front approach and a robust acquisition maximizer to enhance exploration in complex optimization tasks.
• Empirical tests on 108 real-world problems validate HEBO’s superior performance, highlighting its potential for scalable, efficient solutions in machine learning optimization.

An Analysis of HEBO: Pushing the Limits of Sample-Efficient Hyperparameter Optimisation

The paper "HEBO: Pushing the Limits of Sample-Efficient Hyperparameter Optimisation" by Cowen-Rivers et al. offers a comprehensive examination of the challenges and advancements in hyperparameter optimization using Bayesian methodologies. With a focus on addressing issues such as heteroscedasticity and non-stationarity, the authors propose HEBO, an innovative algorithm positioned to enhance the efficiency of hyperparameter tuning tasks.

Key Contributions and Findings

The authors identify significant bottlenecks in black-box optimization tasks, particularly the assumptions of homoscedasticity and stationarity. Their empirical analysis on the Bayesmark benchmark reveals that most hyperparameter tuning tasks exhibit heteroscedastic noise and non-stationary behavior, which can adversely impact optimization performance when using traditional models.

HEBO, or Heteroscedastic Evolutionary Bayesian Optimisation, incorporates several advanced techniques to mitigate these issues:

1. Heteroscedasticity and Non-Stationarity Handling: HEBO employs non-linear input and output transformations, specifically using Box-Cox and Yeo-Johnson strategies, to handle varying noise levels and non-stationary data distributions effectively. Such transformations significantly enhance the flexibility and predictive capabilities of Gaussian Process (GP) models.
2. Acquisition Strategy: Instead of relying on a single acquisition function, the paper suggests a multi-objective approach using ensembles to resolve conflicts in acquisition solutions. This method yields a Pareto front, enabling more robust and varied exploration during the optimization process.
3. Robust Acquisition Maximizer: The authors propose a robust acquisition maximization strategy that leverages stochastic perturbations to approximate worst-case scenarios. This technique minimizes the impact of model misspecification by averaging over multiple model instantiations.

Empirical Validation

The performance of HEBO is validated through rigorous experimentation across 108 real-world problems sourced from the UCI repository. Results showcase HEBO's superiority in handling complex noise patterns and delivering higher mean and median performance scores compared to established optimization algorithms such as TuRBO, PySOT, and Nevergrad.

The experimental results, depicted through mean normalised scores and variance measurements, highlight HEBO's capability in consistently outperforming competitors and achieving exceptional results on a significant portion of the tested tasks.

Implications and Future Directions

The implications of HEBO extend beyond immediate enhancements in hyperparameter tuning. By addressing inherent assumptions in traditional models, HEBO sets a precedent for future advancements in sample-efficient strategies. The robust, adaptive nature of HEBO positions it as a potential framework for tackling a broader array of optimization tasks beyond hyperparameter tuning.

The paper encourages further exploration into asynchronous and synchronous method integration, potentially bridging the gap between real-time applications and model efficiency. Moreover, the development of scalable solutions for deep learning architectures remains a promising direction for subsequent research.

In sum, Cowen-Rivers et al.'s paper provides a substantial contribution to the field of hyperparameter optimization. It prompts a reevaluation of conventional assumptions while offering tangible solutions through HEBO, supporting the continued evolution of optimization algorithms in machine learning contexts.