Risk-Controlling Model Selection via Guided Bayesian Optimization (2312.01692v1)

Abstract: Adjustable hyperparameters of machine learning models typically impact various key trade-offs such as accuracy, fairness, robustness, or inference cost. Our goal in this paper is to find a configuration that adheres to user-specified limits on certain risks while being useful with respect to other conflicting metrics. We solve this by combining Bayesian Optimization (BO) with rigorous risk-controlling procedures, where our core idea is to steer BO towards an efficient testing strategy. Our BO method identifies a set of Pareto optimal configurations residing in a designated region of interest. The resulting candidates are statistically verified and the best-performing configuration is selected with guaranteed risk levels. We demonstrate the effectiveness of our approach on a range of tasks with multiple desiderata, including low error rates, equitable predictions, handling spurious correlations, managing rate and distortion in generative models, and reducing computational costs.

• The paper presents a novel approach that integrates Bayesian Optimization with risk-controlling procedures to select model configurations under statistical constraints.
• It defines a region of interest in the objective space to focus the search on Pareto optimal configurations that are likely to pass subsequent statistical tests.
• Empirical evaluations demonstrate that the method efficiently balances objectives such as fairness, robustness, and cost-accuracy in various machine learning tasks.

Bayesian Optimization (BO) is a widely used method for selecting optimal configurations of functions that are expensive to evaluate, such as hyperparameters in machine learning models. These hyperparameters can significantly influence model performance aspects such as accuracy, fairness, robustness, and computational cost. A fundamental challenge in model selection is finding hyperparameters that not only improve these performance aspects but also adhere to certain user-specified constraints such as error rates or fairness metrics.

To address this challenge, the discussed paper introduces a novel approach that combines Bayesian Optimization with rigorous risk-controlling procedures to guide the search process towards more efficient and statistically valid model configurations. The core idea of the method is to define a "region of interest" in the objective space that reflects the user's constraints and preferences, and then adjust the BO process to focus on finding Pareto optimal configurations within this region.

The adjusted BO procedure is uniquely tailored to recover a focused set of Pareto optimal configurations that are most likely to pass subsequent statistical testing. These configurations are then subjected to a statistical testing framework that provides a way to verify whether they meet the desired constraints with a user-specified level of confidence. Importantly, this enables the selection of models under multiple constraints while maintaining computational efficiency.

The effectiveness of this methodology is demonstrated across a variety of tasks involving different objectives, such as preserving fairness in classification, ensuring robustness against spurious correlations, managing reconstruction quality and latent space complexity in Variational Autoencoders (VAEs), and optimizing cost-accuracy trade-offs in large transformer models. Through empirical evaluations, the authors show that their guided BO method selects efficient configurations that are both high-performing and meet statistical constraints, even when compared to various baselines.

Overall, this work provides a significant contribution by proposing a flexible and efficient framework for model selection under multiple constraints. It offers a pragmatic solution for practitioners to balance diverse objectives and control risk when selecting machine learning model configurations, especially in scenarios with limited computational budgets.