Quantile-Scaled Bayesian Optimization Using Rank-Only Feedback

Abstract: Bayesian Optimization (BO) is widely used for optimizing expensive black-box functions, particularly in hyperparameter tuning. However, standard BO assumes access to precise objective values, which may be unavailable, noisy, or unreliable in real-world settings where only relative or rank-based feedback can be obtained. In this study, we propose Quantile-Scaled Bayesian Optimization (QS-BO), a principled rank-based optimization framework. QS-BO converts ranks into heteroscedastic Gaussian targets through a quantile-scaling pipeline, enabling the use of Gaussian process surrogates and standard acquisition functions without requiring explicit metric scores. We evaluate QS-BO on synthetic benchmark functions, including one- and two-dimensional nonlinear functions and the Branin function, and compare its performance against Random Search. Results demonstrate that QS-BO consistently achieves lower objective values and exhibits greater stability across runs. Statistical tests further confirm that QS-BO significantly outperforms Random Search at the 1\% significance level. These findings establish QS-BO as a practical and effective extension of Bayesian Optimization for rank-only feedback, with promising applications in preference learning, recommendation, and human-in-the-loop optimization where absolute metric values are unavailable or unreliable.