Quantile-Scaled Bayesian Optimization

• QS-BO is a framework that transforms raw objective values into quantile-aligned latent representations, ensuring robustness against scale and outliers.
• It supports diverse formulations including transfer-learning, rank-only feedback, and risk-aware optimization using Gaussian copula models and GP surrogates.
• Empirical studies show QS-BO significantly improves hyperparameter optimization and multi-objective tuning, outperforming standard baselines with enhanced stability.

Searching arXiv for the cited QS-BO-related papers to ground the article in current records. Quantile-Scaled Bayesian Optimization (QS-BO) designates a family of Bayesian optimization procedures that replace raw objective values with quantile-aligned latent representations before surrogate modeling and acquisition. In the transfer-learning formulation of hyperparameter optimization, the objective values of related tasks are mapped through empirical CDFs and a Gaussian copula so that heterogeneous datasets and objectives become comparable in a common approximately Gaussian space (Salinas et al., 2019). In the rank-only formulation, ordinal feedback is converted into Gaussian pseudo-targets with heteroscedastic uncertainty derived from order-statistics, allowing standard GP regression and standard acquisition functions to be used without observing explicit metric scores (Egunjobi, 28 Sep 2025). A related but distinct line directly optimizes conditional quantiles or expectiles of stochastic black-box functions through asymmetric likelihoods and heteroscedastic latent GPs rather than through quantile scaling of observations (Picheny et al., 2020). This suggests that QS-BO is best understood as an umbrella centered on optimization in quantile space, monotone-invariant decision rules, and robustness to heterogeneous scales, ordinal data, or tail-sensitive criteria.

1. Conceptual scope and problem classes

In the transfer-learning setting, QS-BO addresses hyperparameter optimization over related tasks. Let $x \in \mathbb{R}^p$ denote a hyperparameter vector, let $f^j:\mathbb{R}^p \to \mathbb{R}$ be the black-box objective for task $j \in \{1,\dots,M\}$, and let the offline data be

$$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$

with $y_i^j = f^j(x_i^j)$. The goal is to minimize a new task with as few evaluations as possible, potentially with multiple objectives $f_1,\dots,f_L$ (Salinas et al., 2019).

In the rank-only setting, QS-BO addresses problems in which only relative or ordinal feedback is available. The motivating cases include human-in-the-loop settings, systems with ordinal outcomes, and settings where scalar objective values are noisy, unreliable, or poorly calibrated. Standard GP-based BO is ill-posed there because it assumes numeric targets and a Gaussian likelihood, whereas QS-BO uses a monotone rank-to-quantile-to-Gaussian pipeline and retains ordinary GP regression machinery (Egunjobi, 28 Sep 2025).

A third, adjacent problem class is risk-aware stochastic optimization. There, the target is not a transformed observation but a functional of the conditional distribution $Y|x$, such as a $\tau$-quantile or a $\tau$-expectile. That formulation is designed for heteroscedastic and non-Gaussian noise and differs from both transfer-oriented copula scaling and rank-only pseudo-target construction (Picheny et al., 2020).

These three settings share a common structural idea: optimization is conducted in a latent space in which ranking information or tail structure is more stable than raw objective magnitudes. The literature, however, does not use the term identically across all formulations. The transfer-learning paper refers to Copula Thompson Sampling and Gaussian Copula Process with a parametric prior, whereas the 2025 paper uses the label QS-BO explicitly for rank-only feedback. This suggests that the unifying concept is methodological rather than terminological.

2. Quantile scaling via semi-parametric Gaussian copulas

The transfer-learning variant is motivated by the observation that objectives across datasets may differ in scale, noise level, and outlier structure. Naive standardization is therefore fragile. QS-BO instead aligns ranks rather than raw values, making the representation invariant to strictly monotone transformations of marginals and robust to heavy tails and outliers (Salinas et al., 2019).

For a task-specific objective $Y$ with conditional distribution given $f^j:\mathbb{R}^p \to \mathbb{R}$0, let

$f^j:\mathbb{R}^p \to \mathbb{R}$1

denote the $f^j:\mathbb{R}^p \to \mathbb{R}$2-quantile. The key transform is the probability integral transform followed by a probit map:

$f^j:\mathbb{R}^p \to \mathbb{R}$3

where $f^j:\mathbb{R}^p \to \mathbb{R}$4 is the standard normal CDF. In the semi-parametric Gaussian copula view, arbitrary marginals are coupled through a Gaussian dependency structure with density

$f^j:\mathbb{R}^p \to \mathbb{R}$5

The marginals are modeled nonparametrically through empirical CDFs,

$f^j:\mathbb{R}^p \to \mathbb{R}$6

followed by winsorization to avoid infinities at the extrema:

$f^j:\mathbb{R}^p \to \mathbb{R}$7

with

$f^j:\mathbb{R}^p \to \mathbb{R}$8

This maps each task’s $f^j:\mathbb{R}^p \to \mathbb{R}$9 to $j \in \{1,\dots,M\}$0, making $j \in \{1,\dots,M\}$1 across tasks and therefore comparable.

The shared conditional model in copula space is

$j \in \{1,\dots,M\}$2

with

$j \in \{1,\dots,M\}$3

where $j \in \{1,\dots,M\}$4 is an MLP and $j \in \{1,\dots,M\}$5 is softplus. The parameters

$j \in \{1,\dots,M\}$6

are shared across tasks, which is the mechanism by which transfer is induced.

Training minimizes the Gaussian negative log-likelihood over all tasks:

$j \in \{1,\dots,M\}$7

optionally with inverse-$j \in \{1,\dots,M\}$8 weighting to balance tasks of different sizes. Although pinball loss is a natural quantile-regression alternative,

$j \in \{1,\dots,M\}$9

the method optimizes Gaussian NLL in copula space because it directly exploits the normality induced by $$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$0.

Quantiles on the original scale can be recovered from

$$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$1

and

$$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$2

In practice, optimization is performed in $$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$3-space. Since $$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$4 is strictly increasing,

$$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$5

so the optimizer is preserved under the transform.

3. BO mechanisms in quantile space

The transfer-learning framework instantiates two BO strategies: Copula Thompson Sampling (CTS) and a Gaussian Copula Process (GCP) with a parametric prior (Salinas et al., 2019).

CTS is prior-only. Given the learned conditional model $$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$6, one samples candidate points $$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$7, draws

$$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$8

and evaluates the candidate with the smallest sampled value. Because CTS does not condition on observations from the current task, it is computationally light and effective when transfer is strong, but it can degrade when task mismatch is substantial.

The adaptive alternative is GCP with a residual GP prior. Define the transformed target-task objective

$$D^M = \bigcup_{j=1}^M \{(x_i^j, y_i^j)\}_{i=1}^{N_j},$$9

and the residual

$y_i^j = f^j(x_i^j)$0

The implementation uses $y_i^j = f^j(x_i^j)$1 and a Matérn-$y_i^j = f^j(x_i^j)$2 ARD kernel; categorical hyperparameters are one-hot encoded and kernel hyperparameters are fit by type-II maximum likelihood. Given the GP posterior

$y_i^j = f^j(x_i^j)$3

the surrogate for $y_i^j = f^j(x_i^j)$4 has

$y_i^j = f^j(x_i^j)$5

Expected Improvement for minimization is then computed in $y_i^j = f^j(x_i^j)$6-space:

$y_i^j = f^j(x_i^j)$7

with

$y_i^j = f^j(x_i^j)$8

Since $y_i^j = f^j(x_i^j)$9 is undefined before any target-task observation, the method warm-starts with a small CTS batch, for example $f_1,\dots,f_L$0, to estimate the target empirical CDF.

The same framework extends to multi-objective optimization by scalarizing in copula space rather than in raw units. With objectives $f_1,\dots,f_L$1,

$f_1,\dots,f_L$2

Because the $f_1,\dots,f_L$3 align ranks, the scalarized surface approximates the geometry of the Pareto frontier more faithfully than linear scalarization in raw units. The paper primarily uses such scalarization rather than a full multivariate copula over objectives, though it identifies a multivariate Gaussian copula with correlation matrix $f_1,\dots,f_L$4 as a natural extension.

The computational profile is one of the central practical distinctions. Prior fitting scales as $f_1,\dots,f_L$5 when $f_1,\dots,f_L$6 source tasks each have $f_1,\dots,f_L$7 evaluations. CTS has $f_1,\dots,f_L$8 per-iteration sampling cost once $f_1,\dots,f_L$9 and $Y|x$0 are available, giving overall $Y|x$1. GCP refits cost $Y|x$2 in the target-task sample size, for total $Y|x$3. This avoids the $Y|x$4 scaling of multi-task GPs over all source and target evaluations.

4. Rank-only QS-BO

The rank-only formulation begins from the assumption that only ranks are observed. Let $Y|x$5 be the number of observed points so far, and let $Y|x$6 be the rank of $Y|x$7 among those points, with $Y|x$8 best for minimization. The default rank-to-quantile map is

$Y|x$9

with the alternative

$\tau$0

also noted as standard. More generally,

$\tau$1

Practical clipping to $\tau$2, with $\tau$3, prevents divergence of the probit transform. Ties are handled by assigning mid-ranks before transformation (Egunjobi, 28 Sep 2025).

The Gaussian pseudo-targets are

$\tau$4

Because the transform is strictly monotone, the ordering induced by the unknown objective is preserved. The uncertainty in each rank-derived target is obtained from order-statistics. If $\tau$5 is the $\tau$6th order statistic of $\tau$7 i.i.d. $\tau$8 random variables, then

$\tau$9

with

$\tau$0

Applying the delta method to $\tau$1 yields

$\tau$2

The heteroscedasticity is intrinsic: extreme quantiles have larger uncertainty on the probit scale because $\tau$3 is small, while the variance shrinks as $\tau$4 grows.

QS-BO then fits an exact GP with known per-point noise:

$\tau$5

with $\tau$6 in the reported implementation and kernels such as RBF or Matérn. The marginal log-likelihood is

$\tau$7

where $\tau$8. Prediction at $\tau$9 follows standard exact-GP formulas:

$Y$0

Acquisition is carried out entirely on the $Y$1-scale. For minimization, Expected Improvement is

$Y$2

with

$Y$3

Lower Confidence Bound selects the minimizer of

$Y$4

and Thompson Sampling draws posterior function samples on the $Y$5-scale. The sequential algorithm recomputes ranks after each new query, rebuilds the pseudo-targets and heteroscedastic variances, refits the GP, and proposes the next point.

The theoretical interpretation rests on monotone invariance. If $Y$6 is strictly monotone, then $Y$7. Since the rank-to-quantile-to-probit mapping is strictly monotone in the ordering induced by $Y$8, optimizing on the $Y$9-scale is consistent with optimizing the rank of $f^j:\mathbb{R}^p \to \mathbb{R}$00. The paper does not derive regret bounds, and partial rankings or top-$f^j:\mathbb{R}^p \to \mathbb{R}$01 feedback are identified as extensions rather than handled cases.

5. Relation to Bayesian quantile and expectile optimisation

A related but technically distinct research line treats quantiles and expectiles themselves as the BO targets. Let $f^j:\mathbb{R}^p \to \mathbb{R}$02 be an expensive stochastic simulator, and let $f^j:\mathbb{R}^p \to \mathbb{R}$03 have distribution $f^j:\mathbb{R}^p \to \mathbb{R}$04. The objective is to optimize a risk functional

$f^j:\mathbb{R}^p \to \mathbb{R}$05

with $f^j:\mathbb{R}^p \to \mathbb{R}$06 chosen as a quantile or expectile rather than a mean (Picheny et al., 2020).

For the $f^j:\mathbb{R}^p \to \mathbb{R}$07-quantile,

$f^j:\mathbb{R}^p \to \mathbb{R}$08

with pinball loss

$f^j:\mathbb{R}^p \to \mathbb{R}$09

For the $f^j:\mathbb{R}^p \to \mathbb{R}$10-expectile,

$f^j:\mathbb{R}^p \to \mathbb{R}$11

with asymmetric squared loss

$f^j:\mathbb{R}^p \to \mathbb{R}$12

The latent model uses two GPs. One GP models the target functional $f^j:\mathbb{R}^p \to \mathbb{R}$13,

$f^j:\mathbb{R}^p \to \mathbb{R}$14

and a second GP models the log-scale

$f^j:\mathbb{R}^p \to \mathbb{R}$15

so that $f^j:\mathbb{R}^p \to \mathbb{R}$16 captures heteroscedasticity.

Quantile BO uses the Asymmetric Laplace Distribution:

$f^j:\mathbb{R}^p \to \mathbb{R}$17

where $f^j:\mathbb{R}^p \to \mathbb{R}$18. Expectile BO uses an asymmetric Gaussian-like pseudo-likelihood

$f^j:\mathbb{R}^p \to \mathbb{R}$19

with

$f^j:\mathbb{R}^p \to \mathbb{R}$20

Inference is performed with sparse variational GPs and inducing variables for both latent processes, maximizing an ELBO of the form

$f^j:\mathbb{R}^p \to \mathbb{R}$21

The reported acquisition strategies are Quantile-GIBBON, a lower-bound approximation to MES adapted to the asymmetric heteroscedastic setting, and Thompson Sampling with decoupled SVGP-plus-RFF sampling.

This line is often associated with QS-BO because it is quantile-centered and explicitly models input-dependent scale. However, it does not quantile-scale observed outcomes via empirical CDFs or ranks. It instead defines the optimization target as the quantile or expectile itself. The distinction matters: transfer-oriented QS-BO normalizes observations across tasks, rank-only QS-BO manufactures pseudo-targets from order information, and quantile/expectile BO models a tail functional of the data-generating distribution.

6. Empirical behavior, limitations, and recurrent misunderstandings

The transfer-learning copula formulation was evaluated on hyperparameter optimization tasks including DeepAR time-series, FCNET, XGBoost, and NAS-Bench-201, with baselines such as RS, standard GP-BO, warm-start GP, AutoGP, SGPT, ABLR, BOHB, REINFORCE, and REA. The central empirical findings were that copula normalization dramatically improved transfer baselines, GCP was a strong single-task baseline, and GCP combined with the parametric prior yielded the best overall performance. Reported normalized improvement over RS was approximately $f^j:\mathbb{R}^p \to \mathbb{R}$22 for DeepAR, $f^j:\mathbb{R}^p \to \mathbb{R}$23 for FCNET, $f^j:\mathbb{R}^p \to \mathbb{R}$24 for XGBoost, and $f^j:\mathbb{R}^p \to \mathbb{R}$25 for NAS, with average rank approximately $f^j:\mathbb{R}^p \to \mathbb{R}$26 across $f^j:\mathbb{R}^p \to \mathbb{R}$27 datasets. In multi-objective NAS, copula scalarization tracked the Pareto front of error versus latency well, outperformed naive linear scalarization in raw units, and produced faster hypervolume-error reduction; CTS accelerated search when transfer was strong, but GCP with prior dominated as target-task data accumulated (Salinas et al., 2019).

The rank-only formulation was evaluated on a $f^j:\mathbb{R}^p \to \mathbb{R}$28D sinusoidal–quadratic function, the $f^j:\mathbb{R}^p \to \mathbb{R}$29D Forrester function, and the $f^j:\mathbb{R}^p \to \mathbb{R}$30D Branin function. The protocol used $f^j:\mathbb{R}^p \to \mathbb{R}$31 initial random points, then $f^j:\mathbb{R}^p \to \mathbb{R}$32 iterations for a total of $f^j:\mathbb{R}^p \to \mathbb{R}$33 evaluations, with $f^j:\mathbb{R}^p \to \mathbb{R}$34 acquisition candidates per iteration and $f^j:\mathbb{R}^p \to \mathbb{R}$35 seeds per task. QS-BO consistently outperformed Random Search and exhibited lower variance across runs. For Branin, for example, Random Search had mean $f^j:\mathbb{R}^p \to \mathbb{R}$36 and QS-BO had mean $f^j:\mathbb{R}^p \to \mathbb{R}$37; the paired $f^j:\mathbb{R}^p \to \mathbb{R}$38-test gave $f^j:\mathbb{R}^p \to \mathbb{R}$39 and the Wilcoxon signed-rank test gave $f^j:\mathbb{R}^p \to \mathbb{R}$40. The paper states that the superiority over Random Search was statistically significant at the $f^j:\mathbb{R}^p \to \mathbb{R}$41 level on all reported tasks (Egunjobi, 28 Sep 2025).

The quantile/expectile BO line reported that its variational two-GP models outperformed replicate-based GPR-EI and HetGP baselines in heteroscedastic, non-Gaussian settings, with especially strong results on synthetic GLD benchmarks, Lunar Lander RL, and FEL laser tuning. It also reported that Q-GIBBON was strongest in lower dimensions, whereas Thompson Sampling was competitive or superior in higher dimensions and larger-batch settings. At the same time, Q-GIBBON degraded in $f^j:\mathbb{R}^p \to \mathbb{R}$42 dimensions because the Gumbel sampler became sensitive to high-dimensional random multistart behavior (Picheny et al., 2020).

Several limitations recur across the literature. In transfer-oriented QS-BO, CTS does not incorporate target-task observations and deteriorates when transfer is weak; GCP with prior requires empirical CDF estimation on the target task, which can be noisy with very few points; and the reported multi-objective method uses scalarization rather than a full multivariate copula across objectives (Salinas et al., 2019). In rank-only QS-BO, eliciting consistent global ranks over all past points can be cumbersome, early iterations suffer large Beta-derived uncertainty near the extremes, and no regret bounds are provided (Egunjobi, 28 Sep 2025). In direct quantile or expectile BO, ELBO quality depends on Monte Carlo variance, extreme $f^j:\mathbb{R}^p \to \mathbb{R}$43 values demand larger budgets, and information-theoretic acquisition can be fragile in high dimensions (Picheny et al., 2020).

A common misunderstanding is to treat QS-BO as a single canonical algorithm. The literature instead contains at least two substantively different mechanisms under that label or closely adjacent to it. One uses semi-parametric Gaussian copula normalization for transfer across datasets and objectives; another converts ranks into heteroscedastic Gaussian pseudo-targets for ordinal-only optimization. A further nearby line optimizes quantiles or expectiles directly through asymmetric likelihoods. This suggests that the enduring core of QS-BO is not a fixed acquisition rule or surrogate family, but the decision to conduct BO in a quantile-aligned latent representation where ordering, robustness, and scale comparability are primary.