Bayesian Multi-Objective HPO

• Bayesian multi-objective hyperparameter optimization is a probabilistic approach that tunes ML models by searching for Pareto-optimal configurations among conflicting performance metrics.
• It leverages surrogate models like Gaussian and Student-t processes combined with scalarization and hypervolume-based acquisition functions to efficiently navigate complex search spaces.
• Empirical studies show that these methods offer increased sample efficiency and robustness compared to traditional grid or random search techniques.

Bayesian multi-objective hyperparameter optimization (MO-HPO) is a class of methods for efficiently tuning machine learning models when multiple, potentially conflicting performance metrics must be jointly optimized under limited computational budgets. These methods employ probabilistic surrogate models—typically Gaussian processes (GPs), Student-t processes (TPs), or density estimators—to explore the Pareto front of non-dominated configurations, accounting for trade-offs between objectives such as accuracy, inference latency, fairness, model size, or energy cost.

1. Foundations of Bayesian Multi-Objective Optimization

Bayesian MO-HPO generalizes classical Bayesian optimization (BO) to vector-valued black-box functions $f: \mathcal{X} \to \mathbb{R}^m$, where $\mathcal{X}$ is a (possibly mixed-type) hyperparameter search space and $m > 1$ is the number of objectives. The target is the Pareto set

$$\mathcal{P} = \{x^* : \nexists\,x\in\mathcal{X},\,f(x) \preceq f(x^*) \;\text{with strict inequality for at least one component}\}$$

and its image, the Pareto front. Bayesian surrogates, especially GPs and TPs, provide posterior predictive distributions $\mathcal{N}(\mu_i(x), \sigma^2_i(x))$ (GP) or $\text{Student-}t(\mu_i(x), \hat{s}_i^2(x), \nu)$ (TP) over each objective $f_i$, facilitating uncertainty-aware search strategies.

Key acquisition functions for MOBO include:

• Random Scalarization (ParEGO-style): At each iteration, sample a weight vector $\lambda$ on the simplex, form a scalar objective $S_\lambda(f(x))$, and apply a single-objective acquisition function (EI, UCB, etc.) (Paria et al., 2018).
• Hypervolume-based (EHI, HvPoI): Quantify the expected improvement in dominated hypervolume computed from the surrogate posteriors (Herten et al., 2016).
• Information-Theoretic (PESMO/MESMO): Seek evaluations that most reduce uncertainty about the location or values of the Pareto set (Martín et al., 2021).
• Constraint-Handling: Unified extended domination over objectives and inequality constraints, leveraging specialized EHVI and SMC (Feliot et al., 2015, Li et al., 6 Nov 2024).
• Preference Learning: Incorporate decision-maker trade-offs or stability ordering via Bayesian utility/posterior over preferences (Ozaki et al., 2023, Abdolshah et al., 2019).

MOBO methods are particularly effective for expensive, noisy, or non-convex objectives, providing sample-efficient coverage of the Pareto frontier compared to evolutionary algorithms or random search (Karl et al., 2022).

2. Surrogate Models and Acquisition Computation

Surrogate models are at the core of Bayesian MO-HPO. Typical choices include:

Model Class	Posterior Equation	Notable Features
GP	$f_i(x)\sim\mathcal{GP}(\mu_i(x),k_i(x,x'))$	Efficient analytic mean/variance update, assumes Gaussian noise.
TP	$f(x)\sim\text{TP}(\nu, \phi(x), k(x,x'))$	Heavier-tailed prediction, increased robustness to outliers, variance inflation factor (Herten et al., 2016).
Density Estimator (TPE)	$p(x|f(x)\leq \gamma)$, $p(x|f(x)>\gamma)$	Non-parametric, used for fast batch candidate suggestion (Park et al., 7 Mar 2025).

For a candidate $x$, the joint surrogate predictive is typically assumed factorized: $p(f(x)|D) = \prod_{i=1}^m p_i(f_i(x)|D)$. The hypervolume-based acquisition (Herten et al., 2016) decomposes the non-dominated region $A$ into axis-aligned cells $[\ell^{(k)}, u^{(k)}]$ providing a closed-form probability of improvement (under TPs): $$\Pr[f(x) \in A] = \sum_{k=1}^q \prod_{j=1}^m \left\{\Phi_{\nu+n}\left(\frac{u_j^{(k)}-\mu_j(x)}{\hat{s}_j(x)}\right) - \Phi_{\nu+n}\left(\frac{\ell_j^{(k)}-\mu_j(x)}{\hat{s}_j(x)}\right)\right\}$$ where $\Phi_{\nu}$ is the Student-t CDF.

Scalarization methods, notably ParEGO (Karl et al., 2022), randomize $\lambda$ and collapse objectives to $S_\lambda(f(x))$ via weighted linear or Tchebycheff scalarizations, enabling a single GP to manage acquisition optimization. This approach guarantees sublinear preference-weighted regret under regularity assumptions (Paria et al., 2018).

For many-objective settings ($m>3$), notable work introduces objective similarity metrics to identify and remove redundant objectives, trading off resource savings for Pareto frontier fidelity (Martín et al., 2021).

3. Hypervolume-Based and Preference-Aware Acquisition Functions

Hypervolume-based acquisition functions directly target the discovery of Pareto-efficient solutions by modeling the expected gain in the dominated volume of objective space. For TPs, the hypervolume probability of improvement (HvPoI) is analytically tractable and empirically superior in the presence of heteroskedastic or heavy-tailed noise (Herten et al., 2016).

Preference-guided MOBO frameworks allow the decision-maker to specify utility models or order constraints on the objectives. Preference Bayesian optimization (MBO-APL) learns a posterior over utility parameters (e.g., Chebyshev scalarization weights) via interactive pairwise comparisons and improvement requests, integrating decision-maker uncertainty in each acquisition (Ozaki et al., 2023). This yields sharply targeted search—empirically, preference-adaptive BO attains superior regret and requires far fewer expensive function evaluations than Pareto-front recovery (Ozaki et al., 2023).

Stability-order constraints, as in MOBO-PC (Abdolshah et al., 2019), leverage GP-derivative posteriors and Monte Carlo estimation of the probability that a candidate solution has a more stable behavior (smaller gradient magnitude) in preferred objectives. Preference-weighted Expected Hypervolume Improvement ($a_{PEHI}$) further focuses sampling on desired front regions.

4. Handling Constraints and Many-Objective Extensions

Multi-objective Bayesian optimization routinely faces practical constraints, e.g., on model size, energy consumption, or runtime. The extended domination principle (Feliot et al., 2015) unifies objectives and constraints within the acquisition, allowing the (expected) hypervolume improvement to reflect feasibility when solutions may violate constraints: $EHVI_n(x) = \int_{G_n} P_n[\xi(x) \prec_e y] dy$ where $G_n$ is the current non-dominated region and the extension handles both feasibility and Pareto dominance. Practically, sequential Monte Carlo (SMC) and subset simulation provide scalable algorithms for both acquisition computation and maximization in high-dimensional or multi-constraint settings (Feliot et al., 2015).

The COMBOO algorithm combines optimistic constraints estimation (UCBs over constraints) and random-hypervolume scalarization on feasible regions, yielding sublinear cumulative regret and constraint violation guarantees (Li et al., 6 Nov 2024). Empirical results on neural architectures and realistic constraints demonstrate significant improvements in hypervolume and feasibility rates (Li et al., 6 Nov 2024).

Removing redundant objectives in many-objective settings, as in (Martín et al., 2021), relies on GP predictive similarity metrics (correlation, calibrated mean/covariance distances) computed over representative input grids; empirical evidence suggests resource savings (≈1 eval per deleted target per iteration) with negligible impact on Pareto-front coverage.

5. Parallelization and Sample Efficiency

Modern applications demand scalable MO-HPO. Asynchronous parallelization techniques—such as decentralized in-memory sharing of evaluated points and independent local BO loops—deliver $5\times$ speed-ups with $16\times$ more workers (Egele et al., 2023). Uniform normalization (empirical quantile or ECDF scaling) of objectives and randomized-weight scalarization further increase Pareto-front diversity and robustness in distributed settings.

Batch MOBO can be achieved either by q-EHVI (joint expected hypervolume improvement over $q$ batched points), or by employing parallel candidate generation via TPE density estimation (Park et al., 7 Mar 2025). Penalty-based soft constraints on objectives prune uninteresting regions, focusing BO on feasible and high-value trade-off regions.

6. Advanced Algorithms and Empirical Results

Recent innovations integrate deep learning and reinforcement learning methodology. BOFormer (2505.21974)—a Transformer-based, non-Markovian RL framework—models the full history and captures hypervolume identifiability, outperforming qNEHVI and other benchmarks in hyperparameter tuning tasks, with millisecond-scale inference cost. Empirical results substantiate up to $5\%$ higher attained hypervolume compared to state-of-the-art alternatives.

Trajectory-based MOBO algorithms treat epoch as an explicit decision variable, modeling predictive learning trajectories for each proposed hyperparameter configuration. Trajectory-EHVI (TEHVI) acquisition, together with a Pareto-aware epoch-wise early stopping rule, delivers time- and sample-efficient exploration, outperforming qEHVI, NEHVI, and ParEGO in both synthetic and real-world benchmarks (Wang et al., 24 May 2024).

In e-commerce, MOHPER combines Bayesian surrogates (GP or TPE), multi-objective density estimation, meta-configuration voting, and cumulative learning to robustly optimize trade-offs between engagement (CTR) and conversion (CTCVR), demonstrating production-level efficacy (Park et al., 7 Mar 2025).

For fairness- and efficiency-aware ML, multi-source Bayesian optimization (FanG-HPO) leverages information from cheap data subsets, fuses them into augmented GP (AGP) surrogates, and achieves higher Pareto front hypervolume at reduced energy cost compared to constrained fairness-aware methods and single-source MOBO (Candelieri et al., 2022).

7. Implementation and Practical Guidance

Implementing Bayesian MO-HPO requires careful integration of surrogate modeling, acquisition optimization, and domain-specific constraints/preferences. Recommendations include:

• Use ARD squared-exponential or Matérn kernels for GP surrogate models; integrate hyperparameter estimation every 10–20 BO steps.
• For scalarization-based methods, sample direction vectors uniformly or according to user-specified region-of-interest.
• Normalize objectives in real time (e.g., ECDF-based quantile scaling), especially when diverse metric scales and outliers are present.
• Incorporate soft/hard constraints via either joint GP modeling or penalty-based exclusion mechanisms.
• Leverage parallelization and asynchronous acquisition to scale up to large evaluation budgets.
• For batch proposals or preference learning, utilize interactive or feedback-driven utility estimation.
• Monitor hypervolume and feasibility rates; adapt acquisition function complexity according to the number of objectives and computational budget.
• Use SMC or MC-approximation for high-dimensional EHVI wherever necessary.

Empirical studies consistently show Bayesian MOBO outperforms grid/random search, multi-objective evolutionary approaches, and even advanced fairness-aware algorithms, especially when function evaluations are expensive and trade-off management is central (Karl et al., 2022, Park et al., 7 Mar 2025, 2505.21974). The choice of surrogate (GP, TP, TPE), acquisition (scalarization vs hypervolume), and algorithmic detail (preference, constraint, or trajectory) should be tailored to problem structure and resource requirements.