Cost-Aware Multi-Objective Bayesian Optimization

• Cost-aware MOBO is a framework that extends traditional Bayesian optimization by incorporating evaluation costs and managing multiple objectives simultaneously.
• It employs Gaussian Process surrogates and tailored acquisition functions, like scalarized GP-UCB and EHVI, to balance exploration, exploitation, and cost-efficiency.
• Empirical applications in hyperparameter tuning and LLM team design demonstrate significant cost savings and efficient discovery of Pareto-optimal solutions.

Cost-Aware Multi-Objective Bayesian Optimization (MOBO) extends traditional Bayesian optimization (BO) to simultaneously manage multiple black-box objectives and their associated evaluation costs. Unlike classical settings where either cost is assumed uniform or objectives are single-valued, cost-aware MOBO explicitly models and exploits non-uniform input-dependent evaluation costs, resource constraints, and the statistical relationships between objectives. Applications include hyperparameter tuning, black-box function optimization in computational science, and resource-efficient design of AI systems, exemplified by frameworks such as MALBO for assigning LLMs to specialized multi-agent roles (Sabbatella, 14 Nov 2025, Abdolshah et al., 2019, Martín et al., 2021).

1. Formal Definition and Motivation

Let $\mathcal X\subset\mathbb R^N$ denote the input space, and $\mathbf f:\mathcal X\to\mathbb R^M$ define a vector-valued, black-box objective map. In cost-aware multi-objective Bayesian optimization, each evaluation $\mathbf f(\mathbf x)$ at point $\mathbf x$ incurs a potentially non-uniform, input-dependent cost $c(\mathbf x)$. The goal is to efficiently identify the Pareto front

$$\mathcal{P}^* = \left\{ \mathbf f(\mathbf x) : \nexists\, \mathbf x' \; \text{s.t.} \; \mathbf f(\mathbf x')\succeq \mathbf f(\mathbf x),\; \mathbf f(\mathbf x')\neq \mathbf f(\mathbf x) \right\}$$

while explicitly minimizing total incurred cost—either in terms of number and type of queries, or measured by a cost function over the input and objective space (Abdolshah et al., 2019, Sabbatella, 14 Nov 2025).

This paradigm reflects real-world scenarios where resource budgets, evaluation times, and heterogeneous constraints necessitate careful trade-offs between competing objectives (e.g., accuracy vs. cost, speed vs. fidelity), especially when each candidate query is expensive.

2. Gaussian Process Surrogate Models

Cost-aware MOBO generally relies on independent Gaussian Process (GP) surrogates to model each objective $f_m(x)$: $f_m(x)\sim\mathcal{GP}(0,k_m(x,x';\theta_m))$ with additive noise. For a dataset $\mathcal D_t=\{(x_i, y_{i1},...,y_{iM})\}$, the posterior at a candidate $x$ gives mean $m_m(x)$ and variance $\sigma_m^2(x)$: $$m_m(x) = k_m(x,X)K_m^{-1}y_{m}, \qquad \sigma_m^2(x) = k_m(x,x) - k_m(x,X)K_m^{-1}k_m(X,x)$$ where $K_m=k_m(X,X)+\sigma_{n,m}^2I$, $X$ is the queried set, and kernel hyperparameters (e.g., ARD-Matérn(5/2) in MALBO) are fit by maximizing marginal likelihood (Sabbatella, 14 Nov 2025, Martín et al., 2021).

These surrogates support sample-efficient, uncertainty-aware exploration in both objectives and cost spaces, and enable explicit construction of acquisition functions that internalize resource constraints.

3. Cost-Aware Acquisition Functions

Acquisition functions prioritize the next evaluation by balancing exploitation, exploration, and cost-awareness. In cost-aware MOBO, user-supplied (or learned) knowledge of input costs is encoded as a time-varying cost penalization $C(\mathbf x, t)$.

Scalarized GP-UCB with cost penalty (Abdolshah et al., 2019): $$\alpha(\mathbf x,\bm\theta_t,t) = Q(\mathbf x,\bm\theta_t)\cdot (1 - C(\mathbf x, t))$$ where $Q(\mathbf x,\bm\theta_t)$ is a scalarized UCB across objectives, and $C(\mathbf x, t)$ penalizes expensive regions in early iterations, decaying to allow full-space exploration as $t\to\infty$.

Hypervolume Improvement (EHVI and qLogEHVI) (Sabbatella, 14 Nov 2025): $$\alpha_{\text{EHVI}}(x) = \mathbb{E}_{F(x)\sim N(m(x), \Sigma(x))}[\Delta HV]$$ where HV is the dominated hypervolume with respect to a Pareto set and user-defined reference, and $\Delta HV$ is the increment from adding candidate $F(x)$. In bi-objective settings, closed-form analytic EHVI is used; for larger batches, qLogEHVI (as implemented in BoTorch) enables efficient candidate proposal.

4. Algorithmic Frameworks

Several concrete frameworks instantiate cost-aware MOBO principles:

CA-MOBO (Abdolshah et al., 2019)

• Inputs: Search domain, black-box objectives, cost-index tuple $\mathcal I$, budget $T$.
• Loop:

1. Draw random scalarization weights $\bm\theta_t$ on the simplex. 2. Fit/update each GP. 3. For each candidate, compute UCB and cost-penalty, form acquisition $\alpha$. 4. Select $x_t = \arg\max \alpha$. 5. Evaluate and augment dataset.

Cost-aware step:

Penalization via $1-C(\mathbf x, t)$ discourages queries to high-cost subspaces early, decaying over time to ensure asymptotic optimality.

MALBO (Sabbatella, 14 Nov 2025)

• Problem: Assign $N$ LLM agent roles from $M$ pools, each LLM embedded as $\ell_j\in\mathbb R^{D}$.
• Relaxation: Optimize over continuous-team feature space $x\in\mathcal{F}^N$ (convex hull of LLM representatives).
• Inner Loop:

1. Sample initial team assignments, evaluate objectives (accuracy $f_1$, cost $f_2$), and fit GP surrogates. 2. Optimize batch qLogEHVI in feature space. 3. Project candidate "ideal" embeddings to nearest feasible discrete LLM assignments. 4. Evaluate, update data, re-fit GPs. 5. Return non-dominated set as an approximate Pareto front.

This approach enables efficient navigation of combinatorial search over $M^N$ assignments and yields significant cost reduction and high-quality, heterogeneous LLM teams.

Many-Objective BO with Pruning (Martín et al., 2021)

• Each objective is modelled by a GP.
• Pairwise similarity between GP predictive distributions is computed.
• Redundant objectives (with similarity below threshold $\epsilon$) are pruned after an initial phase $\delta$.
• Pruning reduces evaluation and GP model costs without degrading final Pareto front quality ($$|HV_{\text{baseline}} - HV_{\text{pruned}}|/HV_\text{baseline}<10^{-3}$$ in all experiments).

5. Convergence Theory and Performance

Cost-aware MOBO algorithms inherit no-regret guarantees from UCB-type acquisition schemas. For CA-MOBO, the cumulative regret after $T$ iterations is bounded by

$$\mathcal R(T) = \mathcal O\!\left( \sqrt{M\,T\,\beta_T\sum_{m=1}^M \gamma_T^{(m)} + \pi^2 M\,\mathbb E[U_{\bm\theta}]} \right)$$

where $\gamma_T^{(m)}$ is the maximum information gain for each GP, and $\mathbb E[U_{\bm\theta}]$ is from random scalarization. The dynamic cost penalty guarantees that expensive regions are not ignored asymptotically, preserving optimality (Abdolshah et al., 2019).

Empirical benchmarks indicate:

• CA-MOBO reaches 95% of ground-truth dominated hypervolume in 200 evaluations; standard MOBO requires 300+.
• MALBO reduces mean LLM team configuration cost by 45.6% in the BO phase relative to random search, with no accuracy loss. Final Pareto optimal teams achieve up to 65.8% cost savings vs. best homogeneous baseline (Sabbatella, 14 Nov 2025).
• In many-objective BO, heuristic pruning eliminates redundant objectives, saving evaluation cost and model runtime with negligible impact on hypervolume metrics (Martín et al., 2021).

6. Cost Modeling, Acquisition, and Practical Implementation

Cost-awareness is achieved by incorporating explicit cost models into the acquisition function, leveraging user priors or learned input-dependent cost penalties (as sorted index-tuple $\mathcal I$, exponential family marginals, or similar). Practical guidance includes:

• Encode costly variables via sorted indices according to user knowledge.
• Use exponential-family or similar $C(\mathbf x,t)$ to schedule exploration towards costly regions late in the search.
• Tune acquisition and decay parameters (e.g., $\beta_t$, Dirichlet priors) for desired balance between cost-saving and optimality rate.
• In many-objective settings, measure redundancy by weighted distances of predictive means, variances, and correlation, then prune the most similar objectives to minimize surplus cost burden (Martín et al., 2021).

MALBO demonstrates that continuous embedding plus nearest-neighbor rounding allows tractable, sample-efficient exploration of extremely high-dimensional, combinatorial action spaces otherwise intractable for direct combinatorial BO (Sabbatella, 14 Nov 2025).

7. Empirical Evidence and Application Domains

Comprehensive experiments across synthetic and real-world tasks provide the following findings:

Algorithm/Paper	Task Domain	Cost Savings	Pareto Quality
CA-MOBO (Abdolshah et al., 2019)	ZDT-3, Matyas+Booth, Hyperparam tuning	95% HV in 200 evals (vs 300+)	Matches vanilla on convergence
MALBO (Sabbatella, 14 Nov 2025)	LLM team assignment	45.6% less cost in BO phase, 65.8% less vs. baseline	Maintains max performance
Many-Obj BO (Martín et al., 2021)	Synthetic, hyperparam tuning	Cuts per-iteration eval cost by pruning	$$|HV_{\text{baseline}} - HV_{\text{pruned}}|/HV_\text{baseline} < 10^{-3}$$

• CA-MOBO excels in rapid discovery of diverse Pareto fronts, favoring cheaper subspaces early while guaranteeing asymptotic exploration.
• MALBO automates design of LLM-based multi-agent teams for user-defined trade-offs between cost and performance.
• Many-objective BO pruning workflows adapt the number of modeled objectives, reducing cost while preserving solution diversity.

A common thread is the reduction of redundant computation and cost in resource-limited, high-uncertainty, or high-dimensional environments, aligning with practical constraints faced in both machine learning and scientific optimization.

8. Connections, Generalizations, and Outlook

Cost-aware MOBO interfaces naturally with constraint-aware BO, multi-fidelity optimization, and combinatorial BO. Techniques for cost modeling, surrogate construction, hypervolume-based acquisition, and Pareto front estimation translate across domains wherever evaluation cost is not uniform or objectives are numerous and partially correlated.

Theoretical regret bounds, empirically validated trade-offs, and user-interpretable configuration (via cost tuples, penalty functions, PSR-based pruning) establish cost-aware MOBO as a robust and versatile framework for resource-efficient multi-objective optimization in practical, high-stakes applications (Sabbatella, 14 Nov 2025, Abdolshah et al., 2019, Martín et al., 2021).