---
title: Log EI per Cost in Bayesian Optimization
url: https://www.emergentmind.com/topics/log-expected-improvement-per-cost
type: topic
---

# Log EI per Cost in Bayesian Optimization

Searching arXiv for the cited papers to ground the article in current research.
Log Expected Improvement per Cost denotes a family of cost-aware acquisition constructions in Bayesian optimization that combine expected improvement with a logarithmic device and a cost normalization. The recent literature indicates that the phrase does not refer to a single canonical formula. The logarithm may be applied to the objective before Gaussian-process fitting, to the acquisition value itself for numerical stabilization, or to the cost model to enforce positivity and obtain tractable moments. The resulting objects are mathematically distinct, and the distinction is central to correct implementation and interpretation [2411.18095] [2111.06537] [2310.20708].

## 1. Terminological scope and competing meanings

A persistent source of confusion is that closely related expressions use the word “log” in three different ways. In the SMAC / Hutter et al. sense revisited in "Derivation of Closed Form of Expected Improvement for Gaussian Process Trained on Log-Transformed Objective" [2411.18095], a Gaussian process is trained on the transformed objective \(g(x)=\log f(x)\), and expected improvement is then computed on the original objective scale under the induced log-normal predictive distribution. In "Unexpected Improvements to Expected Improvement for Bayesian Optimization" [2310.20708], by contrast, LogEI means a numerically stable computation of \(\log(\mathrm{EI}(x))\) or a closely related smoothed log-domain surrogate. In "Multi-Step Budgeted Bayesian Optimization with Unknown Evaluation Costs" [2111.06537], the logarithm is attached to the cost model through a GP prior on \(\ln c(x)\), while the acquisition itself remains EI, EI-per-unit-cost, or a budget-aware multi-step variant.

| Usage | Representative definition | Role of log |
|---|---|---|
| Log-transformed objective | \(\alpha_{\log \mathrm{EI}}(x)=\mathbb E[(f^\star-e^{g(x)})_+]\) | GP trained on \(g(x)=\log f(x)\) |
| EI-per-cost with log-cost model | \(\mathrm{EI}^f(x\mid D)\exp\!\left(-\mu_D^{\ln c}(x)+\frac{(\sigma_D^{\ln c}(x))^2}{2}\right)\) | GP trained on \(\ln c(x)\) |
| Modern LogEI | \(\mathrm{LogEI}_{y^*}(x)=\mathrm{log\_h}\!\left(\frac{\mu(x)-y^*}{\sigma(x)}\right)+\log \sigma(x)\) | \(\log\) applied to EI for numerical stability |

This terminology matters because “log expected improvement per cost” can denote at least two inequivalent constructions. One is the ratio obtained by taking the Hutter-style log-transformed-objective acquisition and dividing by a positive cost model. Another is the log-domain difference \(\mathrm{LogEI}(x)-\log c(x)\), which follows the monotonicity argument behind modern LogEI. A third nearby object, EI-per-cost with a GP on \(\ln c(x)\), is cost-aware but is not itself a “log expected improvement per cost” acquisition in the sense of applying a logarithm to EI.

## 2. Closed form induced by a GP on the log-transformed objective

The log-transformed-objective construction assumes a positive objective and defines
\[
g(x)=\log f(x), \qquad g(x)\mid D \sim \mathcal N(\mu(x),\sigma(x)^2).
\]
If \(f^\star\) denotes the incumbent on the original scale, the improvement is measured on that same original scale:
\[
\alpha_{\log \mathrm{EI}}(x)=\mathbb E\!\left[(f^\star-e^{g(x)})_+\right].
\]
This is not EI in log space, because the improvement term is not \((\log f^\star-g(x))_+\). It is also not \(\log(\mathrm{EI}(x))\). It is expected improvement on the original objective under a predictive model that is Gaussian in \(g=\log f\) and therefore log-normal in \(f=e^g\) [2411.18095].

With
\[
z(x)=\frac{\log f^\star-\mu(x)}{\sigma(x)},
\]
the paper derives the closed form
\[
\alpha_{\log \mathrm{EI}}(x)
=
f^\star \Phi(z(x))
-
\exp\!\left(\mu(x)+\frac{\sigma(x)^2}{2}\right)\Phi(z(x)-\sigma(x)).
\]
The derivation proceeds by rewriting the expectation as an integral over the latent Gaussian variable \(g\), changing variables to a standard normal, splitting the integral into a tractable term and a nontrivial Gaussian-exponential term, and then evaluating the latter by completing the square. The resulting expression matches the Hutter et al. proposition whose intermediate derivation had not previously been provided in full [2411.18095].

The construction is motivated by the observation that ordinary EI can be sensitive to numerical precision, and that fitting a GP on the log-transformed objective was reported to improve predictive accuracy and substantially improve Bayesian-optimization performance. The rationale is that positive objectives encountered in hyperparameter optimization often vary across orders of magnitude, so modeling \(\log y\) can make the target more Gaussian-like and stabilize variation, thereby improving posterior calibration.

## 3. Direct construction of expected improvement per cost in the log-transformed-objective sense

Once \(\alpha_{\log \mathrm{EI}}(x)\) is available as a scalar expected utility, the most direct cost-aware extension is the ratio
\[
\alpha_{\log \mathrm{EI}/c}(x)=\frac{\alpha_{\log \mathrm{EI}}(x)}{c(x)}
=
\frac{
f^\star \Phi(z(x))
-
\exp\!\left(\mu(x)+\frac{\sigma(x)^2}{2}\right)\Phi(z(x)-\sigma(x))
}{c(x)},
\]
provided \(c(x)>0\). This is the natural analogue of EI-per-cost for the Hutter-style acquisition [2411.18095].

Its interpretation is precise. The numerator remains expected improvement on the original objective scale under a log-normal predictive model. Dividing by \(c(x)\) therefore yields expected original-scale improvement per unit cost, not expected log-space improvement per cost. This distinction is essential because the same label “logEI” is used differently in other parts of the literature.

Several caveats follow immediately from the construction. The objective must be positive so that \(\log f(x)\) is defined. The cost must be positive if one divides by it. If cost is uncertain and modeled probabilistically, one must choose whether the denominator is \(\mathbb E[c(x)]\), whether the acquisition is \(\mathbb E[\alpha(x)/c(x)]\), or whether a joint utility is used; these are not generally equivalent. A further practical implication is that if a software library already uses “logEI” to mean \(\log(\mathrm{EI})\), then dividing that quantity by cost is not the same operation as using the closed form above and normalizing by cost.

## 4. Unknown and heterogeneous costs: EI-per-unit-cost, log-cost modeling, and budgeted BO

In budgeted Bayesian optimization with unknown evaluation costs, the objective is to maximize the best objective value found before the cumulative cost exceeds a budget \(B\). The state is the data set \(D_n\) of observed triples \((x_i,y_i,z_i)\), the state utility is
\[
u(D_n)=\max_{(x,y,z)\in D_n} y,
\]
and the spent budget is
\[
s(D_n)=\sum_{(x,y,z)\in D_n} z.
\]
The stopping time is random because costs are not known until queried [2111.06537].

Within this framework, the paper studies classical EI, EI-per-unit-cost, and non-myopic budget-aware alternatives. Classical EI is
\[
\mathrm{EI}^{f}(x\mid D)
=
\Delta_D(x)\Phi\!\left(\frac{\Delta_D(x)}{\sigma_D^f(x)}\right)
+
\sigma_D^f(x)\varphi\!\left(\frac{\Delta_D(x)}{\sigma_D^f(x)}\right),
\qquad
\Delta_D(x)=\mu_D^f(x)-u(D),
\]
or equivalently
\[
\mathrm{EI}^f(x\mid D)=\mathbb E\bigl[(f(x)-u(D))^+\mid D\bigr].
\]

For unknown costs, the paper defines EI-per-unit-cost as
\[
\mathrm{EI\textnormal{-}PUC}(x\mid D)
=
\mathbb E_D\left[\frac{\{f(x)-f_n^*\}^+}{c(x)}\right].
\]
Under independent GP models for \(f\) and \(\ln c\), a more general moment identity is
\[
\mathbb{E}_D\left[\frac{\{f(x)-f_n^*\}^+}{c(x)^\nu}\right]
=
\mathrm{EI}^{f}(x\mid D)
\exp\!\left(
-\nu \mu_D^{\ln c}(x)
+\frac{\nu^2(\sigma_D^{\ln c}(x))^2}{2}
\right).
\]
Setting \(\nu=1\) yields
\[
\mathrm{EI\textnormal{-}PUC}(x\mid D)
=
\mathrm{EI}^{f}(x\mid D)
\exp\!\left(
-\mu_D^{\ln c}(x)
+\frac{(\sigma_D^{\ln c}(x))^2}{2}
\right).
\]
Here the logarithm belongs to the cost surrogate model, not to the improvement term. The GP prior on \(\ln c(x)\) ensures positivity of \(c(x)\) and makes inverse-cost moments lognormal and therefore analytically tractable.

The same paper shows that myopic cost handling is fundamentally limited. It proves that the approximation ratios of EI and EI-PUC are unbounded: for any arbitrarily large \(\rho>0\), there exists a Bayesian-optimization instance on which the optimal policy value exceeds \(\rho\) times the value obtained by either EI or EI-PUC. This theorem formalizes the failure modes of both “ignore cost” and “divide by cost” heuristics under unknown heterogeneous costs and a budget constraint.

As a principled one-step alternative, the paper defines
\[
Q_1(x\mid D)
=
\mathbb E_{y,z}\left[(y-u(D))^+\,1\{s(D)+z\le B\}\right],
\]
and under independent GP priors for \(f\) and \(\ln c\),
\[
Q_1(x\mid D)
=
\mathrm{EI}^{f}(x\mid D)\,\Phi(\zeta)\,1\{s(D)\le B\},
\qquad
\zeta=
\frac{\ln(B-s(D))-\mu_D^{\ln c}(x)}{\sigma_D^{\ln c}(x)}.
\]
This is not EI divided by cost. It is standard EI multiplied by the posterior probability that the next evaluation cost fits in the remaining budget. The full proposed acquisition, Budgeted Multi-Step Expected Improvement, extends this principle non-myopically through a Bellman recursion over future observations and remaining budget.

## 5. LogEI as \(\log(\mathrm{EI})\) and the difference-of-logs per-cost extrapolation

Modern LogEI begins from the standard analytic EI formula
\[
\mathrm{EI}_{y^*}(x)=\sigma(x)\,h\!\left(\frac{\mu(x)-y^*}{\sigma(x)}\right),
\qquad
h(z)=\phi(z)+z\Phi(z),
\]
and defines
\[
\mathrm{LogEI}_{y^*}(x)
=
\mathrm{log\_h}\!\left(\frac{\mu(x)-y^*}{\sigma(x)}\right)
+
\log \sigma(x),
\]
where \(\mathrm{log\_h}(z)\) is a numerically stable implementation mathematically equivalent to \(\log h(z)\). For analytic EI, this preserves maximizers whenever \(\max_x \mathrm{EI}(x)>0\), because \(\log\) is strictly increasing on \(\mathbb R_{>0}\) [2310.20708].

The numerical motivation is that ordinary EI often becomes exactly zero in floating-point arithmetic for sufficiently negative standardized improvement, and its gradients then vanish over large parts of the domain. The stable log-domain implementation avoids this pathology. The paper derives piecewise formulas for \(\mathrm{log\_h}(z)\), uses \(\mathrm{erfcx}\) and \(\mathrm{log1mexp}\) for stable evaluation, and shows that for large negative \(z\),
\[
\log h(z)\approx -\frac{z^2}{2}-\frac{\log(2\pi)}{2}-2\log|z|,
\]
which produces a numerically healthier optimization surface than underflowed EI.

The paper does not define a cost-aware LogEI acquisition. A plausible implication, however, is immediate from the same monotonicity argument: if \(c(x)>0\), then
\[
\arg\max_x \frac{\mathrm{EI}(x)}{c(x)}
=
\arg\max_x \bigl(\log \mathrm{EI}(x)-\log c(x)\bigr).
\]
This suggests the sequential construction
\[
\mathrm{LogEI\mbox{-}per\mbox{-}Cost}(x)
=
\mathrm{LogEI}(x)-\log c(x)
=
\mathrm{log\_h}\!\left(\frac{\mu(x)-y^*}{\sigma(x)}\right)
+
\log \sigma(x)
-
\log c(x),
\]
for deterministic positive cost or a positive plug-in estimate \(\hat c(x)\). In this sense, “log expected improvement per cost” means a difference of logs rather than a ratio in primal space. This extrapolation is mathematically natural but is not itself proposed as an acquisition in the paper [2310.20708].

## 6. Assumptions, convention mismatches, and implementation constraints

**Objective positivity.** The log-transformed-objective construction requires \(f(x)>0\), because \(g(x)=\log f(x)\) is modeled directly. If the objective can be nonpositive, a shift or a different transform would be required, but that case is not treated in the derivation note [2411.18095].

**Cost positivity.** Cost-aware formulations based on division by cost require \(c(x)>0\). Modeling \(\ln c(x)\) with a GP automatically enforces this and also yields closed forms for inverse-cost moments and budget-feasibility probabilities. For log-domain per-cost constructions such as \(\mathrm{LogEI}(x)-\log c(x)\), positivity is again essential because the logarithm must be defined [2111.06537] [2310.20708].

**Objective orientation.** The derivation note states a maximization convention with \(y^\star=\max_n y_n\), yet its formulas are written using
\[
(y^\star-y)_+
\quad\text{and}\quad
\int_{-\infty}^{y^\star}(y^\star-y)p(y)\,dy,
\]
which are structurally the usual improvement expressions for minimization. The mathematics is internally consistent once the improvement convention is fixed, but any implementation must make explicit whether the problem is a minimization or maximization problem and adapt signs or thresholds accordingly [2411.18095].

**Gaussianity and independence assumptions.** The closed form for \(\alpha_{\log \mathrm{EI}}(x)\) depends on the posterior assumption
\[
g(x)\mid D\sim \mathcal N(\mu(x),\sigma(x)^2).
\]
If the surrogate posterior is not Gaussian in log space, the derivation does not apply exactly. In the budgeted-BO analysis, closed forms for EI-PUC and \(Q_1\) use independent GP models for \(f\) and \(\ln c\), although the Monte Carlo implementation discussed there does not require independence and could accommodate correlated models [2411.18095] [2111.06537].

**Numerical stability.** Direct computation of the log-transformed-objective acquisition requires care when \(\sigma(x)=0\), when \(\exp(\mu(x)+\sigma(x)^2/2)\) may overflow, and when \(c(x)\) is very small in a per-cost ratio. Modern LogEI avoids different pathologies by never computing tiny EI values in primal space; instead it works directly with \(\mathrm{log\_h}(z)\), \(\mathrm{log1mexp}\), \(\mathrm{erfcx}\), and related log-domain primitives. A plausible implication is that the two traditions solve different numerical problems: the Hutter-style construction changes the surrogate model, whereas modern LogEI changes the numerical representation of the acquisition [2411.18095] [2310.20708].

Source: https://www.emergentmind.com/topics/log-expected-improvement-per-cost