BAKER: BIC-Acquisition Kernel Ranking

• BAKER is a kernel selection mechanism for Bayesian optimization that integrates BIC-based model fit with expected improvement acquisition to choose optimal Gaussian process kernels.
• It computes an exponential BIC weight for each candidate kernel and multiplies it by a normalized acquisition value to balance model fidelity and immediate optimization utility.
• Integrated within the CAKE framework, BAKER outperforms BIC-only and utility-only strategies, demonstrating improved BO performance across hyperparameter tuning, controller tuning, and design tasks.

Searching arXiv for the cited BAKER/CAKE and related BO kernel–acquisition selection papers. BIC-Acquisition Kernel Ranking (BAKER) is a kernel-selection mechanism for Bayesian optimization (BO) introduced within the CAKE framework, where a population of candidate Gaussian process (GP) kernels is evolved and then ranked using a joint criterion that combines the Bayesian Information Criterion (BIC) with an acquisition value, by default expected improvement (EI). Its defining purpose is to convert a set of candidate kernels into an actual BO decision rule at each iteration: for each candidate kernel, a GP is fitted to the current data, BIC is computed as a model fit and complexity term, the acquisition function is optimized under that kernel to obtain a proposed query point, and the kernel is selected according to a weighted acquisition score derived from both ingredients (Suwandi et al., 22 Sep 2025). In the formulation reported for CAKE, this ranking is intended to balance model fidelity and BO utility, addressing the observation that a kernel with strong fit need not propose a query with strong actual improvement, while a kernel with a high acquisition peak may be poorly calibrated or overly flexible (Suwandi et al., 22 Sep 2025).

1. Definition and role within Bayesian optimization

BAKER operates in a BO setting with current dataset $\mathcal{D}$ and a population of candidate kernels $\mathbb{K}$. For each kernel $k \in \mathbb{K}$, a GP model is fitted to $\mathcal{D}$, a model-selection score based on BIC is evaluated, and an acquisition function $\alpha(\cdot; \mathcal{D}, k)$ is optimized to obtain a candidate point $\mathbf{x}_{t,k}$ and its associated acquisition value (Suwandi et al., 22 Sep 2025). BAKER then ranks kernels by combining a model fit and complexity term derived from BIC with the acquisition value, and selects the kernel whose weighted acquisition score is maximal (Suwandi et al., 22 Sep 2025).

The mechanism is designed for the CAKE setting, where the candidate set $\mathbb{K}$ is not fixed but is instead an evolving population of kernels generated by an LLM-guided procedure over a compositional kernel grammar with base kernels $$\{\text{SE},\text{PER},\text{LIN},\text{RQ},\text{M3},\text{M5}\}$$ and operators $\{+, \times\}$ (Suwandi et al., 22 Sep 2025). Within that architecture, CAKE is responsible for generating and evolving the kernel population, while BAKER is the decision rule that chooses which kernel’s proposed point is actually evaluated at the current BO iteration (Suwandi et al., 22 Sep 2025).

This division of labor distinguishes BAKER from approaches that either use a single fixed kernel throughout BO or perform kernel selection using only one criterion. The underlying motivation is explicit: a kernel that appears favorable under a model-selection criterion alone may not generate a useful BO step, while a kernel selected solely by acquisition can favor misspecified or over-flexible models that produce optimistic acquisition peaks (Suwandi et al., 22 Sep 2025). A plausible implication is that BAKER should be understood as a one-step policy for kernel-conditioned query selection rather than as a standalone kernel-learning framework.

2. Mathematical formulation

The core ranking rule is defined by first converting each kernel’s BIC into a weight: $$w_k = \frac{\exp(-\mathrm{BIC}_k)}{\sum_{k' \in \mathbb{K}} \exp(-\mathrm{BIC}_{k'})},$$ where $\mathbb{K}$0 is the BIC of the GP model fitted with kernel $\mathbb{K}$1 (Suwandi et al., 22 Sep 2025). For each kernel, the acquisition function is then optimized to obtain a candidate point $\mathbb{K}$2, and the weighted acquisition score is

$\mathbb{K}$3

BAKER selects

$\mathbb{K}$4

and the BO query is set to $\mathbb{K}$5 (Suwandi et al., 22 Sep 2025).

The BIC term is motivated as a simpler proxy for marginal log likelihood based on the Laplace approximation. The reported standard form is

$\mathbb{K}$6

where $\mathbb{K}$7 is the likelihood under a GP with kernel $\mathbb{K}$8 and fitted hyperparameters $\mathbb{K}$9, $k \in \mathbb{K}$0 is the number of kernel hyperparameters, and $k \in \mathbb{K}$1 is the number of observations (Suwandi et al., 22 Sep 2025). The corresponding Laplace-approximation expression is given as

$k \in \mathbb{K}$2

with $k \in \mathbb{K}$3 (Suwandi et al., 22 Sep 2025).

The default acquisition is expected improvement. In the BO preliminaries associated with CAKE, EI is written as

$k \in \mathbb{K}$4

with

$k \in \mathbb{K}$5

where $k \in \mathbb{K}$6 and $k \in \mathbb{K}$7 are the GP posterior mean and variance and $k \in \mathbb{K}$8 is the best posterior mean so far (Suwandi et al., 22 Sep 2025). In the BAKER step, EI is normalized to $k \in \mathbb{K}$9 to ensure comparability across different kernels before it is multiplied by the BIC-derived weight (Suwandi et al., 22 Sep 2025).

The interpretation supplied in the source is that $\mathcal{D}$0 is proportional to an approximate marginal likelihood, so $\mathcal{D}$1 can be viewed as an approximate model posterior over kernels (Suwandi et al., 22 Sep 2025). This suggests that BAKER approximates a model-probability-weighted utility comparison over candidate kernels, but resolves the choice by a hard $\mathcal{D}$2 rather than by model averaging.

3. Algorithmic integration with CAKE

Within CAKE, BAKER is executed after the kernel population has been evolved and filtered. At each BO iteration, CAKE updates the LLM context with the current data $\mathcal{D}$3, performs $\mathcal{D}$4 LLM-driven crossovers and optional mutation with probability $\mathcal{D}$5, evaluates all kernels by fitting a GP and computing BIC-based fitness, normalizes that fitness to $\mathcal{D}$6, and retains the top $\mathcal{D}$7 kernels in $\mathcal{D}$8 (Suwandi et al., 22 Sep 2025). BAKER then ranks these surviving kernels using the BIC–acquisition rule and chooses the actual query point (Suwandi et al., 22 Sep 2025).

The stepwise role of BAKER inside one BO iteration is reported as follows. For each surviving kernel $\mathcal{D}$9, compute $\alpha(\cdot; \mathcal{D}, k)$0, form the weight $\alpha(\cdot; \mathcal{D}, k)$1, optimize the acquisition function to obtain $\alpha(\cdot; \mathcal{D}, k)$2, evaluate the normalized acquisition value at that point, and select the kernel maximizing $\alpha(\cdot; \mathcal{D}, k)$3 (Suwandi et al., 22 Sep 2025). The selected point $\alpha(\cdot; \mathcal{D}, k)$4 is then evaluated with the true objective, producing $\alpha(\cdot; \mathcal{D}, k)$5, and the dataset is updated by $\alpha(\cdot; \mathcal{D}, k)$6 (Suwandi et al., 22 Sep 2025).

A notable design property is that BAKER does not influence how the LLM proposes kernels; it is downstream of the kernel-generation stage (Suwandi et al., 22 Sep 2025). It also does not maintain an ensemble prediction model: despite the use of normalized BIC weights, it selects a single kernel per BO iteration rather than averaging predictions or acquisitions over the kernel population (Suwandi et al., 22 Sep 2025).

This architecture places BAKER in a specific operational niche. It is neither a pure model-selection criterion nor a pure acquisition-based selector. Instead, it is the rule that maps a multi-kernel candidate set into a single BO action. A plausible implication is that its behavior depends jointly on three factors: the quality of hyperparameter fitting for each GP, the shape and calibration of the acquisition under each kernel, and the diversity and expressiveness of the current candidate population.

4. Relation to other kernel and acquisition selection strategies

The immediate contrast in the CAKE work is with two single-axis selection rules. A BIC-only strategy chooses

$\alpha(\cdot; \mathcal{D}, k)$7

while a utility-only strategy chooses

$\alpha(\cdot; \mathcal{D}, k)$8

without regard to global fit or complexity (Suwandi et al., 22 Sep 2025). BAKER is introduced because these alternatives can fail in opposite ways: BIC-only selection is agnostic to whether the corresponding acquisition suggests an effective next point, whereas utility-only selection can prefer overly optimistic kernels whose high acquisition values are not supported by model adequacy (Suwandi et al., 22 Sep 2025).

A useful point of comparison is BOOST, which also performs joint kernel–acquisition selection in BO but does not use BIC, AIC, marginal likelihood, or any explicit information criterion (Park et al., 4 Aug 2025). BOOST ranks each candidate $\alpha(\cdot; \mathcal{D}, k)$9 pair by retrospective internal BO performance: it partitions the current dataset into a reference subset and a query subset, runs an internal BO simulation over the held-out query subset, and scores each pair by the number of internal steps needed to hit a target value, selecting the pair with minimal internal effort (Park et al., 4 Aug 2025). The score is therefore explicitly behavioral and optimization-oriented rather than information-criterion-based (Park et al., 4 Aug 2025).

This yields a sharp conceptual distinction. BOOST asks, in effect, how fast a given kernel–acquisition pair would have found a good point on retrospectively hidden data (Park et al., 4 Aug 2025). BAKER asks which kernel currently best balances approximate model evidence, via BIC, with the acquisition value of the next point that kernel proposes (Suwandi et al., 22 Sep 2025). Both use only already observed data, but they represent different ranking philosophies: BOOST is a retrospective simulation of search efficiency, whereas BAKER is an analytical model-selection criterion coupled to a one-step acquisition utility term (Park et al., 4 Aug 2025, Suwandi et al., 22 Sep 2025).

Another relevant line of work concerns the interaction between kernel structure and acquisition optimization in high-dimensional BO. The study on composition of kernel and acquisition functions for high-dimensional BO argues that additive kernel structure can make acquisition optimization substantially more tractable by decomposing the GP and the acquisition over low-dimensional subspaces, particularly for Thompson sampling and also for EI and LCB/UCB under the additive assumption (Candelieri et al., 2020). That work does not propose BIC-based ranking, but it establishes that kernel structure affects not only model fit but also the geometry and optimization difficulty of the acquisition function (Candelieri et al., 2020). This supports the broader premise behind BAKER that kernel choice should not be evaluated in isolation from acquisition behavior.

5. Experimental behavior and empirical comparisons

The ablation experiments reported for CAKE compare five kernel-selection variants over 60 HPOBench tasks and 5 model families: Random Sampling, Genetic Algorithm, CAKE + BIC, CAKE + Utility, Adaptive + BAKER, CKS + BAKER, and CAKE + BAKER (Suwandi et al., 22 Sep 2025). The average ranks, with lower better, are reported as 6.80 for Random Sampling, 2.70 for Genetic Algorithm, 3.02 for CAKE + BIC, 2.40 for CAKE + Utility, 4.60 for Adaptive + BAKER, 3.12 for CKS + BAKER, and 1.04 for CAKE + BAKER (Suwandi et al., 22 Sep 2025).

These results are used to support three empirical claims. First, CAKE + BAKER outperforms CAKE + BIC and CAKE + Utility, indicating that the combination of BIC and acquisition is stronger than either criterion alone when the underlying kernel evolution mechanism is held fixed (Suwandi et al., 22 Sep 2025). Second, BAKER improves existing adaptive or compositional strategies when inserted as their kernel-selection rule, as shown by the Adaptive + BAKER and CKS + BAKER variants (Suwandi et al., 22 Sep 2025). Third, CAKE + BAKER is reported to consistently outperform established baselines across hyperparameter optimization, controller tuning, and photonic chip design tasks (Suwandi et al., 22 Sep 2025).

The main experiments further report that CAKE-based BO consistently outperforms baselines on real-world tasks including hyperparameter optimization, controller tuning, and photonic chip design (Suwandi et al., 22 Sep 2025). The article attributes the performance gains among CAKE + BIC, CAKE + Utility, and CAKE + BAKER specifically to the kernel-ranking rule, since that is the only component differing across those variants (Suwandi et al., 22 Sep 2025). This supports the interpretation of BAKER as an effective selection criterion for transforming an evolving kernel population into BO actions.

The evidence should nonetheless be interpreted in the scope of the reported experimental design. The candidate populations were generated by CAKE’s LLM-driven kernel evolution, with population size $\mathbf{x}_{t,k}$0, and BAKER ranked at most those surviving kernels at each iteration (Suwandi et al., 22 Sep 2025). A plausible implication is that BAKER’s observed gains depend not only on its ranking formula but also on the quality and diversity of the upstream kernel population.

6. Design assumptions, computational profile, and limitations

BAKER assumes that BIC is a meaningful proxy for predictive quality and model adequacy for the GP kernels under consideration (Suwandi et al., 22 Sep 2025). The source notes that this is often reasonable for moderately sized data and typical GP kernel families, but also identifies circumstances in which BIC may be less reliable, such as very small data regimes, where BIC may heavily penalize more expressive kernels, or highly noisy or misspecified settings (Suwandi et al., 22 Sep 2025). It also assumes that the acquisition function is sufficiently calibrated under each kernel so that its normalized value is informative about near-term BO utility (Suwandi et al., 22 Sep 2025).

The computational burden of BAKER is dominated by fitting multiple GPs and optimizing the acquisition under each kernel. The reported implementation notes that each kernel requires GP fitting, approximately 0.5 seconds per GP on the hardware used in the study, BIC computation, which is cheap once the GP is fitted, and acquisition optimization, whose overhead is similar across kernels (Suwandi et al., 22 Sep 2025). With population size $\mathbf{x}_{t,k}$1, this implies about 10 GP fits per BO iteration (Suwandi et al., 22 Sep 2025). The authors characterize BIC weighting itself as negligible in cost relative to the multi-kernel GP fits (Suwandi et al., 22 Sep 2025).

Several limitations are stated explicitly. BAKER selects a single kernel per iteration rather than performing full Bayesian model averaging over kernels, even though the normalized BIC weights $\mathbf{x}_{t,k}$2 have an approximate posterior interpretation (Suwandi et al., 22 Sep 2025). It relies specifically on BIC rather than alternatives such as AIC, cross-validated log likelihood, or fully Bayesian marginal likelihood, and the exploration of other criteria is left open (Suwandi et al., 22 Sep 2025). It also uses a fixed exponential mapping from BIC to weights with no tempering parameter; the source notes that if BIC values differ drastically, the resulting softmax may become overly peaked, effectively reverting toward BIC-only behavior (Suwandi et al., 22 Sep 2025).

A related limitation, implied by the broader BO literature, is that BAKER’s score is one-step and kernel-local. It measures approximate model evidence and the acquisition value of the current next point, but it does not directly simulate longer-horizon BO behavior. BOOST provides a contrasting design in which the ranking score is precisely the number of retrospective internal BO steps needed to hit a target on held-out data (Park et al., 4 Aug 2025). This suggests a methodological distinction rather than a defect: BAKER favors analytical simplicity and per-iteration decision making, while retrospective methods favor explicit search-efficiency proxies.

7. Position within the broader BO literature

Within the BO literature, BAKER occupies a space between classical GP model selection and adaptive multi-kernel BO. Traditional BO often uses a single fixed kernel such as SE or Matérn-5/2, which is simple but can be brittle when the chosen kernel is poorly matched to the underlying objective (Suwandi et al., 22 Sep 2025). Adaptive multi-kernel methods have instead maintained several GPs with different kernels and selected among them using random selection, BIC-only, or utility-only rules (Suwandi et al., 22 Sep 2025). Compositional kernel search methods, including CKS and related grammar-based approaches, search over kernel structures using marginal likelihood or BIC but do not explicitly interleave BO acquisition values with kernel ranking in the manner BAKER does (Suwandi et al., 22 Sep 2025).

The conceptual novelty claimed for BAKER is that kernel choice at each BO step is treated as a trade-off between approximate model evidence and acquisition utility, rather than as a pure model-selection problem or a pure acquisition maximization problem (Suwandi et al., 22 Sep 2025). The product $\mathbf{x}_{t,k}$3 is presented as a simple mechanism for implementing that trade-off (Suwandi et al., 22 Sep 2025). In this sense, BAKER can be read as an operational synthesis of two principles: kernels should be penalized for poor fit or excessive complexity, and kernels should also be evaluated by the quality of the BO action they induce.

Connections to work on additive kernels in high-dimensional BO sharpen this interpretation. The study on composition of kernel and acquisition functions argues that additive kernel structure can reduce acquisition optimization difficulty by lowering effective dimensionality and enabling subspace-wise optimization (Candelieri et al., 2020). Although that work does not propose BIC-Acquisition Kernel Ranking, it provides direct evidence that kernel structure influences acquisition tractability and BO behavior, not just surrogate-model fit (Candelieri et al., 2020). BAKER is consistent with that broader view because it explicitly incorporates acquisition value into kernel ranking.

BOOST offers another complementary perspective. Its retrospective scoring of joint $\mathbf{x}_{t,k}$4 pairs shows that kernel–acquisition interactions can be evaluated directly through simulated search behavior on observed data, and that such joint adaptive selection can outperform fixed configurations across synthetic and real-world tasks (Park et al., 4 Aug 2025). BAKER differs in mechanism but shares the premise that ranking kernels independently of acquisition is often insufficient for BO (Park et al., 4 Aug 2025, Suwandi et al., 22 Sep 2025).

Taken together, these threads position BAKER as a specifically BIC-centered answer to a broader methodological problem: BO kernel choice should be evaluated not only as a statistical modeling decision but also as a decision about how the acquisition function will behave under that kernel. In the CAKE framework, BAKER is the mechanism that formalizes that joint evaluation and converts an evolving kernel population into a concrete BO query rule (Suwandi et al., 22 Sep 2025).