Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs (2509.17998v2)

Abstract: The efficiency of Bayesian optimization (BO) relies heavily on the choice of the Gaussian process (GP) kernel, which plays a central role in balancing exploration and exploitation under limited evaluation budgets. Traditional BO methods often rely on fixed or heuristic kernel selection strategies, which can result in slow convergence or suboptimal solutions when the chosen kernel is poorly suited to the underlying objective function. To address this limitation, we propose a freshly-baked Context-Aware Kernel Evolution (CAKE) to enhance BO with LLMs. Concretely, CAKE leverages LLMs as the crossover and mutation operators to adaptively generate and refine GP kernels based on the observed data throughout the optimization process. To maximize the power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to select the most effective kernel through balancing the model fit measured by the Bayesian information criterion (BIC) with the expected improvement at each iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO method consistently outperforms established baselines across a range of real-world tasks, including hyperparameter optimization, controller tuning, and photonic chip design. Our code is publicly available at https://github.com/richardcsuwandi/cake.

• The paper demonstrates that CAKE uses LLMs as genetic operators to dynamically evolve kernel structures for Bayesian optimization.
• It employs a kernel grammar and Bayesian Information Criterion to refine operations and balance exploration with exploitation.
• Experiments show significant improvements in hyperparameter tuning, controller tasks, and photonic chip design compared to fixed kernel methods.

Adaptive Kernel Design for Bayesian Optimization: A Context-Aware Kernel Evolution (CAKE) Approach

Introduction

The paper "Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs" presents an innovative approach to enhancing the efficacy of Bayesian optimization (BO) by focusing on improving kernel selection strategies. The central premise is that traditional methods often rely on fixed or heuristic kernel choices, which can lead to inefficiencies and suboptimal results. The authors propose the Context-Aware Kernel Evolution (CAKE) system that integrates LLMs to dynamically generate and refine Gaussian Process (GP) kernels based on continuous observation data. This adaptive framework aims to optimize BO performance across diverse tasks by tailoring the kernel design process to the specific characteristics of the optimization problem at hand.

Context-Aware Kernel Evolution

CAKE utilizes LLMs to serve as genetic operators—crossover and mutation—thereby creating and evolving kernels in an adaptive manner throughout the BO process. The system starts with an initial population of kernel candidates and leverages the LLM to propose new kernels iteratively.

• Kernel Grammar Utilization: By employing a kernel grammar-based approach, CAKE constructs a search space that allows the combination of base kernels like squared exponential, linear, and periodic through operations that maintain positive definiteness, such as addition and multiplication.
• Adaptive Kernel Generation: The LLM processes observational data to suggest kernel modifications, which are further assessed for fitness using the Bayesian Information Criterion (BIC). This process refines the model to balance exploration and exploitation effectively.

  Figure 1: Overview of CAKE. Starting with an initial population of kernels, the LLM acts as crossover and mutation operators, proposing new kernels based on the given prompts. The proposed kernels are then evaluated using a fitness calculator, and the fittest ones advance to the next generation.

Experimental Evaluation

The authors conduct extensive experiments to validate the effectiveness of CAKE, comparing it against several baseline methods across different real-world optimization tasks, including hyperparameter optimization, controller tuning, and photonic chip design.

• Hyperparameter Optimization: CAKE outperforms baseline methods in optimizing the hyperparameters of machine learning models, demonstrating significant improvements in test accuracy over 20 random seeds.

  Figure 2: Average test accuracy over 20 random seeds for different ML models.

• Controller Tuning and Photonic Chip Design: Experiments in dynamic environments and multi-objective settings show CAKE’s superior adaptability and efficiency in advancing to high-scoring solutions with fewer trials, thus indicating its robustness and generalization capability.

  Figure 3: Average reward for the controller tuning tasks over 10 different initial conditions.

  

  Figure 4: Average score and hypervolume of the designed chip over 250 trials.

The results emphasize CAKE's advantage in efficiently discovering optimal solutions by iteratively refining its kernel choice through data-driven insights. The experiments confirm CAKE's utility in rapidly converging to high-performance solutions compared to static or heuristic kernel selection methods.

Ablation Study

An ablation paper is conducted to assess the importance of different components of CAKE. This paper reveals that both the LLM-driven kernel generation and the BIC-Acquisition Kernel Ranking (BAKER) contribute significantly to the approach’s success. The paper highlights the necessity of each component, confirming that the integrated method surpasses its individual parts.

Implications and Future Directions

The implications of this research are significant for both theoretical exploration and practical application in machine learning and optimization fields. The integration of LLMs in adaptive kernel generation suggests a new direction for designing models that can dynamically adapt to diverse and complex data environments.

Future research could explore the extension of CAKE’s kernel grammar to include additional operations like convolution and composition to further augment the expressiveness of the kernel search space. Moreover, adapting CAKE for broader machine learning tasks outside of BO, such as SVM-based regression or kernel PCA, could open new avenues for research.

Conclusion

The CAKE methodology represents a promising advancement in kernel design for Bayesian optimization, leveraging the power of LLMs to autonomously evolve and optimize kernel structures. The ability to adaptively generate expressive kernels on-the-fly marks a significant shift from conventional fixed kernel approaches, leading to enhanced performance across various optimization tasks. As LLM capabilities continue to improve, CAKE and related techniques are poised to become integral components of advanced optimization toolsets.