Multi-fidelity Bayesian Optimisation with Continuous Approximations (1703.06240v1)

Abstract: Bandit methods for black-box optimisation, such as Bayesian optimisation, are used in a variety of applications including hyper-parameter tuning and experiment design. Recently, \emph{multi-fidelity} methods have garnered considerable attention since function evaluations have become increasingly expensive in such applications. Multi-fidelity methods use cheap approximations to the function of interest to speed up the overall optimisation process. However, most multi-fidelity methods assume only a finite number of approximations. In many practical applications however, a continuous spectrum of approximations might be available. For instance, when tuning an expensive neural network, one might choose to approximate the cross validation performance using less data $N$ and/or few training iterations $T$. Here, the approximations are best viewed as arising out of a continuous two dimensional space $(N,T)$. In this work, we develop a Bayesian optimisation method, BOCA, for this setting. We characterise its theoretical properties and show that it achieves better regret than than strategies which ignore the approximations. BOCA outperforms several other baselines in synthetic and real experiments.

• The paper presents a novel multi-fidelity framework that uses continuous approximations to balance evaluation cost and information gain.
• It develops a Gaussian process-based method that adaptively selects fidelity levels and achieves better regret bounds.
• Empirical results on synthetic and real-world tasks demonstrate BOCA’s efficiency in reducing computational costs while maintaining optimisation accuracy.

Multi-fidelity Bayesian Optimisation with Continuous Approximations

The paper "Multi-fidelity Bayesian Optimisation with Continuous Approximations" presents a comprehensive paper of a Bayesian optimisation method, BOCA, designed to address optimisation scenarios where a continuous spectrum of approximations is available. This research is particularly relevant for applications where function evaluations are computationally expensive, such as hyper-parameter tuning and experiment design.

Key Contributions

1. Novel Optimisation Setting: The paper introduces a multi-fidelity optimisation framework that leverages continuous approximations. This is a significant departure from conventional methods that assume a finite set of fidelities. The approach is particularly suited for scenarios where fidelity can be adjusted continuously, such as in machine learning applications where fewer training data points or iterations can be used as cheaper approximations.
2. Theoretical Characterisation: The authors provide a theoretical analysis showing that their algorithm, BOCA, achieves better regret bounds compared to strategies that ignore fidelity approximations. The regret, a measure of the algorithm's performance over time, is shown to be reduced by leveraging the fidelity space's smoothness. This analysis includes evaluating the informational benefits of lower-fidelity approximations and balancing them against their computational costs.
3. Empirical Evaluation: The paper discusses empirical results demonstrating that BOCA outperforms existing methods on both synthetic and real-world problems, including astrophysics applications and hyper-parameter tuning tasks. These results illustrate BOCA's practical efficacy in reducing computational costs while maintaining optimisation accuracy.

Methodology

The theoretical foundation of BOCA is built on a multi-fidelity Gaussian process framework. It utilises a fidelity space and models the function as a slice of a higher-dimensional function defined over this space. The proposed method introduces a mechanism for selecting fidelity levels based on information gain and computational cost, enabling the algorithm to adaptively query lower fidelities to guide the search for the function's optimum effectively.

The algorithmic process involves constructing an upper confidence bound based on the Gaussian process posterior and selecting the next point to evaluate by maximising this bound. The choice of fidelity is driven by balancing the fidelity's cost and informational value, ensuring that cheaper fidelities are explored before more expensive ones.

Implications and Future Directions

The research has several notable implications:

• Efficiency in High-cost Applications: BOCA presents a significant opportunity for resource-efficient optimisation in machine learning and scientific computation, where evaluation costs are prohibitive.
• General Applicability: While the paper primarily focuses on continuous fidelity spaces, the approach could be extended to handle discrete fidelities and other forms of fidelity spaces, broadening its applicability.
• Robustness to Fidelity Design: Unlike some existing methods, BOCA's performance does not critically depend on the precise design or discretisation of fidelity levels, which is advantageous in complex real-world scenarios where such design might be non-trivial.

Future research could focus on extending BOCA's applicability to more general fidelity spaces and refining theoretical results to consider finite dimensional kernels. The exploration of kernel choice's impact on performance and further empirical validation in varied domains would further strengthen the understanding and utility of multi-fidelity Bayesian optimisation frameworks like BOCA.