Adaptive multi-fidelity optimization with fast learning rates

Abstract: In multi-fidelity optimization, biased approximations of varying costs of the target function are available. This paper studies the problem of optimizing a locally smooth function with a limited budget, where the learner has to make a tradeoff between the cost and the bias of these approximations. We first prove lower bounds for the simple regret under different assumptions on the fidelities, based on a cost-to-bias function. We then present the Kometo algorithm which achieves, with additional logarithmic factors, the same rates without any knowledge of the function smoothness and fidelity assumptions, and improves previously proven guarantees. We finally empirically show that our algorithm outperforms previous multi-fidelity optimization methods without the knowledge of problem-dependent parameters.

• The paper introduces the Kometo algorithm that robustly attains minimax regret rates in multi-fidelity optimization under general cost-to-bias conditions.
• It leverages hierarchical partitioning and rank-based selection to adaptively balance budget allocation without prior smoothness or bias knowledge.
• Empirical results demonstrate significant improvements over existing methods in tasks such as SVM hyperparameter tuning and complex simulation benchmarks.

Adaptive Multi-Fidelity Optimization with Fast Learning Rates: An Expert Analysis

Introduction and Problem Formulation

This paper addresses adaptive derivative-free optimization in the multi-fidelity setting, where access to the objective function is provided through biased approximations of varying costs, with the ultimate aim of minimizing simple regret under a fixed evaluation budget. A central challenge in this paradigm is the necessity to allocate budget judiciously across function evaluations of different fidelities, in the absence of knowledge about the bias-cost relationship or the smoothness of the true function.

The authors introduce a framework that generalizes prior multi-fidelity literature by assuming only local smoothness of the target and positing a highly flexible, essentially unknown cost-to-bias function $\Phi(c)$, encompassing polynomial, exponential, and finite-support decay. This relaxation is practically significant, as it reflects the realistic scenario in hyperparameter tuning and scientific simulation where the bias decrease in function approximations with computation is problem-dependent and not a priori characterized.

Lower Bound Analysis

The paper provides the first minimax lower bounds on simple regret in adaptive multi-fidelity optimization under general assumptions about the cost-to-bias decay. For a target function $f$ with near-optimality dimension $d$ and a cost-to-bias function $\Phi(c)$, the authors rigorously establish:

• For polynomial decay ($\Phi(c) \leq A/c^\alpha$): minimax simple regret decays as $\Omega(\Lambda^{-1/(d + 1/\alpha)})$
• For exponential decay ($\Phi(c) \leq B\exp(-c^\beta/\sigma)$): for $d=0$, the minimax rate is exponential in the budget, specifically $e^{-\Theta(\Lambda^{\beta/(1+\beta)})}$.
• For the finite-support case: regret is at least $\Omega(e^{-D\Lambda})$ for $f$0.

These constructions are non-trivial, leveraging tailored trees of partitioned input spaces and distributions over approximations, and they extend prior theory that was largely restricted to known parametric bias models or smoothness.

The KOMETO Algorithm

The principal algorithmic contribution is Kometo, a rank-based multi-fidelity optimization method that does not rely on any knowledge of the bias-cost mapping or on smoothness parameters. Unlike preceding works (e.g., MFPDOO), Kometo only requires the fidelity cost function as input and adapts automatically to unknown fidelity approximation behavior.

The core insight of Kometo is to operate on hierarchical partitions of the input space, dynamically allocating budget to cells and fidelities in a manner inspired by Zipf-sampling and variants of StroquOOL. At every iteration, the algorithm prioritizes points purely based on ranking evaluations within the same fidelity level, eschewing reliance on direct bias estimation.

Figure 1: Empirical evaluation of Kometo versus prior multi- and single-fidelity algorithms across synthetic and real-world tasks with varying dimensions and fidelity structures.

This design ensures theoretical robustness even when the shape of the cost-to-bias function is highly irregular, and it circumvents issues that plague value-based approaches when the additive error can have arbitrary monotonic, strictly increasing transformations.

Theoretical Guarantees

Kometo achieves, up to logarithmic factors, the minimax rates prescribed by the lower bounds, regardless of the particular decay structure of $f$1 and without access to smoothness or bias parameters. In detail:

• For polynomial decays, Kometo attains $f$2 simple regret.
• For exponential cost-to-bias ($f$3), simple regret decays as $f$4 up to log terms.
• In the finite-support regime, Kometo matches the exponential decay rate achieved by smoothness-aware single-fidelity algorithms, without requiring prior smoothness knowledge (in contrast to, e.g., DOO or MFPDOO).

The analysis hinges on a careful coupling of the hierarchical cell activation with fidelity selection, and uses the fact that Kometo's open-cell actions are determined by within-fidelity ranks, making the algorithm invariant to monotonic distortions in $f$5.

Empirical Evaluation

The empirical results demonstrate Kometo's consistent performance and adaptivity in both synthetic and real-world (SVM hyperparameter tuning) scenarios. Kometo significantly outperforms MFPDOO in several low- and moderate-dimensional tasks, especially where the relationship between fidelity and bias is complex and unknown, as typified by Hartman3d, Branin, and Curin benchmarks.

Interestingly, on some instances where high-fidelity evaluations dominate (Borehole, Hartman6d), single-fidelity-optimized SequOOL is competitive or even superior, reflecting the theoretical additional logarithmic factor in Kometo’s guarantees for those regimes.

In the real-world SVM experiment, Kometo’s adaptivity to fidelity cost—balanced exploration at lower fidelities with targeted exploitation—leads to superior accuracy at lower cost, highlighting the practical utility of the approach in settings where evaluation is both expensive and uncertainty in bias structure is pronounced.

Discussion, Limitations, and Future Directions

The fully parameter-free and rank-based design of Kometo supports robust application to novel domains where fidelity structure is a priori unknown or difficult to model. Because Kometo does not require knowledge of smoothness or bias, it can be expected to have broad impact in derivative-free optimization settings encountered in large-scale machine learning, simulation-based optimization, and scientific computing.

The discrete and adaptive nature of partitioning and cell evaluation supports direct generalization to stochastic feedback cases, assuming the noise can be controlled at higher fidelities by increasing sample counts per cell. Hence, a promising direction is to extend the analysis to cumulative regret minimization or settings with heavy-tailed/intractable noise distributions, refining the $f$6 adaptation accordingly.

Another critical direction, as noted by the authors, is to further close the remaining gap in settings where cumulative (not simple) regret is of interest and to characterize the conditions that distinguish minimax optimality for cumulative versus simple regret in multi-fidelity environments.

Conclusion

This work provides a rigorous and general answer to the fundamental question of optimal budget allocation for black-box optimization when only cost-annotated, biased approximations are available. The Kometo algorithm advances the state-of-the-art by dispensing with restrictive bias and smoothness knowledge, robustly attaining minimax regret rates in a practical and parameter-free manner, as validated both in theory and empirical analysis. The findings have substantial implications for robust, scalable hyperparameter tuning and more generally for automated scientific discovery where ground-truth access is infeasible and approximation biases defy parametric modeling.