Freeze-Thaw Bayesian Optimization

• Freeze-thaw Bayesian Optimization is a dynamic method that uses partial training data to predict full performance, balancing exploration and exploitation in iterative learning.
• It employs bespoke probabilistic surrogates with custom temporal covariance kernels to accurately model learning curves and quantify uncertainty.
• FT-BO uses cost-aware, entropy-reduction acquisition functions to adaptively allocate resources, enabling faster convergence with reduced computational overhead.

Freeze-thaw Bayesian Optimization (FT-BO) is a dynamic approach to hyperparameter optimization for iterative learning algorithms. It strategically exploits partial training information by alternately freezing and thawing model configurations, aiming to accelerate identification of high-performing hyperparameters while reducing computational cost. FT-BO is characterized by bespoke probabilistic surrogates, novel covariance kernels tailored for temporal learning curves, and acquisition functions based on information gain or cost-awareness.

1. Foundations of Freeze-Thaw Bayesian Optimization

FT-BO is designed for settings where model training is performed iteratively (e.g., stochastic gradient descent), providing partial information about model performance before completion. Unlike conventional Bayesian optimization—which treats each completed model run as a single black-box observation—FT-BO leverages intermediate loss values to forecast asymptotic performance and allocate resources adaptively.

The method maintains a pool of candidate configurations, some partially trained ("frozen") and others yet-to-be-evaluated. Decision-making alternates between starting new configurations and resuming training of promising ones. By dynamically reallocating epochs, FT-BO efficiently explores and exploits the hyperparameter space, quickly abandoning poor candidates and intensively refining those with favorable early signals (Swersky et al., 2014).

2. Probabilistic Surrogates and Temporal Covariance Kernels

A central technical innovation is hierarchical modeling of learning curves with Gaussian Processes (GPs):

• Temporal GP for training curves: For each hyperparameter setting, the evolution of loss (or another objective) over epochs is modeled with a custom covariance kernel. The canonical kernel is an infinite mixture of exponential decays $\exp(-\lambda t)$, parameterized by a gamma mixing measure $$\psi(\lambda) = (\beta^\alpha/\Gamma(\alpha)) \lambda^{\alpha-1} e^{-\beta\lambda}$$, yielding:

$$k(t, t') = (\beta^\alpha) / (t + t' + \beta)^\alpha$$

This captures the converging and noisy nature of typical training curves; additional noise processes (e.g., Ornstein-Uhlenbeck) may be added for further realism (Swersky et al., 2014).

• Global GP over hyperparameters: The predicted asymptotic value of each temporal GP is itself the output of a global GP defined on the hyperparameter space, typically using a Matérn-5/2 kernel with Beta cumulative distribution function (BetaCDF) input warping for each dimension.

This hierarchical construction allows joint inference over both the trajectory of each individual training curve and the relationships between curves for different hyperparameter configurations.

3. Acquisition Functions and Decision Mechanisms

Resource allocation within FT-BO hinges on information-theoretic or cost-aware acquisition functions. The classical criterion of Expected Improvement (EI) can be suboptimal because it over-prioritizes unexplored candidates in settings with large uncertainty. FT-BO adopts an entropy reduction framework:

• It constructs $P_{min}$, the distribution over the minimum (best configuration).
• For each candidate, the method simulates ("fantasizes") possible future observations and computes the expected reduction in entropy: $H(P_{min}^y) - H(P_{min})$.
• The next action is chosen to maximize expected information gain (Algorithm 1 in (Swersky et al., 2014)). This decision mechanism balances exploration (initiating new candidates) and exploitation (extending promising ones) adaptively.

Recent work has introduced cost-aware acquisition functions that integrate the computational cost of further training into the selection criteria, such as maximizing $\alpha(x,t) = EI(x,t) / c(x,t)$ (Nguyen et al., 2019), where $c(x,t)$ models cost as a function of the candidate and additional steps.

4. Scalability, Surrogate Advances, and Warm Starting

Historically, FT-BO relied on GP surrogates, whose computational complexity scales cubically with the number of evaluations. Multiple Adaptive Bayesian Linear Regression (ABLRS) models have been deployed as scalable alternatives, paired with a shared neural network that learns nonlinear basis expansions jointly across tasks, greatly reducing computational overhead and enabling warm starts from previous partial or full evaluations (Perrone et al., 2017).

Recent advances include deep ensemble and transformer-based surrogates such as the prior-data fitted network (PFN), which exploit in-context learning. PFNs are trained on large synthetic or real learning curve datasets to extrapolate future performance directly from partial histories via a single forward pass, eliminating online refitting (Rakotoarison et al., 25 Apr 2024). Specialization via pretraining on realistic (e.g., Adam-optimized) tasks, combined with learning curve augmentation (e.g., CDF-augment based on Beta CDFs), further improves extrapolation accuracy and sample efficiency (Athanasiadis et al., 27 Aug 2025).

Surrogate Type	Computational Scaling	Knowledge Transfer	Online Updates Required
Gaussian Process	$\mathcal{O}(N^3)$	Limited	Yes
Bayesian Linear Reg.	$\mathcal{O}(N)$	Via NN sharing	No
Transformer PFN	$\mathcal{O}(1)$	In-context	No

5. Empirical Performance and Applications

FT-BO methods have achieved strong empirical results on a range of tasks:

• Logistic Regression (MNIST), Online LDA (Wikipedia), and PMF (MovieLens): FT-BO reached superior best objectives versus cumulative training cost compared to standard BO, especially in settings where early training iterations are predictive of final performance (Swersky et al., 2014).
• Deep Reinforcement Learning (DRL) and CNNs: Cost-aware, fidelity-aware FT-BO models outperform baselines by quickly terminating suboptimal candidates and reallocating resources to trajectories with higher early promise (Nguyen et al., 2019).
• Hyperparameter Optimization (HPO) on TaskSet and Out-of-distribution Tasks: PFN surrogates, particularly Adam-PFN trained on real learning curves and augmented via CDF-augment, demonstrated faster convergence (e.g., comparable regret in 150 epochs vs. 750 for prior methods) and robustness to OOD generalization (Athanasiadis et al., 27 Aug 2025).

6. Implications, Extensions, and Recommendations

FT-BO offers several advantages:

• Efficiency: Aggressive early stopping reallocates computation away from poor candidates.
• Scalability: Hierarchical GP models are designed to scale gracefully with increased temporal observations, while BLR and PFN surrogates further reduce computational costs.
• Automation: Entropy-reduction and cost-aware acquisition functions minimize expert tuning.
• Flexibility: FT-BO can be adapted to leverage prior knowledge via surrogate specialization, yielding strong generalization across optimizer families and tasks.
• Uncertainty Quantification: Fully Bayesian surrogate treatments, e.g., marginalizing hyperparameters via HMC (Hamiltonian Monte Carlo) and employing ARD kernels, produce less overconfident predictions and improve decision making in freeze/thaw cycles (Ath et al., 2021).

Best practices, based on empirical results, suggest using EI with a fully Bayesian hyperparameter treatment and ARD kernels for the GP surrogate when applicable, and considering transformer-based PFNs for large-scale, low-budget HPO (Ath et al., 2021, Rakotoarison et al., 25 Apr 2024).

7. Current Limitations and Future Directions

Key limitations include surrogate misspecification and the challenge of modeling highly nonstationary or task-dependent learning curves. While PFN surrogates trained on synthetic curves offer generality, specialization via real learning curve datasets significantly improves performance, as evidenced by Adam-PFN (Athanasiadis et al., 27 Aug 2025). The introduction of learning curve augmentation, such as CDF-augment, addresses data scarcity and supports rank preservation—an essential property for robust extrapolation.

A plausible implication is that further research may examine PFN surrogates tailored to specific optimizers or model families and the integration of richer cost models for more finely tuned resource allocation. The extension of FT-BO to multi-task or transfer learning frameworks also appears viable given the demonstrated efficacy of neural mechanisms for knowledge transfer (Perrone et al., 2017).

---

In summary, Freeze-thaw Bayesian Optimization represents a dynamic, highly adaptive framework for hyperparameter optimization in iterative learning settings. Through bespoke probabilistic surrogates, temporal modeling, and information-driven acquisition functions, it achieves both rapid convergence and high computational efficiency, with ongoing innovations in neural surrogates and acquisition strategies driving state-of-the-art results in deep learning optimization.