Hyperparameter Transfer & Power-Law Extrapolation

• The paper demonstrates that combining hyperparameter transfer with power-law extrapolation enhances sample efficiency and guides resource allocation in ML systems.
• It details methodologies such as surrogate initialization, multi-task Bayesian optimization, and fitting power-law models to predict scaling behavior.
• Empirical evidence shows improved optimization performance and adaptive surrogate refinement in high-dimensional, costly evaluation settings.

Hyperparameter transfer and power-law extrapolation are two methodological paradigms that have emerged as key techniques in sample-efficient hyperparameter optimization and predictive scaling analysis of machine learning systems. Both address the challenges posed by high-dimensional search spaces and computationally expensive black-box objectives, but from distinct perspectives: transfer learning attempts to leverage prior optimization experience from related tasks or domains, while power-law extrapolation seeks to model and extend observable scaling trends to guide resource allocation or predict model performance.

1. Conceptual Foundations

Hyperparameter transfer refers to strategies that use optimization results from source tasks (e.g., smaller models, datasets, or pretraining domains) to warm-start, restrict, or bias hyperparameter search on a target, more expensive or higher-fidelity task. The core idea is to reduce redundant exploration by identifying promising regions or configurations a priori. Transfer can be implemented as direct parameter reuse, prior construction for Bayesian optimization surrogates, or as meta-learning policies.

Power-law extrapolation is a statistical approach aimed at modeling the dependence of evaluation metrics (such as model accuracy, loss, or even optimal hyperparameter values) on proxy variables such as data set size, model size, training compute, or wall-clock time. Empirical and theoretical studies have identified that many metrics follow power-law behavior, $y = a x^\alpha + b$, within certain scaling regimes, enabling accurate prediction beyond observed ranges and informing optimal resource specification.

These frameworks are complementary: power-law models are often used for extrapolative prediction of hyperparameter-performance landscapes, while transfer methods help tailor surrogate-based optimization to specific scaling regimes, reducing sample complexity.

2. Hyperparameter Transfer in Bayesian Optimization

Recent surrogate-based optimization pipelines (e.g., Gaussian-process-driven Bayesian optimization) have incorporated hyperparameter transfer via multitask surrogates, hierarchical kernel construction, or meta-prior pooling. A prototypical system involves:

• Construction of a surrogate $\mathcal{S}_{\text{source}}$ using historical evaluations (e.g., lower-fidelity, small-scale, or similar-task runs).
• Initialization or conditioning of the target-task surrogate $\mathcal{S}_{\text{target}}$ with information (means, kernel hyperparameters, or direct observations) transferred from $\mathcal{S}_{\text{source}}$.
• Restriction of the search space or shaping of the acquisition function (e.g., via refined subspace design, as in divide-and-refine heuristics) based on observed cross-task hyperparameter correspondences.
• Sequential updating as new target-task data are collected, allowing the transfer effect to adaptively wane as sufficient target-specific evidence accrues (Nomura et al., 2019, Leenders et al., 16 Dec 2025).

A key challenge is the design of effective domain adaptation strategies: direct transfer may fail when task dissimilarity is large or the conditional distribution shift is significant, requiring robust domain similarity measures or meta-learnt task embeddings.

Within multi-objective or hierarchical optimization (e.g., process-parameter tuning in industrial settings), hyperparameter transfer is further complicated by conflicting target objectives and hierarchical design variable activation (e.g., in system architecture optimization under hidden constraints) (Bussemaker et al., 11 Apr 2025).

3. Power-Law Extrapolation for Hyperparameter and Performance Prediction

Empirical analysis of ML system scaling (notably in deep learning, Gaussian processes, or random forests) reveals that optimal validation/test metrics, loss curves, or sometimes even optimal hyperparameter values exhibit predictable power-law behavior in system size, data quantity, or computational budget (Karlsson et al., 2020, Lu et al., 2022).

In surrogate-based optimization, power-law extrapolation is operationalized in two ways:

1. Direct performance modeling: fitting $\hat{y}$ to a power-law function of evaluated resource $x$ (e.g., model size, epoch, subsample count) and predicting further improvements under extrapolated budgets.
2. Hyperparameter landscape modeling: estimating how the optimum of $f(\lambda; x)$ (i.e., the best hyperparameter $\lambda^*$ as a function of system size $x$) shifts with $x$. This can be used to restrict or shift the hyperparameter search region as the task scales.

Explicit power-law fits are used to inform early stopping or to guide the allocation of future computational resources. In design-of-experiment settings, they support adaptive allocation between fidelity levels in multi-fidelity Bayesian optimization.

4. Methodologies: Surrogate Construction and Extrapolation Workflow

Synergistic workflows often proceed as follows:

• Initial exploration: Use prior low-fidelity runs or small data/models to locate promising hyperparameter subspaces via transfer.
• Joint surrogate training: Fit a hierarchical surrogate or multitask Gaussian process that models $\mathcal{S}_{\text{source}}$0 jointly over hyperparameters $\mathcal{S}_{\text{source}}$1 and scaling variable $\mathcal{S}_{\text{source}}$2.
• Power-law fitting: For each fixed $\mathcal{S}_{\text{source}}$3, or for optimal observed value(s), fit a power-law function $\mathcal{S}_{\text{source}}$4 using least squares or maximum likelihood.
• Extrapolation: Predict future metric values or best hyperparameter regions at larger $\mathcal{S}_{\text{source}}$5 (e.g., future compute budgets, larger datasets), compute acquisition functions (e.g., UCB, EI) with bias towards extrapolated optima.
• Optimization: Restrict or bias candidate proposals around extrapolation-informed regions, enabling one-shot or batch design for large-scale or high-fidelity evaluations (Nomura et al., 2019, Neufang et al., 2024, Li et al., 2023).
• Iterative refinement: As further high-fidelity data arrive, update both the surrogate and the extrapolation model, yielding an adaptive, sample-efficient workflow.

5. Empirical Performance and Best Practices

Empirical studies demonstrate that hyperparameter transfer and power-law extrapolation—individually or jointly—systematically improve optimization efficiency, especially under constrained evaluation budgets or in hyperparameter-rich regimes:

• Refine-then-optimize (RTO) heuristics: Partitioning the input space based on coarse evaluations, then running BO/SMBO on promising subregions, yields significant gains over plain BO in low-to-moderate budget regimes (Nomura et al., 2019).
• Surrogate parameterizations: Finite-width Bayesian neural network surrogates or ensemble-Gaussian-process mixtures can better adapt to non-stationarity and varying scaling regimes, outperforming canonical stationary GPs or tree-based surrogates when paired with transfer/extrapolation (Li et al., 2023, Lu et al., 2022).
• Case studies: In process engineering, power-law extrapolation guides experiment allocation under cost constraints, while warm-starting surrogates with historical or simulated data accelerates convergence to optimal settings (Neufang et al., 2024, Kronenwett et al., 30 Jul 2025).

A major limitation is that power-law regimes may break down at scale extremities, or when the task undergoes a qualitative regime change (e.g., phase transition in model behavior). In such cases, adaptive diagnostics are needed to guard against over-extrapolation.

6. Table: Methods and Representative Applications

Methodology	Transfer Mode/Extrapolation	Representative Use Cases / Results
Refine-then-Optimize (RTO) (Nomura et al., 2019)	Search-space reduction before BO	Low-budget HP tuning, CNN/MLP hyperparams
Multi-task/Meta-prior BO (Leenders et al., 16 Dec 2025)	Surrogate initialization from source data	Preference optimization with warm-start
Power-law Surrogate Scaling (Karlsson et al., 2020)	Predictive scaling of performance/HP optima	Discrete/Binary/Ordinal HP landscapes
Hierarchical/mixed-discrete BO (Bussemaker et al., 11 Apr 2025)	PoV model transfer, batch/f