---
title: Cross-Dataset Hyperparameter Transfer
url: https://www.emergentmind.com/topics/cross-dataset-hyperparameter-transfer
type: topic
---

# Cross-Dataset Hyperparameter Transfer

Cross-dataset hyperparameter transfer denotes the practice of leveraging hyperparameter optimization or search results from related datasets to accelerate, warm-start, or otherwise improve hyperparameter selection on a new dataset. This paradigm is fundamental across transfer learning, AutoML, and meta-learning, wherein prior computational effort or empirical knowledge is adaptively reused or encoded. Research spans principled transfer in Bayesian optimization, neural/surrogate-based alignment, copula models, kernel embeddings, meta-feature-driven surrogates, combinatorial portfolio selection, multi-source covariate shift, and ordered optimization in continual settings.

## 1. Problem Formulation and Conceptual Foundations

Cross-dataset hyperparameter transfer is rigorously defined in terms of transferring knowledge (surrogate models, priors, optimal configurations, meta-features, or empirical mappings) from a set of “source” tasks (datasets) to a new “target” task. The core problem is to estimate, for the target dataset $D^t$, the mapping from hyperparameter configuration $x \in \mathcal{X}$ to objective value $y = f^t(x)$, using available data $\{(x^{s}_i, y^{s}_i)\}_{i,s}$ from other datasets.

A simple formalization, as in ensemble Bayesian optimization [1802.02219], is:
- Given $T$ source tasks, each with evaluations $\{\mathcal{D}^{(s)}\}_{s=1}^T$, learn a surrogate or set of priors $M^s$.
- For a new target, combine $M^s$ with an online-updated $M^t$ to form transfer recommendations or surrogates driving acquisition functions.

Within multi-objective or multi-fidelity settings (cf. [2202.07631], [2007.13382]), the objective may further include coherence, diversity, or multi-task performance metrics, and transfer solutions may be structured as portfolios, embedding-based surrogates, or ensembles.

## 2. Bayesian Optimization and Surrogate-Based Transfer

Bayesian optimization (BO) underpins much of the technical literature (e.g., [1802.02219], [1810.06305], [2206.02663], [1909.13595], [2306.16916]). Transfer is enacted either by:
- Constructing ensemble surrogates, such as the ranking-weighted Gaussian process ensemble (RGPE) [1802.02219], which aggregates GP predictions from each source task weighted by performance on the target.
- Building kernel-augmented GPs, e.g., embedding datasets into an RKHS via mean-embedding representations and conditioning the BO surrogate on both hyperparameters and data-embedding [1810.06305], or using learned meta-feature extractions [2102.03776].
- Implementing parametric or semi-parametric copula models mapping source empirical quantiles to a latent normal, allowing robust pooling across tasks of different scales and variances [1909.13595].

For task-weighting, RGPE uses ranking statistics and bootstrap aggregation, while TransBO optimizes source and target combination weights jointly through supervised ranking losses and cross-validation [2206.02663].

A general BO loop incorporating transfer typically: initializes with transferred priors or configurations, fits joint or ensemble surrogates, proposes new hyperparameter candidates via acquisition maximization (e.g., Expected Improvement), and updates models with target-evaluated results [1802.02219].

## 3. Surrogate Alignment and Nonparametric Mapping Approaches

Methods such as surrogate alignment (HTS) [1608.00218] learn a direct nonlinear mapping $g:\mathcal{X}\to\mathcal{X}$ from source to target hyperparameter optima. HTS leverages surrogate models (radial-basis function regressors with polynomial tail) of error landscapes from source and target, and trains a neural network to align them by minimizing rank correlation loss between surrogate predictions. This method is effective for DNNs where exact modeling is impractical due to high training cost, and does not require meta-features—only $(x, f(x))$ pairs from both domains.

Experiments confirm substantial reductions (3–5×) in required target evaluations compared to non-transfer surrogates (e.g., HORD).

## 4. Meta-Feature and Kernel Embedding Techniques

Recent approaches employ learned dataset representations or meta-features to enable cross-dataset transfer even across heterogeneous domains [2102.03776], [1810.06305]. In DMFBS [2102.03776], a differentiable deep-set architecture extracts embeddings from each dataset, which are jointly optimized for response regression, manifold regularization that encourages similarity across surrogate predictions for similar datasets, and a dataset-identification auxiliary task. The output embedding is concatenated with hyperparameter configuration to condition the surrogate predictor, driving acquisition and ranking for new dataset evaluation.

Distributional transfer [1810.06305] utilizes kernel mean embedding of distributions $P_{X,Y}$ into RKHS, and places a GP prior over $(\theta, \psi(D), s)$, facilitating transfer by relating hyperparameter performance across similar datasets in feature-space.

## 5. Portfolio Selection and Zero-Shot Cross-Dataset Transfer

Zero-shot HPO approaches such as [2007.13382], [1710.04725], and [2202.07631] demonstrate that a small portfolio of hyperparameter configurations can "cover" a large set of future datasets: for any unseen dataset, at least one configuration performs near-optimally.

Portfolio selection algorithms solve a combinatorial, submodular minimization of mean regret over meta-datasets, employing greedy augmentation and surrogate modeling or multi-fidelity evaluations. This yields lookup tables of recommended default configurations.

Table: Portfolio Construction Summary

| Method           | Portfolio Construction | Evaluation Approach |
|------------------|-----------------------|--------------------|
| Greedy Submodular [2007.13382] | Greedy K-set minimizing mean meta-loss | Direct empirical or surrogate-based |
| Surrogate Adaptive Query [2007.13382] | Surrogate model learned over $(d,\theta)$ | Bayesian optimization on meta-table |
| Multi-Objective BO [2202.07631] | Pareto front for coherence/diversity/classification | Multi-output random scalarization |

Empirical evidence attests that such portfolios, constructed from hundreds of datasets and tens of thousands of configurations, allow practitioners to skip or dramatically reduce hyperparameter search on new datasets by evaluating only the top-K recommended options.

## 6. Task and Dataset Similarity: Ordered and Distributional Transfer

OTHPO [2306.16916] introduces ordered transfer for sequential tasks (e.g., increasing data sizes, continual learning), positing that recent tasks are more strongly correlated. The approach models transfer via GP surrogates over the joint space of hyperparameters and an ordered context variable (e.g., time, index, data fraction). Warm-start heuristics select best configurations from immediately prior tasks, attesting to improved "first-evaluation" regret.

Distributional embeddings or meta-feature kernels generalize this approach for unordered, meta-feature-equipped cross-dataset scenarios [1810.06305], [2102.03776], [2202.07631].

## 7. Practical Guidelines and Empirical Impact

Effective cross-dataset transfer depends on principled weighting of sources, careful model aggregation, attention to distributional shift, and adaptive regularization:
- Always validate transferred configurations or priors on a held-out split to account for outliers, semantic shift, or domain distance [1807.11459].
- Graduated (layer-wise) hyperparameter schedules and regression-based prediction of transfer scales yield superior results in deep networks [1807.11459].
- Embedding-based or copula-normalized surrogates avoid pitfalls from raw value pooling across tasks with disparate objective scales [1909.13595].
- Variance-minimizing importance weighting is critical under multi-source covariate shift [2006.10600].

Empirical studies consistently report 2–10× reductions in search time, substantial improvements in accuracy or regret over random search, and resilience to outlier tasks and distributional noise.

Cross-dataset hyperparameter transfer remains a cornerstone enabling scalable, sample-efficient, and adaptive automated model tuning in contemporary machine learning pipelines.

Source: https://www.emergentmind.com/topics/cross-dataset-hyperparameter-transfer