Overtuning in Hyperparameter Optimization

• Overtuning is a phenomenon in hyperparameter optimization where the search process overfits validation data, causing selected configurations to generalize poorly on unseen test sets.
• Empirical studies show overtuning is prevalent and can occasionally be severe, sometimes losing all HPO progress, especially with flexible models and small datasets.
• Robust resampling like repeated k-fold cross-validation is the most effective strategy to mitigate overtuning, especially on small datasets.

Overtuning in hyperparameter optimization is a phenomenon wherein the hyperparameter search process adapts excessively to the validation error, resulting in the selection of hyperparameter configurations that perform well on the validation data but generalize poorly to unseen, independent data. This form of overfitting is distinct from classical model training overfitting, as it arises specifically from the meta-level process of validation-based selection during hyperparameter search and is exacerbated in settings with flexible models, small datasets, or weak resampling.

1. Formal Definition and Distinction from Related Concepts

The formalism introduced in recent work defines overtuning as follows. Let $(\lambda_1, \ldots, \lambda_T)$ be the sequence of hyperparameter configurations (HPCs) evaluated during an HPO run, with each incumbent at step $t$ being $$\lambda^*_t = \arg\min_{\lambda \in \{\lambda_1, \ldots, \lambda_t\}} \widehat{\mathrm{val}}(\lambda)$$, the configuration attaining lowest observed validation error up to time $t$. Overtuning at step $t$ is mathematically quantified as

$$\mathrm{ot}_{t}(\lambda_1, ..., \lambda_t) = \mathrm{test}(\lambda^*_t) - \min_{\lambda^*_{t'} \in \{\lambda^*_1, ..., \lambda^*_t\}} \mathrm{test}(\lambda^*_{t'}),$$

where $\mathrm{test}(\cdot)$ represents (unknown) generalization error measured on an untouched test set. This measures how much worse, in terms of test error, the final configuration is compared to the best previously seen incumbent, despite always having minimized observed validation error.

Relative overtuning normalizes this value by the total achievable test improvement: $$\tilde{\mathrm{ot}}_t = \frac{\mathrm{ot}_t}{\mathrm{test}(\lambda^*_1) - \min_{t'\leq t} \mathrm{test}(\lambda^*_{t'})}$$ so that $0$ corresponds to no overtuning and values greater than $1$ indicate all test gains were lost (i.e., performance is no better, or worse, than the starting point). This distinguishes overtuning from "meta-overfitting," which commonly refers to overfitting validation splits in repeated resampling structures, as it focuses specifically on loss of generalization caused by the selection process in HPO (not only by data reuse).

2. Empirical Prevalence and Severity

A large-scale reanalysis covering public HPO trajectory repositories (including FCNet, LCBench, TabZilla, TabRepo, reshuffling, WDTB, PD1) demonstrates that overtuning is both prevalent and occasionally severe:

• No Overtuning: Approximately 60% of HPO runs conclude with no overtuning, meaning the final selected configuration generalizes at least as well as any earlier configuration.
• Mild Overtuning: About 70% of runs have relative overtuning below $0.1$ (i.e., less than 10% of possible test improvement lost due to overtuning).
• Severe Overtuning: In roughly 10% of cases, severe overtuning is observed (relative overtuning $>1$), meaning HPO produced a solution that actually generalizes worse than the initial configuration.
• Heterogeneity exists across benchmarks and algorithms: settings such as TabRepo or certain reshuffling runs exhibit overtuning rates above 50%, with severe cases exceeding 15%. Neural networks, CatBoost, and other flexible models are at greatest risk, especially when combined with small datasets and holdout validation.

This meta-evaluation highlights that while overtuning is typically mild, it is not rare and can occasionally nullify all HPO progress, particularly in challenging regimes.

3. Factors Influencing Overtuning

The severity and prevalence of overtuning depend on several interrelated technical factors:

• Performance Metric: Overtuning is more pronounced when optimizing binary metrics such as accuracy and ROC AUC, as these are high-variance and less smooth. Log loss and $R^2$ are less susceptible.
• Resampling Scheme: Holdout validation—especially on small datasets—shows the highest overtuning rates. k-fold cross-validation (especially repeated CV) dramatically reduces overtuning, both in odds and severity.
• Dataset Size: Overtuning risk is much higher on small datasets. For example, increasing training set size from 500 to 5,000 instances robustly reduces overtuning.
• Learning Algorithm: More flexible learning algorithms (e.g., CatBoost, deep feedforward MLPs, XGBoost) are more prone to overtuning in small-data settings. Elastic Net exhibits substantially less overtuning.
• HPO Algorithm: Random search is somewhat more overtuning-prone than Bayesian optimization, though even sophisticated methods may suffer if validation is weak. Early stopping helps, but its impact is incremental once robust resampling is already used.
• HPO Budget: Larger HPO budgets increase overtuning, though the effect saturates.

The following table summarizes key influences:

Factor	Effect on Overtuning
Metric	ROC AUC > Accuracy > Log Loss
Resampling	Holdout > k-fold CV > Repeated CV
Dataset Size	Small size: High; Large size: Low
Algorithm	Flexible models: High
HPO Method	Random Search > Bayesian Opt

4. Mitigation Strategies

Based on analyses and prior work, several practical strategies can reduce overtuning in HPO:

• Robust Resampling: Employ repeated k-fold cross-validation (e.g., $5 \times 5$-fold) over holdout. This is the most effective empirical guard.
• Larger Training Sets: Where feasible, increasing the data size directly reduces overtuning risk.
• Advanced HPO Algorithms: Bayesian optimization (especially when noise-aware) reduces overtuning severity compared to random search.
• Reshuffling Splits: Regularly reshuffling resampling splits (e.g., for holdout or ROC AUC settings) can reduce overtuning by breaking spurious configuration–split pairings.
• Early Stopping and Conservative Incumbent Selection: Incorporating early stopping or using the incumbent predicted by a surrogate's posterior mean (rather than naïve best-validation) can curb overtuning at some cost to attainable performance.
• Objective Modification and Regularization: While variance-regularization approaches are less explored in real HPO for complex models, explicitly regularizing the hyperparameter search or model-averaging across top configurations may help.
• Monitor Validation vs. Test Error: During method development or meta-benchmarking, always compare the validation curve to true (held-out) generalization error. Don’t rely solely on validation improvement as an indicator of progress.

Practitioners are advised to design HPO pipelines so that selection is based on robust validation and not aggressive exploitation, especially in small-sample or high-variance settings.

5. Illustrative Examples and Quantitative Formulation

Let $GE(\lambda)$ denote the expected generalization error of a model trained with HPC $\lambda$: $$GE(\lambda) = E_{D \sim \mathbb{P}_{xy}^n, (x, y) \sim \mathbb{P}_{xy}} \left[L(y, \hat{f}_\lambda^D(x))\right]$$ Overtuning at HPO step $t$ is then

$$\mathrm{ot}_t = \mathrm{test}(\lambda^*_t) - \min_{t' \leq t} \mathrm{test}(\lambda^*_{t'})$$

with relative overtuning: $$\tilde{\mathrm{ot}}_t = \frac{\mathrm{ot}_t}{\mathrm{test}(\lambda^*_1) - \min_{t' \le t}\mathrm{test}(\lambda^*_{t'})}$$ Empirically, most HPO runs exhibit small $\tilde{\mathrm{ot}}_t$, but a non-negligible minority reach or exceed $1.0$ (full performance lost), underscoring the need for careful search design.

6. Recommendations and Practical Guidelines

The following guidelines are supported by the empirical paper and prior literature:

• Favor repeated CV or robust resampling over single holdout partitions, especially for small datasets and flexible models.
• Limit HPC search budget in low-sample, high-variance regimes, or increase repeats as the HPO budget (number of tried HPCs) increases.
• Carefully select performance metrics; using log loss or $R^2$ may reduce overtuning impact compared to accuracy or AUC in certain contexts.
• Employ noise-aware surrogate modeling and conservative selection in Bayesian optimization (e.g., incumbent on surrogate posterior mean).
• When possible, avoid reporting only validation-optimal results—rely on outer-split held-out performance or, in benchmarking, monitor test error on independent splits for overtuning awareness.

7. Outlook and Research Directions

Overtuning in HPO is empirically relatively common and, while typically mild, can be severe under certain conditions: small datasets, holdout validation, flexible models, and when optimizing high-variance metrics. Mitigation via resampling and robust search strategies is effective, but open research questions remain, such as:

• How to further regularize hyperparameter search processes systematically.
• How to best allocate computational resources between search breadth, repeat validation, and early stopping in automated ML systems.
• Development of theoretical frameworks and robust estimation schemes for the generalization error of the selected HPC under arbitrary search.

Continued development of HPO algorithms, resampling-aware pipelines, and overtuning diagnostics will be essential as machine learning pipelines become more automated and widely deployed.