Quantify selection bias as candidate pipelines increase

Determine how quickly selection bias grows as the number of candidate machine-learning pipelines increases relative to the effective sample size, and establish when a large untouched test set is sufficient versus when nested assessment remains necessary.

Background

The paper discusses model-selection optimism in biomedical and applied machine learning, particularly when many candidate features, hyperparameters, model families, thresholds, or preprocessing alternatives are evaluated. It emphasizes that nested cross-validation separates model selection from performance estimation, whereas reusing assessment results for selection can produce optimistic performance estimates.

The unresolved issue is to obtain quantitative guidance relating the number of candidate pipelines to the effective sample size, especially when observations are clustered within subjects, patients, centers, or other groups. Such guidance would help determine when an untouched test set can provide adequate protection against selection bias and when nested assessment is still required.

References

A related open question is how quickly selection bias grows as the number of candidate pipelines increases relative to N_eff. More quantitative guidance is needed to separate settings where a large untouched test set is sufficient from those where nested assessment remains necessary.

— Model validation in machine learning: A scenario-based guide from hold-out splits to nested group cross-validation in biomedical and applied research  (2610.01284 - Baygin et al., 1 Oct 2026) in Section 12.3, “Open questions”

Although the differences across encoder families (Tab.~\ref{tab:audit}) suggest broader implications for architecture exploration, whether reuse of evaluation data in related optimization procedures produces similar optimism remains an open question.

— When Layer Selection Misleads Speech Depression Detection  (2610.06465 - Perez-Toro et al., 5 Oct 2026) in Section: Conclusions and Limitations