Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Well

Abstract: Modern machine learning progresses through empirical work, benchmarking new methods to evaluate relative performance. However, the statistical variability inherent to evaluation - exacerbated by the stochastic nature of many algorithms - often makes performance estimation unreliable due to the limited test samples available, leading to a validation crisis in which genuine advances are difficult to discern. In this work, we show that cross-validation improves markedly confidence when evaluating and comparing learning algorithm performances. We introduce the concept of sample gain, which quantifies the virtual data augmentation achieved by using multiple cross-validation splits to reduce benchmarking variance. Experiments on both synthetic and real-world datasets (histopathologic scans and NLP fine-tuning) demonstrate that multiple splits can substantially improve the reliability and stability of performance estimates, with diminishing returns often setting in later than expected. We also introduce a procedure to dynamically early-stop cross-validation by estimating from the first few folds if subsequent folds will bring large sample gains. Our findings highlight the value of pushing cross-validation on available samples to achieve robust and reliable benchmarking.

• The paper introduces variance-equivalent test sample gain, showing that repeated cross-validation can deliver effective test-set gains of roughly 10–15 times and that benefits often continue well beyond five splits.
• The experiments show that multi-split cross-validation improves ranking reliability, raising correct retrieval from 60% to 100% in one histopathology comparison and from 70% to 92% in a Yelp comparison.
• The paper proposes an early-stopping redundancy statistic that uses only initial splits to identify when further cross-validation is unlikely to provide substantial variance reduction, helping balance statistical power against computational cost.

The validation crisis and the role of cross-validation

Empirical benchmarking is the primary engine of progress in machine learning, yet it rests on statistical foundations that are frequently too weak to support the conclusions drawn from them. This paper, by Eve, Varoquaux, and Moreau (2606.12552), addresses the "validation crisis": the situation in which limited test samples, combined with the stochasticity of learning algorithms (data splits, weight initialization, hyperparameter optimization), render performance estimates and algorithm rankings unreliable. The authors document that small-sample regimes are pervasive — more than half of the 20% most-used OpenML datasets contain fewer than 1,000 samples, and widely cited benchmarks such as TruthfulQA (817 samples), CUAD (510), or MedMNISTv2 (first quartile at 780–1,600) operate far below the sample sizes needed for stable comparisons. A systematic review of ML-for-healthcare studies found that fewer than 14% performed cross-validation (CV), despite its classical status as a variance-reduction tool.

The paper's central claim is deliberately provocative: cross-validation reduces benchmarking variance "surprisingly well", with diminishing returns setting in much later than standard heuristics suggest. In particular, the common practice of stopping CV once every observation has been held out once (e.g., $K$-fold with test fraction $1/K$) is shown to be statistically unjustified.

Formal framework: benchmarking learning algorithms

The paper carefully distinguishes two estimands. Benchmarking decision functions concerns ranking fixed predictors $g$ by their population risk $\mathcal{R}^*(g)$, estimated unbiasedly by empirical scores on finite data. Benchmarking learning algorithms instead targets the oracle score of an algorithm $F_\lambda$ at a fixed training size $n_{\text{tr}}$: the expectation of the population risk over training sets of size $n_{\text{tr}}$ and internal random states $\xi$. This three-level expectation (test data, training data, internal randomness) formalizes what a benchmark should measure: the capacity of an algorithm to consistently produce high-performing predictors.

Estimation proceeds via Monte-Carlo Cross-Validation (MCCV): draw $K$ i.i.d. splits $(\eta_k)$ and random states $1/K$0, train and evaluate per split, and average. Two classical results anchor the analysis. First, MCCV is unbiased for the oracle score at training size $1/K$1, independent of $1/K$2 [(2606.12552), following Arlot & Celisse]. Second, following Nadeau & Bengio, the estimator's variance decomposes as

$1/K$3

where $1/K$4 is the single hold-out variance and $1/K$5 is the covariance between evaluations on different splits, induced by reusing the finite dataset. As $1/K$6, variance converges to the irreducible floor $1/K$7. Under Nadeau–Bengio's approximation $1/K$8, a 20% test split would cap gains at a factor of five — a prediction the paper's experiments contradict in practice.

The paper also derives an exact, unconditional covariance expansion for the benchmark-adjusted estimation error $1/K$9 (the difference between the CV estimate and the average oracle-score estimate), showing $g$0, where $g$1 isolates evaluation-side dependence after cancelling training-side drift. This identity is exact under exchangeability of fold errors.

The sample gain metric

The paper's main methodological contribution is the variance-equivalent test sample gain $g$2: the factor by which a single hold-out test set would need to be enlarged to match the evaluation variance achieved by $g$3-split CV. Operationally, it is estimated via ANOVA-type method-of-moments estimators of $g$4 and $g$5 from multiple seeds, with non-parametric bootstrap confidence intervals. An intraclass-correlation form makes the mechanics transparent:

$g$6

Gains approach $g$7 when folds are weakly correlated and collapse toward 1 when they are redundant; the asymptotic ceiling is $g$8. Notably, the estimators do not conflict with the Bengio–Grandvalet impossibility result, which applies only to single-run ($g$9) variance estimation; here $\mathcal{R}^*(g)$0 independent seed replications are required.

Empirical findings

Experiments span synthetic Gaussian regression (with Ridge, GradientBoosting, ExtraTrees, MLP, and seven additional solvers), PatchCamelyon histopathology classification (DenseNet121, MobileNetV2, WideResNet101_2, ViT-B/16), and Yelp Review Full fine-tuning (BERT vs. XLM-RoBERTa). Each uses a large held-out benchmarking set (100k–600k samples) as a high-precision oracle proxy, with study sets restricted to realistic benchmark sizes.

Headline numbers: on simulated data at training size $\mathcal{R}^*(g)$1, Ridge reaches $\mathcal{R}^*(g)$2 and MLP reaches $\mathcal{R}^*(g)$3, with diminishing returns for MLP only after roughly 80 splits. Even the weakest cases (ExtraTrees, GradientBoosting) had not converged by $\mathcal{R}^*(g)$4, with $\mathcal{R}^*(g)$5. These results directly contradict both the Nadeau–Bengio asymptotic prediction (a ceiling near 5 for a 20% test split) and the "hold out each observation once" heuristic. Results are robust to switching from MCCV to repeated 5-fold CV.

Ranking reliability: using paired t-tests across 100 seeds against quasi-oracle rankings, multi-split CV substantially improves retrieval of the correct ordering when differences are statistically significant — e.g., DenseNet121 vs. WideResNet101_2 on PCam improves from 60% single-split retrieval to 100% at $\mathcal{R}^*(g)$6 ($\mathcal{R}^*(g)$7); BERT vs. XLM-RoBERTa on Yelp improves from 70% to 92% ($\mathcal{R}^*(g)$8). When differences are not significant, retrieval stays near chance, correctly reflecting statistical indistinguishability.

A subtle and practically important finding concerns the distinction between ranking means versus averaging rankings: on Yelp at $\mathcal{R}^*(g)$9, XLM-RoBERTa beats BERT in 70–80% of individual splits despite a lower mean score, because severe low-score outliers (learning failures) drag down its mean. Single splits systematically miss these failures; only larger $F_\lambda$0 exposes them. This implies that single-split benchmarks can be structurally biased toward unstable-but-often-winning algorithms.

An appendix extends the framework to a ranking-equivalent sample gain: for families with hyperparameter-pinned oracle orderings (e.g., ExtraTrees with $F_\lambda$1), $F_\lambda$2 splits already multiply the effective study size roughly tenfold at the 95% full-ranking-retrieval level, approaching two orders of magnitude by $F_\lambda$3.

Early-stopping cross-validation

Because CV cost scales linearly in $F_\lambda$4, the paper introduces a study-only redundancy statistic $F_\lambda$5, computable from the first $F_\lambda$6 splits (as few as two or three) of a single run without any outer or benchmarking set. It combines the pairwise covariance of split-specific predictions, the correlation of out-of-fold losses on shared test observations, and the mean overlap size. Validated prospectively on 10,800 partial synthetic runs spanning 216 configurations, the statistic is strongly informative: high redundancy almost rules out large gains (probability of $F_\lambda$7 drops from 16.4% in the low-redundancy tertile to 0.0% in the high-redundancy tertile at $F_\lambda$8), while low redundancy identifies settings where continuing CV plausibly pays off. The rule is explicitly asymmetric: low redundancy does not guarantee large gains, it only keeps them plausible.

A complementary calibrated non-benchmark proxy preserves the theoretical gain formula's structure using raw-score quantities alone, though the authors are careful to present it as a diagnostic rather than an unbiased estimator.

Limitations and open questions

The paper concedes several constraints plainly. Its real-data experiments cover only two domains (histopathology and sentiment classification), constrained by the requirement of very large benchmarking sets (244,912 and 600,000 samples respectively) — a requirement that itself limits applicability to domains where no large evaluation pool exists. The evaluation-noise approximation assumes the oracle estimator's noise is negligible relative to test-set noise, justified by benchmarking-set size but unavailable in genuinely small-data settings. The non-benchmark proxy contains empirically fitted constants (e.g., the 0.60 shrinkage factor and the 1/4–3/4 variance weighting) whose generality beyond the studied setups is unestablished. The early-stopping rule is validated on synthetic regression; its behavior on deep-learning workloads, where training instability interacts differently with split overlap, remains open. Finally, the analysis is confined to single-dataset benchmarking; across-dataset comparisons call for different methodology, and whether the observed sample-gain magnitudes transfer to other modalities is left unresolved.

Conclusion

This paper quantifies, through a well-defined and estimable metric, how much repeated resampling improves the statistical power of machine-learning benchmarks in small-sample regimes. Its strongest empirical claims — sample gains around 10–15 at practical split counts, and diminishing returns arriving far later than folklore predicts — carry a direct prescription: when benchmark data are scarce and comparisons are close, single train/test splits should be avoided, and the number of CV repetitions should be treated as an explicit statistical-computational trade-off governed by measurable fold-redundancy diagnostics rather than by the heuristic that each observation need only be held out once.