Heterogeneous Cross-Validation

• Heterogeneous cross-validation is a set of methods that adapt standard validation techniques to account for non-homogeneous data, such as spatial or cluster-dependent structures.
• It redesigns the validation process by modifying split geometry, loss functions, and data representation to ensure the out-of-sample criterion aligns with the underlying data heterogeneity.
• Applications include Gabriel-style CV for clustering, spatial stratification, and advanced Bayesian methods, all aimed at improving model evaluation under varied noise and dependence conditions.

Heterogeneous cross-validation refers to cross-validation schemes that are explicitly adapted to non-homogeneous data-generating or evaluation structures rather than to a single i.i.d. prediction problem with a uniform loss. In the literature, the relevant heterogeneity may be feature-level, cluster-level, spatial, hierarchical, region-specific, study-specific, or algorithmic, and it may also arise from short-range dependence, heavy-tailed noise, fold-specific difficulty, or domain shift. The common objective is to redesign the validation target—by changing the split geometry, the loss, the unit of validation, or even the data representation—so that the estimated out-of-sample criterion remains aligned with the scientific task under heterogeneity (Fu et al., 2017, Lobo et al., 2018, Wood, 2024).

1. Conceptual foundations

Standard cross-validation presumes that a model is trained on one subset of observations, evaluated on another, and summarized by a prediction loss whose interpretation is stable across folds. Several strands of the literature show that this premise is often inadequate. In benchmarking, variability arises not only from finite test samples but also from training-set randomness and internal stochasticity, and the resulting fold-to-fold dependence induces an irreducible covariance floor even as the number of splits grows (Eve et al., 10 Jun 2026). In model selection, traditional CV can also ignore “validation uncertainty,” so that a single minimizer of empirical fold loss is treated as if it were known without error; “Cross-Validation with Confidence” formalizes this as a hypothesis-testing problem over models and constructs a confidence set of competitive candidates rather than a single unqualified winner (Lei, 2017).

The term itself is not uniformly standardized across subfields. In some papers it is used directly for clustering under heterogeneous or heavy-tailed noise, while in others it is an apt description for procedures that average over heterogeneous splits, stratify by spatial region, leave out dependence neighborhoods, or target only a subset of the predictor space (Fu et al., 2017, Lobo et al., 2018, Wood, 2024, Zhang et al., 2021). A plausible implication is that heterogeneous cross-validation is best understood as a family resemblance concept: the essential feature is not a single algorithmic template, but explicit accommodation of heterogeneity in the estimand, the folds, or both.

2. Unsupervised and structural reformulations

A canonical example is Gabriel-style cross-validation for clustering, developed to estimate the number of clusters in settings where ordinary supervised CV is inapplicable because there is no observed response variable (Fu et al., 2017). The procedure randomly splits columns into predictor and response subsets, splits rows into training and test subsets, clusters the training responses with $k$-means, trains a nearest-centroid classifier from predictors to cluster labels, predicts response-space cluster centers for the test rows, and scores each candidate $k$ by mean squared prediction error in response space. The resulting criterion

$$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$

turns an unsupervised problem into a supervised prediction problem without imposing homogeneous Gaussian assumptions (Fu et al., 2017).

In that framework, heterogeneity is explicit: features may have different variances, clusters may be heteroscedastic, and coordinates may be heavy-tailed. The method is reported to be largely insensitive to variance ratios in the variance-heterogeneity setting and comparatively robust as $t_\nu$ tails become heavier, whereas competitors such as Gap and Jump deteriorate under very heavy tails (Fu et al., 2017). The same paper also identifies strong correlation as a failure mode for uncorrected Gabriel CV and introduces a two-stage correction based on estimating a common noise covariance,

$$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$

followed by sphering and random rotation,

$\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$

before reapplying cross-validation (Fu et al., 2017). In noiseless settings with distinct centers, the same procedure is self-consistent: $CV(k)>0$ for $k<K$ and $CV(k)=0$ for $k\ge K$, so the true number of clusters is selected (Fu et al., 2017).

3. Dependence, spatiality, and hierarchical structure

When heterogeneity is induced by dependence or multilevel structure, the unit of splitting itself must change. For nonlinear mixed-effects models, out-of-sample prediction is ambiguous because random effects blur the distinction between new observations within known subjects and observations from entirely new subjects (Colby et al., 2013). The proposed subject-level CV variants therefore operate by leaving out whole subjects rather than individual measurements. For structural model selection, the criterion

$k$0

uses post hoc random effects for held-out subjects; for covariate selection, the criterion

$k$1

targets unexplained between-subject heterogeneity directly (Colby et al., 2013). The paper shows that prediction-error CV can fail completely for covariate detection because random effects compensate for omitted covariates, whereas $k$2 is correct in $k$3 to $k$4 of several simulated scenarios (Colby et al., 2013).

Spatial and dependent-data settings motivate analogous redesigns. In geostatistics, model assessment can vary substantially with the locations selected for validation, so Bayesian cross-validation is formulated over a prior distribution on split vectors $k$5, and the expected predictive discrepancy

$k$6

is averaged over many training/validation configurations via sampling importance resampling (Lobo et al., 2018). A stratified cross-validation scheme partitions the spatial domain into strata and combines stratum-specific discrepancies as

$k$7

thereby reducing variance and making spatial heterogeneity explicit (Lobo et al., 2018). For penalized regression with un-modelled short-range autocorrelation, neighbourhood cross-validation defines omitted sets $k$8 and evaluation sets $k$9 and minimizes

$$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$0

with leave-out-neighbourhood choices such as $$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$1, $$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$2 (Wood, 2024). This construction is designed precisely for non-i.i.d. data with local dependence, and the paper shows how to evaluate and optimize such criteria at leading-order cost comparable to a single model fit (Wood, 2024).

4. Region-specific and study-specific heterogeneity

A different notion of heterogeneity arises when only part of the predictor space matters. Targeted cross-validation replaces global squared-error assessment by a weighted criterion

$$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$3

where $$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$4 integrates to one under $$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$5 and can concentrate on a region, a low-variance subdomain, or even a shrinking neighborhood of a single point (Zhang et al., 2021). The corresponding target is the weighted risk $$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$6, and the paper proves $$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$7-consistency even when the identity of the best candidate changes with $$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$8 (Zhang et al., 2021). A closely related extreme-value formulation defines

$$CV(k) = \frac{1}{KL}\sum_{r=1}^K \sum_{s=1}^L CV_{r,s}(k), \qquad \hat{k} = \arg\min_k CV(k),$$9

for a rare region $t_\nu$0 and studies CV estimators whose effective sample size is $t_\nu$1, obtaining non-asymptotic bounds of order $t_\nu$2 for K-fold and leave-$t_\nu$3-out procedures (Aghbalou et al., 2022).

At the study level, heterogeneity is encoded through differing joint distributions $t_\nu$4. Bayesian nonparametric cross-study validation constructs the array $t_\nu$5, where a model trained on study $t_\nu$6 is validated on study $t_\nu$7, and then models

$t_\nu$8

under a random partition $t_\nu$9 of studies (Trippa et al., 2015). The latent clusters $$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$0 identify subsets of studies with similar validation profiles and provide the basis for cluster-specific training pools and algorithm comparison under study heterogeneity (Trippa et al., 2015). In multi-study stacking, the same issue appears as a generalization problem to unseen studies. The cross-study CV approach evaluates stacked predictors by leaving one study out,

$$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$1

and aggregates these quantities to estimate generalist utility for new studies (Ren et al., 2020). Under substantial between-study heterogeneity, this leave-one-study-out construction improves prediction accuracy relative to stacking with data reuse (Ren et al., 2020).

5. Variance control, confidence procedures, and repeated validation

Once folds themselves are heterogeneous, inference for CV becomes a first-order issue rather than an afterthought. “Cross-Validation with Confidence” treats each candidate model $$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$2 through the null hypothesis that its predictive risk is no larger than that of every alternative, and constructs the confidence set

$$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$3

using a maximum of studentized loss differences and a Gaussian multiplier bootstrap that explicitly accommodates heterogeneous scales and correlations across model comparisons (Lei, 2017). In classical linear regression, choosing the most parsimonious model inside $$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$4 yields consistent variable selection even with conventional split ratios (Lei, 2017).

A complementary line of work derives CLTs and variance estimators for cross-validated test error itself. Under weak stability conditions, $$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$5-fold and leave-one-out CV admit asymptotically exact confidence intervals for test error and hypothesis tests comparing algorithms (Bayle et al., 2020). The CV estimate can also be bootstrapped efficiently through a random-effects decomposition of bootstrap-by-split replicates, which separates between-bootstrap from within-bootstrap variability and accommodates complex functionals such as the c-index, ROC curves, and subgroup treatment effects (Cai et al., 2023). At the benchmarking level, repeated CV has been shown to reduce variance substantially, with

$$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$6

and the corresponding sample-gain approximation

$$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$7

where $$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$8 is the fold-error intraclass correlation (Eve et al., 10 Jun 2026). Low redundancy across folds implies large gains, and the study-only redundancy score $$\hat{\Sigma} = \frac{1}{N-\hat{k}_0}\sum_{i=1}^N (X_i-\hat{\mu}_i)(X_i-\hat{\mu}_i)^\top,$$9 is proposed as an early-stopping diagnostic after only two or three splits (Eve et al., 10 Jun 2026).

A further refinement distinguishes the average performance of a modeling procedure from the performance of the particular model actually deployed. Under a random-effects model

$\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$0

conventional CV estimates $\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$1, whereas a naive single-split estimate targets the specific $\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$2 of the chosen training set (Pan et al., 9 Oct 2025). Hierarchical Bayesian and empirical Bayes estimators then shrink the naive estimate toward the CV average,

$\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$3

so that information from heterogeneous CV splits is used to improve estimation of the specific deployed model’s performance (Pan et al., 9 Oct 2025).

6. Emerging algorithmic designs and recurring limitations

Recent work pushes heterogeneous cross-validation beyond ordinary splitting. For robust penalized regression with non-convex losses, ordinary CV is unreliable because different folds can converge to different local minima. Robust Information Sharing Cross-Validation addresses this by tracking multiple minima for each $\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$4, reusing full-data minima as fold initializations, and matching fold-specific minima to full-data minima via the Pearson correlation of robustness-weight vectors,

$\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$5

thereby producing smoother CV curves and more stable hyper-parameter selection (Kepplinger et al., 2024). In settings where sample splitting itself is infeasible, antithetic Gaussian randomization constructs pseudo train–test pairs

$\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$6

with equicorrelated Gaussian perturbations satisfying

$\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$7

so that $\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$8 almost surely (Liu et al., 2024). In theory, this antithetic correlation is what keeps the estimator’s variance bounded while letting the bias vanish as $\tilde{X} = X \Gamma \Lambda^{-1/2} Q,$9 (Liu et al., 2024).

The literature also identifies recurring limitations. Gabriel cross-validation can overestimate $CV(k)>0$0 under strong correlation without its sphering correction, and its correction assumes a shared noise covariance and remains tied to $CV(k)>0$1-means (Fu et al., 2017). In nonlinear mixed-effects models, $CV(k)>0$2 can be distorted by $CV(k)>0$3-shrinkage in sparse designs (Colby et al., 2013). Bayesian nonparametric cross-study validation reduces a study pair to a one-dimensional validation statistic $CV(k)>0$4, which can miss richer forms of heterogeneity even when clustering is informative (Trippa et al., 2015). A plausible implication is that heterogeneous cross-validation is best viewed not as a finished theory but as a design principle: choose the validation unit, the target loss, and the dependence correction so that the resulting out-of-sample criterion matches the heterogeneity that matters scientifically.