Cluster-Based Cross-Validation

• Cluster-based cross-validation is a technique that splits data into folds based on inherent clusters, ensuring unbiased error estimation in structured datasets.
• It employs methods such as leave-one-cluster-out, cluster-aware folding, and bi-cross-validation to tailor evaluation for diverse data types.
• This approach reduces overfitting and variance by respecting subpopulation structure, spatial autocorrelation, and other dependencies in the data.

Cluster-based cross-validation refers to a collection of cross-validation (CV) methodologies in which data are split into folds or blocks that respect, utilize, or are otherwise organized around latent or explicit clusters. These methods have emerged in response to challenges such as intra-cluster correlation, domain/group imbalance, subpopulation structure, and non-exchangeability in supervised or unsupervised learning. They aim to mitigate bias, variance, and overfitting associated with classical i.i.d.-based CV. Applications span machine learning, spatial statistics, network analysis, multi-domain generalization, heterogeneous studies, and more. Cluster-based CV methods encompass hold-out or leave-group-out blocks, constrained validation splitting, cluster-informed model evaluation, and block-based estimation of predictive error, each with tailored algorithms, theoretical guarantees, and problem-specific design choices.

1. Foundational Principles and Motivations

Cluster-based cross-validation schemes arise primarily to address two shortcomings of classical fold-based CV: (i) the underestimation of generalization error due to train/test leakage among similar (intra-cluster) units, and (ii) the poor representativeness or high variance of standard random folds when there is strong sub-group structure, spatial autocorrelation, or other forms of dependency in the data (Spezia et al., 30 Jul 2025).

In supervised settings, random splits may place nearly identical correlated observations in both train and test folds, producing optimistically biased performance estimates. In structured data—such as multi-domain, spatial, or networked datasets—naïve splitting disregards clustering or blockwise correlation, violating independence assumptions and mischaracterizing model performance (Yuval et al., 20 Feb 2025). Cluster-based CV thus seeks to:

• Partition data using explicit clusters (e.g., patients, domains, spatial neighborhoods, communities) derived from metadata or unsupervised clustering.
• Evaluate performance on left-out blocks or whole clusters to assess extrapolative, out-of-cluster generalization.
• Control for cluster size, balance, and heterogeneity to reduce estimation variance and avoid over- or underrepresentation.

The motivation extends to unsupervised model selection (e.g., the number of clusters), where explicit cluster-based splits are required to define predictive tasks in the absence of labels [(Fu et al., 2017); (Chen et al., 2014)].

2. Methodological Classes and Algorithms

A broad taxonomy of cluster-based cross-validation schemes includes:

A. Leave-One-Cluster-Out (LOCO-CV) and Block CV:

Entire clusters (subject groups, spatial blocks, communities, or discovered clusters) are left out in turn, assessing model performance on data not seen in any form during training. Used in regression/classification with cluster structure (Qiu et al., 2024, Yuval et al., 20 Feb 2025, Durdy et al., 2022), network analysis [(Chen et al., 2014); (Kawamoto et al., 2016)], materials science (Durdy et al., 2022), and spatial modeling (Cooper et al., 22 Apr 2025).

B. Cluster-Aware Fold Formation and Validation Splits:

Folds are constructed to maximize or preserve inter-fold cluster diversity or explicitly maximize statistical discrepancies (e.g., through MMD) between training and validation (Napoli et al., 2024). This includes domain-aware splits using metadata or unsupervised kernel k-means, subject to balance and label constraints.

C. Bi-Cross-Validation and Gabriel-Type CV:

Specialized for unsupervised learning, notably k-means and spectral clustering. Rows and columns are split for cross-prediction, or block matrices are held out to select the number of clusters and tuning parameters (Zohar et al., 2019, Fu et al., 2017).

D. Bayesian and Hierarchical Cluster-Informed CV:

Cross-validation is integrated with cluster estimation or uncertainty quantification, as in Bayesian cross-study validation with random study partitions (Trippa et al., 2015), hierarchical discriminant analysis (Hirose et al., 2021), or block-level model averaging (Yu et al., 2024).

E. Cluster-Validated Unsupervised Discovery:

Cluster structures estimated on a discovery set are validated using held-out data, either by refitting (method-based) or classifying new data using cluster centroids/memberships learned in the discovery set (result-based) (Ullmann et al., 2021).

3. Theoretical Guarantees and Performance Implications

Cluster-based CV procedures often come with specific theoretical properties and empirical advantages:

• Bias correction and generalization: In settings with cluster, spatial, or other non-i.i.d. dependencies, standard CV can be severely biased. Bias-corrected estimators using leave-one-cluster-out or explicit covariance corrections restore unbiasedness and consistency in estimating mean generalization error, applicable to regression, GLMMs, deep networks, and more (Yuval et al., 20 Feb 2025).
• Model selection and under/overfitting: When used for hyperparameter selection (e.g., number of clusters $k$ in k-means or communities $K$ in SBMs), cluster-based CV yields consistent estimators, provably avoids underfitting (with $P\{\hat K < K\}\to 0$) in networks (Chen et al., 2014), or identifies saturating regimes for model complexity using cross-entropy errors (Kawamoto et al., 2016). In unsupervised CV, explicit error-minimization on held-out clusters recovers true $K$ in homoskedastic or highly-structured settings (Fu et al., 2017).
• Variance reduction: Block/joint scoring for spatial and group-based CV reduces the variance of predictive estimates compared to pointwise scoring, improving reliability of model selection in the presence of strong intra-block correlation (Cooper et al., 22 Apr 2025).
• Domain shift and distributional robustness: Holding out clusters representing distinct latent domains enhances out-of-distribution (OOD) model evaluation and generalization diagnostics (Napoli et al., 2024). Maximizing MMD between training and validation sets correlates with higher test-domain accuracy (Spearman $\rho=0.63$), establishing it as a diagnostic for robust model selection under domain shift.

4. Algorithmic Patterns and Practical Implementation

Implementation details vary across domains and methodologies:

• Cluster Assignment: Clusters can be explicit (based on metadata like patient or site) or learned using k-means, kernel k-means, DBSCAN, agglomerative, or other clustering algorithms. For spatial data or networks, clusters may respect natural contiguity or community structure [(Chen et al., 2014); (Cooper et al., 22 Apr 2025)].
• Block/Fold Construction: In cluster-based fold formation, each fold may include points from all clusters (to guarantee stability of representation) or sample entire clusters per fold (for extrapolation/OOD evaluation) (Spezia et al., 30 Jul 2025). Algorithms for constrained clustering, such as linear programming to enforce sample size or label balance, are used in advanced settings (Napoli et al., 2024).
• Computational Complexity: Cluster-based CV can be substantially more intensive (often requiring $M$ or $n\times S$ refits). Fast approximations such as the Network Information Criterion (NICc) (Qiu et al., 2024) or second-order Taylor expansions (SEAL for model averaging CV) (Yu et al., 2024) mitigate this by providing leave-one-cluster analogs to classical AIC/BIC or analytic CV approximations.
• Diagnostics and Hyperparameter Tuning: Cluster size unevenness and spread must be controlled to ensure stable CV outcomes (Durdy et al., 2022). Cluster size and number may be chosen via data-driven surrogate metrics (e.g., "elbow" in cluster size variance). In unsupervised CV, row-column splits and bi-cross-validation guide $k$ selection and denoising (Fu et al., 2017, Zohar et al., 2019).

5. Empirical Performance and Comparative Analyses

Extensive benchmarks and comparative studies reveal nuanced strengths and actionable recommendations:

• Bias and variance: Mini-Batch K-Means with class stratification (SCBCV Mini) achieves lowest bias and variance in balanced datasets, though not in imbalanced settings, where stratified CV is superior (Spezia et al., 30 Jul 2025). Pure cluster-based CV methods show subtle dataset-dependent differences, and no one clustering algorithm is uniformly best.
• Domain adaptation and generalization: Maximizing MMD between train/validation, as implemented via constrained kernel k-means, yields superior model selection and test-domain accuracy for domain generalization and UDA tasks, outperforming standard random and leave-one-domain-out splits (Napoli et al., 2024).
• Utility in non-i.i.d. and block-correlated data: In regression, spatial modeling, and deep learning with correlated data, cluster-based (leave-one-group-out) CV with explicit bias correction eliminates underestimation of error and leads to substantially improved model selection and evaluation (Yuval et al., 20 Feb 2025, Yu et al., 2024, Qiu et al., 2024).
• Unsupervised learning: In cluster analysis and spectral clustering, Gabriel and bi-cross-validation strategies successfully select $k$ in settings with complex structure, high-dimensional noise, or domain shift, outperforming traditional gap statistic, BIC, and other unsupervised selection methods, and providing interpretable and parsimonious clusterizations (Fu et al., 2017, Zohar et al., 2019).

6. Domain-Specific Extensions and Advanced Directions

Cluster-based cross-validation is a modular principle with multiple domain applications:

• Networks and community detection: Network cross-validation (NCV; blockwise node-pair splitting) provides scalable and consistent model selection for both SBM and DCBM, with theoretical guarantees of asymptotic no-underfitting and practical block-wise pseudocode for implementation (Chen et al., 2014). Related LOOCV strategies using belief-propagation provide edge-prediction-based cluster selection down to the information-theoretic detectability threshold (Kawamoto et al., 2016).
• Spatial statistics and Bayesian modeling: Joint block-scoring for spatial CV using leave-group-out schemes leads to increased Z-ratios (effect-size of model selection) when comparing Bayesian spatial models, especially in the presence of strong spatial dependence (Cooper et al., 22 Apr 2025).
• Multi-study/multi-domain analysis: Bayesian nonparametric CV, as in cross-study validation, clusters heterogeneous datasets and quantifies performance as a function of the posterior study partition, facilitating robust model evaluation under heterogeneity and integrating uncertainty in cluster assignments (Trippa et al., 2015).
• Model averaging and computational efficiency: Unified model averaging estimators can be constructed through leave-cluster-out CV, with the computational burden addressed by the SEAL approximation. This ensures risk-optimality and scalability in non-i.i.d. clustering contexts (Yu et al., 2024).

7. Best Practices, Limitations, and Recommendations

Best practices and limitations are context-dependent:

• Stratify by class in presence of imbalance; use SCBCV Mini or analogous hybrid cluster-stratified techniques for balanced datasets to further reduce bias and variance.
• Choose block size and cluster number to balance independence versus sufficient sample size, referencing spatial/geometric range in spatial models or empirical diagnostics (e.g., cluster size variance) in tabular data.
• Apply leave-one-cluster-out or block-based CV whenever train/test dependencies or domain shifts are present, as standard CV will tend to underestimate error.
• Control for cluster assignment variability by averaging across multiple random restarts or using methods robust to stochasticity (e.g., kernelized clustering with hyperparameter tuning) (Durdy et al., 2022).
• Employ analytic or efficient approximations (NICc, SEAL, fast leave-one-out updates) when computational cost is prohibitive.
• Acknowledge caveats: Small numbers of clusters, noisy or misaligned block definitions, and highly imbalanced block sizes can all degrade the validity of cluster-based cross-validation (Qiu et al., 2024).

These principles enable statistically valid model selection and evaluation across a range of structured, correlated, and heterogeneous data environments, providing robust alternatives to traditional cross-validation strategies.