Biased Train-Test Splitting Strategies

• Biased train-test splitting strategies are systematic deviations from random splitting that induce distributional mismatches and expose model bias in various learning scenarios.
• Methods include adversarial, class-balancing, and group-aware partitions, each employing rigorous algorithmic formulations and novel metrics to control bias and assess generalization.
• These strategies improve model evaluation by revealing failure modes, optimizing worst-case accuracy, and offering actionable insights for bias correction and robust performance.

A biased train–test splitting strategy is any systematic deviation from random, exchangeable splitting that induces distributional or structural mismatches between the train and test sets, with explicit implications for model selection, performance evaluation, and assessment of generalization in supervised, semi-supervised, or relational learning. These strategies serve to reveal, measure, or mitigate failure modes associated with under-represented data regions, class imbalance, model misspecification, or methodological leakage. Approaches include adversarial splitting, class-balancing, group- or structure-aware partitioning, similarity- or dissimilarity-based exclusion, temporal separation, and optimal subsampling frameworks. Each method is operationalized through rigorous algorithmic and theoretical formulations that expose or control the sources and magnitudes of bias, frequently accompanied by reporting protocols, new evaluation metrics, or explicit correction procedures.

1. Mathematical Foundations and Taxonomy of Splitting Bias

Random train–test splitting is exchangeable over the input–label pairs and preserves marginal population statistics in expectation. Bias is introduced when the assignment mechanism renders the conditional $\mathbb{P}(z_i = 1 \mid x_i, y_i)$ non-uniform, uses side information (temporal, structural, group), or prioritizes under- or over-represented data subdomains. The sources of bias include:

• Coverage Bias: The failure of the split to reflect rare or boundary-case data regions, impacting class inclusion and support (Khan, 2022).
• Representation Bias: Disparities between train/test class proportions, label distributions, or feature-marginals (Khan, 2022, Jami et al., 25 Sep 2025).
• Overfitting-by-Selection Bias: Downward test error bias caused by hyperparameter or model selection performed directly on the test or validation data (Guan, 2018, Jiao et al., 8 Nov 2025).
• Structural Leakage: Usage of structurally non-independent splits (e.g., overlapping groups, temporal leaks, near-duplicate propagation) resulting in artificially optimistic estimates (Jiao et al., 8 Nov 2025, Laroca et al., 2023).

Formally, given data $D = \{(x_i, y_i)\}_{i=1}^n$, a splitting policy is a mapping $S: D \mapsto (D^{\text{train}}, D^{\text{test}})$ (or $S(\phi)$ parameterized by $\phi$ as in adversarial splits (Bao et al., 2022)), with associated measures of optimality or representativeness, such as class coverage rates, KL-divergence regularizers, Wasserstein distance between distributions, or support-point energy minimization (Bao et al., 2022, Jami et al., 25 Sep 2025, Joseph et al., 2020).

2. Adversarial and Bias-Maximizing Splits

Adversarial splitting strategies are designed to construct train/test partitions that maximize generalization error or reveal spurious correlations. "Learning to Split" (LS) (Bao et al., 2022) operationalizes this by defining a bipartition $z_i \in \{0,1\}$ to maximize the predictor's generalization gap $$\Delta(\theta, z) = L_{\text{test}}(\theta; z) - L_{\text{train}}(\theta; z)$$, subject to regularizers for train/test size and label-balance (KL-divergence constraints). LS iteratively alternates between inner-loop optimization of the splitter (minimizing cross-entropy losses favoring misclassified test examples) and outer-loop retraining of the predictor, producing challenging splits that isolate under-represented or "hard" data slices known to induce model bias. Empirical evidence confirms that LS splits correlate with human-identified bias in vision, NLU, and molecular domains, dramatically reducing worst-group performance relative to random splits, and when used as pseudo-group labels for Group DRO, substantially improve the worst-case accuracy (+23.4% averaged gain across tasks).

3. Class-Imbalance and Representation-Aware Splitting

For classification under severe class imbalance, random and stratified splits allow majority-class dominance and minority-class exclusion. The "Balanced Split" (Khan, 2022) enforces exact per-class equality in training size by sampling $k=\lfloor \text{TrS}/N \rfloor$ examples per class, given $$\text{TrS} = \lfloor \text{train-ratio} \times m \rfloor$$ and $N$ classes, subject to $\text{TrS}/N < \min_i m_i$. This construction ensures zero coverage and representation bias in training, eliminates cases where minority classes are absent from train, and directly neutralizes the prior-probability effect, although at the cost of increased estimator variance in small-sample regimes. Empirical studies show Balanced Split yields clear improvements in F1 and accuracy over random/stratified alternatives, especially as class-imbalance severity grows, with gains up to 0.628 F1 versus 0.51 (Random Forest, $tr=0.90$), and typical accuracy improvements of 3–8 points.

4. Structure- and Group-Aware Splitting: Handling Rare, Boundary, or Correlated Data

When data exhibit group, temporal, or relational structure, or where rare/outlier regions are critical for risk assessment, structure-aware and group-aware splitting strategies are essential. Dissimilarity-based, informed (group-holdout), and clustering-based splits (Catania et al., 2022) construct test sets to stress the model on rare, boundary, or unobserved subspaces, using explicit groupings, clustering algorithms, or greedy max-dissimilarity assignment. Monte Carlo (random) splits serve only as best-case, whereas informed and clustering-based splits simulate "novel domain" or "worst-case" scenarios, exposing brittleness to unseen modes and yielding pessimistic but robust generalization estimates. These approaches are validated by pronounced shifts in median balanced accuracy (e.g., -16pp dissimilarity vs. -6pp informed vs. MC), with variance reflecting the induced distributional gap between train/test.

Special attention is required for time-dependent and multi-relational datasets (e.g., in link prediction (Jiao et al., 8 Nov 2025) or exchangeable bipartite graphs (Veitch et al., 2017)). In link prediction, two-set splits with hyperparameter tuning on the test set lead to measurable performance inflation (Loss Ratio $\overline{\mathcal{L}} = 3.6\%$, up to 15%), especially for parameterized and deep models. Only three-set (nested) splits with strict separation of training, validation, and test guarantee unbiased hyperparametric evaluation.

In sparse network settings, the only non-biased splitting is (p,q)-sampling: users and items are retained independently Bernoulli with rates $p, q$, producing induced subgraphs that preserve the original model’s exchangeable law and marginal degree distributions (Veitch et al., 2017). Naïve edge or node holdouts systematically alter degree spectra and inflate/deflate ranking metrics, especially for high-degree or "long tail" nodes.

5. Instance- and Similarity-Based Stratified Splits

Similarity-Based Stratified Splitting (SBSS) (Farias et al., 2020) and support-point-based SPlit (Joseph et al., 2020) exploit the feature geometry beyond label stratification. SBSS forms folds such that the $k$ most-similar (by specified metric) samples of each class are guaranteed to be in different folds, thereby enforcing coverage and reducing the likelihood that a localized region of feature space is unseen in training. Empirically, SBSS reduced train–test gap by 11.5% (correlation similarity) and improved test accuracy in 75% of scenarios across 22 UCI datasets with multiple classifier families.

The SPlit method applies the support-point optimal design theory, selecting test subsamples that minimize energy distance between test and full-data distributions. This minimizes both mean and worst-case test error, with empirical improvements in worst-case out-of-sample error of up to 18%, and halves the standard deviation relative to random splits. Support-point construction can be adapted to categorical variables via Helmert coding, ensuring balanced coverage in mixed-type domains.

6. Segmentation, Multi-Label, and Complex Structure Splits

In pixel-level or multi-label image segmentation, random splits produce folds with pronounced distributional shifts in per-class pixel occupancy, leading to high fold-to-fold dispersion in IoU or F1 for rare classes (Jami et al., 25 Sep 2025). Iterative Pixel Stratification (IPS) and Wasserstein-Driven Evolutionary Stratification (WDES) address this by minimizing deviation in global and fold-wise label proportions. WDES, in particular, uses a genetic algorithm to minimize 1-Wasserstein distance between fold and global label distributions, achieving globally optimal splits that are empirically superior across metrics—PLD and LWD for PascalVOC improved from 955±137 (random) to 456±78 (WDES), with IoU standard deviation dropping from 0.0343 (random) to 0.0242 (WDES). These advanced stratification methods are critical for low-entropy, highly imbalanced segmentation tasks, where random splits dramatically misrepresent model robustness.

7. Bias Correction, Overfitting, and Empirical Risk Adjustment

Biased splitting not only inflates or underestimates model performance but also induces optimistic selection bias in test error, especially after hyperparameter model selection (Guan, 2018). Bias-corrected estimators—including two-fold bias-contrast and randomized error estimators—achieve $o(1/\sqrt{n})$ bias and allow for valid bootstrap confidence intervals without additional model refitting, even after model selection steps that use validation errors.

In generalized empirical risk minimization under covariate shift or selection bias, weighted ERM with density-ratio correction $w(x)=\frac{dP_{\text{test}}}{dP_{\text{train}}}(x)$ is essential to recover unbiased test risk from biased splits (Clémençon et al., 2019). The learning rate remains optimal $O(1/\sqrt{n})$ provided weights are bounded and well-estimated.

For inference and hypothesis testing with multiple random splits, averaged split-sample estimators (repeated cross-fitting) vastly improve reproducibility and power, and their asymptotic distributions are controlled by recently established CLTs with explicit variance inflation factors accounting for overlap in train/test partitions (Fava, 7 Nov 2025).

8. Practical Guidance and Implications

The rigorous application of splitting strategies must be tailored to the data domain, task structure, and known sources of bias. In large-scale evaluation or benchmarking, strict partitioning protocols (e.g., three-set splits for link prediction (Jiao et al., 8 Nov 2025), group-holdout for instance-leakage (Laroca et al., 2023)) are mandatory to avoid information leakage and ensure faithful generalization estimation. For highly imbalanced, structured, or rare-event detection regimes, only balance- or coverage-optimizing splits yield stable and non-optimistic error estimates. Further, best practices demand reporting of fold-specific dispersion, worst-case performance, and group-level metrics (e.g., worst-group accuracy), not just aggregate means, and adoption of explicit bias-correction procedures when performing model selection by validation error. The methods summarized constitute the contemporary arsenal for controlling and leveraging splitting bias in machine learning research and practice.