---
title: 'Cross-Fitting: Sample Splitting'
url: https://www.emergentmind.com/topics/cross-fitting-sample-splitting
type: topic
---

# Cross-Fitting: Sample Splitting

Cross-fitting, also known as sample splitting or out-of-fold evaluation, is a statistical methodology designed to eliminate empirical-process bias that arises when estimating target functionals using data-adaptive nuisance parameter estimates. Cross-fitting is now central to high-dimensional and nonparametric inference, particularly in causal machine learning, semiparametric estimation, changepoint detection with flexible models, high-dimensional two-sample testing, and variance-debiased estimation in complex dependence structures.

## 1. Foundational Definition and Rationale

In the canonical semiparametric or causal inference paradigm, the parameter of interest is often of the form $\theta = \E[f_{\eta_0}(X)]$, where $f_{\eta_0}$ depends on one or multiple nuisance functions $\eta_0$ (such as propensity scores or outcome regressions). Plug-in estimators that use machine-learned $\hat{\eta}$ trained on the entire sample induce a bias term of the form $(P_n - P)[f_{\hat{\eta}} - f_{\eta_0}]$. This empirical-process bias can dominate stochastic fluctuation, especially with high-dimensional or highly adaptive $\hat{\eta}$, and may violate the conditions for root-$n$ consistency and valid CLT-based inference [2601.10899][1801.09138][2201.12692][2012.00745].

Cross-fitting eliminates this bias by splitting the sample into $K$ folds and, for each fold, evaluating $f_{\hat{\eta}^{(-k)}}(x_i)$ using a nuisance function trained only on the complement of that fold. The cross-fitted estimator aggregates the out-of-fold functionals:
$$
\hat\theta_{\text{CF}} = \frac{1}{n} \sum_{k=1}^K \sum_{i\in I_k} f_{\hat{\eta}^{(-k)}}(X_i)
$$
where $I_1, ..., I_K$ are the index sets for the $K$ folds. This ensures that $f_{\hat{\eta}^{(-k)}}(X_i)$ and $X_i$ are independent conditional on the training split, neutralizing own-observation bias and permitting use of complex ML methods for $\hat{\eta}$ [2201.12692][2601.10899][2207.11932].

## 2. Cross-Fitting Protocols and Theoretical Guarantees

The essential algorithm for $K$-fold cross-fitting is as follows:
1. Partition $\{1,...,n\}$ into $K$ disjoint (roughly equal-sized) folds $I_1,...,I_K$.
2. For each $k$:
    - Train nuisance $\hat{\eta}^{(-k)}$ on $\bigcup_{\ell\neq k} I_\ell$.
    - Compute $f_{\hat{\eta}^{(-k)}}(X_i)$ for $i\in I_k$.
3. Aggregate across folds as above.

This protocol applies to doubly robust estimation, meta-learners for CATE, sample selection models, AIPW/TMLE, and many others [2601.10899][2411.07874][2012.00745][2201.12692][1801.09138][2405.15242], with extensions to three-way cross-fitting (using separate folds for each nuisance and the final regression) when higher-order orthogonality is required [2306.07230][2403.15175].

### Main Theoretical Results
- **Empirical-process bias elimination:** Cross-fitting ensures $(P_{S_2} - P)[f_{\hat{\eta}_{S_1}} - f_{\eta_0}] = o_P(1/\sqrt{n})$ under $\sqrt{n}$-rate CLTs for $P_n$ and mild conditions on nuisance estimation error and dependence structure [2601.10899].
- **Orthogonal (doubly robust) scores:** With Neyman-orthogonal estimating equations (as in DML or DR learners), cross-fitting yields root-$n$ consistency and asymptotic normality for $\hat\theta_{\text{CF}}$, provided $\hat{\eta}$ converges at $o_P(n^{-1/4})$ rates in $L_2$ [2012.00745][2207.11932][2201.12692][1801.09138].
- **Variance estimation:** Cross-fitted influence functions (scores evaluated using out-of-fold nuisances) are used for asymptotic variance estimation and Wald-type confidence intervals [2405.15242][2012.00745][2207.11932].
- **Correlated and dependent data:** The method remains valid under weak dependence—including m-dependence, $\beta$-mixing, timeseries, clusters, spatial/network structures—so long as the number of correlated pairs is $o(n^2)$ at the relevant rate [2601.10899][1902.07425].

## 3. Cross-Fitting Under Correlation, Dependence, and Non-IID Sampling

Although cross-fitting was originally motivated in the IID setting, its applicability extends to dependent data. The core result [2601.10899] demonstrates that cross-fitting as if the data were IID (randomly assigning units to folds, ignoring dependence) remains asymptotically valid for debiasing the empirical process under:
- A dependent-data CLT for $P_n f_{\eta_0}$
- Well-behaved $L_2$ error for nuisance fits
- Subquadratic growth of the number of correlated pairs, i.e. $o(n^2)$

This obviates the need for bespoke fold construction in most clustered, time-series, or networked settings. In fact, purposely packing highly correlated units into the same fold typically *increases* the variance of the empirical process term, and therefore generic K-fold partitioning is recommended for bias elimination [2601.10899].

**Limitation:** If correlation is extremely dense (e.g., long-range dependence where most pairs are correlated), specialized splitting may still be needed. Further, cross-fitting does not address variance estimation in the presence of correlation; cluster-robust or HAC variance estimators must still be used [2601.10899][1902.07425].

In randomized experiments under a design-based framework (fixed covariates and outcomes; randomization only in treatment), standard cross-fitting ceases to be unbiased because splitting induces dependence between folds. Conditional cross-fitting employs splits defined *conditional on the treatment assignment* so that for each fold, the assignment mechanism remains valid and independent across splits; this preserves unbiasedness and efficiency [2508.15664].

## 4. Advanced Cross-Fitting Designs: Double/Triple Splitting and Averaging

### Double and Triple Cross-Fitting
Double cross-fitting (DCDR)—using independent folds for separate nuisance functions—tightens control of higher-order bias and achieves faster rates and minimax-optimality in non-$\sqrt n$ regimes under Hölder smoothness. Triple (or more generally, $M$-way) cross-fitting separates folds for each of $M$ nuisance components as well as the main regression, removing both first- and second-order bias terms and accelerating convergence [1801.09138][2306.07230][2403.15175].

### Repeated Splitting and Aggregation
To reduce estimator variance and increase inferential reproducibility, practitioners often repeat the cross-fitting procedure across $B$ random partitions and aggregate (typically by mean or median) the resulting estimator values. Aggregating multiple splits stabilizes estimators and p-values, mitigates the "p-value lottery" seen with a single split, and is formally justified via central limit theorems that account for the dependence structure of overlapping splits [2511.04957][2007.02852][1210.4584].

| Aggregation Technique       | Main Purpose                          | Empirical Impact                 |
|----------------------------|---------------------------------------|----------------------------------|
| Mean over $B$ splits       | Lowered variance, higher efficiency   | Generally reduced MSE            |
| Median over $B$ splits     | Robustness to outliers/split artifacts| Estimator less sensitive to tails|

## 5. Application Domains and Empirical Behavior

Cross-fitting is now integral to modern double/debiased machine learning (DML) [2012.00745][2207.11932], heterogeneous treatment effect meta-learners (e.g., DR-learner, R-learner, T-learner) [2201.12692][2007.02852][2306.07230], high-dimensional two-sample testing [1210.4584], changepoint detection in complex models [2411.07874], post-selection inference [1908.03967], and variance-debiased evaluation in A/B testing and sequential trials [2512.03366].

**Empirical findings include:**
- In large-sample and high-dimensional settings, cross-fitting robustly eliminates overfitting bias, enabling theoretically justified use of highly adaptive or black-box ML models for nuisance fitting [2201.12692][2405.15242].
- In small samples, repeated or full-sample estimation may outperform due to reduced size of the training set per fold [2201.12692][2007.02852].
- When deployed with doubly robust scores (Neyman orthogonality), cross-fitting delivers root-$n$ rates and semiparametric efficiency bounds for a wide variety of estimands [2012.00745][1801.09138].
- In time-series and spatial contexts, standard cross-fitting maintains inferential validity under mild mixing or dependence conditions and is simpler than block or buffer techniques [2601.10899][1902.07425].

## 6. Limitations, Alternatives, and Open Problems

- **Variance Estimation:** Cross-fitting only debiases point estimates; standard errors must account for cross-sectional, cluster, or time-series dependence using robust estimators separate from the cross-fitting protocol [2601.10899][1902.07425][2405.15242].
- **Dense Dependence:** When the number of correlated pairs is $O(n^2)$ (e.g., no decay in dependence), cross-fitting as-IID may incur inflated variance. Research into fold designs that explicitly minimize finite-sample variance in these settings is ongoing [2601.10899].
- **Low Sample Regimes & Stability:** Where sample sizes are small or cross-fitted nuisances are highly variable, leave-one-out–stable ML algorithms (e.g., bagging, ensemble methods) can recover root-$n$ inference without cross-fitting, provided stability conditions hold [2206.01825][2602.11333].
- **Conditional Cross-Fitting in Experiments:** In non-IID randomized experiments, carefully designed conditional cross-fitting ensures unbiasedness and asymptotic normality without assuming a super-population model [2508.15664].
- **Changepoint Detection and Model Selection:** For complex fitting tasks (e.g., high-dimensional changepoint detection), out-of-sample (cross-fit) loss targeting is essential to avoid degenerate or biased minima induced by overfitting on in-sample loss [2411.07874].
- **Hyperparameter Sensitivity and Averaging:** Aggregating results over multiple random splits by median is empirically more robust than mean in finite samples, especially in the presence of outlier nuisance fits [2007.02852].

## 7. Summary Table: Theoretical and Practical Aspects

| Aspect                       | Result/Implication                                              | Reference      |
|------------------------------|-----------------------------------------------------------------|----------------|
| Bias removal                 | Cross-fitting yields $o_P(1/\sqrt n)$ empirical-process bias    | [2601.10899]   |
| Efficiency                   | DR cross-fitting achieves semiparametric efficiency under $o_P(n^{-1/4})$ rates for each nuisance | [2012.00745][2207.11932] |
| Dependent data               | As-IID cross-fitting valid under CLT, weak dependence           | [2601.10899]   |
| Small $n$                    | Full-sample or out-of-bag preferable for nuisance estimation    | [2201.12692]   |
| Double/triple cross-fitting  | Accelerates rates, achieves efficiency under minimal smoothness | [1801.09138][2403.15175][2306.07230] |
| Variance estimation          | Requires robust estimators (not handled by cross-fitting per se)| [2601.10899][2405.15242] |
| Conditional design           | Conditional cross-fitting restores unbiasedness under finite populations | [2508.15664]   |

Cross-fitting is a statistically rigorous, robust, and computationally tractable method for bias correction and valid inference in data-adaptive estimation with complex nuisance fitting, extending to correlated and dependent data, with wide adoption across modern causal and high-dimensional inference. Current methodological frontiers include optimizing cross-fitting for dense dependence structures, formalizing stability-based alternatives, and developing efficient implementations for highly structured experimental designs.

Source: https://www.emergentmind.com/topics/cross-fitting-sample-splitting