---
title: Cross-Fitting in Semiparametric Estimation
url: https://www.emergentmind.com/topics/cross-fitting
type: topic
---

# Cross-Fitting in Semiparametric Estimation

Cross-fitting is a sample-splitting methodology fundamental to semiparametric statistical estimation and high-dimensional causal inference. It separates model fitting (“nuisance” estimation) from evaluation to eliminate own-observation bias and relax empirical-process complexity assumptions, permitting the use of highly adaptive machine learning regressors. Cross-fitting underlies debiased/double machine learning, efficient estimation of average and heterogeneous effects, and general plug-in estimation of linear functionals. Its rigorous asymptotic properties hold for independent, clustered, time-series, networked, and even non-i.i.d. randomized design data, and, under appropriate conditions, lead to root-n consistency and semiparametric efficiency.

## 1. Cross-Fitting: Formalism and Algorithms

Cross-fitting partitions observed units $\{Z_i\}_{i=1}^n$ into $K$ disjoint folds $I_1, \ldots, I_K$ of size $n/K$. For each $k$, a model for nuisance parameter(s) $\hat{\eta}^{I_k^c}$ is trained on the complement $I_k^c$, and its predictions are plugged into the estimating equations evaluated on the held-out fold $I_k$. The generic K-fold cross-fitted estimator for a statistical functional $\beta_0 = \mathbb{E}[m(Z, \eta_0)]$ is
\[
\hat\beta_{\mathrm{CF}} = \frac{1}{n} \sum_{k=1}^K \sum_{i\in I_k} m(Z_i, \hat{\eta}^{I_k^c}).
\]
This construction ensures that $\hat{\eta}^{I_k^c}$ is independent of the data used for evaluation, breaking the dependency that causes overfitting bias.

For doubly robust functionals involving multiple nuisances (e.g., regression and propensity), cross-fitting can be extended to “doubly cross-fitted” estimators, where different nuisance components are fit on non-overlapping partitions, completely eliminating higher-order bias terms associated with own-observation and estimator nonlinearity. The paradigm generalizes to conditional functionals and heterogeneous effect estimation, with further modifications for multiple treatments and complexities of the design, as in randomized experiments or changepoint models [2207.11932][1801.09138][2007.02852][2306.07230][2411.07874].

## 2. Bias Elimination, Rate Conditions, and Efficiency

Cross-fitting decomposes the estimation error of plug-in estimators into:
1. A main term corresponding to the mean-zero empirical process.
2. A low-order nuisance-bias, $\mathbb{E}[m(Z, \hat{\eta}) - m(Z, \eta_0)]$, which is second order when the functional is Neyman-orthogonal.
3. The empirical-process bias, $(\mathbb{P}_n - \mathbb{P})[m(Z, \hat{\eta}) - m(Z, \eta_0)]$, which is the principal obstacle and is eliminated by cross-fitting because fitting and evaluation occur on independent data.

Under appropriate consistency rates for the nuisance estimators—e.g., product of $L_2$ risks $o_p(n^{-1/2})$ for doubly robust estimators or pointwise risk $o_p(n^{-1/4})$ for plug-in—the cross-fit estimator attains root-n consistency, asymptotic linearity, and achieves the semiparametric efficiency bound:
\[
\sqrt{n} (\hat\beta_{\mathrm{CF}} - \beta_0) = \frac{1}{\sqrt{n}}\sum_{i=1}^n \mathrm{IF}(Z_i) + o_p(1)
\]
where $\mathrm{IF}$ is the efficient influence function for the parameter of interest. The smallest achievable remainder rate for average linear functionals with multi-stage series estimation is 
\[
A_n^* = n^{-1/2} K^{-(s_y+s_a)/r} + K^{-s_y/r} + K^{-s_a/r} + K/n
\]
with $K$ the basis size, $s_y, s_a$ the Hölder smoothness of the respective nuisances, and $r$ the covariate dimension [1801.09138].

## 3. Extensions to Complex Dependence Structures

Contrary to prior intuition, cross-fitting remains effective for dependent data—spatial, clustered, networked, and time-series settings—so long as an appropriate central limit theorem for averages over independent units (or effective sample size) holds and the total number of strongly correlated pairs grows no faster than $O(n^2 / r_n^2)$, with $r_n$ the CLT rate [2601.10899]. Standard random K-fold splits suffice; there is no need to “block” or “cluster” the folds to minimize dependence. The main theoretical guarantee is that the empirical-process term is still controlled at the parametric rate $o_p(1/r_n)$, so estimator validity and efficiency are preserved.

Practical guidelines are summarized in the table below:

| Setting         | Cross-Fitting Required? | Modifications Needed       |
|-----------------|------------------------|---------------------------|
| IID data        | Yes                    | Standard random folds     |
| Clustered data  | Yes                    | No cluster-specific folding needed |
| Network data    | Yes                    | As above                  |
| Time series (m-dep) | Yes                | Standard folding sufficient|

Empirically, cross-fitting without correlation-aware splitting yields equivalent or improved bias/variance compared to more complex partitioning, particularly in finite samples [2601.10899].

## 4. Conditional and Adaptive Cross-Fitting in Experimental Designs

In classical randomized experiments with dependent assignments (e.g., block, stratified, or completely randomized designs), traditional cross-fitting based on i.i.d. random splits is invalid, as assignment vectors across folds are not independent. Conditional cross-fitting solves this by selecting splits such that, *given* the split, assignments in different folds are conditionally independent under the design [2508.15664]. Examples include:
- Bernoulli randomized experiments: split units independently.
- Completely randomized experiments: split within each arm to preserve treatment group sizes.
- Stratified randomized experiments: split within strata or by strata groups.

These procedures restore unbiasedness of estimator plug-in corrections, yielding unbiased covariate-adjusted ATE estimators even in finite samples, regardless of the accuracy of machine-learning predictions. Valid confidence intervals are constructed using design-based variance estimators, and the method accommodates arbitrary regression learners with only mild stability requirements [2508.15664].

## 5. Multi-Way and Three-Way Cross-Fitting Strategies

While classical two-way (or K-fold) cross-fitting suffices for standard bias reduction, further splitting is sometimes used to disentangle dependencies among multiple nuisance estimators, accelerating remainder decay and enhancing finite-sample performance.
- Three-way cross-fitting partitions the data into three equal parts; each is used exclusively for a single stage (e.g., one for estimating the propensity, one for the outcome regression, and one for final pseudo-outcome regression). Bias terms associated with dependence among nuisance estimators are thereby reduced from $O(k_n/n)$ to $O(k_n^{1/2-s/r}/\sqrt{n})$ or better under smoothness conditions [2306.07230].
- In plug-in estimation of general linear functionals, double cross-fitting ensures that remainder rates match the best known minimax-optimal rates in high-dimensional estimation [1801.09138].

However, increasing the number of splits reduces the data available in each partition for model fitting, potentially compromising nuisance estimation accuracy. In practice, one rotates splits and averages over multiple random partitions to mitigate efficiency loss [2306.07230][2007.02852].

## 6. Applications and Scope

Cross-fitting is used widely for:
- Doubly robust estimation of average and heterogeneous treatment effects with flexible (non-Donsker) machine learners [2207.11932][2007.02852].
- Covariate adjustment in randomized controlled trials, meta-learners (DR, R, T, X-learners), and distributional functionals under missing data.
- Changepoint detection, wherein cross-fitting replaces in-sample loss minimization to counteract overfitting bias and guarantees recovery of changepoints with high probability, even under highly adaptive ML estimators and automated hyperparameter tuning [2411.07874].

In all these domains, cross-fitting enables aggressive use of black-box predictive models while preserving rigorous inferential validity, provided rate conditions and design-consistency properties are verified in context.

## 7. Limitations and Alternatives

Despite its foundational role, cross-fitting can present computational burdens in complex models, especially in clustered data settings, where re-fitting nuisances on multiple splits significantly increases required resources. In high-dimensional dependence regimes (multiway clustering, separately exchangeable arrays), recent advances prove that Neyman-orthogonal moment conditions and empirical-process localization arguments can eliminate the need for cross-fitting entirely, without loss of asymptotic linearity or validity of standard error estimators [2602.11333]. In such cases, full-sample estimation of nuisance parameters is provably justified, regaining efficiency and computational practicality.

Practical limitations include:
- Need for sufficiently large data within each fold to fit stable nuisance estimators.
- Correct design-based splitting mechanisms for conditional cross-fitting in complex experimental designs.
- Trade-off between split count (K) and variance in final estimators; typical values are $K=2$–$5$.
- Absence of off-the-shelf cross-fitting folding mechanisms for certain adaptive or rerandomized designs [2508.15664].

In summary, cross-fitting represents a central tool in the modern theory and practice of semiparametric estimation under flexible machine learning, with formal theoretical guarantees, broad applicability, and proven efficiency recovery in a range of complex data-generating scenarios [2207.11932][2601.10899][1801.09138][2508.15664][2007.02852][2602.11333][2306.07230][2411.07874].

Source: https://www.emergentmind.com/topics/cross-fitting