Cross-Fitting Algorithm

• Cross-Fitting is a sample-splitting method that estimates nuisance functions out-of-fold to reduce overfitting bias.
• It enables unbiased and efficient estimation in semiparametric models, causal inference, changepoint detection, and spatial statistics.
• The algorithm partitions data into folds, fits models on complementary subsets, and aggregates results to achieve oracle-level performance.

The cross-fitting algorithm is a sample-splitting method for unbiased nuisance regression and efficient estimation of functionals and predictive targets. Cross-fitting addresses the distortion and bias that arise when flexible learners (e.g., adaptive ML or hyperparameter-tuned models) are evaluated on data used for their own training. It is foundational in modern semiparametric estimation, causal inference, high-dimensional prediction, changepoint detection, spatial statistics, and robust randomized experiment analysis. The algorithmic principle involves partitioning the sample, fitting nuisance functions or predictive models out-of-fold, and evaluating their predictive or influence contributions on held-out samples, often cycling and averaging over folds to mitigate own-observation and nonlinear bias.

1. Motivation and Problem Statement

Cross-fitting was introduced to overcome finite-sample overfitting bias in complex model estimation and inference. In settings where data-adaptive or hyperparameter-optimized learners are deployed, naive in-sample evaluation severely understates predictive error, leading to inconsistent estimation and unreliable inference. This is exemplified in changepoint analysis, where in-sample loss minimization under flexible modeling can yield spurious changepoint selection due to overfitting the error criterion to segment idiosyncrasies (Qian et al., 2024). Similarly, causally robust estimation with ML nuisance fits, spatial intensity models, or covariate-adjusted ATE estimation in randomized trials requires unbiased estimation of nuisance or adjustment functions. Cross-fitting enables consistent estimation aligned with oracle/objective targets regardless of model adaptivity or data structure (Zeng, 2022, Lin et al., 2024, Lu et al., 21 Aug 2025).

2. Formal Algorithmic Definition and Notation

The canonical cross-fitting procedure operates on a partitioned sample, evaluating nuisance-regression and target estimation on held-out observations. Let $\mathcal{D} = \{z_i \in \mathcal{Z}: i=1,...,n\}$ denote a data sample.

• Partition $\{1,...,n\}$ into $L$ disjoint folds $I_1,...,I_L$.
• For each fold $\ell$, fit the nuisance estimator $\widehat{\nu}_{(\ell)}$ (e.g., regression, propensity score, intensity function) on $I_\ell^{c}$ (complement of $I_\ell$).
• Evaluate $\widehat{\nu}_{(\ell)}$ on $I_\ell$ to obtain out-of-sample risk or pseudo-outcomes.
• Average pointwise influence or contrast expressions to construct the estimate of interest.

For changepoint detection, cross-fitting operates on candidate segmentations $T$ with $M$ folds $J_1,...,J_M$ ordered as in (Qian et al., 2024). For spatial processes, V-fold thinning splits observed events, fitting nuisance components on $X_v^c$ and evaluating parametric likelihood on $X_v$ (Lin et al., 2024). In randomized experiment analysis under design-based inference, conditional independence of treatment assignments post-sample split is crucial to ensure unbiasedness (Lu et al., 21 Aug 2025).

3. Step-by-Step Algorithms Across Domains

Changepoint Detection (Qian et al.):

• For each candidate segment $(s,e)$, aggregate over $M$ folds:
  1. Fit $f_{I\setminus J_m}$ on $z_{I\setminus J_m}$.
  2. Compute $$\ell_m(s,e) = \sum_{i\in I\cap J_m} \ell(z_i; f_{I\setminus J_m})$$.
  3. Set $\mathsf{cost}(s,e) = \sum_{m=1}^M \ell_m(s,e)$.

• Global search for changepoints by minimizing $$\sum_{k} \mathsf{cost}(\tau_{k-1},\tau_k)+\gamma |T|$$ (Qian et al., 2024).

Semiparametric Causal Estimation (Zeng, Jacob, 3-way CF):

• Partition sample into $K$ folds; for fold $k$, fit nuisance regressions on $I_k^c$, evaluate AIPW/doubly robust scores on $I_k$.
• Average foldwise estimates: $$\hat\tau_{GCF, j, j'} = \sum_{k=1}^K \frac{|I_k|}{n} \hat\tau_{j,j'}^{I_k}$$ (Zeng, 2022).
• For heterogeneous effect estimation, repeat cross-fitting over $B$ randomizations, aggregate by median or mean for stable inference (Jacob, 2020).
• For three-way CF: partition into three folds, each estimating distinct nuisance component (e.g., propensity, outcome regression, pseudo-outcome regression), rotate and aggregate (Fisher et al., 2023).

Doubly Robust Estimation:

• Repeat sample splitting independently for different nuisance functions to avoid nonlinear bias terms (Newey et al., 2018).
• Construct plug-in or doubly robust estimators by averaging influence expressions, ensuring that each estimator is applied only out-of-fold.

Randomized Experiments (Conditional Cross-Fitting):

• Under design-based randomization, split units to ensure independent assignment vectors in each fold; fit ML predictors on one half, plug into Horvitz-Thompson ATE formula evaluated on the other (Lu et al., 21 Aug 2025).

4. Optimization Objectives and Consistency Theory

Cross-fitting seeks to minimize out-of-sample risk or construct asymptotically unbiased (often semiparametric efficient) estimates:

• For changepoint detection, the criterion is $$\min_T \sum_{k=1}^{K+1}\sum_{m=1}^M L(z_{(\tau_{k-1},\tau_k]\cap J_m}; f_{(\tau_{k-1},\tau_k]\setminus J_m}) + \gamma K$$, which aligns with oracle population loss (Qian et al., 2024).
• In semiparametric functional estimation, cross-fitted plug-in estimators and doubly robust estimators achieve root-$n$ remainder rates and efficiency bounds under minimal smoothness (Newey et al., 2018, Fisher et al., 2023).
• For causal inference and treatment effect heterogeneity, product-of-MSE rates for nuisance estimators guarantee validity even with aggressive ML methods, bypassing Donsker constraints (Zeng, 2022).
• For randomized experiments, cross-fitted adjusted ATE estimators are provably unbiased for the finite population effect, with conservative variance estimation and valid inference (Lu et al., 21 Aug 2025).

Consistency results are formalized via high-level conditions on tail control, predictive accuracy, signal separation (changepoints), uniform rates for ML nuisances, and cross-fold independence. In all settings, cross-fitting eliminates own-observation and nonlinear bias, ensuring rates matching oracle or population-level estimators.

5. Computational Complexity and Implementation Considerations

• Full cross-fitting over all segments or folds can be computationally intensive: $O(Mn^2)$ model fits for changepoint detection, $O(V)$ fits for spatial point process folds, $O(K)$ fits for ATE/CATE estimation (Qian et al., 2024, Lin et al., 2024, Zeng, 2022).
• Pruning or partitioning algorithms (PELT, seeded intervals, binary segmentation) reduce complexity in changepoint detection (Qian et al., 2024).
• Recycled cross-validation can reduce the number of required fits: using outer splits for inner CV, lowering model fits per segment from $M^2$ to $M$.
• Median aggregation over repeated splits mitigates outlier sensitivity in heterogeneous effect estimation, empirically reducing MSE by 10–30% (Jacob, 2020).
• For design-based randomized experiments, splits must preserve conditional independence, often requiring stratified or arm-balanced splitting (Lu et al., 21 Aug 2025).

Best practices include selecting $M=5$ or $10$ for folding, utilizing ML methods with proven stability for prediction, and exploiting efficient numerical techniques (quadrature, kernels, etc.) for likelihood approximation in complex models.

6. Theoretical Guarantees and Efficiency Bounds

Cross-fitting yields the following guarantees across domains:

• Changepoint detection: with proper spacing and jump-size conditions, the cross-fit estimator exactly recovers the number of changepoints ($\hat{K}_{cf}=K^*$) and localizes each up to vanishing error $O(\Delta_k^{-1})$ (Qian et al., 2024).
• Spatial processes: cross-fitted parametric estimators are consistent and, under Poisson or log-linear intensity, attain the semiparametric Cramér–Rao lower bound (Lin et al., 2024).
• Causal/functional estimation: cross-fit (plug-in and doubly robust) estimators achieve root-$n$ rates and efficiency under minimal smoothness via sample-splitting (Newey et al., 2018, Zeng, 2022). Three-way cross-fitting accelerates bias decay, replacing $O(k_n/n)$ with $o(n^{-1/2})$ remainder terms (Fisher et al., 2023).
• Randomized experiments: conditional cross-fitting yields exact unbiasedness for finite-sample ATE, with valid asymptotic normality and conservative variance estimation (Lu et al., 21 Aug 2025).

7. Domain-Specific Extensions and Practical Recommendations

Cross-fitting’s modularity supports diverse models and inferential frameworks:

• In changepoint analysis, it aligns empirical loss closely with population risk, correcting oversegmentation artifacts from in-sample loss minimization (Qian et al., 2024).
• For spatial semiparametric processes, V-fold random thinning generalizes cross-fitting to dependent spatial observations, enabling unbiased parameter estimation even with complex dependency (Lin et al., 2024).
• In multi-treatment causal inference, generalized cross-fitting allows robust estimation and inference with flexible ML for nuisance parts, even in non-Donsker, weak-overlap regimes (Zeng, 2022).
• In randomized designs, conditional splitting adapts cross-fitting to the design-based context, with explicit unbiasedness and conservative inference (Lu et al., 21 Aug 2025).

Best practice recommendations are fold selection ($M=5$ or $10$), median aggregation over $B=20$–$50$ splits for robustness, model fitting restricted to out-of-fold data for each nuisance, internal cross-validation conducted strictly within training partitions, and careful stratified or treatment-wise splitting in finite-population experiments.

---

Cross-fitting is thus recognized as a foundational algorithmic strategy for mitigating bias and achieving efficiency in modern statistical learning, causal inference, adaptive changepoint detection, spatial modeling, and design-based randomized experiment analysis. It provides guarantees of unbiasedness, consistency, and valid inference in the presence of highly flexible, adaptive, and potentially misspecified model components. The technique is supported by a rich literature demonstrating its efficacy, extensibility, and theoretical optimality under minimal assumptions (Qian et al., 2024, Lin et al., 2024, Zeng, 2022, Fisher et al., 2023, Newey et al., 2018, Lu et al., 21 Aug 2025, Jacob, 2020).