---
title: Cross-Conformal Prediction
url: https://www.emergentmind.com/topics/cross-conformal-prediction-4be27eb8-03ce-40f7-a0d9-3df559d1df38
type: topic
---

# Cross-Conformal Prediction

Searching arXiv for recent and foundational papers on cross-conformal prediction to ground the article.
Cross-conformal prediction is a family of conformal inference methods that combines the sample-efficiency of cross-validation with the calibration logic of inductive conformal prediction. In its basic form, the training sample is partitioned into \(K\) folds, each fold is used in turn as a calibration set while the remaining folds serve as the proper training set, and the resulting foldwise conformity-based quantities are aggregated into a final predictor or predictive distribution. The method was introduced as a hybrid of inductive conformal prediction and cross-validation, with the aim of improving predictive efficiency relative to single-split inductive conformal prediction while remaining computationally tractable [1208.0806]. Subsequent work extended the framework from set prediction to predictive distributions, from ordinary conformity scores to fully adaptive predictive systems, from supervised classification and regression to multi-label classification and anomaly detection, and from classical p-value aggregation to e-value aggregation and cross-validation-based risk control [1902.06579].

## 1. Origins and basic motivation

Cross-conformal prediction was introduced by Vladimir Vovk as “a hybrid of the methods of inductive conformal prediction and cross-validation” [1208.0806]. The starting point is the standard trade-off in conformal inference. Full conformal prediction uses all available data symmetrically and is computationally expensive, whereas inductive conformal prediction uses a single split into proper training and calibration subsets, which is much cheaper but statistically less efficient because each observation contributes either to model fitting or calibration, but not both [1208.0806].

The basic motivation for cross-conformal prediction is therefore to use more of the data for both fitting and calibration without incurring the full cost of transductive conformal prediction. In the original formulation, the training set of size \(l\) is partitioned into \(K\) equal or nearly equal folds \(S_1,\dots,S_K\). For each fold \(k\), the complement \(S_{-k}\) is treated as the proper training set and \(S_k\) as the calibration set. Foldwise p-values are then computed and aggregated across folds [1208.0806].

This design addresses the main deficiency of inductive conformal prediction: the loss of predictive efficiency due to a single fixed split. Each observation serves as calibration in one fold and as training data in the remaining folds, so the method uses the sample more intensively than split conformal procedures [1208.0806]. A plausible implication is that cross-conformal prediction is best viewed not as a single algorithm but as an aggregation principle within conformal inference.

## 2. Core construction and formalism

In the original classification-oriented presentation, for each candidate label \(y\) and fold \(k\), one computes conformity scores
\[
\alpha_{i,k} = A(z_{S_{-k}}, z_i), \qquad i \in S_k,
\]
and the test conformity score
\[
\alpha_k^y = A(z_{S_{-k}}, (x,y)).
\]
The foldwise p-value is then
\[
p_k^y = \frac{\left| \{ i \in S_k \mid \alpha_{i,k} \leq \alpha_k^y \} \right| + 1}{|S_k| + 1},
\]
and the aggregated cross-conformal p-value is obtained by averaging:
\[
p^y = \frac{1}{K}\sum_{k=1}^K p_k^y.
\]
For unequal fold sizes, a weighted average is used [1208.0806].

At significance level \(\epsilon\), the prediction set is
\[
\Gamma_{\epsilon}(z_1,\ldots,z_l,x) = \{ y \mid p^y > \epsilon \}.
\]
In binary classification, one may also report confidence and credibility, respectively \(1-\min(p^0,p^1)\) and \(\max(p^0,p^1)\) [1208.0806].

Later work generalized this scheme from set predictors to conformal predictive systems. In the regression-oriented predictive-distribution setting, cross-conformal predictive systems compute foldwise randomized predictive distributions and aggregate them. One explicit formulation is
\[
Q(z_1,\ldots,z_n,(x,y),\tau) =
\frac{1}{n+1}\sum_{k=1}^K |\{i \in S_k : \alpha_{i,k} < \alpha^y_k\}|
+ \frac{\tau}{n+1}\sum_{k=1}^K |\{i \in S_k : \alpha_{i,k} = \alpha^y_k\}|
+ \frac{\tau}{n+1},
\]
which can also be expressed as an approximate weighted average of per-fold p-values [1911.00941].

The 2019 “Conformal calibrators” framework generalized the input object even further. There, a predictive system is a map
\[
Q:\mathbf{Z}^{n+1}\to[0,1],
\]
and a randomized predictive system is
\[
Q:\mathbf{Z}^{n+1}\times[0,1]\to[0,1].
\]
The key relaxation is that the base predictive system need not already be calibrated; only monotonicity in \(y\) is required. Split-conformal and cross-conformal predictive systems are then obtained by calibrating this arbitrary input predictive system [1902.06579]. This is the basis for the paper’s claim that the resulting split-conformal and cross-conformal predictive systems are “fully adaptive” [1902.06579].

## 3. Validity, calibration, and the main theoretical fault line

The central theoretical issue in cross-conformal prediction is validity. For standard split conformal predictors, validity is finite-sample and follows from exchangeability. For the original cross-conformal predictor, the 2012 paper states explicitly that no theoretical proof of validity is given; validity remains an open problem there, although the empirical evidence is favorable [1208.0806].

The reason is the dependence across foldwise p-values. The original paper also reports that Fisher’s method, which assumes independence, leads to miscalibration when used to combine foldwise p-values [1208.0806]. Averaging p-values was proposed precisely because it behaved well empirically, but this did not amount to a proof [1208.0806].

The predictive-distribution counterpart makes the same distinction. Split conformal predictive systems are described as having guaranteed validity, whereas cross-conformal predictive systems have validity “only empirically and in the absence of excessive randomization”; their main advantage is greater predictive efficiency [1911.00941]. That paper further notes that cross-conformal predictive distributions are “well calibrated empirically” across a range of datasets and algorithms, even though they are not formally randomized predictive systems in the same sense as split-conformal predictive systems [1911.00941].

A different line of theory appeared in “Conformal calibrators,” where the split-conformalized predictive system \(C^A\) is guaranteed to be calibrated in probability in the IID case:
\[
\forall \alpha\in[0,1]: \quad \mathbb{P}\left( Q(Z_1,\ldots,Z_n, Z, \tau) \le \alpha \right) = \alpha.
\]
The same work states that cross-conformal calibrators are obtained by aggregating multiple split-conformal predictive systems built from different folds, exactly in the cross-validation style of earlier cross-conformal work, but now with a fully adaptive base predictive system [1902.06579].

That paper also gives an asymptotic approximation result: when the base predictive system is already ideal, the conformalized version approaches it at rate \(O(n^{-1/2})\), with
\[
\sqrt{n}\left( C^A_{Z_1,\dots,Z_n,X,\tau} \circ A_{Z_1,\dots,Z_n,X}^{-1} - I \right) \Rightarrow U,
\]
where \(U\) is a Brownian bridge [1902.06579]. This suggests that conformalization is asymptotically non-destructive when the starting predictive system is already well specified.

A more recent theoretical development concerns training-conditional validity. A 2025 paper shows that, under a stability condition and stochastic boundedness of the conformity score, \(n\)-fold cross-conformal prediction has the same asymptotic training conditional coverage guarantees as full conformal prediction [2508.05272]. In the notation of that paper, asymptotic training conditional validity means
\[
\sup_{\alpha \in [0,1]} \left| \mathbb{P}\left\{y_{n+1} \notin \mathrm{PS}_\alpha \mid T\right\} - \alpha \right| \stackrel{p}{\to} 0.
\]
The result is asymptotic rather than finite-sample and depends on stability and boundedness assumptions [2508.05272]. This is important because it reframes cross-conformal prediction not only as an empirically useful aggregation scheme but also, under conditions, as asymptotically equivalent to full conformal prediction from the viewpoint of training-conditional coverage.

## 4. Efficiency gains, data usage, and computational trade-offs

The principal practical argument for cross-conformal prediction has always been statistical efficiency under limited computational budget. The original paper reports that cross-conformal predictors produce smaller prediction sets and more stable predictions than inductive conformal prediction, with much lower standard deviation of confidence, because more of the data is used for both model fitting and calibration [1208.0806].

In conformal predictive systems for regression, the same pattern appears. Split conformal predictive systems are finite-sample valid but less efficient because only part of the data is used in calibration, whereas cross-conformal predictive systems are more efficient and less sensitive to the choice of the split ratio or fold number [1911.00941]. Empirical evaluation there uses the continuous ranked probability score,
\[
\mathrm{CRPS}(F, y_i) = \int_{-\infty}^\infty [F(y) - \mathbf{1}_{y \geq y_i}]^2 dy,
\]
and reports that cross-conformal predictive systems consistently achieve lower median CRPS than split conformal predictive systems on several datasets [1911.00941].

The efficiency rationale is especially prominent in low-data regimes. In neural-network regression, the proposed \(k\)-fold prediction interval method uses out-of-fold residuals from all observations:
\[
D_i = Y_i - \hat{m}_j(X_i), \qquad i \in L_j,
\]
and constructs intervals for a new input \(X\) from the empirical distribution of all out-of-fold residuals [2006.16941]. The paper reports that the method tends to produce narrower prediction intervals than split conformal while maintaining the same coverage probability, and that the advantage is strongest when the number of training observations is limited [2006.16941]. Although the paper distinguishes its \(k\)-fold interval method from standard cross-conformal variants, its motivation and data-usage logic are clearly aligned with the cross-conformal paradigm.

The computational profile is intermediate between split and full conformal methods. Cross-conformal prediction requires \(K\) model fits rather than one, but each fit uses approximately \((1-1/K)\) of the sample, and the folds can be processed independently [1911.00941]. This makes the method substantially cheaper than full conformal prediction while exploiting the data more fully than single-split procedures [1208.0806].

A closely related computational development is the nested conformal framework, which starts from nested set families rather than nonconformity scores and extends smoothly to cross-conformal, jackknife+, and out-of-bag aggregation [1910.10562]. For interval-based nested families, that paper gives an efficient cross-conformal style aggregation algorithm reducing naïve \(O(n^2)\) computation to \(O(n \log n)\) per test point [1910.10562]. This suggests that some of the historical computational disadvantages of cross-conformal procedures are more contingent on implementation than on the aggregation principle itself.

## 5. Variants, extensions, and neighboring frameworks

Cross-conformal prediction has been extended in several distinct directions.

First, “Conformal calibrators” develops split-conformal and cross-conformal predictive systems that are “fully adaptive” in the sense that they calibrate arbitrary predictive systems rather than requiring restrictive conformity measures tied weakly to the test object [1902.06579]. The calibration map operates on arbitrary predictive systems, even if the input predictive system is not calibrated, and yields output calibrated in probability [1902.06579].

Second, cross-validation ideas have been generalized from ordinary miscoverage control to broader risk control. Cross-Validation Conformal Risk Control introduces CV-CRC, a cross-validation version of conformal risk control that extends jackknife-minmax ideas from conformal prediction to arbitrary bounded, monotonic risk functions [2401.11974]. The predictive set is
\[
\Gamma^\text{CV}_\lambda(x \mid \mathcal{D}) = \left\{ y' \in \mathcal{Y} : \min_{k=1,\ldots,K} \mathrm{NC}((x, y') \mid \mathcal{D}_{-k}) \leq \lambda \right\},
\]
with threshold selected by an augmented cross-validation risk estimator [2401.11974]. This is not ordinary cross-conformal prediction in the original p-value-averaging sense, but it occupies the same methodological space: using cross-validation to recover sample efficiency while preserving distribution-free guarantees.

Third, few-shot meta-learning has been combined with cross-validation-based conformal prediction. The meta-XB approach builds on cross-validation-based conformal prediction rather than validation-based conformal prediction and is designed to preserve formal per-task calibration guarantees while reducing prediction set size [2210.03067]. The paper describes the resulting guarantee as stronger than task-marginal validity because it applies to each task individually [2210.03067].

Fourth, the framework has been adapted to specific structured-output settings. In multi-label classification, cross-conformal prediction is used over candidate labelsets rather than single labels, with a multi-label nonconformity score of the form
\[
a_i = \sum_{j=1}^n |t_j - o'_j|^d + \lambda \sum_{j<r} t_j t_r p_{j,r},
\]
where the second term penalizes label co-occurrences never seen in the training data [2211.16238]. The paper reports that this cross-conformal predictor provides informative confidence measures in a setting where the label space is combinatorial and uncertainty is high [2211.16238].

Fifth, cross-conformal procedures have been carried into anomaly detection. A 2024 paper formally defines leave-one-out, bootstrap, and cross-conformal anomaly detectors, where out-of-fold or out-of-bag anomaly scores are converted into conformal p-values [2402.16388]. The authors state that these resampling-conformal methods make more efficient use of the available data than split-conformal anomaly detection and improve power in low-data regimes [2402.16388].

## 6. Aggregation, controversies, and newer theoretical refinements

The main controversy around cross-conformal prediction concerns p-value aggregation. In the classical p-value setting, averaging foldwise p-values is intuitive and empirically effective, but dependence across folds prevents a clean finite-sample validity theorem of the usual conformal type [1208.0806]. This has motivated several refinements.

One line replaces p-values with e-values. “Conformal e-prediction” argues that cross-conformal e-predictors are always valid because the arithmetic mean of e-values is itself an e-value:
\[
f(z_1,\ldots,z_n,x,y) := \frac{1}{K}\sum_{k=1}^{K} \alpha_k,
\qquad
\mathbb{E}\left[\frac{1}{K}\sum_{k=1}^{K} E_k\right] \le 1.
\]
The paper contrasts this with p-value averaging, for which no analogous preservation principle holds [2001.05989]. A 2026 experimental study of conformal e-classification develops this further under the name cross-conformal e-prediction and its variants, including inverse CCEP, repeated ICP, and BICEP, emphasizing that averaging e-values preserves validity while allowing flexible aggregation [2605.07963]. This suggests a conceptual bifurcation: p-value-based cross-conformal prediction prioritizes the conventional coverage interpretation, while e-value-based aggregation prioritizes algebraic stability under averaging.

Another line of refinement keeps p-values but improves the combining rule. A 2025 paper studies “improving the statistical efficiency of cross-conformal prediction” and states that original cross-conformal prediction, trained at miscoverage \(\alpha\) with \(n \gg K\), guarantees marginal coverage at least
\[
1 - 2\alpha - 2(1-\alpha)\frac{K-1}{n+K}.
\]
A simple modified version achieves coverage at least \(1-2\alpha\), and the paper then proposes sharper variants based on exchangeability and randomization [2503.01495]. One such improved predictor is
\[
\hat C_{n,K,\alpha}^{\mathrm{e~mod~cross}}(X_{n+1})
=
\left\{ y \in \mathcal{Y} :
\min_{\ell \in [K]}
\left(\frac{1}{\ell}\sum_{k=1}^{\ell} P_{(k)}(y)\right)
> \alpha \right\},
\]
where \(P_{(1)}(y)\le \cdots \le P_{(K)}(y)\) are the ordered foldwise p-values [2503.01495]. The paper states that this set is always smaller than modified CCP while retaining the same \(1-2\alpha\) guarantee [2503.01495]. A randomized variant and a combined exchangeable-randomized variant are also proposed [2503.01495].

These developments complicate a common misconception: that cross-conformal prediction is simply “average the foldwise p-values.” Historically that description is accurate for the original method [1208.0806], but recent work shows that aggregation rules can be modified in principled ways to improve efficiency or validity properties under additional structure such as exchangeability among foldwise p-values [2503.01495].

## 7. Relations to full conformal, jackknife+, and training-conditional perspectives

Cross-conformal prediction is often situated between split conformal and full conformal prediction, but later work has also placed it alongside jackknife and jackknife+ procedures. The nested conformal framework explicitly subsumes cross-conformal, jackknife+, and out-of-bag methods under a unified view based on nested set families [1910.10562]. In that formulation, leave-one-out cross-conformal prediction takes the form
\[
C^{LOO}(x) :=
\left\{ y :
\sum_{i=1}^n \mathbbm{1}\{ r_i(X_i, Y_i) < r_i(x, y) \}
< (1-\alpha)(n+1)
\right\},
\]
with \(r_i(x,y)\) defined by the smallest nesting parameter \(t\) for which \(y\) belongs to the leave-one-out nested set \(\mathcal{F}_t^{-i}(x)\) [1910.10562]. The paper states that the resulting prediction set is always contained in the jackknife+ interval and can be strictly smaller [1910.10562].

The 2025 training-conditional validity paper strengthens the connection further. Under stability and boundedness assumptions on the conformity score, full conformal prediction, \(n\)-fold cross-conformal prediction, the jackknife, the Jackknife+, and a shortcut approximation all yield the same asymptotic training conditional coverage guarantees when the conformity score is the out-of-sample prediction error [2508.05272]. The paper’s cross-conformal set is written as
\[
\left\{ y:
1 + \#\{i: \mathcal{C}_n((y, x_{n+1}), T \setminus t_i)
\leq \mathcal{C}_n(t_i, T \setminus t_i)\}
> \alpha (n+1)
\right\}.
\]
This equivalence result does not erase the methodological distinctions among these procedures, but it suggests that, under stability, several apparently different resampling-based conformal procedures can converge to the same training-conditional behavior [2508.05272].

A plausible implication is that “cross-conformal prediction” now names both a historical construction based on p-value averaging and a broader class of cross-validation-style conformal aggregation schemes whose exact finite-sample guarantees depend on the calibration object, the aggregation rule, and the regularity assumptions imposed.

## 8. Empirical behavior and domain-specific uses

Empirically, cross-conformal methods have repeatedly been reported to improve efficiency relative to split conformal baselines. The original 2012 study used the Spambase binary classification dataset and found calibration plots close to the ideal diagonal, lower standard deviation in confidence and credibility than inductive conformal prediction, and severe miscalibration for a naïve Fisher-style combination rule [1208.0806].

In regression with predictive distributions, experiments on Boston Housing, Diabetes, Yacht Hydrodynamics, Wine Quality, and Naval Propulsion are reported to show lower median CRPS for cross-conformal predictive systems than for split-conformal predictive systems across multiple underlying regressors [1911.00941].

In neural networks, the \(k\)-fold prediction interval method is reported to produce narrower intervals than split conformal while maintaining the same coverage probability, with the advantage being especially pronounced when the training sample is small [2006.16941].

In multi-label classification, cross-conformal prediction has been used to assign p-values to candidate labelsets and to form informative prediction sets despite the exponential size of the output space. The paper reports that confidence sets matched empirical error rates closely to prescribed confidence levels and became more informative when the co-occurrence penalty was enabled [2211.16238].

In anomaly detection, cross-conformal and related resampling-conformal detectors are reported to provide higher statistical power and more stable performance than split-conformal procedures while maintaining false discovery rate control through superuniform p-values [2402.16388].

These results should be interpreted carefully. Several papers emphasize empirical calibration and efficiency gains without finite-sample proofs of validity in the original p-value-based cross-conformal setting [1208.0806; 1911.00941]. By contrast, the e-value and risk-control extensions are designed specifically to restore provable guarantees under aggregation [2001.05989; 2401.11974]. The recurring pattern is therefore not a single universal theorem, but a family of methods trading off conventional validity statements, data efficiency, computational burden, and the type of uncertainty object being calibrated.

## 9. Conceptual significance

Cross-conformal prediction occupies a central position in the evolution of conformal inference because it isolates a key methodological principle: repeated splitting and aggregation can recover much of the data efficiency lost in single-split calibration. In the original formulation, that principle was operationalized through foldwise conformal p-values averaged across folds [1208.0806]. In later work, the same principle was extended to predictive distributions, arbitrary predictive systems, nested sets, risk control, meta-learning, anomaly detection, and e-values [1902.06579].

Its theoretical status is correspondingly layered. In its classical p-value form, cross-conformal prediction remains associated with strong empirical performance and a historically incomplete finite-sample validity theory [1208.0806]. In its predictive-system form, it can be calibrated in probability through conformal calibrators [1902.06579]. Under stability assumptions, it can inherit asymptotic training-conditional validity comparable to full conformal prediction [2508.05272]. Under e-value aggregation, validity is preserved exactly by averaging [2001.05989]. Under improved combining rules, prediction sets can be made smaller without weakening the best available \(1-2\alpha\) style guarantees [2503.01495].

For that reason, cross-conformal prediction is best understood not merely as an approximation to full conformal prediction, nor merely as a refinement of inductive conformal prediction, but as a broad aggregation paradigm within distribution-free predictive inference.

Source: https://www.emergentmind.com/topics/cross-conformal-prediction-4be27eb8-03ce-40f7-a0d9-3df559d1df38