CV-CRC: Cross-Validation Conformal Risk Control

• The paper introduces CV-CRC as a cross-validation analogue of conformal risk control that converts a point predictor into a set predictor with calibrated risk.
• CV-CRC replaces a single holdout calibration split with K-fold cross-validation, improving data efficiency and yielding smaller predictive sets especially when data are limited.
• The method guarantees finite-sample average risk control under bounded, monotonic loss functions while requiring permutation-invariant nonconformity scores and multiple training runs.

Cross-validation conformal risk control (CV-CRC) is a cross-validation analogue of conformal risk control (CRC) that applies post hoc to a conventional point predictor in order to produce a set predictor with guarantees on a user-chosen risk function. CRC generalizes conformal prediction from miscoverage to the expected value of any monotone loss function, while CV-CRC replaces the training/validation split of validation-based CRC with $K$-fold cross-validation. The stated motivation is that the original split can be problematic when data availability is limited, resulting in inefficient set predictors; CV-CRC is introduced to use the data more efficiently while retaining theoretical guarantees on the average risk of the set predictor (Angelopoulos et al., 2022, Cohen et al., 2024).

1. Origin within conformal risk control

CRC was introduced as an extension of conformal prediction that controls the expected value of any monotone loss function. In the base formulation, one starts from a pretrained model $f$ and a post-processing rule $C_\lambda$ indexed by a threshold or conservativeness parameter $\lambda$, with larger $\lambda$ giving more conservative outputs and a loss that is non-increasing in $\lambda$. The central CRC guarantee is finite-sample expected-risk control, and the method is described as finite-sample, distribution-free, and tight up to an $\mathcal{O}(1/n)$ factor (Angelopoulos et al., 2022).

The original CRC construction is split-based. In the formulation emphasized by CV-CRC, this validation-based CRC (VB-CRC) divides the available data into a training set $\mathcal D_{\mathrm{tr}}$, used to fit the predictor, and a validation set $\mathcal D_{\mathrm{val}}$, used to calibrate the threshold $\lambda$. The CV-CRC paper identifies the cost of this arrangement directly: the model is trained on fewer samples, and the calibration threshold is estimated from a smaller validation set, which can lead to overly conservative, larger predictive sets. CV-CRC is therefore introduced as a cross-validation-based CRC method, and the paper states that it extends a version of the jackknife-minmax from conformal prediction to CRC so as to allow control of a broader range of risk functions (Cohen et al., 2024).

2. Statistical formulation and admissible losses

The CV-CRC formulation considers $f$0 exchangeable samples

$f$1

with the first $f$2 samples forming the dataset

$f$3

and the last sample $f$4 serving as the test point. Exchangeability is sufficient; independence is not needed (Cohen et al., 2024).

A loss function

$f$5

is fixed, bounded as

$f$6

and monotone in the prediction set in the sense that

$f$7

This monotonicity condition is the structural assumption that underlies CRC. The paper gives miscoverage,

$f$8

and false negative rate in multilabel prediction as examples (Cohen et al., 2024).

A nonconformity score is chosen,

$f$9

and for a threshold $C_\lambda$0 the induced set predictor is

$C_\lambda$1

These sets are nested in $C_\lambda$2:

$C_\lambda$3

The corresponding population risk is

$C_\lambda$4

and the target of CV-CRC is average-risk control:

$C_\lambda$5

By contrast, the guarantee highlighted for VB-CRC is conditional on a fixed training set (Cohen et al., 2024).

3. Cross-validation construction

The dataset $C_\lambda$6 is partitioned into $C_\lambda$7 equal-sized folds,

$C_\lambda$8

and $C_\lambda$9. A separate model is trained on each leave-fold-out dataset $\lambda$0, so the method requires $\lambda$1 training rounds (Cohen et al., 2024).

For a candidate threshold $\lambda$2, CV-CRC defines the predictive set

$\lambda$3

Equivalently,

$\lambda$4

A label is therefore included if it is admitted by at least one leave-fold-out model. This is the CRC analogue of jackknife-minmax in conformal prediction (Cohen et al., 2024).

Threshold selection is based on a cross-validation estimate of the risk,

$\lambda$5

The final term $\lambda$6 is described as an extra dummy “$\lambda$7-th fold” of maximal loss, and it functions as a regularizer needed for finite-sample control. CV-CRC then chooses the smallest threshold satisfying the empirical constraint,

$\lambda$8

and outputs

$\lambda$9

In procedural terms, the method splits the data into $\lambda$0 folds, trains $\lambda$1 leave-fold-out models, evaluates the foldwise losses for candidate thresholds, forms $\lambda$2, and selects the smallest threshold whose cross-validation estimate is below $\lambda$3 (Cohen et al., 2024).

4. Average-risk guarantee and proof structure

The main theorem of CV-CRC fixes a bounded and monotonic loss $\lambda$4 satisfying

$\lambda$5

together with a nonconformity score that is permutation-invariant with respect to the ordering of examples in the training set. For any number of folds satisfying

$\lambda$6

the CV-CRC predictive set satisfies

$\lambda$7

This is the central CV-CRC guarantee: average risk control over the full dataset and test point (Cohen et al., 2024).

The proof is organized through an augmented dataset containing the $\lambda$8 data folds plus a $\lambda$9-th test fold, together with a leave-two-fold-out construction. The argument defines a leave-two-fold-out threshold $\lambda$0, shows

$\lambda$1

and then combines exchangeability with an auxiliary lemma: if $\lambda$2 are exchangeable and

$\lambda$3

then $\lambda$4 for every $\lambda$5. This turns a deterministic empirical-average bound into the desired expectation bound for the test fold (Cohen et al., 2024).

A technical distinction from VB-CRC is explicit. Unlike VB-CRC, the CV theorem requires the nonconformity score to be permutation-invariant in the training data. The paper notes that this is the same kind of condition needed in full conformal prediction and cross-validation conformal prediction, and cites full-batch gradient descent as an example of a training procedure whose final predictor does not depend on data ordering (Cohen et al., 2024).

5. Relation to VB-CRC, conformal prediction, and later CRC variants

VB-CRC and CV-CRC share the same objective—post-hoc conversion of a point predictor into a set predictor with a calibrated risk level—but they differ in calibration architecture. VB-CRC uses one training split and one validation split, calibrates on $\lambda$6, and carries a guarantee conditional on $\lambda$7. CV-CRC uses all samples through $\lambda$8-fold cross-validation, trains $\lambda$9 models on leave-fold-out datasets, calibrates using all folds, and carries a guarantee on the overall expected risk over $\mathcal{O}(1/n)$0. The practical tradeoff stated in the paper is direct: VB-CRC requires only one training run, whereas CV-CRC requires $\mathcal{O}(1/n)$1 training runs but uses the data more efficiently and can yield smaller predictive sets (Cohen et al., 2024).

Within the broader conformal literature, CRC is the generalization step and CV-CRC is the cross-validation step. CRC extends conformal prediction to any monotone loss and includes split conformal prediction as a special case via the miscoverage loss. The base CRC paper states no algorithm called CV-CRC and no explicit cross-validation construction; CV-CRC is introduced later as a separate development (Angelopoulos et al., 2022, Cohen et al., 2024).

The relation to conformal prediction is especially transparent in the miscoverage case. The CV-CRC paper states that when the loss is miscoverage and $\mathcal{O}(1/n)$2, CV-CRC reduces to the jackknife-minmax scheme. This locates CV-CRC as a strict extension of cross-validation conformal prediction ideas from coverage control to broader bounded monotonic risks (Cohen et al., 2024).

Later CRC work does not uniformly adopt the cross-validation construction. For example, the semantic CT method “Conformal Risk Control for Semantic Uncertainty Quantification in Computed Tomography” is explicitly described as not using CV-CRC, jackknife+, or any cross-validation conformal calibration scheme; instead it uses a train/calibration/test split and an internal split of the calibration set for optimization and backtracking. This indicates that CV-CRC is a particular calibration strategy within the CRC family rather than a universal template for subsequent CRC methods (Teneggi et al., 28 Feb 2025).

6. Numerical studies and reported empirical behavior

The CV-CRC paper reports two synthetic experiments comparing VB-CRC and CV-CRC. The first is vector regression under a hierarchical Gaussian model,

$\mathcal{O}(1/n)$3

with

$\mathcal{O}(1/n)$4

and a Gaussian-mixture prior on $\mathcal{O}(1/n)$5. The reported parameters are

$\mathcal{O}(1/n)$6

The predictor is linear regression trained by maximum likelihood, with nonconformity score

$\mathcal{O}(1/n)$7

The controlled loss is the per-coordinate fraction of missed components,

$\mathcal{O}(1/n)$8

and inefficiency is average interval size,

$\mathcal{O}(1/n)$9

With target risk $\mathcal D_{\mathrm{tr}}$0, $\mathcal D_{\mathrm{tr}}$1 test points, and $\mathcal D_{\mathrm{tr}}$2 independent simulations, both VB-CRC and CV-CRC satisfy the target risk empirically, while CV-CRC gives smaller predictive sets or lower inefficiency, especially when $\mathcal D_{\mathrm{tr}}$3 is small. The paper further reports that $\mathcal D_{\mathrm{tr}}$4 strikes a good balance between inefficiency and computational burden (Cohen et al., 2024).

The second experiment concerns temporal point process prediction. Given the past $\mathcal D_{\mathrm{tr}}$5 event times $\mathcal D_{\mathrm{tr}}$6, the task is to predict intervals for the next $\mathcal D_{\mathrm{tr}}$7 event times. Data are generated by a Hawkes process with intensity

$\mathcal D_{\mathrm{tr}}$8

with

$\mathcal D_{\mathrm{tr}}$9

A recurrent neural network outputs a predictive density, and the median is used as the point estimate. The reported setup uses observed length $\mathcal D_{\mathrm{val}}$0, prediction horizon $\mathcal D_{\mathrm{val}}$1, target risk $\mathcal D_{\mathrm{val}}$2, $\mathcal D_{\mathrm{val}}$3 simulations, and $\mathcal D_{\mathrm{val}}$4 test points per run. Again, both VB-CRC and CV-CRC satisfy the desired risk level, and CV-CRC with $\mathcal D_{\mathrm{val}}$5 yields smaller average interval sizes than VB-CRC (Cohen et al., 2024).

Across these experiments, the reported practical pattern is consistent: the main advantage of CV-CRC is better data efficiency, which can translate into smaller predictive sets when the available data are limited, at the expense of training $\mathcal D_{\mathrm{val}}$6 models instead of one. The same paper also notes an explicit design constraint,

$\mathcal D_{\mathrm{val}}$7

which in the common case $\mathcal D_{\mathrm{val}}$8 and $\mathcal D_{\mathrm{val}}$9 becomes

$\lambda$0

For small target risks, enough folds are therefore required (Cohen et al., 2024).

7. Conceptual significance and boundaries

CV-CRC occupies a specific position in the conformal literature. It is neither a reformulation of split CRC nor a mere restatement of jackknife-minmax. Its distinctive contribution is to retain conformal-style risk calibration for general bounded monotonic losses while replacing a single holdout calibration split with a cross-validation estimator that includes a finite-sample correction through the dummy maximal-loss fold. In the terminology of the paper, the result is an average-risk guarantee for the set predictor rather than only a conditional guarantee tied to a fixed training split (Cohen et al., 2024).

At the same time, the method inherits the structural restrictions of CRC. The loss must be bounded and monotonic in the prediction set, and the cross-validation theorem additionally requires permutation invariance of the nonconformity score with respect to the ordering of training examples. The computational cost is also intrinsic: CV-CRC trades one training run for $\lambda$1 training runs. Within those boundaries, the paper’s stated conclusion is that CV-CRC can reduce the average set size with respect to CRC when the available data are limited (Cohen et al., 2024).