--- title: 'CV-CRC: Cross-Validation Conformal Risk Control' url: https://www.emergentmind.com/topics/cross-validation-conformal-risk-control-cv-crc type: topic --- # CV-CRC: Cross-Validation Conformal Risk Control Cross-validation conformal risk control (CV-CRC) is a cross-validation analogue of conformal risk control (CRC) that applies post hoc to a conventional point predictor in order to produce a set predictor with guarantees on a user-chosen risk function. CRC generalizes conformal prediction from miscoverage to the expected value of any monotone loss function, while CV-CRC replaces the training/validation split of validation-based CRC with $K$-fold cross-validation. The stated motivation is that the original split can be problematic when data availability is limited, resulting in inefficient set predictors; CV-CRC is introduced to use the data more efficiently while retaining theoretical guarantees on the average risk of the set predictor [2208.02814, 2401.11974]. ## 1. Origin within conformal risk control CRC was introduced as an extension of conformal prediction that controls the expected value of any monotone loss function. In the base formulation, one starts from a pretrained model $f$ and a post-processing rule $C_\lambda$ indexed by a threshold or conservativeness parameter $\lambda$, with larger $\lambda$ giving more conservative outputs and a loss that is non-increasing in $\lambda$. The central CRC guarantee is finite-sample expected-risk control, and the method is described as finite-sample, distribution-free, and tight up to an $\mathcal{O}(1/n)$ factor [2208.02814]. The original CRC construction is split-based. In the formulation emphasized by CV-CRC, this validation-based CRC (VB-CRC) divides the available data into a training set $\mathcal D_{\mathrm{tr}}$, used to fit the predictor, and a validation set $\mathcal D_{\mathrm{val}}$, used to calibrate the threshold $\lambda$. The CV-CRC paper identifies the cost of this arrangement directly: the model is trained on fewer samples, and the calibration threshold is estimated from a smaller validation set, which can lead to overly conservative, larger predictive sets. CV-CRC is therefore introduced as a cross-validation-based CRC method, and the paper states that it extends a version of the jackknife-minmax from conformal prediction to CRC so as to allow control of a broader range of risk functions [2401.11974]. ## 2. Statistical formulation and admissible losses The CV-CRC formulation considers $N+1$ exchangeable samples $$ z[i]=(x[i],y[i])\in \mathcal X\times \mathcal Y,\qquad i=1,\dots,N+1, $$ with the first $N$ samples forming the dataset $$ \mathcal D=\{z[i]\}_{i=1}^N, $$ and the last sample $z[N+1]=(x,y)$ serving as the test point. Exchangeability is sufficient; independence is not needed [2401.11974]. A loss function $$ \ell:\mathcal Y\times 2^{\mathcal Y}\to \mathbb R $$ is fixed, bounded as $$ b\le \ell(y,\Gamma)\le B,\qquad B<\infty,\quad b\in\{-\infty\}\cup\mathbb R, $$ and monotone in the prediction set in the sense that $$ \Gamma_1\subseteq \Gamma_2 \Rightarrow \ell(y,\Gamma_1)\ge \ell(y,\Gamma_2),\qquad \forall y\in\mathcal Y. $$ This monotonicity condition is the structural assumption that underlies CRC. The paper gives miscoverage, $$ \ell(y,\Gamma)=\mathbf 1\{y\notin \Gamma\}, $$ and false negative rate in multilabel prediction as examples [2401.11974]. A nonconformity score is chosen, $$ \mathrm{NC}((x,y)\mid \mathcal D_{\mathrm{tr}}), $$ and for a threshold $\lambda$ the induced set predictor is $$ \Gamma_{\lambda}(x\mid \mathcal D_{\mathrm{tr}})=\{y'\in \mathcal Y:\mathrm{NC}((x,y')\mid \mathcal D_{\mathrm{tr}})\le \lambda\}. $$ These sets are nested in $\lambda$: $$ \lambda_1<\lambda_2 \Rightarrow \Gamma_{\lambda_1}(x\mid \mathcal D_{\mathrm{tr}})\subseteq \Gamma_{\lambda_2}(x\mid \mathcal D_{\mathrm{tr}}). $$ The corresponding population risk is $$ R(\lambda\mid \mathcal D_{\mathrm{tr}})=\mathbb E_{(x,y)\sim p_0(x,y)}\big[\ell(y,\Gamma_\lambda(x\mid \mathcal D_{\mathrm{tr}}))\big], $$ and the target of CV-CRC is average-risk control: $$ \mathbb E_{\mathcal D,(x,y)\sim p_0(\mathcal D,x,y)}\big[\ell(y,\Gamma(x\mid \mathcal D))\big]\le \alpha. $$ By contrast, the guarantee highlighted for VB-CRC is conditional on a fixed training set [2401.11974]. ## 3. Cross-validation construction The dataset $\mathcal D$ is partitioned into $K$ equal-sized folds, $$ \mathcal D=\{\mathcal D_k\}_{k=1}^K,\qquad |\mathcal D_k|=N/K, $$ and $\mathcal D_{-k}=\mathcal D\setminus \mathcal D_k$. A separate model is trained on each leave-fold-out dataset $\mathcal D_{-k}$, so the method requires $K$ training rounds [2401.11974]. For a candidate threshold $\lambda$, CV-CRC defines the predictive set $$ \Gamma_{\lambda}^{\mathrm{CV}}(x\mid \mathcal D) = \left\{ y'\in\mathcal Y: \min_{k\in\{1,\dots,K\}} \mathrm{NC}((x,y')\mid \mathcal D_{-k})\le \lambda \right\}. $$ Equivalently, $$ \Gamma_{\lambda}^{\mathrm{CV}}(x\mid \mathcal D) = \bigcup_{k=1}^K \Gamma_\lambda(x\mid \mathcal D_{-k}). $$ A label is therefore included if it is admitted by at least one leave-fold-out model. This is the CRC analogue of jackknife-minmax in conformal prediction [2401.11974]. Threshold selection is based on a cross-validation estimate of the risk, $$ \hat R^{\mathrm{CV}}(\lambda\mid \mathcal D) = \frac{1}{K+1} \left( \sum_{k=1}^K \frac{K}{N}\sum_{j=1}^{N/K} \ell\big(y_k[j],\Gamma_\lambda(x_k[j]\mid \mathcal D_{-k})\big) + B \right). $$ The final term $B$ is described as an extra dummy “$(K+1)$-th fold” of maximal loss, and it functions as a regularizer needed for finite-sample control. CV-CRC then chooses the smallest threshold satisfying the empirical constraint, $$ \lambda^{\mathrm{CV}}(\mathcal D) = \inf_\lambda \left\{ \lambda: \hat R^{\mathrm{CV}}(\lambda\mid \mathcal D)\le \alpha \right\}, $$ and outputs $$ \Gamma^{\mathrm{CV}}(x\mid \mathcal D) = \Gamma^{\mathrm{CV}}_{\lambda^{\mathrm{CV}}(\mathcal D)}(x\mid \mathcal D). $$ In procedural terms, the method splits the data into $K$ folds, trains $K$ leave-fold-out models, evaluates the foldwise losses for candidate thresholds, forms $\hat R^{\mathrm{CV}}$, and selects the smallest threshold whose cross-validation estimate is below $\alpha$ [2401.11974]. ## 4. Average-risk guarantee and proof structure The main theorem of CV-CRC fixes a bounded and monotonic loss $\ell$ satisfying $$ b\le \ell(y,\Gamma)\le B \quad\text{and}\quad \Gamma_1\subseteq \Gamma_2\Rightarrow \ell(y,\Gamma_1)\ge \ell(y,\Gamma_2), $$ together with a nonconformity score that is permutation-invariant with respect to the ordering of examples in the training set. For any number of folds satisfying $$ K \ge \frac{B}{\alpha-b}-1, $$ the CV-CRC predictive set satisfies $$ \mathbb E_{\mathcal D,x,y\sim p_0(\mathcal D,x,y)} \big[ \ell(y,\Gamma^{\mathrm{CV}}(x\mid \mathcal D)) \big] \le \alpha. $$ This is the central CV-CRC guarantee: average risk control over the full dataset and test point [2401.11974]. The proof is organized through an augmented dataset containing the $K$ data folds plus a $(K+1)$-th test fold, together with a leave-two-fold-out construction. The argument defines a leave-two-fold-out threshold $\lambda_{\mathrm{L2O}}^{\mathrm{CV}}(\tilde{\mathcal D})$, shows $$ \lambda_{\mathrm{L2O}}^{\mathrm{CV}}(\tilde{\mathcal D})\le \lambda^{\mathrm{CV}}(\mathcal D), $$ and then combines exchangeability with an auxiliary lemma: if $v_1,\dots,v_M$ are exchangeable and $$ \Pr\!\left(\frac1M\sum_{i=1}^M v_i\le \alpha\right)=1, $$ then $\mathbb E[v_m]\le \alpha$ for every $m$. This turns a deterministic empirical-average bound into the desired expectation bound for the test fold [2401.11974]. A technical distinction from VB-CRC is explicit. Unlike VB-CRC, the CV theorem requires the nonconformity score to be permutation-invariant in the training data. The paper notes that this is the same kind of condition needed in full conformal prediction and cross-validation conformal prediction, and cites full-batch gradient descent as an example of a training procedure whose final predictor does not depend on data ordering [2401.11974]. ## 5. Relation to VB-CRC, conformal prediction, and later CRC variants VB-CRC and CV-CRC share the same objective—post-hoc conversion of a point predictor into a set predictor with a calibrated risk level—but they differ in calibration architecture. VB-CRC uses one training split and one validation split, calibrates on $\mathcal D_{\mathrm{val}}$, and carries a guarantee conditional on $\mathcal D_{\mathrm{tr}}$. CV-CRC uses all samples through $K$-fold cross-validation, trains $K$ models on leave-fold-out datasets, calibrates using all folds, and carries a guarantee on the overall expected risk over $(\mathcal D,x,y)$. The practical tradeoff stated in the paper is direct: VB-CRC requires only one training run, whereas CV-CRC requires $K$ training runs but uses the data more efficiently and can yield smaller predictive sets [2401.11974]. Within the broader conformal literature, CRC is the generalization step and CV-CRC is the cross-validation step. CRC extends conformal prediction to any monotone loss and includes split conformal prediction as a special case via the miscoverage loss. The base CRC paper states no algorithm called CV-CRC and no explicit cross-validation construction; CV-CRC is introduced later as a separate development [2208.02814, 2401.11974]. The relation to conformal prediction is especially transparent in the miscoverage case. The CV-CRC paper states that when the loss is miscoverage and $K=N$, CV-CRC reduces to the jackknife-minmax scheme. This locates CV-CRC as a strict extension of cross-validation conformal prediction ideas from coverage control to broader bounded monotonic risks [2401.11974]. Later CRC work does not uniformly adopt the cross-validation construction. For example, the semantic CT method “Conformal Risk Control for Semantic Uncertainty Quantification in Computed Tomography” is explicitly described as not using CV-CRC, jackknife+, or any cross-validation conformal calibration scheme; instead it uses a train/calibration/test split and an internal split of the calibration set for optimization and backtracking. This indicates that CV-CRC is a particular calibration strategy within the CRC family rather than a universal template for subsequent CRC methods [2503.00136]. ## 6. Numerical studies and reported empirical behavior The CV-CRC paper reports two synthetic experiments comparing VB-CRC and CV-CRC. The first is vector regression under a hierarchical Gaussian model, $$ p_0(\mathcal D,x,y)=\int p_0(\phi)\prod_{i=1}^{N+1} p_0(x[i])\,p_0(y[i]\mid x[i],\phi)\,d\phi, $$ with $$ p_0(x)=\mathcal N(x\mid 0,d^{-1}I_d),\qquad p_0(y\mid x,\phi)=\mathcal N(y\mid \phi^\top x,\beta_0^{-1}I_m), $$ and a Gaussian-mixture prior on $\phi$. The reported parameters are $$ \mu_0=10,\quad \gamma_0=1,\quad \beta_0=4,\quad d=50,\quad m=30. $$ The predictor is linear regression trained by maximum likelihood, with nonconformity score $$ \mathrm{NC}((x,y)\mid \mathcal D_{\mathrm{tr}})=2\|y-\hat y(x\mid \mathcal D_{\mathrm{tr}})\|_\infty. $$ The controlled loss is the per-coordinate fraction of missed components, $$ \ell(y,\Gamma)=\frac{1}{m}\sum_{j=1}^m \mathbf 1\{y_j\notin \Gamma_j\}, $$ and inefficiency is average interval size, $$ \mathrm{ineff}(\Gamma)=\frac{1}{m}\sum_{j=1}^m |\Gamma_j|. $$ With target risk $\alpha=0.1$, $N_{\mathrm{te}}=200$ test points, and $50$ independent simulations, both VB-CRC and CV-CRC satisfy the target risk empirically, while CV-CRC gives smaller predictive sets or lower inefficiency, especially when $N$ is small. The paper further reports that $K=20$ strikes a good balance between inefficiency and computational burden [2401.11974]. The second experiment concerns temporal point process prediction. Given the past $d$ event times $x=\{t_1,\dots,t_d\}$, the task is to predict intervals for the next $m$ event times. Data are generated by a Hawkes process with intensity $$ \lambda(t\mid \mathcal H_t) = \mu + \sum_{i:t_i