Conformal Prediction Assessment (CPA)

• Conformal Prediction Assessment is a framework that evaluates conformal predictors by examining validity, efficiency, and conditional coverage.
• It diagnoses model performance using reliability estimators and metrics like the Conditional Validity Index to guide model selection and calibration.
• CPA further assesses adaptiveness and robustness by analyzing prediction set variability under perturbations, long-tail distributions, and noisy labels.

Conformal Prediction Assessment (CPA) denotes the evaluation of conformal predictors beyond the bare statement of marginal coverage. In the literature summarized here, CPA includes the classical assessment of validity and efficiency, the diagnosis of prediction-set composition in binary classification, the estimation of instance-level conditional coverage, the measurement of adaptiveness to example difficulty, and the analysis of robustness under score variability, long-tail label distributions, perturbations, noisy labels, structured outputs, and downstream decision pipelines (Krstajic, 2020, Zhou et al., 28 Mar 2026). One explicit formulation defines CPA as a framework for conditional coverage evaluation and selection via a learned reliability estimator and the Conditional Validity Index (CVI), but the broader literature treats CPA more generally as the methodological study of what conformal prediction sets mean operationally, where they fail locally, and how they should be compared.

1. Foundational assessment criteria: validity, efficiency, and binary prediction-set semantics

The classical basis of CPA is the conformal guarantee that, under exchangeability, a prediction set $\Gamma_\epsilon(x)$ contains the true label with probability at least $1-\epsilon$. In the original online formulation, conformal prediction produces prediction regions sequentially from a nonconformity measure, and assessment begins with two quantities: validity, meaning coverage of the true label, and efficiency, meaning that prediction regions are small and therefore informative (0706.3188, Krstajic, 2020).

In binary classification, this distinction becomes especially sharp. A binary conformal predictor returns one of four regions: $\{\text{positive}\}$, $\{\text{negative}\}$, $\{\text{positive},\text{negative}\}$ (“both”), or $\varnothing$ (“empty”). Validity counts a prediction region as successful whenever it contains the true label, so “both” always counts as valid. This yields the central binary CPA warning: a model that predicts “both” for every sample is $100\%$ valid yet $0\%$ practically useful. The same nominal validity can arise from radically different compositions of single correct predictions, single incorrect predictions, “both”, and “empty” (Krstajic, 2020).

This makes binary CPA irreducibly multidimensional. Reporting only validity is insufficient. A rigorous assessment must distinguish validity from accuracy or usefulness, report an efficiency measure such as the proportion of single-label predictions, and break down prediction types into single correct, single incorrect, “both”, and “empty”. The same paper argues that CPA should also report standard predictive metrics for the underlying model on the proper training set, calibration set, and test or external validation set, because a conformal predictor built on a near-random model may still be valid in the label-set sense while remaining practically weak (Krstajic, 2020).

A further foundational issue concerns the nonconformity measure itself. In conformal prediction, nonconformity is a real-valued function measuring how different a test sample is from training samples, and in classical constructions it may depend on both inputs $X$ and labels $Y$. CPA therefore should not conflate conformal nonconformity with applicability-domain distances defined only in feature space. The binary assessment literature also emphasizes that using classifier probabilities as nonconformity scores, especially random-forest probabilities, requires explicit theoretical or empirical justification rather than being treated as automatically conformal (Krstajic, 2020).

2. CPA as conditional coverage estimation: reliability estimators, CVI, and model selection

A recent explicit formulation of CPA reframes conditional coverage evaluation as a supervised learning problem (Zhou et al., 28 Mar 2026). For a conformal procedure at level $1-\epsilon$0, the target object is the instance-wise conditional coverage function

$1-\epsilon$1

Rather than stratifying $1-\epsilon$2-space into bins, CPA constructs a new binary learning problem on an evaluation split: for each point $1-\epsilon$3, it computes the indicator

$1-\epsilon$4

and trains a probabilistic classifier on $1-\epsilon$5. The output $1-\epsilon$6 is interpreted as an estimated local coverage probability.

This produces a direct diagnostic of conditional validity. The central summary is the Conditional Validity Index

$1-\epsilon$7

CVI decomposes into undercoverage risk and overcoverage cost: $1-\epsilon$8 where $1-\epsilon$9 averages the shortfall below $\{\text{positive}\}$0 and $\{\text{positive}\}$1 averages the excess above $\{\text{positive}\}$2. The framework further defines the undercoverage rate $\{\text{positive}\}$3, overcoverage rate $\{\text{positive}\}$4, Conditional Mean Undercoverage (CMU), and Conditional Mean Overcoverage (CMO), together with a Conditional Validity Profile (CVP) curve obtained by sorting $\{\text{positive}\}$5 over the evaluation set (Zhou et al., 28 Mar 2026).

The corresponding selection rule, CC-Select, estimates CVI for multiple candidate conformal procedures across repeated train/evaluation splits, selects the model with the smallest average CVI, retrains that conformal method on the full dataset, and outputs both its prediction set and a deployment-time trust score

$\{\text{positive}\}$6

The theory establishes convergence of the reliability estimator, consistency of CVI as an estimator of oracle conditional miscalibration, and consistency of CVI-based model selection under stated stability assumptions (Zhou et al., 28 Mar 2026).

3. Adaptiveness assessment: difficulty-aware binning, T-CV, and T-SS

A separate line of CPA work focuses on adaptiveness: easy examples should receive smaller prediction sets and hard examples larger ones, while maintaining valid coverage (Jang et al., 14 Nov 2025). The critique is that earlier assessment procedures such as SSCV, ESCV, Deficit, Excess, Rank-CV, and Rank-SS rely on imbalanced binning, which makes per-bin coverage and set-size estimates unstable.

The proposed remedy is Transformation-based Binning (T-Binning). For an input $\{\text{positive}\}$7, one computes an “ease” score

$\{\text{positive}\}$8

where $\{\text{positive}\}$9 are small input transformations, typically Gaussian-noise perturbations. Difficulty is then $\{\text{negative}\}$0. Sorting by $\{\text{negative}\}$1 and using uniform-mass bins yields balanced groups across the difficulty spectrum (Jang et al., 14 Nov 2025).

Two CPA metrics are built on this construction. Transformation-based Coverage Violation (T-CV) is the maximum absolute deviation between empirical bin-wise coverage and the target $\{\text{negative}\}$2. Transformation-based Set Size relationship (T-SS) is the signed $\{\text{negative}\}$3 between average ground-truth rank and average set size across bins. Large positive T-SS indicates that average set size increases strongly and monotonically with difficulty; small T-CV indicates stable coverage across difficulty levels (Jang et al., 14 Nov 2025).

This framework is not only evaluative. It also motivates difficulty-adaptive conformal algorithms based on Mondrian or group-conditional calibration within T-Binning groups, instantiated as O-LAC and O-SAPS. On ImageNet and a visual acuity prediction task, these methods improve T-CV and T-SS relative to their non-grouped counterparts. This suggests that CPA is not merely post hoc auditing; it can also drive algorithm design by identifying when prediction-set size fails to track example difficulty (Jang et al., 14 Nov 2025).

4. Assessing variability, ranking stability, and multivariate structure

In modern vision and LLMs, candidate scores are often variable across posterior samples, prompt paraphrases, stochastic forward passes, or evaluator judgments. A recent assessment framework argues that standard CP uses a single unstable score realization, while average-then-calibrate variants smooth multiple realizations but discard the variance information that helps distinguish stable signals from noise-driven fluctuations (Zeng et al., 22 May 2026).

The proposed remedy is an empirical Bayes conformal framework based on r-values. For each candidate, multiple scores $\{\text{negative}\}$4 are used to estimate a latent stable score $\{\text{negative}\}$5, and the r-value encodes both the posterior mean and the posterior uncertainty. Used as a nonconformity score, the r-value preserves marginal conformal coverage while reducing the inclusion of high-variance false candidates under mild regularity conditions. The assessment consequences are explicit: ranking stability, average rank of the true label within the set, and set size become first-class CPA criteria, not merely auxiliary observations (Zeng et al., 22 May 2026).

A related issue arises for multivariate outputs. Scalar nonconformity scores can be geometry-blind in multi-output regression and multiclass classification, because they collapse vector-valued residual structure into a single number. Optimal-transport conformal procedures address this by using Monge–Kantorovich vector ranks and quantile regions. Their empirical assessment emphasizes marginal coverage together with multivariate efficiency measures such as average region volume, average set size, informativeness, worst-set coverage, and label-wise coverage (Thurin et al., 31 Jan 2025). This suggests that CPA for multivariate outputs must inspect whether prediction regions track the geometry of the residual distribution rather than only whether they achieve nominal coverage.

5. Heterogeneity and robustness: long-tail classes, perturbations, and noisy calibration labels

CPA becomes especially stringent when marginal validity masks systematic failures on structured subpopulations. In long-tail classification, standard CP may overcover head classes and undercover tail classes. The corresponding assessment toolkit introduces Cov-head, Cov-tail, CovGap-HT, CovGap, and AvgSize. Tail-Aware Conformal Prediction (TACP) and soft TACP (sTACP) use rank-based penalties to reduce the head-tail coverage gap and improve class-wise balance while preserving marginal coverage (Liu et al., 15 Aug 2025). In this regime, CPA explicitly asks “for whom is the guarantee valid?”

Under test-time perturbations, probabilistically robust conformal prediction broadens the assessment target from nominal coverage on clean inputs to

$\{\text{negative}\}$6

The adaptive procedure aPRCP uses a quantile-of-quantile design: an inner quantile over perturbations and an outer quantile over samples. Assessment then compares clean coverage, robust coverage, and prediction-set size under natural or adversarial perturbations. Empirically, aPRCP achieves better robustness–efficiency trade-offs than both standard CP and adversarially robust CP baselines (Ghosh et al., 2023).

CPA must also handle corrupted calibration data. Noise-Aware Conformal Prediction (NACP) studies threshold estimation when the validation labels are noisy. Under uniform label noise, it reconstructs clean coverage from noisy coverage and random coverage through

$\{\text{negative}\}$7

and estimates the noise-free conformal threshold accordingly. The associated assessment remains the familiar pair of empirical coverage and average set size, but now interpreted relative to the latent clean-label target. This is especially important in many-class problems, where naive noisy-label calibration can produce extremely conservative, practically useless prediction sets (Penso et al., 22 Jan 2025).

6. Domain-specific CPA: dynamic systems, data assimilation, fairness, and loss control

Several application domains expand CPA beyond classical coverage–size summaries. In neural-operator forecasting for dynamical systems, conformal uncertainty is assessed through sharpness, miscalibration area (MA), recalibration area (RA) after isotonic regression, calibration curves, and symmetry tests such as rotation invariance. In that setting, CP acts as a post-hoc wrapper around operator-learning models, enabling model comparison through calibration and sharpness rather than point accuracy alone (Liang et al., 2024).

In data assimilation, CP is evaluated through average empirical coverage, average interval length, miss low, miss high, and average interval score loss (AISL), and compared with standard deviation intervals and ensemble spread. Standard CP, Normalized CP, and Conformalized Quantile Regression are assessed both as post-hoc uncertainty sets and as sources of perturbations fed back into the assimilation cycle. The results show variable-dependent behavior: for precipitation, CP-based intervals maintain near-nominal coverage while Gaussian standard-deviation intervals severely under-cover extremes (George et al., 25 Jun 2026).

Fairness-oriented CPA distinguishes procedural fairness from substantive fairness. Procedural metrics include marginal coverage, group-conditional coverage, average prediction-set size, and prediction-set size disparity; substantive fairness is quantified through downstream decision improvements and the maxROR statistic. A theoretical decomposition shows that prediction-set size disparity can be separated into intra-cluster label heterogeneity, cross-cluster spread, and intra-label cross-group disparity. Empirically, label-clustered CP variants often achieve lower maxROR than group-conditional methods, and equalized set sizes rather than equalized coverage correlate more strongly with improved substantive fairness (Liu et al., 18 Feb 2026).

A further generalization replaces coverage by arbitrary monotone losses. Conformal Loss-Controlling Prediction selects a nested set predictor $\{\text{negative}\}$8 so that

$\{\text{negative}\}$9

This shifts CPA from the miscoverage indicator to application-specific loss control, including class-varying classification loss and point-wise regression or segmentation losses in weather forecasting (Wang et al., 2023). A plausible implication is that CPA is best viewed not as a single metric but as a hierarchy of assessment targets: validity, efficiency, conditional reliability, structural robustness, and task-specific utility.

Across these strands, the common lesson is that conformal prediction cannot be assessed adequately by marginal coverage alone. CPA has therefore evolved into a broader evaluative program in which validity is necessary but never sufficient; what matters equally is how coverage is distributed across instances, groups, perturbations, label frequencies, latent score variability, and downstream decisions (Zhou et al., 28 Mar 2026).