Conditional Coverage Evaluation

• Conditional coverage evaluation is a framework that ensures prediction sets meet target probabilities conditionally on covariate values or group attributes.
• It employs methods such as group-weighted quantile regression and importance-weighted calibration to adjust for covariate shifts and heterogeneous subpopulations.
• The approach tackles shortcomings of marginal guarantees by introducing relaxed targets and novel algorithms to detect and mitigate under-coverage in critical groups.

Conditional coverage evaluation concerns the quantification and certification of predictive coverage rates not just on average (marginal coverage), but conditional on various sources of heterogeneity, such as subpopulations, covariate values, groups, or shifts in the underlying data distribution. Achieving reliable conditional coverage is central in high-stakes applications (e.g., medicine, fairness, resource allocation), as marginal coverage guarantees can systematically mask under-coverage in critical subgroups. Conditional coverage evaluation synthesizes a spectrum of theoretical notions, algorithmic tools, and empirical criteria for assessing and improving distributional validity beyond the marginal case.

1. Core Notions and Definitions

Conditional coverage formalizes the requirement that a prediction set $C(X)$ for $Y$ should contain $Y$ with at least probability $1-\alpha$ given side information, typically $X$ or a group attribute $G$. Let $P$ denote the joint distribution of covariates $X$ and label $Y$.

• Marginal coverage: $P(Y \in C(X)) \geq 1-\alpha$.
• Pointwise conditional coverage: $P(Y \in C(X) \mid X=x) \geq 1-\alpha$ for all $x$.
• Group-conditional coverage: $P(Y \in C(X) \mid G=g) \geq 1-\alpha$ for all $g$.
• Selection-conditional coverage: Guarantees coverage for test points that have been adaptively selected by a data-dependent rule.
• Weighted/functional conditionality: Coverage controlled over a function class $F$; e.g., $$|E[f(X)(\mathbf{1}\{Y \in C(X)\}-(1-\alpha))]| \leq \epsilon$$ for all $f \in F$.

Pointwise conditional coverage is impossible in finite samples without trivial solutions, which forces attention to relaxed and approximate guarantees (Lee et al., 25 Sep 2025, Duchi, 28 Feb 2025, Gibbs et al., 2023).

2. Impossibility Results and Relaxed Targets

The impossibility of universal distribution-free conditional coverage for nontrivial prediction sets is now classical: for continuous $X$, any finite-sample, distribution-free procedure that achieves $P(Y \in C(X) \mid X=x) \geq 1-\alpha$ for all $x$ must output sets of infinite measure (Lee et al., 25 Sep 2025, Gibbs et al., 2023). This motivates a taxonomy of relaxed targets:

• Group-conditional or subpopulation coverage over pre-defined or learned groups (Alpay et al., 29 Sep 2025, Bairaktari et al., 24 Feb 2025, Zhou et al., 23 May 2024, Jaubert et al., 4 Jun 2025).
• $L^k$-norm control: Control the $L^k$-norm $\|C-\gamma\|_{L^k}$ (e.g., $k=2$) of the conditional coverage function as a surrogate for worst-case error (Lee et al., 25 Sep 2025).
• Coverage under covariate shift: Guarantee $P_f(Y \in C(X)) \geq 1-\alpha$ for $X \sim f(x)P_X$ for $f$ in a restricted function class (Gibbs et al., 2023, Alpay et al., 29 Sep 2025).
• Selection-conditional coverage: Exact coverage on units selected by an arbitrary (but exchangeability-respecting) procedure (Jin et al., 6 Mar 2024).
• Training-conditional (PAC) coverage: With high probability over the random training data, the resulting coverage on new test points is at least $1-\alpha-\epsilon$ (Bian et al., 2022, Pournaderi et al., 21 Apr 2024, Liang et al., 2023, Duchi, 28 Feb 2025, Pournaderi et al., 26 May 2024).

3. Algorithmic Frameworks for Conditional Coverage Evaluation

Group-Conditional and Shift-Aware Calibration

Procedures such as Calibrated Counterfactual Conformal Fairness (C³F) (Alpay et al., 29 Sep 2025) and Kandinsky Conformal Prediction (Bairaktari et al., 24 Feb 2025) guarantee group-conditional or group-weighted coverage via:

• Importance-weighted conformal calibration: Calibration scores are weighted by the likelihood ratio $w(x_i) = p_\text{test}(x_i) / p_\text{train}(x_i)$, yielding group-specific thresholds ensuring

$$P_{\text{test}}(Y \in C(X)\mid G=g) \geq 1-\alpha - \text{error}(n_g, B_g, K, \delta)$$

where $n_g$ is the group calibration sample size, $B_g$ is a second-moment bound, and $K$ is the number of groups (Alpay et al., 29 Sep 2025, Pournaderi et al., 26 May 2024).

• Group-weighted quantile regression: Quantile thresholds are learned to guarantee coverage with respect to arbitrary weights $w(x,y)$ representing overlapping/fractional group membership (Bairaktari et al., 24 Feb 2025, Gibbs et al., 2023).
• Counterfactual regularization: Path-specific effect regularizers penalize unfair changes in nonconformity scores from interventions along unfair causal paths, shrinking counterfactual coverage gaps (Alpay et al., 29 Sep 2025).

Cluster- and Locally-Conditional Calibration

Cluster-based conditional conformal prediction uses clustering (e.g., via histograms or learned summaries) to partition data into approximately homogeneous subpopulations, guaranteeing per-cluster coverage calibrated at the empirical $(1-\alpha)$ quantile (Jaubert et al., 4 Jun 2025, Kaur et al., 17 Jan 2025).

Functional Control of Conditional Coverage

$L^k$-norm control defines coverage targets over function spaces, e.g., control the $L^2$-norm of the deviation between actual and nominal conditional coverage. Calibration proceeds by ensuring that, for a rich class of test functions $f$ (kernels or indicator balls),

$$E_f [ (E[ f(X) ( \mathbf{1}\{Y \notin C(X)\} - \alpha ) ] )^2 ] \leq \alpha^2$$

achieving strong local or kernel-smoothed control without requiring unattainable pointwise validity (Lee et al., 25 Sep 2025).

Adaptive and Fair Conformal Prediction

Adaptively Fair Conformal Prediction (AFCP) (Zhou et al., 23 May 2024) adaptively selects sensitive covariates for each test instance and dynamically enforces group-conditional coverage where bias is detected, yielding coverage guarantees conditional on adaptively chosen protected features.

Selection-Conditional Calibration

The JOMI framework (Jin et al., 6 Mar 2024) computes conformal sets that achieve exact $1-\alpha$ coverage conditional on a unit being selected for inference by a (permutation-invariant) selection rule. Coverage is thus guaranteed at the level of data-driven focal units, even for complex selection rules (top-K, threshold, p-value, knapsack).

4. Theoretical Guarantees and Bounds

All approaches rely on the uniform convergence of empirical quantiles, often with empirical process inequalities in the presence of weighting or clustering:

• Weighted DKW inequalities control the deviation between the weighted empirical CDF and the target CDF under covariate shift, leading to explicit group-conditional or training-conditional PAC bounds (Alpay et al., 29 Sep 2025, Pournaderi et al., 26 May 2024, Pournaderi et al., 21 Apr 2024).
• Functional and group-optimal error rates scale with $(\#\text{groups}/n)^{1/2}$ for finite subgroups (Bairaktari et al., 24 Feb 2025, Gibbs et al., 2023); regularization in infinite-dimensional function classes yields explicit, tunable error bounds (Gibbs et al., 2023).
• Counterfactual regularization bounds the coverage gap via the first derivative of a smooth surrogate for path-specific effect violation (Alpay et al., 29 Sep 2025).
• Algorithmic stability is essential for ensuring training-conditional coverage in full conformal and jackknife+ methods. Uniform stability (or weaker $m$-stability) yields high-probability bounds on the deviation between empirical and expected coverage, quantified in terms of stability coefficients and (possibly) model dimension (Liang et al., 2023, Pournaderi et al., 21 Apr 2024, Bian et al., 2022).
• Empirical evaluation metrics for practical conditional coverage checking include worst-slab coverage (minimum coverage over axis-aligned feature slices), coverage error across slices, triage metrics (for instance, confident & accurate rates), and coverage gap histograms in partitions of relevant summary statistics (Jaubert et al., 4 Jun 2025, Kaur et al., 17 Jan 2025, Bairaktari et al., 24 Feb 2025, Zhou et al., 23 May 2024).

5. Empirical Protocols and Applications

Conditional coverage methods have been applied to a broad set of scenarios:

• Fairness-critical classification: C³F achieves improved group-conditional validity and parity with minimal loss in set efficiency across fairness benchmarks (Adult, COMPAS, Law School, German Credit) (Alpay et al., 29 Sep 2025, Zhou et al., 23 May 2024).
• Image and risk prediction: Conditional conformal prediction methods utilizing trust scores, cluster stratification, or local quantile regression demonstrate improved local and subgroup coverage in computer vision and medical risk assessment settings (Kaur et al., 17 Jan 2025, Jaubert et al., 4 Jun 2025, Jin et al., 6 Mar 2024).
• High-dimensional multiple testing: Rank-conditional coverage (RCC) addresses systematic under-coverage of extreme parameters in large-scale inference, using bootstrap intervals that calibrate bias as a function of estimator rank (Morrison et al., 2017).
• Regression calibration: Techniques such as rectified conformity scores (Plassier et al., 22 Feb 2025), orthogonal quantile regression (Feldman et al., 2021), and KS-penalized model training (Gao et al., 26 Sep 2024) offer calibrated intervals with improved conditional coverage under heteroscedasticity and distributional drift.

6. Limitations, Open Questions, and Future Directions

Despite advances, important limitations remain:

• Impossibility boundaries: Nontrivial, exact $X$-conditional coverage remains unattainable in finite samples for arbitrary continuous covariate spaces, motivating ongoing investigation of functional/relaxed coverage control (Lee et al., 25 Sep 2025, Duchi, 28 Feb 2025).
• Data and computational requirements: Group-conditional or clusterwise calibration requires sufficient calibration data per group or cluster. In high-dimensional or ultra-sparse regimes, proper regularization or dimension reduction is necessary (Gibbs et al., 2023, Plassier et al., 22 Feb 2025).
• Design of group bases and regularization: Choice of overlapping/fractional group representation, the dimension of basis functions, and regularization strength directly affect power, error, and computational cost (Bairaktari et al., 24 Feb 2025).
• Stability dependency of PAC guarantees: Achieving training-conditional coverage with tight excess relies on strong algorithmic stability or regularization; loosely regularized or highly adaptive models remain problematic (Pournaderi et al., 21 Apr 2024, Liang et al., 2023).
• Selection-conditional methods: General JOMI-style selection-conditional coverage is computationally demanding in the worst case and dependent on exchangeability assumptions, prompting future research into streaming, online, or covariate-shift contexts (Jin et al., 6 Mar 2024).
• Higher-dimensional and structured outputs: Ongoing extensions tackle multi-output regression, structured prediction, and high-cardinality label spaces where efficient subpopulation coverage remains challenging (Plassier et al., 22 Feb 2025, Kaur et al., 17 Jan 2025, Bairaktari et al., 24 Feb 2025).

7. Summary Table: Representative Conditional Coverage Evaluation Methods

Method/Paper	Target Conditionality	Coverage Guarantee
C³F (Alpay et al., 29 Sep 2025)	Group-conditional, covariate shift	Finite-sample lower bound
Kandinsky CP (Bairaktari et al., 24 Feb 2025)	Overlapping/fractional group-conditional	High-probability, minimax
Trust Score CP (Kaur et al., 17 Jan 2025)	Confidence/trust score strata	Empirical, binned
Cluster CP (Jaubert et al., 4 Jun 2025)	Per-cluster conditional	Empirical, clusterwise
$L^k$-CP (Lee et al., 25 Sep 2025)	Functional ($L^k$ norm) control	Exact, finite-sample
Rectified CP (Plassier et al., 22 Feb 2025)	Estimated conditional quantiles	Marginal, improved approx
Selection-Cond. (JOMI) (Jin et al., 6 Mar 2024)	Selection-adaptive units	Exact, finite-sample
Adaptively Fair CP (Zhou et al., 23 May 2024)	Adaptive, group via feature selection	Exact, adaptive

Conditional coverage evaluation occupies a central role in the modern theory and practice of uncertainty quantification, connecting statistical learning theory, algorithmic fairness, robust statistics, and the deployment of machine-learned prediction rules in critical domains. The focus of current work is on quantifying, localizing, and reducing conditional miscoverage via functional, group, and distributional relaxations that maintain distribution-free validity and interpretable guarantees.