On the Burden of Achieving Fairness in Conformal Prediction

Abstract: Conformal prediction is often calibrated with a single pooled threshold, but this can hide cross-group heterogeneity in score distributions and distort group-wise coverage. We study this phenomenon through the population score distributions underlying split conformal calibration. First, we derive a conservation law and lower bound showing that pooled calibration incurs irreducible group-wise coverage distortion at a scale set by cross-group quantile heterogeneity. Second, we demonstrate that the two leading fairness definitions for conformal prediction, Equalized Coverage and Equalized Set Size, are fundamentally in tension. Third, we quantify the cost of moving between policies which treat groups separately or pool them. Experiments on synthetic and real data confirm the same bidirectional trade-off after finite-sample calibration. Our results show that, for the policy families studied here, calibration choice does not remove cross-group heterogeneity; it determines whether the resulting distortion appears in the coverage or size dimension, providing a principled lens for analyzing fairness-oriented calibration choices in practice.

• The paper presents impossibility results showing that achieving both equalized coverage and equalized expected set size is structurally infeasible due to group heterogeneity.
• It derives quantitative lower bounds on the distortion costs when transitioning between coverage and size calibration policies, guided by the stiffness of score CDFs.
• Empirical validations on synthetic and real data benchmarks confirm that naive pooled calibration induces significant fairness distortions in heterogeneous settings.

Structural Trade-offs in Group-fair Conformal Prediction

Introduction

Conformal prediction (CP) provides finite-sample, model-agnostic, distribution-free prediction intervals with coverage guarantees. However, standard CP approaches typically calibrate a single threshold on the pooled calibration set, delivering marginal coverage but potentially inducing fairness concerns when population heterogeneity exists across pre-specified groups. The paper "On the Burden of Achieving Fairness in Conformal Prediction" (2605.14260) thoroughly analyzes the structural tension between two group-level fairness notions—Equalized Coverage and Equalized Expected Set Size—in the context of split conformal prediction, establishing quantitative impossibility results and offering a unifying lens for evaluating fairness-centric calibration strategies.

Pooled Calibration and Its Intrinsic Group-wise Distortion

A standard split-CP protocol, via marginal thresholding on pooled calibration data, achieves global coverage but fails to guarantee group-wise coverage unless all group-conditional nonconformity score quantiles coincide. The paper formalizes this with a conservation law stating the weighted sum of group-level coverage distortions exactly equals the difference between the realized marginal coverage and the target level. Under mild regularity conditions, this conservation law establishes a zero-sum exchange between over-covered and under-covered groups. Crucially, the paper proves a lower bound on the root mean square (RMS) group-level miscoverage caused by cross-group quantile heterogeneity: pooled calibration incurs unavoidable group-level distortion at a scale determined by both the quantile variance and the local density (“stiffness”) of the score CDFs.

Coverage-Size Trade-off: Incompatibility of Equalized Coverage and Equalized Set Size

The centerpiece of the analysis is a rigorous demonstration that Equalized Coverage and Equalized Expected Set Size are generally incompatible calibration objectives in multi-group settings. Achieving exact group-wise coverage via group-specific thresholds transmits the population heterogeneity into the set size dimension, producing non-zero, quantifiable expected set size disparity across groups. Enforcing equalized expected set sizes, by contrast, requires perturbing thresholds away from their group-optimal levels, inevitably introducing measurable coverage disparities. This bidirectional impossibility extends the algorithmic fairness literature's core impossibility results (e.g., Chouldechova and Kleinberg-Mullainathan-Raghavan) to the conformal prediction framework, with group-conditional conformal quantile heterogeneity playing the structural role of base rate heterogeneity.

Figure 1: Bidirectional policy conversion in the synthetic study, showing coverage-to-size and size-to-coverage directions. Panels demonstrate that correcting one fairness dimension inevitably induces disparity in the other.

Quantitative Bounds on Policy Conversion

The authors derive sharp, interpretable lower bounds quantifying the distortion costs incurred when shifting between coverage and set size calibration policies. The effective magnitude of these costs is controlled by local regularity of the conditional score CDFs (density "stiffness") and the quantile or set size heterogeneity across groups. These bounds hold at the population level and are empirically robust even under finite-sample noise, provided calibration sample sizes are adequate. The results clarify that calibration policy selection determines which fairness dimension absorbs structural heterogeneity: coverage or set size.

Empirical Validation on Synthetic and Real Data

A comprehensive suite of experiments validates the theoretical predictions in both synthetic mixtures and real-world settings, including the Bias in Bios, MultiNLI, and FACET benchmarks:

• Synthetic mixtures robustly reproduce the bidirectional trade-off at varying heterogeneity scales, with observed group-wise coverage and size disparities matching lower-bound estimates derived from population parameters.
• Bias in Bios: Male and female group disparities under pooled calibration persist at scales predicted by the theory. Transitioning to group-wise thresholds removes coverage disparity but introduces expected set size disparity, and vice versa.

  Figure 2: Bias in Bios mechanism view at $\alpha = 0.1$ for the simple score. Panel A shows the marginal pooling-induced disparity; Panels B/C illustrate the induced coverage/set-size trade-offs under group-wise and size-equalizing thresholds.

• MultiNLI genres: Robustly display the same qualitative and quantitative policy trade-off, as evidenced by positive and negative coverage distortions under pooled calibration and compensatory set size or coverage disparities under alternative policies.

Figure 3: MultiNLI at $\alpha=0.1$ using both simple and RAPS scores. Panel A: group-wise coverage distortion under pooling; B: set size changes after enforcing equalized coverage; C: coverage disparities after equalized set size.

• FACET (computer vision): Even under severe group imbalance, group-specific disparities and induced trade-offs persist, confirming the structural nature of the phenomenon.

Robustness analyses show that the observed policy-conversion distortions scale predictably with increases in group-quantile heterogeneity (via synthetic or real perturbations) and remain stable under varying calibration sample sizes once the structural signal-to-noise ratio threshold is surpassed.

Implications and Future Directions

The findings entail that, for all finite, pre-specified groups in exchangeable data with heterogeneous nonconformity score distributions, no calibration policy can simultaneously guarantee group-level coverage and set size equality. Therefore, fairness-aware CP deployment requires value-driven choices regarding which dimension (coverage or size) should absorb unresolvable heterogeneity, aligning with use-case priorities and downstream impacts (e.g., human-in-the-loop scenarios).

Theoretically, the results parallel and extend the classical base-rate impossibility theorems, but in the conformal setting, with quantile heterogeneity as the source of non-reconcilable fairness tension. Practically, the work provides empirical diagnostics and calibration guidelines for practitioners, discouraging naïve belief that threshold pooling or groupwise splitting alone “solves fairness” in predictive confidence quantification.

Extensions to settings with adaptively selected groups, continuous attributes, and data-adaptive calibration policies remain for future study. The finite-sample diagnostic framework introduced here may also inform statistical and algorithmic advances in conditional validity, group-adaptive conformal inference, and calibration strategies for structured or massive group spaces.

Conclusion

This work delivers an integrated theoretical and empirical account of the irreducible trade-offs inherent in fairness-aware conformal prediction. It establishes that group-level coverage and set size parity are structurally in tension whenever group-conditional quantiles differ, and quantifies the cost of pursuing one fairness criterion over the other. The findings impose a fundamental constraint on the practice of uncertainty quantification in heterogeneous settings, motivating more nuanced and context-aware use of conformal methods in fair machine learning research and deployment.