Label-Weighted Conformal Prediction

• Label-weighted conformal prediction is an advanced framework that adjusts prediction set thresholds with label-specific weights to better handle imbalanced and long-tailed data.
• It employs weighted quantile calibration and prevalence-adjusted scores to interpolate between marginal and class-conditional coverage, ensuring calibrated macro-coverage with moderate set sizes.
• The method is particularly useful in domains like species identification and federated learning, offering practical solutions for fair uncertainty quantification and human-in-the-loop verification.

Label-weighted conformal prediction is a generalization of the conformal prediction framework that modulates the prediction set construction to address heterogeneity in label frequencies, costs, or reliability—particularly when class distributions are long-tailed, imbalanced, or when per-class error rates are of specific interest. It enables practitioners to produce prediction sets that interpolate between purely marginal and strictly class-conditional conformal inference, providing more nuanced control over statistical coverage and prediction set efficiency in practical classification problems with many or rare classes.

1. Foundations of Conformal Prediction and Label-Weighted Extensions

Conformal prediction is a method that creates set-valued predictions with calibrated coverage guarantees under minimal assumptions. Given a sequence of samples assumed to be exchangeable, and a nonconformity measure $s(x, y)$ quantifying how unusual (or "nonconforming") a candidate label $y$ is for features $x$, a conformal predictor outputs the set

$$\mathcal{C}(x; q) = \{ y \in \mathcal{Y} : s(x, y) \leq q \}$$

for a score cutoff $q$ chosen so that the prediction set contains the true label with a prespecified probability (typically $1 - \alpha$) (0706.3188). In standard conformal prediction, $q$ is usually a common quantile threshold over all labels, ensuring marginal coverage.

Label-weighted conformal prediction introduces label-specific cutoffs or weights into the prediction set construction. The general form is

$$\mathcal{C}(x; \mathbf{q}) = \{ y \in \mathcal{Y} : s(x, y) \leq q_y \}$$

where $\mathbf{q} = (q_1, \ldots, q_{|\mathcal{Y}|})$ is a vector of cutoffs, and each $q_y$ is computed as a weighted quantile of the calibration nonconformity scores, possibly with weights depending on both observed and candidate labels. This mechanism allows for per-class or interpolated coverage guarantees (Ding et al., 9 Jul 2025).

This framework subsumes several special cases:

• Standard CP: All weights identical, a common threshold $q$.
• Classwise CP: Weights are indicators for each class, yielding separate thresholds and class-conditional coverage.
• Label-weighted CP: Weights interpolate between these extremes, offering a continuum of possible trade-offs.

2. Methodologies for Label-Weighted Conformal Prediction

In long-tailed classification scenarios, simply applying a uniform cutoff for all labels (standard CP) leads to poor coverage on rare classes, while classwise CP ensures coverage for rare classes but creates very large prediction sets due to limited calibration data per class (Ding et al., 9 Jul 2025).

Label-weighted conformal prediction achieves an intermediate regime through weighted quantile calibration. For each label $y$, one computes the cutoff as

$$q^{w}_y = \mathrm{Quantile}_{1-\alpha} \left[ \sum_{i=1}^n \frac{w(Y_i, y)}{W_y} \delta_{s(X_i, Y_i)} + \frac{w(y, y)}{W_y} \delta_{\infty} \right]$$

where $w(Y_i, y)$ is the weight of the calibration sample with label $Y_i$ relative to candidate label $y$, $W_y = \sum_{i=1}^n w(Y_i, y) + w(y, y)$, and $\delta$ denotes the Dirac measure (Ding et al., 9 Jul 2025).

Specific choices include:

• Indicator weights $w(y', y) = \mathbf{1}\{y' = y\}$: recovers classwise CP.
• Uniform weights over all classes: standard CP.
• Intermediate or kernel-based weights: "fuzzy" or interpolated CP, blending per-class and global calibration by setting

$$q_y^{(\mathrm{IQ})} = \tau \cdot q_y^{(\mathrm{CW})} + (1 - \tau) \cdot q$$

with interpolation parameter $\tau \in [0, 1]$.

A further refinement uses kernel functions over class embeddings to share calibration data among similar classes, improving estimation accuracy for rare labels.

Another complementary approach is to design the nonconformity score itself to be label-weighted, such as the prevalence-adjusted softmax (PAS) score

$$s_{\mathrm{PAS}}(x, y) = -\frac{\hat{p}(y|x)}{\hat{p}(y)}$$

where $\hat{p}(y|x)$ is the classifier’s predicted probability and $\hat{p}(y)$ is the estimated prevalence of class $y$ (Ding et al., 9 Jul 2025). This adjustment mimics the oracle that would threshold the likelihood ratio $p(y|x)/p(y)$, directly targeting improved macro-coverage (unweighted per-class average coverage).

3. Coverage Properties and Theoretical Guarantees

Standard CP guarantees marginal coverage: $P(y \in \mathcal{C}(X)) \geq 1 - \alpha,$ but this marginalization may result in much lower coverage for classes underrepresented in the data. Classwise CP guarantees for all $y \in \mathcal{Y}$: $$P(y \in \mathcal{C}(X) \mid Y = y) \geq 1 - \alpha.$$ Label-weighted CP interpolates between these principles: by choosing the weighting appropriately, it can guarantee average (macro) coverage,

$$\mathrm{MacroCov} = \frac{1}{|\mathcal{Y}|} \sum_{y \in \mathcal{Y}} P(y \in \mathcal{C}(X) \mid Y = y) \geq 1 - \alpha,$$

while keeping prediction set size moderate (Ding et al., 9 Jul 2025).

When using prevalence-adjusted scores and global thresholds, the resulting coverage guarantee remains marginal, but macro-coverage and fairness with respect to rare labels can be sharply improved relative to unadjusted standard CP.

4. Applications in Long-Tailed and Imbalanced Classification

Label-weighted conformal prediction is particularly suited for domains with severe class imbalance, such as species identification (e.g., Pl@ntNet, iNaturalist), where thousands of species may have highly uneven representation (Ding et al., 9 Jul 2025). In such contexts:

• Standard CP (softmax-based, marginal) under-covers rare classes, as common classes dominate the quantile estimation.
• Classwise CP can ensure fair per-class coverage but at the cost of extremely large and often impractical prediction sets for rare classes due to few calibration points.
• Label-weighted and PAS-based methods provide a continuum: rare classes attain substantially higher coverage without the excessive set sizes of the strict classwise approach.

Empirical studies show that for Pl@ntNet (1,081 classes) and iNaturalist (8,142 classes), prevalence-adjusted and label-weighted CP achieve a more equitable balance between prediction set size and per-class coverage, making the sets more practical for human-in-the-loop verification.

5. Practical Considerations and Implementation

Implementing label-weighted conformal prediction involves:

• Selection of an appropriate nonconformity score (e.g., softmax, PAS score).
• Estimation of class prevalence $\hat{p}(y)$, typically from the calibration set.
• Weight design: for interpolation, a parameter $\tau$ or kernel bandwidth decides the strength of borrowing across classes. For large label spaces, using class embeddings $\Pi(y)$, derived from external knowledge or model features, enables similarity-based sharing.
• Computation of weighted quantiles using the empirical distribution formed by calibration samples, which may require efficient data structures in large multi-class settings.

Computational cost increases with the number of classes and calibration samples, especially when kernel or fuzzy weights are used. Optimizations such as batching, approximate nearest neighbors, or parallelization can mitigate resource demands.

6. Extensions, Challenges, and Research Directions

Label-weighted conformal prediction supports several further developments:

• Macro-coverage optimization: The method aligns with the goal of maximizing average per-class coverage, relevant in fairness-driven or rare-class-focused applications.
• Hybrid strategies: The approach allows for context-sensitive adaptation; users may optimize set size, macro-coverage, or even other fairness criteria as required.
• Uncertainty quantification for structured or clustered labels: Weighting schemes that leverage hierarchy, taxonomy, or semantic similarity can further improve effectiveness in high-dimensional, structured label spaces (Ding et al., 2023).
• Integration with federated or distributed settings: Importance weighting can correct for label shift in federated conformal prediction (Plassier et al., 2023).
• Online, weakly labeled, or noisy settings: Label-weighted constructions are also foundational to advances in weakly supervised or partial-label conformal prediction frameworks (Cauchois et al., 2022, Javanmardi et al., 2023, Fuchs et al., 11 Feb 2025).

Open challenges include:

• Optimal design of weighting and similarity functions, particularly in extremely high-dimensional or sparsely labeled scenarios.
• Statistical efficiency and robustness under severe scarcity of calibration data for rare classes.
• Finite-sample analysis ensuring valid coverage and minimal set size with adaptive, data-driven weighting.

7. Summary Table: Three Modes of Conformal Prediction in Multi-Class Long-Tailed Settings

Approach	Coverage Guarantee	Typical Set Size	Strengths
Standard CP	Marginal (overall)	Small	Efficient for common classes
Classwise CP	Per-class (strict)	Large for rare	High for all classes, but impractical size
Label-Weighted CP	Macro / Interpolated	Intermediate	Balanced set size and rare-class coverage

This construction enables practical uncertainty quantification in real-world classification problems where both coverage fairness and interpretability are critical, bridging the trade-off between prediction set size and per-class reliability (Ding et al., 9 Jul 2025).