---
title: Split & Weighted Conformal Prediction
url: https://www.emergentmind.com/topics/split-and-weighted-conformal-prediction
type: topic
---

# Split & Weighted Conformal Prediction

Split and weighted conformal prediction refer to a class of nonparametric, distribution-free methods for uncertainty quantification in statistical prediction, specifically designed to provide finite-sample coverage guarantees even in the presence of covariate shift, class imbalance, missing data, or heterogeneity across groups. These methods generalize traditional conformal prediction by replacing the standard empirical quantiles of nonconformity scores with weighted quantiles, adjusting for discrepancies between the calibration and target (test) distributions. This framework includes compatibility with data-splitting (split conformal) and various weighting schemes, leveraging importance sampling and group membership to correct for sampling biases and distributional shifts.

## 1. Foundations of Split Conformal Prediction

The split conformal prediction procedure operates by dividing the available data into a training set and a calibration set. A regression or classification model $\mu_0$ (or $\hat{f}$ in some treatments) is fitted using the training set. For each example $(X_i, Y_i)$ in the calibration set, a nonconformity score is computed; in regression, this is often the absolute residual $R_i = |Y_i - \mu_0(X_i)|$. 

To make a prediction at a new covariate $x$, the interval
$$
\widehat C_n(x) = [\mu_0(x) - Q_{1-\alpha},\; \mu_0(x) + Q_{1-\alpha}]
$$
is output, where $Q_{1-\alpha}$ is the empirical $1-\alpha$ quantile of $\{R_1,\dots,R_n,\infty\}$. By exchangeability of the scores on the calibration set and the new test point’s score, this construction satisfies the marginal coverage guarantee
$$
P\{Y_{n+1} \in \widehat C_n(X_{n+1})\} \ge 1 - \alpha
$$
without assumptions on the model $\mu_0$ [1904.06019, 2401.17452].

This structure is central to subsequent weighted generalizations.

## 2. Weighted Conformal Prediction: Covariate Shift and Weighted Exchangeability

When the distribution of covariates in the calibration set ($P_X$) differs from that in the target/test set ($\widetilde P_X$), i.e., under covariate shift, marginal validity can be lost. Weighted conformal prediction restores validity by incorporating importance weights. Specifically, the weight function is defined as
$$
w(x) = \frac{d\widetilde P_X}{dP_X}(x)
$$
Under the covariate shift model, the (augmented) nonconformity scores plus the test point are "weighted exchangeable": their joint probability law can be factorized as a product of functions of each coordinate (the weights) times a symmetric function. This generalizes exchangeability and enables extending the conformal argument to the new setting [1904.06019].

The weighted quantile replaces the ordinary empirical quantile, with
$$
\widehat Q_{1-\alpha} = \inf\left\{ t: \sum_{i=1}^n w(x_i)\,\mathbf{1}\{R_i \le t\} \ge (1-\alpha) \sum_{i=1}^n w(x_i) \right\}
$$
This weighted estimator guarantees
$$
P\{Y_{n+1} \in \widehat C_n(X_{n+1})\} \ge 1 - \alpha
$$
provided the weights are known or accurately estimated [1904.06019, 2401.17452].

The weighted-conformal procedure is summarized in the following steps [1904.06019]:

- Compute residuals and weights $(R_i, w(x_i))$ for the calibration set.
- For each test $x$, compute the weighted quantile $\widehat Q_{1-\alpha}$.
- Output $[\mu_0(x)-\widehat Q_{1-\alpha}, \mu_0(x)+\widehat Q_{1-\alpha}]$.

Empirical results demonstrate restoration of nominal coverage under simulated covariate shift.

## 3. Generalizations: Group, Mask, and Frequency-based Weighting

Beyond classical covariate shift, weighted conformal prediction encompasses structures such as:

**Group-weighted conformal prediction**: When the population is partitioned into $K$ discrete groups with group distribution shifting from $P$ to $Q$, the weights are $w(x) = q_k/p_k$ for group $k$. The prediction set is calibrated using a group-weighted mixture distribution over conformal scores:
$$
\widehat P_{\mathrm{score}} = \sum_{k=1}^K q_k \widehat P_{\mathrm{score}}^{(k)}
$$
Coverage guarantees improve drastically over general weighting in settings with fixed, well-sampled groups [2401.17452].

**Mask-weighted conformal prediction under missing data**: Given general missings given by mask patterns $M \in \{0,1\}^d$, weighted conformal prediction with mask-conditional weighting $\omega_m$ is deployed after calibration set imputation and masking:
$$
\omega_m(x_{obs(m)},y) = \frac{dP_m}{dQ_m}(x_{obs(m)},y)
$$
Prediction sets constructed using these weights achieve both marginal and mask-conditional validity, extending guarantees to data missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR) [2512.14221].

**Class-label and frequency-based weighting in imbalanced/open-set settings**: Selective sample splitting with probabilistic inclusion functions $\pi(k)$ (as a function of class frequency) is used to prioritize rare classes in calibration. The resulting nonconformity scores are weighted according to explicit formulas based on label frequencies, restoring validity under severe class imbalance or when the label space expands at test time [2510.13037].

## 4. Weighted Aggregation of Multiple Conformity Scores

Weighted conformal prediction is not restricted to covariate-based weighting. In multi-class classification, prediction set size can be drastically reduced by aggregating multiple nonconformity (or conformity) scores in a weighted fashion. For a collection of $k$ scores $s_1,\dots,s_k$, the aggregated score $S_w(x,y) = \sum_{j=1}^k w_j s_j(x,y)$ is used in the split-conformal construction. Weights are chosen—via constrained optimization—to minimize average prediction set size subject to the coverage constraint:
$$
\min_{w \in \Delta_k} \; \mathbb{E}_{(X,Y)}\left[|C_w(X)|\right] \quad \text{s.t.} \quad P\left\{Y \notin C_w(X)\right\} \le \alpha
$$
where $C_w(x)$ denotes the conformal prediction set generated with $S_w$ [2407.10230]. Coverage is preserved marginally, and theoretical analysis establishes connections to subgraph classes in VC theory.

Empirical results on image classification demonstrate substantial set-size reductions for the same coverage, with the method outperforming all single-score conformal predictors.

## 5. Theoretical Guarantees and Effective Sample Size

Weighted conformal prediction provides strong, minimally-assumptive finite-sample guarantees:

- **Marginal coverage**: When the appropriate weights are known or accurately estimated, coverage at the nominal level $1-\alpha$ is attained.
- **Conditional/structured coverage**: For group-weighted or mask-conditional weighting, coverage bounds match or improve on classical conformal prediction, with precise finite-sample deficits characterized in terms of group sizes or estimation error [2401.17452, 2512.14221].
- **Robustness to estimated weights**: Miscoverage due to estimated weights is bounded by a function of the error between estimated and Oracle weight functions; the effect can be quantitatively evaluated [2401.17452].
- **Effective sample size**: The variance of weights influences the effective sample size $\widehat n_{\rm eff} = (\sum_i w_i)^2/\sum_i w_i^2$, impacting prediction set width particularly under severe covariate shift or imbalanced groups [1904.06019].

## 6. Practical Considerations and Empirical Illustration

- **Weight estimation**: In practice, test-to-train likelihood ratios for covariate shift can be estimated by probabilistic classifiers trained on labeled mixture of calibration and test covariates, e.g., logistic regression or random forests [1904.06019]. For group or mask settings, frequencies or mask-pattern classifiers are used.
- **Computational cost**: Split and weighted conformal prediction yield $O(n)$ per-query complexity, as only calibration scores need be processed, in contrast to full conformal methods, which can be $O(n^2)$ or higher [1904.06019].
- **Empirical performance**: On both regression and classification tasks with covariate shift, group shift, imbalance or missingness, weighted procedures restore target coverage and often yield more informative (narrower) intervals or smaller prediction sets compared to standard split conformal, with efficiency scaling with the quality of weighting and sample size [1904.06019, 2512.14221, 2401.17452].

The table below gives a summary of practical aspects:

| Scenario                      | Weight Type (w)        | Coverage Guarantee                |
|-------------------------------|------------------------|------------------------------------|
| Covariate Shift               | $w(x)=dQ_X/dP_X(x)$    | $1-\alpha$ marginal               |
| Group Shift                   | $w(x)=q_k/p_k$         | $1-\alpha - O(\max_k q_k/n_k)$    |
| Mask Missingness              | $\omega_m$ (mask dep.) | $1-\alpha$ mask-conditional       |
| Score Aggregation (multi-class)| $w_j$ (score weights)  | $1-\alpha$ marginal               |

## 7. Extensions and Connections

Weighted and split conformal prediction form the technical foundation for multiple recent developments in distribution-free predictive inference:

- Selective splitting and weighting for open-set and class-imbalanced problems, incorporating new label detection and Good-Turing estimation [2510.13037].
- Mask-conditional valid sets via reweighting and acceptance-rejection calibrated on imputed datasets for arbitrary missing data patterns [2512.14221].
- Localized, kernel-based conformal methods (split localized conformal prediction) delivering nearly conditional coverage through kernel weighted quantiles [2206.13092].
- Group-weighted approaches yielding sharp finite-sample coverage in stratified or grouped data structures, outperforming generic density-ratio-based weighting when group membership drives the shift [2401.17452].

These advances demonstrate the broad applicability of split and weighted conformal prediction, with theoretical and empirical support for nearly all modern predictive settings involving distribution shift, heterogeneity, imbalance, or missingness.

Source: https://www.emergentmind.com/topics/split-and-weighted-conformal-prediction