---
title: Weighted Quantiles in Conformal Prediction
url: https://www.emergentmind.com/topics/weighted-quantiles-in-conformal-prediction
type: topic
---

# Weighted Quantiles in Conformal Prediction

Weighted quantiles in conformal prediction are a rigorous extension of classical quantile-calibration methods, designed to address violations of exchangeability and to enable locally or conditionally valid coverage. They provide a mechanism for incorporating informative, context-dependent weights when computing the empirical distribution of calibration nonconformity scores. This approach is technically central to robust uncertainty quantification in spatial, temporal, distribution-shifted, and group-heterogeneous settings, as well as in domains such as automated valuation models, covariate shift adaptation, and fair or localized inference.

## 1. Formal Definition of the Weighted Quantile in Conformal Calibration

Let $\{s_i\}_{i=1}^n$ denote nonconformity scores derived from a calibration sample and $\{w_i\}_{i=1}^n$ be associated nonnegative weights, potentially dependent on covariates, location, group, or other structure. The weighted empirical cumulative distribution function (CDF) is defined as
\[
F_w(t) = \frac{\sum_{i=1}^n w_i\,\mathbf{1}\{s_i \le t\}}{\sum_{i=1}^n w_i}.
\]
The $(1-\alpha)$ weighted quantile is
\[
q_{1-\alpha} = \inf\left\{ t \in \mathbb{R} : F_w(t) \ge 1-\alpha \right\},
\]
which generalizes the classical empirical quantile (recovered when $w_i \equiv 1$) [2312.06531, 1904.06019, 2401.17452].

This construction admits several equivalent forms in the literature:
- As the smallest $t$ such that the cumulative sum of sorted weights at scores $s_i \le t$ is at least $(1-\alpha)$ of the total;
- As a solution to $q_{1-\alpha} = \min\{ s_{(k)} : \sum_{i: s_i \le s_{(k)}} w_i \ge (1-\alpha) \sum_{i=1}^n w_i \}$, with $s_{(k)}$ the order statistics;
- Including a weight for an "infinite residual" (dummy calibration score), to ensure randomization and exactness at the boundary [2202.13415].

## 2. Weighted Quantile Algorithms Across Conformal Prediction Extensions

The weighted quantile is integrated into the CP calibration process in several settings, each motivating a particular choice of weight:

- **Spatial Conformal Prediction:** Weights are kernelized spatial distances, typically $w_i(x) = \exp(-\|loc(x_i) - loc(x)\|^2/\eta)$, enhancing coverage in local neighborhoods for spatially nonstationary data [2312.06531].
- **Covariate Shift:** Weights are formed from importance ratios $w_i = dP_{\text{test}}/dP_{\text{calib}}(x_i)$ (often estimated via classification odds), aligning the calibration distribution to the test distribution [1904.06019, 2505.19587, 2512.03738].
- **Group-Weighted Conformal Prediction:** Each group $k$ has a specified prevalence $q_k$, and samples in that group are weighted by $q_k/n_k$, where $n_k$ is the calibration count in group $k$ [2401.17452, 2509.25295].
- **Adaptive/Local Conformal Prediction:** Weights are computed by QRF, kernel estimators, or other regression-derived similarity metrics, focusing the quantile on calibration points nearest in feature space to the test input [2303.12695, 2405.16828].
- **Model-Uncertainty or Distribution/Shift Diagnostics:** Weights reflect model-based uncertainties such as VAE-based reconstruction loss ratios between calibration and test data [2505.19587].

The core algorithmic steps—computation of scores, derivation of weights, sorting and cumulative summation, and weighted threshold selection—remain structurally similar across domains.

## 3. Theoretical Guarantees and Coverage Properties

Weighted quantile-based conformal prediction achieves a variety of finite-sample and asymptotic coverage guarantees, contingent on the underlying statistical structure:

- **Weighted Exchangeability:** Under weighted exchangeability [1904.06019], coverage is $\Pr\{Y_{n+1} \in C_{1-\alpha}(X_{n+1})\} \geq 1-\alpha$, matching the classical result except weights replace permutations in the symmetric group action.
- **Conditional and Local Validity:** If weights concentrate around regions where the calibration and test distributions match (e.g., under a spatial Gaussian process or in local exchangeability regimes), coverage becomes locally calibrated—exact in the asymptotic neighborhood shrinkage limit [2312.06531].
- **Robustness to Non-exchangeability:** For arbitrary non-exchangeable sequences, coverage loss is upper bounded by a weighted sum of total variation distances between original and "swapped" datasets, scaled by the magnitude of weights [2202.13415].
- **Group-adaptive and Fairness-aware Bounds:** In group-weighted settings, under covariate shift and conditional exchangeability within groups, coverage loss per group is $O(1/n)$ with a sharp explicit bound tied to maximal group weight and calibration count, outperforming infinite-dimensional bounds for arbitrary weights [2401.17452, 2509.25295].
- **Covariate-shift with Distribution Estimation Error:** When importance weights are estimated, coverage bounds degrade gracefully with the $L_2$ error of the weight estimator [2505.19587, 2512.03738].

These results extend across conformal methodologies, including survival analysis with censoring, localized regression, and fairness-constrained learning.

## 4. Bayesian and Empirical Distributions: Uncertainty in Weighted Quantiles

Weighted quantiles, by themselves, provide point estimates for cutoff thresholds. Recent work generalizes the frequentist approach to a Bayesian perspective:

- The **Weighted Bayesian Conformal Prediction (WBCP)** framework interprets the quantile threshold as a Dirichlet posterior over spacing-probabilities between sorted scores, parameterized by effective sample size $n_{\rm eff} = (\sum w_i)^2 / \sum w_i^2$ and normalized weights $\tilde w_i$ [2604.06464]. This yields posterior distributions for the quantile threshold and corresponding meta-uncertainty diagnostics (posterior sd, $n_{\rm eff}$), and affords sharper, data-conditional coverage guarantees.
- The resulting stochastic dominance and conditional coverage theorems provide $O(1/\sqrt{n_{\rm eff}})$ improvement in the miscoverage rate via highest posterior density (HPD) thresholding, directly reflecting information-theoretic content in the weight profile.

## 5. Empirical and Practical Impact

Empirical studies consistently demonstrate key benefits from weighted quantile calibration in conformal prediction:

| Scenario                  | Weighted Quantile Effects                                      | Coverage / Efficiency Tradeoff   |
|---------------------------|---------------------------------------------------------------|----------------------------------|
| Spatial heterogeneity     | Flattens local coverage discrepancy, minor width increase     | ±1% local gap, 1–3% width rise  |
| Covariate shift           | Maintains coverage under shift, reduces set inflation         | Robust to moderate misweighting  |
| Group/fairness settings   | Achieves uniform group-wise coverage with small excess error  | O(1/n) excess for finite groups  |
| Adaptive/local calibration| Interval width adapts to uncertainty, improved conditional fit| Achieves local PAC coverage      |

Weighted quantiles are crucial in environments with systematic variation (spatial, temporal, covariate-dependent), enabling predictive intervals to adapt accurately to underlying nonuniformity without sacrificing finite-sample guarantees [2312.06531, 1904.06019, 2401.17452, 2303.12695, 2512.24139].

## 6. Variants and Extensions: Differential Weight Construction

The literature proposes multiple functionals for weight construction, matched to problem structure:
- Spatial kernels, covariate similarity metrics, or leverage scores [2312.06531, 2303.12695, 2602.12693].
- Density ratio or likelihood-based importance functions under known or estimated covariate shift [1904.06019, 2512.03738].
- Group-level proportions for stratified or fairness-driven inference [2401.17452, 2509.25295].
- Model-based uncertainty as in VAE reconstruction losses or residual variance proxies [2505.19587].
- Data-adaptive weights from nonparametric regression or random forest partitions [2303.12695, 2405.16828].

Critically, the calibration weighting must be independent of the test label, and effective sample size diagnostics or regularization (e.g., weight clipping, normalization) are recommended for numerical and statistical stability.

## 7. Limitations, Open Questions, and Extensions

Weighted quantiles introduce several practical and theoretical considerations:
- **Weight estimation error and variance inflation** degrade coverage guarantees; group-weighted schemes offer improved rates when the group structure captures the dominant heterogeneity [2401.17452].
- **Strong weighting (e.g., one dominant weight)** reduces effective sample size and can yield wide, conservative intervals; Bayesian posterior diagnostics, such as $n_{\rm eff}$, allow explicit quantification of this reliability loss [2604.06464].
- **Adaptivity vs. fixed weighting:** Many effective adaptive methods (e.g., QRF or RNW kernels) depart from fixed, a priori weighting but retain similar coverage properties via local exchangeability or bagging arguments [2303.12695, 2405.16828].
- **Conditional coverage impossibility:** Finite-sample exact conditional coverage is unattainable; however, weighted quantile approaches minimize mean squared conditional error and worst-slice undercoverage, as seen in density-weighted pinball methods [2512.24139].
- **Algorithmic complexity:** While weighted quantile computation requires additional sorting and prefix sums, it is computationally negligible in most high-dimensional or large-scale settings, with groupwise or local partitioning offering further improvements [2303.12695].

Weighted quantiles operationalize the extension of conformal prediction to nonstationary, complex, and high-stakes domains by providing a principled mechanism for localizing coverage and quantifying model reliability. These techniques underpin modern robust predictive inference under realistic data-generating assumptions.

Source: https://www.emergentmind.com/topics/weighted-quantiles-in-conformal-prediction