Principled Weight Selection for Nonexchangeable Conformal Prediction

Determine a principled procedure for choosing the calibration weights w1,…,wn in nonexchangeable conformal prediction for nonexchangeable data, replacing heuristic choices so the method can be applied systematically in settings with covariate or distribution shift.

Background

Nonexchangeable conformal prediction extends conformal prediction to nonexchangeable settings by constructing an empirical score distribution with fixed weights on calibration points. In practice, these weights are often chosen heuristically to favor calibration samples deemed similar to the test point.

The authors emphasize that coverage discrepancies in nonexchangeable CP depend on the chosen weights and on the total variation distance between calibration and test score distributions, which is not estimable without ground truth labels in the test domain. They explicitly note that choosing weights remains an open question, motivating their development of a systematic, data-dependent weighting approach in the DS-CP framework.

References

Choosing the weights is based on heuristic and remains a open question.

— Domain-Shift-Aware Conformal Prediction for Large Language Models  (2510.05566 - Lin et al., 7 Oct 2025) in Section 2.2 (Conformal Prediction), Nonexchangeable CP paragraph

Finally, because we measure coverage and set size rather than downstream task harm, the safety framing concerns the statistical safety promise under shift rather than a measured task-level failure; and size-thresholded selective prediction recovers retained coverage only empirically, since the retained population is selected using the model's own output, so it is not a new finite-sample conformal guarantee. The source-invisibility conclusion is limited to the diagnostics we test; it is not an impossibility result for all nonlinear or multivariate source-derived predictors.

— Does Marginal Coverage Guarantee Class-Conditional Safety for Zero-Shot VLMs Under Shift?  (2608.19376 - Sharma et al., 19 Aug 2026) in Section 5, “Limitations,” paragraph “Scope of repair and safety”