Principled Weight Selection for Nonexchangeable Conformal Prediction

Determine a principled procedure for choosing the calibration weights w1,…,wn in nonexchangeable conformal prediction for nonexchangeable data, replacing heuristic choices so the method can be applied systematically in settings with covariate or distribution shift.

Background

Nonexchangeable conformal prediction extends conformal prediction to nonexchangeable settings by constructing an empirical score distribution with fixed weights on calibration points. In practice, these weights are often chosen heuristically to favor calibration samples deemed similar to the test point.

The authors emphasize that coverage discrepancies in nonexchangeable CP depend on the chosen weights and on the total variation distance between calibration and test score distributions, which is not estimable without ground truth labels in the test domain. They explicitly note that choosing weights remains an open question, motivating their development of a systematic, data-dependent weighting approach in the DS-CP framework.