Conformal prediction without knowledge of labeled calibration data

Abstract: We extend the method of conformal prediction beyond the case relying on labeled calibration data. Replacing the calibration scores by suitable estimates, we identify conformity sets $C$ for classification and regression models that rely on unlabeled calibration data. Given a classification model with accuracy $1-\beta$, we prove that the conformity sets guarantee a coverage of $P(Y \in C) \geq 1-\alpha-\beta$ for an arbitrary parameter $\alpha \in (0,1)$. The same coverage guarantee also holds for regression models, if we replace the accuracy by a similar exactness measure. Finally, we describe how to use the theoretical results in practice.