Coverage-Guaranteed Prediction Sets for Out-of-Distribution Data
Abstract: Out-of-distribution (OOD) generalization has attracted increasing research attention in recent years, due to its promising experimental results in real-world applications. In this paper,we study the confidence set prediction problem in the OOD generalization setting. Split conformal prediction (SCP) is an efficient framework for handling the confidence set prediction problem. However, the validity of SCP requires the examples to be exchangeable, which is violated in the OOD setting. Empirically, we show that trivially applying SCP results in a failure to maintain the marginal coverage when the unseen target domain is different from the source domain. To address this issue, we develop a method for forming confident prediction sets in the OOD setting and theoretically prove the validity of our method. Finally, we conduct experiments on simulated data to empirically verify the correctness of our theory and the validity of our proposed method.
- Achim, K. 2013. Probability theory: a comprehensive course. Springer Science & Business Media.
- Alfréd, R. 1961. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, volume 4, 547–562.
- Concrete Problems in AI Safety. CoRR, abs/1606.06565.
- Uncertainty Sets for Image Classifiers using Conformal Prediction. In ICLR.
- Bin, Y. 1994. Rates of convergence for empirical processes of stationary mixing sequences. The Annals of Probability, 94–116.
- Convex optimization. Cambridge university press.
- Robust Validation: Confident Predictions Even When Distributions Shift. CoRR, abs/2008.04267.
- Few-Shot Conformal Prediction with Auxiliary Tasks. In ICML, 3329–3339.
- Domain-Adversarial Training of Neural Networks. J. Mach. Learn. Res., 17: 59:1–59:35.
- Adversarially Robust Conformal Prediction. In ICLR.
- Adaptive Conformal Inference Under Distribution Shift. In NeurIPS, 1660–1672.
- Explaining and Harnessing Adversarial Examples. In ICLR.
- To Trust Or Not To Trust A Classifier. In NeurIPS, 5546–5557.
- Calibrating predictive model estimates to support personalized medicine. J. Am. Medical Informatics Assoc., 19(2): 263–274.
- Minimax Statistical Learning with Wasserstein distances. In NeurIPS, 2692–2701.
- Learning to Generalize: Meta-Learning for Domain Generalization. In AAAI, 3490–3497.
- Domain Generalization With Adversarial Feature Learning. In CVPR, 5400–5409.
- Deep Domain Generalization via Conditional Invariant Adversarial Networks. In ECCV, volume 11219, 647–663.
- Towards Deep Learning Models Resistant to Adversarial Attacks. In ICLR.
- Massart, P. 1990. The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality. The annals of Probability, 1269–1283.
- VC Classes are Adversarially Robustly Learnable, but Only Improperly. In COLT, 2512–2530.
- Understanding the failure modes of out-of-distribution generalization. In ICLR.
- Split Conformal Prediction for Dependent Data. arXiv:2203.15885.
- Patrick, B. 2008. Probability and measure. John Wiley & Sons.
- A Tutorial on Conformal Prediction. J. Mach. Learn. Res., 9: 371–421.
- Towards Out-Of-Distribution Generalization: A Survey. CoRR, abs/2108.13624.
- On information and sufficiency. The annals of mathematical statistics, 22(1): 79–86.
- Deep CORAL: Correlation Alignment for Deep Domain Adaptation. In ECCV, volume 9915, 443–450.
- Intriguing properties of neural networks. In ICLR.
- Conformal Prediction Under Covariate Shift. In NeurIPS, 2526–2536.
- Vovk, V. 2013. Conditional validity of inductive conformal predictors. Mach. Learn., 92(2-3): 349–376.
- Algorithmic Learning in a Random World. Springer-Verlag. ISBN 0387001522.
- Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priors. CoRR, abs/2210.06807.
- Nonlinearity and temporal dependence. Journal of Econometrics, 155: 155–169.
- Domain-wise Adversarial Training for Out-of-Distribution Generalization.
- On the Adversarial Robustness of Out-of-distribution Generalization Models. In Thirty-seventh Conference on Neural Information Processing Systems.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
