Papers
Topics
Authors
Recent
Search
2000 character limit reached

Not all distributional shifts are equal: Fine-grained robust conformal inference

Published 20 Feb 2024 in stat.ME | (2402.13042v3)

Abstract: We introduce a fine-grained framework for uncertainty quantification of predictive models under distributional shifts. This framework distinguishes the shift in covariate distributions from that in the conditional relationship between the outcome ($Y$) and the covariates ($X$). We propose to reweight the training samples to adjust for an identifiable covariate shift while protecting against worst-case conditional distribution shift bounded in an $f$-divergence ball. Based on ideas from conformal inference and distributionally robust learning, we present an algorithm that outputs (approximately) valid and efficient prediction intervals in the presence of distributional shifts. As a use case, we apply the framework to sensitivity analysis of individual treatment effects with hidden confounding. The proposed methods are evaluated in simulation studies and four real data applications, demonstrating superior robustness and efficiency compared with existing benchmarks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Conformal prediction: A gentle introduction. Foundations and Trends® in Machine Learning, 16(4):494–591.
  2. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455–482.
  3. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816–845.
  4. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):565–600.
  5. Breiman, L. (2001). Random forests. Machine learning, 45:5–32.
  6. Assessing treatment effect variation in observational studies: Results from a data challenge.
  7. Robust validation: Confident predictions even when distributions shift. Journal of the American Statistical Association, (just-accepted):1–22.
  8. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794.
  9. Distributional conformal prediction. Proceedings of the National Academy of Sciences, 118(48):e2107794118.
  10. Cover, T. M. (1999). Elements of information theory. John Wiley & Sons.
  11. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34:6478–6490.
  12. Distributionally robust losses for latent covariate mixtures. Operations Research, 71(2):649–664.
  13. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378–1406.
  14. Elie-Dit-Cosaque, K. (2020). qosa-indices.
  15. Adversarially robust conformal prediction. In International Conference on Learning Representations.
  16. Probabilistically robust conformal prediction. In Uncertainty in Artificial Intelligence, pages 681–690. PMLR.
  17. Guan, L. (2023). Localized conformal prediction: A generalized inference framework for conformal prediction. Biometrika, 110(1):33–50.
  18. Conformalized survival analysis with adaptive cutoffs. Biometrika, page asad076.
  19. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 127:108496.
  20. Hastie, T. J. (2017). Generalized additive models. In Statistical models in S, pages 249–307. Routledge.
  21. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press.
  22. Diagnosing the role of observable distribution shift in scientific replications. arXiv preprint arXiv:2309.01056.
  23. Sensitivity analysis of individual treatment effects: A robust conformal inference approach. Proceedings of the National Academy of Sciences, 120(6):e2214889120.
  24. Sensitivity analysis under the f𝑓fitalic_f-sensitivity models: a distributional robustness perspective. arXiv preprint arXiv:2203.04373.
  25. Confounding-robust policy evaluation in infinite-horizon reinforcement learning. Advances in neural information processing systems, 33:22293–22304.
  26. Minimax-optimal policy learning under unobserved confounding. Management Science, 67(5):2870–2890.
  27. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111.
  28. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(5):911–938.
  29. Policy learning under biased sample selection. arXiv preprint arXiv:2304.11735.
  30. On the need for a language describing distribution shifts: Illustrations on tabular datasets. arXiv preprint arXiv:2307.05284.
  31. The effect of natural distribution shift on question answering models. In International conference on machine learning, pages 6905–6916. PMLR.
  32. Factored DRO: Factored distributionally robust policies for contextual bandits. Advances in Neural Information Processing Systems, 35:8318–8331.
  33. Diagnosing model performance under distribution shift. arXiv preprint arXiv:2303.02011.
  34. Neyman, J. (1923). Sur les applications de la théorie des probabilités aux experiences agricoles: Essai des principes. Roczniki Nauk Rolniczych, 10(1):1–51.
  35. Inductive confidence machines for regression. In Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19–23, 2002 Proceedings 13, pages 345–356. Springer.
  36. PAC prediction sets under covariate shift. In International Conference on Learning Representations.
  37. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830.
  38. Fighting covid-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention. Psychological science, 31(7):770–780.
  39. Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in Artificial Intelligence, pages 844–853. PMLR.
  40. Distribution-free prediction sets adaptive to unknown covariate shift. arXiv preprint arXiv:2203.06126.
  41. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659.
  42. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389–5400. PMLR.
  43. Conformalized quantile regression. Advances in neural information processing systems, 32.
  44. Classification with valid and adaptive coverage.
  45. How accurate are accuracy-nudge interventions? a preregistered direct replication of pennycook et al.(2020). Psychological science, 32(7):1169–1178.
  46. Rosenbaum, P. R. (1987). Sensitivity analysis for certain permutation inferences in matched observational studies. Biometrika, 74(1):13–26.
  47. Learning from a biased sample. arXiv preprint arXiv:2209.01754.
  48. Distributionally robust logistic regression. Advances in Neural Information Processing Systems, 28.
  49. Distributionally robust batch contextual bandits. Management Science.
  50. Tan, Z. (2006). A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(476):1619–1637.
  51. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267–288.
  52. Conformal prediction under covariate shift. Advances in neural information processing systems, 32.
  53. Algorithmic learning in a random world, volume 29. Springer.
  54. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine, 181(8):1065–1070.
  55. Bounds on the conditional and average treatment effect with unobserved confounding factors. arXiv preprint arXiv:1808.09521.
  56. Doubly robust calibration of prediction sets under covariate shift. arXiv preprint arXiv:2203.01761.
  57. Yeager, D. S. (2019). The National Study of Learning Mindsets,[United States], 2015-2016.
  58. A national experiment reveals where a growth mindset improves achievement. Nature, 573(7774):364–369.
  59. Conformal sensitivity analysis for individual treatment effects. Journal of the American Statistical Association, pages 1–14.
  60. Optimal multi-distribution learning. arXiv preprint arXiv:2312.05134.
  61. Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Statistical Society Series B: Statistical Methodology, 81(4):735–761.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Authors (2)

  1. Jiahao Ai 
  2. Zhimei Ren 

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free