Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift (2405.10302v2)

Published 16 May 2024 in stat.ME, cs.LG, math.ST, stat.ML, and stat.TH

Abstract: As machine learning models are increasingly deployed in dynamic environments, it becomes paramount to assess and quantify uncertainties associated with distribution shifts. A distribution shift occurs when the underlying data-generating process changes, leading to a deviation in the model's performance. The prediction interval, which captures the range of likely outcomes for a given prediction, serves as a crucial tool for characterizing uncertainties induced by their underlying distribution. In this paper, we propose methodologies for aggregating prediction intervals to obtain one with minimal width and adequate coverage on the target domain under unsupervised domain shift, under which we have labeled samples from a related source domain and unlabeled covariates from the target domain. Our analysis encompasses scenarios where the source and the target domain are related via i) a bounded density ratio, and ii) a measure-preserving transformation. Our proposed methodologies are computationally efficient and easy to implement. Beyond illustrating the performance of our method through real-world datasets, we also delve into the theoretical details. This includes establishing rigorous theoretical guarantees, coupled with finite sample bounds, regarding the coverage and width of our prediction intervals. Our approach excels in practical applications and is underpinned by a solid theoretical framework, ensuring its reliability and effectiveness across diverse contexts.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Density ratio estimation via infinitesimal classification. In International Conference on Artificial Intelligence and Statistics, pages 2552–2573. PMLR.
  2. Joint distribution optimal transportation for domain adaptation. Advances in neural information processing systems, 30.
  3. Domain adaptation with regularized optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part I 14, pages 274–289. Springer.
  4. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 39(9):1853–1865.
  5. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385.
  6. Rates of estimation of optimal transport maps using plug-in estimators via barycentric projections. Advances in Neural Information Processing Systems, 34:29736–29753.
  7. Optimal transport map estimation in general function spaces. arXiv preprint arXiv:2212.03722.
  8. Uci machine learning repository. https://archive.ics.uci.edu.
  9. Probable domain generalization via quantile risk minimization. Advances in Neural Information Processing Systems, 35:17340–17358.
  10. Utopia: Universally trainable optimal prediction intervals aggregation. arXiv preprint arXiv:2306.16549.
  11. A brief review of domain adaptation. Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, pages 877–894.
  12. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34:1660–1672.
  13. Covariate shift by kernel mean matching.
  14. A two-sample conditional distribution test using conformal prediction and weighted rank sum. Journal of the American Statistical Association, pages 1–19.
  15. Hull, J. J. (1994). A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550–554.
  16. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324.
  17. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111.
  18. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(5):911–938.
  19. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28–51.
  20. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning, pages 6672–6681. PMLR.
  21. Maurer, A. (2016). A vector-contraction inequality for rademacher complexities. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pages 3–17. Springer.
  22. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 7. Granada, Spain.
  23. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345–1359.
  24. Qin, J. (1998). Inferences for case-control and semiparametric two-sample density ratio models. Biometrika, 85(3):619–630.
  25. Theoretical analysis of domain adaptation with optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18–22, 2017, Proceedings, Part II 10, pages 737–753. Springer.
  26. Large-scale optimal transport and mapping estimation. arXiv preprint arXiv:1711.02283.
  27. Correlation alignment for unsupervised domain adaptation. Domain adaptation in computer vision applications, pages 153–171.
  28. Conformal prediction under covariate shift. Advances in neural information processing systems, 32.
  29. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167–7176.
  30. Generative adversarial nets from a density ratio estimation perspective. arXiv preprint arXiv:1610.02920.
  31. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018–5027.
  32. Generalizing to unseen domains: A survey on domain generalization. IEEE transactions on knowledge and data engineering, 35(8):8052–8072.
  33. A survey of transfer learning. Journal of Big data, 3:1–40.
  34. Reliable weighted optimal transport for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4394–4403.
  35. Doubly robust calibration of prediction sets under covariate shift. arXiv preprint arXiv:2203.01761.
  36. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):5586–5609.
  37. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4396–4415.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now