2000 character limit reached
Invariance assumptions for class distribution estimation (2311.17225v1)
Published 28 Nov 2023 in cs.LG
Abstract: We study the problem of class distribution estimation under dataset shift. On the training dataset, both features and class labels are observed while on the test dataset only the features can be observed. The task then is the estimation of the distribution of the class labels, i.e. the estimation of the class prior probabilities, in the test dataset. Assumptions of invariance between the training joint distribution of features and labels and the test distribution can considerably facilitate this task. We discuss the assumptions of covariate shift, factorizable joint shift, and sparse joint shift and their implications for class distribution estimation.
- Quantification via probability estimators. In Data Mining (ICDM), 2010 IEEE 10th International Conference on, pages 737–742. IEEE, 2010.
- Discriminative Learning Under Covariate Shift. The Journal of Machine Learning Research, 10:2137–2155, 2009.
- D. Card and N.A. Smith. The Importance of Calibration for Estimating Proportions from Annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1636–1646, 2018. doi: 10.18653/v1/N18-1148.
- Estimating and Explaining Model Performance When Both Covariates and Labels Shift. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, NeurIPS 2022, volume 35, pages 11467–11479. Curran Associates, Inc., 2022.
- M.C. Du Plessis and M. Sugiyama. Semi-supervised learning of class balance under class-prior change by distribution matching. Neural Networks, 50:110–119, 2014.
- C. Elkan. The foundations of cost-sensitive learning. In B. Nebel, editor, Seventeenth International Joint Conference on Artificial Intelligence, IJCAI 2001, pages 973–978. Morgan Kaufmann, 2001.
- Learning to Quantify. Springer Cham, 2023. doi: https://doi.org/10.1007/978-3-031-20467-8.
- T. Fawcett and P.A. Flach. A response to Webb and Ting’s On the Application of ROC Analysis to Predict Classification Performance under Varying Class Distributions. Machine Learning, 58(1):33–38, 2005.
- G. Forman. Counting Positives Accurately Despite Inaccurate Classification. In European Conference on Machine Learning (ECML 2005), pages 564–575. Springer, 2005.
- Comparison of a screening test and a reference test in epidemiologic studies. II. A probabilistic model for the comparison of diagnostic tests. American Journal of Epidemiology, 83(3):593–602, 1966.
- A Review on Quantification Learning. ACM Comput. Surv., 50(5):74:1–74:40, 2017.
- Domain Adaptation with Factorizable Joint Shift. Presented at the ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning, 2021.
- V. Hofer. Adapting a classification rule to local and global shift when only unlabelled data are available. European Journal of Operational Research, 243(1):177–189, 2015.
- Mapping conditional distributions for domain adaptation under generalized target shift, 2021. URL https://arxiv.org/abs/2110.15057. Presented at ICLR 2022.
- F.C. Klebaner. Introduction to Stochastic Calculus with Applications. Imperial College Press, second edition, 2005.
- A. Klenke. Probability Theory: A Comprehensive Course. Springer Science & Business Media, 2013.
- S. Kpotufe and G. Martinet. Marginal singularity and the benefits of labels in covariate-shift. The Annals of Statistics, 49(6):3299–3323, 2021. doi: 10.1214/21-AOS2084.
- Temporal density extrapolation using a dynamic basis approach. Data mining and knowledge discovery, 33:1323–1356, 2019.
- Understanding new tasks through the lens of training data via exponential tilting. In The Eleventh International Conference on Learning Representations (ICLR 2023), 2023. URL https://openreview.net/forum?id=DBMttEEoLbw.
- Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure. Neural Computation, 14(1):21–41, 2001.
- C. Scott. A Generalized Neyman-Pearson Criterion for Optimal Domain Adaptation. In Proceedings of Machine Learning Research, 30th International Conference on Algorithmic Learning Theory, volume 98, pages 1–24, 2019.
- H. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227–244, 2000.
- Low-Dimensional Density Ratio Estimation for Covariate Shift Correction. In K. Chaudhuri and M. Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 3449–3458. PMLR, 2019.
- Density ratio estimation in machine learning. Cambridge University Press, 2012.
- D. Tasche. The art of probability-of-default curve calibration. Journal of Credit Risk, 9(4):63–103, 2013. doi: 10.21314/JCR.2013.169.
- D. Tasche. Calibrating sufficiently. Statistics, 55(6):1356–1386, 2021. doi: 10.1080/02331888.2021.2016767.
- D. Tasche. Class Prior Estimation under Covariate Shift: No Problem? Working paper, presented at ECML/PKDD 2022 workshop Learning to Quantify: Methods and Applications (LQ 2022), 2022a.
- D. Tasche. Factorizable Joint Shift in Multinomial Classification. Machine Learning and Knowledge Extraction, 4(3):779–802, 2022b. doi: 10.3390/make4030038.
- D. Tasche. Sparse joint shift in multinomial classification. arXiv preprint arXiv:2303.16971, 2023.
- Domain Adaptation Under Target and Conditional Shift. In Proceedings of the 30th International Conference on International Conference on Machine Learning – Volume 28, ICML’13, pages III–819–III–827. JMLR.org, 2013.