Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Making Binary Classification from Multiple Unlabeled Datasets Almost Free of Supervision (2306.07036v1)

Published 12 Jun 2023 in cs.LG

Abstract: Training a classifier exploiting a huge amount of supervised data is expensive or even prohibited in a situation, where the labeling cost is high. The remarkable progress in working with weaker forms of supervision is binary classification from multiple unlabeled datasets which requires the knowledge of exact class priors for all unlabeled datasets. However, the availability of class priors is restrictive in many real-world scenarios. To address this issue, we propose to solve a new problem setting, i.e., binary classification from multiple unlabeled datasets with only one pairwise numerical relationship of class priors (MU-OPPO), which knows the relative order (which unlabeled dataset has a higher proportion of positive examples) of two class-prior probabilities for two datasets among multiple unlabeled datasets. In MU-OPPO, we do not need the class priors for all unlabeled datasets, but we only require that there exists a pair of unlabeled datasets for which we know which unlabeled dataset has a larger class prior. Clearly, this form of supervision is easier to be obtained, which can make labeling costs almost free. We propose a novel framework to handle the MU-OPPO problem, which consists of four sequential modules: (i) pseudo label assignment; (ii) confident example collection; (iii) class prior estimation; (iv) classifier training with estimated class priors. Theoretically, we analyze the gap between estimated class priors and true class priors under the proposed framework. Empirically, we confirm the superiority of our framework with comprehensive experiments. Experimental results demonstrate that our framework brings smaller estimation errors of class priors and better performance of binary classification.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (81)
  1. Learning from noisy examples. Machine Learning, 2(4):343–370, 1988.
  2. Unsupervised label noise modeling and loss correction. In ICML, pages 312–321. PMLR, 2019.
  3. A closer look at memorization in deep networks. In ICML, pages 233–242. PMLR, 2017.
  4. Learning from rules generalizing labeled exemplars. arXiv preprint arXiv:2004.06025, 2020.
  5. Learning the structure of generative models without labeled data. In ICML, pages 273–282. PMLR, 2017.
  6. Me-momentum: Extracting hard confident examples from noisily labeled data. In ICCV, pages 9312–9321, 2021.
  7. Semi-supervised novelty detection. The Journal of Machine Learning Research, 11:2973–3009, 2010.
  8. Interactive weak supervision: Learning useful heuristics for data labeling. In ICLR, 2020.
  9. Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pages 421–436. Springer, 2012.
  10. The balanced accuracy and its posterior distribution. In 2010 20th ICPR, pages 3121–3124. IEEE, 2010.
  11. Cluster kernels for semi-supervised learning. NeurIPS, 15, 2002.
  12. On symmetric losses for learning from corrupted labels. In ICML, pages 961–970. PMLR, 2019.
  13. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pages 1062–1070. PMLR, 2019.
  14. Deep learning for classical japanese literature, 2018.
  15. Learning from partial labels. The Journal of Machine Learning Research, 12:1501–1536, 2011.
  16. Urban-rural county and state differences in chronic obstructive pulmonary disease—united states, 2015. Morbidity and Mortality Weekly Report, 67(7):205, 2018.
  17. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, series B, 39(1):1–38, 1977.
  18. Learning from positive and unlabeled examples. Theoretical Computer Science, 348(1):70–83, 2005.
  19. Clustering unclustered data: Unsupervised binary labeling of two datasets having different class balances. In TAAI, pages 1–6. IEEE, 2013.
  20. Jack Edmonds. Maximum matching and a polyhedron with 0, 1-vertices. Journal of research of the National Bureau of Standards B, 69(125-130):55–56, 1965.
  21. Learning classifiers from only positive and unlabeled data. In KDD, pages 213–220, 2008.
  22. psvm for learning with label proportions. 2013.
  23. Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7(2):179–188, 1936.
  24. Mixture proportion estimation and pu learning: A modern approach. In NeurIPS, 2021.
  25. Deep learning. MIT press, 2016.
  26. An instance-dependent simulation framework for learning with label noise. Machine Learning, pages 1–26, 2022.
  27. On calibration of modern neural networks. In ICML, pages 1321–1330. PMLR, 2017.
  28. Co-teaching: Robust training of deep neural networks with extremely noisy labels. NeurIPS, 31, 2018.
  29. Improving generalization by controlling label-noise information in neural network weights. In ICML, pages 4071–4081, 2020.
  30. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.
  31. Dmitry Ivanov. Dedpul: Method for mixture proportion estimation and positive-unlabeled classification based on density estimation. arXiv preprint arXiv:1902.06965, 2019.
  32. Dmitry Ivanov. Dedpul: Difference-of-estimated-densities-based positive-unlabeled learning. In ICMLA, pages 782–790. IEEE, 2020.
  33. Nonparametric semi-supervised learning of class proportions. arXiv preprint arXiv:1601.01944, 2016.
  34. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, pages 2304–2313. PMLR, 2018.
  35. Fine samples for learning with noisy labels. NeurIPS, 34, 2021.
  36. Learning multiple layers of features from tiny images. 2009.
  37. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  38. Robust inference via generative classifiers for handling noisy labels. In ICML, pages 3763–3772. PMLR, 2019.
  39. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3):447–461, mar 2016. doi: 10.1109/tpami.2015.2456899. URL https://doi.org/10.1109%2Ftpami.2015.2456899.
  40. On the minimal supervision for training any binary classifier from only unlabeled data. In ICLR, page 18 pages, New Orleans, Louisiana, USA, May 6–9 2019.
  41. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. In AISTATS, pages 1115–1125. PMLR, 2020.
  42. Binary classification from multiple unlabeled datasets via surrogate set classification. In ICML, pages 7134–7144. PMLR, 2021.
  43. Does label smoothing mitigate label noise? In ICML, pages 6448–6458. PMLR, 2020.
  44. Simple, robust, scalable semi-supervised learning via expectation regularization. In ICML, pages 593–600, 2007.
  45. Adversarial multi class learning under weak supervision with performance guarantees. In ICML, pages 7534–7543. PMLR, 2021.
  46. Learning from corrupted binary labels via class-probability estimation. In ICML, pages 125–134. PMLR, 2015.
  47. Can gradient clipping mitigate label noise? In ICLR, 2019.
  48. Foundations of Machine Learning. MIT Press, 2018.
  49. Learning with noisy labels. NeurIPS, 26, 2013.
  50. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373–1411, 2021a.
  51. Learning with confident examples: Rank pruning for robust classification with noisy labels. arXiv preprint arXiv:1705.01936, 2017.
  52. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103.14749, 2021b.
  53. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pages 1944–1952, 2017.
  54. A study of gaussian mixture models of color and texture features for image classification and segmentation. Pattern recognition, 39(4):695–706, 2006.
  55. Identifying mislabeled data using the area under the margin ranking. In NeurIPS, pages 17044–17056, 2020.
  56. Mixture proportion estimation via kernel embeddings of distributions. In ICML, pages 2052–2060. PMLR, 2016.
  57. Stefan Rueping. Svm classifier estimation from group probabilities. In ICML, 2010.
  58. End-to-end weak supervision. NeurIPS, 34:1845–1857, 2021.
  59. Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In AISTATS, pages 838–846, 2015.
  60. Learning from label proportions: A mutual contamination framework. In NeurIPS, pages 22256–22267, 2020.
  61. Classification with asymmetric label noise: Consistency and maximal denoising. In COLT, pages 489–511. PMLR, 2013.
  62. How does early stopping help generalization against label noise? arXiv preprint arXiv:1911.08059, 2019.
  63. Machine learning in non-stationary environments: Introduction to covariate shift adaptation. MIT press, 2012.
  64. Machine Learning from Weak Supervision: An Empirical Risk Minimization Approach. MIT Press, 2022.
  65. Learning from label proportions with consistency regularization. CoRR, abs/1910.13188, 2019. URL http://arxiv.org/abs/1910.13188.
  66. Learning from label proportions with consistency regularization. In ACML, pages 513–528. PMLR, 2020.
  67. Learning dependency structures for weak supervision models. In ICML, pages 6418–6427. PMLR, 2019.
  68. Pico: Contrastive label disambiguation for partial label learning. In ICLR, 2022.
  69. Be confident! towards trustworthy graph neural networks via confidence calibration. NeurIPS, 34:23768–23779, 2021.
  70. Combating noisy labels by agreement: A joint training method with co-regularization. In CVPR, 2020.
  71. Mitigating memorization of noisy labels by clipping the model prediction. In ICML, 2023.
  72. Learning from noisy pairwise similarity and unlabeled data. Journal of Machine Learning Research, 23(307):1–34, 2022.
  73. Robust early-learning: Hindering the memorization of noisy labels. In ICLR, 2020a.
  74. Part-dependent label noise: Towards instance-dependent label noise. In NeurIPS, 2020b.
  75. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017. URL http://arxiv.org/abs/1708.07747.
  76. Partial multi-label learning. In AAAI, 2018.
  77. Objects in semantic topology. In ICLR, 2022.
  78. Rethinking class-prior estimation for positive-unlabeled learning. In ICLR, 2022.
  79. On learning from label proportions, 2014. URL https://arxiv.org/abs/1402.5902.
  80. Exploiting class activation value for partial-label learning. In ICLR, 2022.
  81. A one-step approach to covariate shift adaptation. In ACML, pages 65–80. PMLR, 2020.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now