Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
131 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Convergence Behavior of an Adversarial Weak Supervision Method (2405.16013v1)

Published 25 May 2024 in cs.LG

Abstract: Labeling data via rules-of-thumb and minimal label supervision is central to Weak Supervision, a paradigm subsuming subareas of machine learning such as crowdsourced learning and semi-supervised ensemble learning. By using this labeled data to train modern machine learning methods, the cost of acquiring large amounts of hand labeled data can be ameliorated. Approaches to combining the rules-of-thumb falls into two camps, reflecting different ideologies of statistical estimation. The most common approach, exemplified by the Dawid-Skene model, is based on probabilistic modeling. The other, developed in the work of Balsubramani-Freund and others, is adversarial and game-theoretic. We provide a variety of statistical results for the adversarial approach under log-loss: we characterize the form of the solution, relate it to logistic regression, demonstrate consistency, and give rates of convergence. On the other hand, we find that probabilistic approaches for the same model class can fail to be consistent. Experimental results are provided to corroborate the theoretical results.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. YouTube Spam Collection. UCI Machine Learning Repository, 2017. DOI: https://doi.org/10.24432/C58885.
  2. SMS Spam Collection. UCI Machine Learning Repository, 2012. DOI: https://doi.org/10.24432/C5CC84.
  3. Adversarial Labeling for Learning Without Labels. CoRR, abs/1805.08877, 2018.
  4. A General Framework for Adversarial Label Learning. Journal of Machine Learning Research, 22(118):1–33, 2021.
  5. Learning from Rules Generalizing Labeled Exemplars. In International Conference on Learning Representations, 2020.
  6. Optimally Combining Classifiers Using Unlabeled Data. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages 211–225, Paris, France, 03–06 Jul 2015a. PMLR.
  7. Scalable Semi-Supervised Aggregation of Classifiers. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1351–1359, Cambridge, MA, USA, 2015b. MIT Press.
  8. Optimal Binary Classifier Aggregation for General Losses. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, pages 5032–5039. Curran Associates, Inc., 2016.
  9. Combining Labeled and Unlabeled Data With Co-Training. In Proceedings of the Workshop on Computational Learning Theory, pages 92–100, 1998.
  10. Convex Optimization. Cambridge University Press, 2004. 10.1017/CBO9780511804441.
  11. Interval Estimation for a Binomial Proportion. Statist. Sci., 16(2):101–133, 05 2001. 10.1214/ss/1009213286.
  12. Experiments With Query Acquisition and Use in Document Retrieval Systems. In Proceedings of the 13th International Conference on Research and Development in Information Retrieval, pages 349–368, 1990.
  13. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20–28, 1979. ISSN 00359254, 14679876.
  14. Cardiotocography. UCI Machine Learning Repository, 2010. DOI: https://doi.org/10.24432/C51S4N.
  15. CVXPY: A Python-embedded Modeling Language for Convex Optimization. Journal of Machine Learning Research, 17(83):1–5, 2016.
  16. Rekall: Specifying Video Events using Compositions of Spatiotemporal Labels. CoRR, abs/1910.02993, 2019.
  17. Fast and Three-rious: Speeding up Weak Supervision with Triplet Methods. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.
  18. Minimax Optimal Convergence Rates for Estimating Ground Truth from Crowdsourced Labels, 2013.
  19. Game Theory, Maximum Entropy, Minimum Discrepancy and Robust Bayesian Decision Theory. The Annals of Statistics, 32(4):1367–1433, 2004. ISSN 00905364.
  20. LLC Gurobi Optimization. Gurobi Optimizer Reference Manual, 2021.
  21. Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods. Machine Learning, 110(3):457–506, Mar 2021. ISSN 1573-0565. 10.1007/s10994-021-05946-3.
  22. Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing, 2014.
  23. Exploiting Worker Correlation for Label Aggregation in Crowdsourcing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3886–3895. PMLR, 09–15 Jun 2019.
  24. Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, page 142–150, USA, 2011. Association for Computational Linguistics. ISBN 9781932432879.
  25. Minimax Classification with 0-1 Loss and Performance Guarantees. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 302–312. Curran Associates, Inc., 2020.
  26. Generalized Maximum Entropy for Supervised Classification. IEEE Transactions on Information Theory, 68(4):2530–2550, 2022. 10.1109/TIT.2022.3143764.
  27. Adversarial Multi Class Learning Under Weak Supervision with Performance Guarantees. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7534–7543. PMLR, 18–24 Jul 2021.
  28. Foundations of Machine Learning. The MIT Press, 2nd edition, 2018. ISBN 0262039400.
  29. MOSEK ApS. MOSEK Optimizer API for Python. Release 9.3.20, 2022. URL https://docs.mosek.com/9.3/pythonapi.pdf.
  30. Hukukane Nikaidô. On von Neumann’s Minimax Theorem. Pacific J. Math, 4(1):65–72, 1954.
  31. Moment Matching for Multi-Source Domain Adaptation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1406–1415, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. 10.1109/ICCV.2019.00149.
  32. Adel Rajab. Burst Header Packet (BHP) flooding attack on Optical Burst Switching (OBS) Network. UCI Machine Learning Repository, 2017. DOI: https://doi.org/10.24432/C51C81.
  33. Snorkel: Rapid Training Data Creation with Weak Supervision. The VLDB Journal, 29(2):709–730, May 2020. ISSN 0949-877X. 10.1007/s00778-019-00552-1.
  34. Data Programming: Creating Large Training Sets, Quickly. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, pages 3567–3575. Curran Associates, Inc., 2016.
  35. Denoising Multi-Source Weak Supervision for Neural Text Classification. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3739–3754, Online, November 2020. Association for Computational Linguistics. 10.18653/v1/2020.findings-emnlp.334.
  36. Dependency Structure Misspecification in Multi-Source Weak Supervision Models. ICLR Workshop on Weakly Supervised Learning, 2021a.
  37. End-to-End Weak Supervision. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1845–1857. Curran Associates, Inc., 2021b.
  38. Snorkel AI Inc. Snorkel tutorials. https://github.com/snorkel-team/snorkel-tutorials, 2022.
  39. Snuba: Automating Weak Supervision to Label Training Data. Proceedings VLDB Endowment, 12(3):223–236, November 2018.
  40. A Clinical Text Classification Paradigm Using Weak Supervision and Deep Representation. BMC Medical Informatics and Decision Making, 19, 2019.
  41. Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository, 1995. DOI: https://doi.org/10.24432/C5DW2B.
  42. Learning Hyper Label Model for Programmatic Weak Supervision. In The Eleventh International Conference on Learning Representations, 2023.
  43. Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(09):2251–2265, sep 2019. ISSN 1939-3539. 10.1109/TPAMI.2018.2857768.
  44. David Yarowsky. Unsupervised Word Sense Disambiguation Rivaling Supervised Methods. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics, pages 189–196, 1995.
  45. Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1063–1077. Association for Computational Linguistics, 2021.
  46. WRENCH: A Comprehensive Benchmark for Weak Supervision. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
  47. A Survey on Programmatic Weak Supervision, 2022.
  48. Character-level Convolutional Networks for Text Classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  49. Spectral Methods Meet EM: A Provably Optimal Algorithm for Crowdsourcing. Journal of Machine Learning Research, 17(102):1–44, 2016.
  50. Introduction to Semi-Supervised Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2009.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets