Multiclass Boosting: Simple and Intuitive Weak Learning Criteria (2307.00642v1)
Abstract: We study a generalization of boosting to the multiclass setting. We introduce a weak learning condition for multiclass classification that captures the original notion of weak learnability as being "slightly better than random guessing". We give a simple and efficient boosting algorithm, that does not require realizability assumptions and its sample and oracle complexity bounds are independent of the number of classes. In addition, we utilize our new boosting technique in several theoretical applications within the context of List PAC Learning. First, we establish an equivalence to weak PAC learning. Furthermore, we present a new result on boosting for list learners, as well as provide a novel proof for the characterization of multiclass PAC learning and List PAC learning. Notably, our technique gives rise to a simplified analysis, and also implies an improved error bound for large list sizes, compared to previous results.
- Reducing multiclass to binary: A unifying approach for margin classifiers. Journal of machine learning research, 1(Dec):113–141, 2000.
- A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
- A simple multi-class boosting framework with theoretical guarantees and empirical proficiency. In International Conference on Machine Learning, pages 186–194. PMLR, 2017.
- Guess-averse loss functions for cost-sensitive multiclass boosting. In International Conference on Machine Learning, pages 586–594. PMLR, 2014.
- A characterization of multiclass learnability. arXiv preprint arXiv:2203.01550, 2022.
- Online boosting with bandit feedback. In Algorithmic Learning Theory, pages 397–420. PMLR, 2021.
- Multiclass boosting and the cost of weak learning. In NIPS, 2021.
- A characterization of list learnability. arXiv preprint arXiv:2211.04956, 2022.
- Optimal learners for multiclass problems. In COLT, pages 287–316, 2014.
- Sample compression, learnability, and the vapnik-chervonenkis dimension. Machine learning, 21(3):269–304, 1995.
- Yoav Freund. Boosting a weak learning algorithm by majority. In Mark A. Fulk and John Case, editors, Proceedings of the Third Annual Workshop on Computational Learning Theory, COLT 1990, University of Rochester, Rochester, NY, USA, August 6-8, 1990, pages 202–216. Morgan Kaufmann, 1990.
- A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci., 55(1):119–139, 1997.
- Predicting {{\{{0, 1}}\}}-functions on randomly drawn points. Information and Computation, 115(2):248–292, 1994.
- M. Kearns. Thoughts on hypothesis boosting. Unpublished, December 1988.
- Balázs Kégl. The return of adaboost. mh: Multi-class hamming trees. ICLR, 2014.
- Multi-class deep boosting. In Advances in Neural Information Processing Systems, pages 2501–2509, 2014.
- Relating data compression and learnability. Unpublished manuscript, 1986.
- List online classification. arXiv preprint arXiv:2303.15383, 2023.
- A theory of multiclass boosting. Journal of Machine Learning Research, 14:437–497, 2011.
- Shifting, one-inclusion mistake bounds and tight multiclass expected risk bounds. In NIPS, pages 1193–1200, 2006.
- Multiclass boosting: Theory and algorithms. In Advances in Neural Information Processing Systems, pages 2124–2132, 2011.
- Robert E. Schapire. The strength of weak learnability. Machine Learning, 5(2):197–227, 1990.
- Boosting: Foundations and algorithms. Cambridge university press, 2012.
- Improved boosting algorithms using confidence-rated predictions. Machine learning, 37(3):297–336, 1999.
- A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
- Understanding machine learning: From theory to algorithms. Cambridge University Press, 2014.
- John von Neumann. Zur theorie der gesellschaftsspiele. Mathematische Annalen, 100(1):295–320, 1928.
- Algorithmic learning in a random world, volume 29. Springer, 2005.
- A multi-class boosting method with direct optimization. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 273–282, 2014.