Linear Classifier: An Often-Forgotten Baseline for Text Classification (2306.07111v1)
Abstract: Large-scale pre-trained LLMs such as BERT are popular solutions for text classification. Due to the superior performance of these advanced methods, nowadays, people often directly train them for a few epochs and deploy the obtained model. In this opinion paper, we point out that this way may only sometimes get satisfactory results. We argue the importance of running a simple baseline like linear classifiers on bag-of-words features along with advanced methods. First, for many text data, linear methods show competitive performance, high efficiency, and robustness. Second, advanced models such as BERT may only achieve the best results if properly applied. Simple baselines help to confirm whether the results of advanced models are acceptable. Our experimental results fully support these points.
- A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pages 144–152. ACM Press.
- LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 4310–4330.
- BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, pages 4171–4186.
- LIBLINEAR: a library for large linear classification. Journal of Machine Learning Research, 9:1871–1874.
- Rong-En Fan and Chih-Jen Lin. 2007. A study on threshold selection for multi-label classification. Technical report, Department of Computer Science, National Taiwan University.
- On the cost-effectiveness of stacking of neural and non-neural methods for text classification: Scenarios and performance prediction. In Findings of the Association for Computational Linguistics: ACL/IJCNLP, pages 4003–4014.
- Deep Learning. The MIT Press.
- Karen S. Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11–20.
- Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning, pages 331–339.
- RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361–397.
- On the use of unrealistic predictions in hundreds of papers evaluating graph representations. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI).
- Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of Research and Development, 2(2):159–165.
- Optimizing f-measures by cost-sensitive classification. In Advances in Neural Information Processing Systems, volume 27.
- A comparison of svm against pre-trained language models (plms) for text classification tasks. In Machine Learning, Optimization, and Data Science, pages 304–313. Springer Nature Switzerland.
- Yiming Yang. 2001. A study on thresholding strategies for text categorization. In Proceedings of the 24th ACM International Conference on Research and Development in Information Retrieval, pages 137–145, New Orleans, US. ACM Press, New York, US.
- PECOS: Prediction for enormous and correlated output spaces. Journal of Machine Learning Research, 23(98):1–32.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
