2000 character limit reached
Small Language Models are Good Too: An Empirical Study of Zero-Shot Classification (2404.11122v1)
Published 17 Apr 2024 in cs.AI
Abstract: This study is part of the debate on the efficiency of large versus small LLMs for text classification by prompting.We assess the performance of small LLMs in zero-shot text classification, challenging the prevailing dominance of large models.Across 15 datasets, our investigation benchmarks LLMs from 77M to 40B parameters using different architectures and scoring functions. Our findings reveal that small models can effectively classify texts, getting on par with or surpassing their larger counterparts.We developed and shared a comprehensive open-source repository that encapsulates our methodologies. This research underscores the notion that bigger isn't always better, suggesting that resource-efficient small models may offer viable solutions for specific data classification challenges.
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. Technical report. ArXiv:2304.01373 [cs] type: article.
- Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
- PaLM: Scaling Language Modeling with Pathways. Technical report. ArXiv:2204.02311 [cs] type: article.
- Ronen Eldan and Yuanzhi Li. 2023. Tinystories: How small can language models be and still speak coherent english?
- Beyond prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8560–8579, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Training Compute-Optimal Large Language Models. Technical report. ArXiv:2203.15556 [cs] type: article.
- Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right. Technical report. ArXiv:2104.08315 [cs] type: article.
- Scaling Laws for Neural Language Models. Technical report. ArXiv:2001.08361 [cs, stat] type: article.
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
- Reconstructing Capsule Networks for Zero-shot Intent Classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4799–4809, Hong Kong, China. Association for Computational Linguistics.
- The flan collection: Designing data and methods for effective instruction tuning.
- What makes pre-trained language models better zero-shot learners? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2288–2303, Toronto, Canada. Association for Computational Linguistics.
- Text Classification Using Label Names Only: A Language Model Self-Training Approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9006–9017, Online. Association for Computational Linguistics.
- Crosslingual Generalization through Multitask Finetuning. Technical report. ArXiv:2211.01786 [cs] type: article.
- Costs to Consider in Adopting NLP for Your Business. Technical report. ArXiv:2012.08958 [cs] type: article.
- OpenAI. 2023. GPT-4 Technical Report. Technical report. ArXiv:2303.08774 [cs] type: article.
- The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. Technical report. ArXiv:2306.01116 [cs] type: article.
- Rwkv: Reinventing rnns for the transformer era.
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Technical report. ArXiv:1910.10683 [cs, stat] type: article.
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. Working paper or preprint.
- Language Models in the Loop: Incorporating Prompting into Weak Supervision. Technical report. ArXiv:2205.02318 [cs] type: article.
- Retentive network: A successor to transformer for large language models.
- Llama: Open and efficient foundation language models.
- Lamini-lm: A diverse herd of distilled models from large-scale instructions. CoRR, abs/2304.14402.
- Zero-shot User Intent Detection via Capsule Neural Networks. Technical report. ArXiv:1809.00385 [cs] type: article.
- Instruction Tuning for Large Language Models: A Survey. Technical report. ArXiv:2308.10792 [cs] type: article.
- Pre-trained language models can be fully zero-shot learners. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15590–15606, Toronto, Canada. Association for Computational Linguistics.
- TubeSpam: Comment Spam Filtering on YouTube. 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pages 138–143.
- Tiago Almeida and Jos Hidalgo. 2012. SMS Spam Collection. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5CC84.
- SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33–38, Uppsala, Sweden. Association for Computational Linguistics.
- Overview of the BioCreative VI chemical-protein interaction Track.
- ETHOS: an Online Hate Speech Detection Dataset. Complex & Intelligent Systems, 8(6):4663–4678. ArXiv:2006.08328 [cs, stat].
- Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
- WRENCH: A Comprehensive Benchmark for Weak Supervision. ArXiv.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.