Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data (2404.02422v1)

Published 3 Apr 2024 in cs.CL and cs.LG

Abstract: LLMs operating in 0-shot or few-shot settings achieve competitive results in Text Classification tasks. In-Context Learning (ICL) typically achieves better accuracy than the 0-shot setting, but it pays in terms of efficiency, due to the longer input prompt. In this paper, we propose a strategy to make LLMs as efficient as 0-shot text classifiers, while getting comparable or better accuracy than ICL. Our solution targets the low resource setting, i.e., when only 4 examples per class are available. Using a single LLM and few-shot real data we perform a sequence of generation, filtering and Parameter-Efficient Fine-Tuning steps to create a robust and efficient classifier. Experimental results show that our approach leads to competitive results on multiple text classification datasets.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (27)
  1. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  2. Universal sentence encoder.
  3. One-for-all: Generalized lora for parameter-efficient fine-tuning.
  4. Scaling instruction-finetuned language models.
  5. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 575–593, Toronto, Canada. Association for Computational Linguistics.
  6. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  7. Annollm: Making large language models to be better crowdsourced annotators.
  8. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685.
  9. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  10. Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING.
  11. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.
  12. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems, volume 35, pages 1950–1965. Curran Associates, Inc.
  13. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. CoRR, abs/2110.07602.
  14. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61–68, Dublin, Ireland. Association for Computational Linguistics.
  15. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  16. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft.
  17. Training language models to follow instructions with human feedback.
  18. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL. Association for Computational Linguistics.
  19. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only.
  20. Exploring the limits of transfer learning with a unified text-to-text transformer.
  21. LINGUIST: Language model instruction tuning to generate annotated utterances for intent classification and slot tagging. In Proceedings of the 29th International Conference on Computational Linguistics, pages 218–241, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  22. Llama: Open and efficient foundation language models.
  23. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  24. Large language model as attributed training data generator: A tale of diversity and bias. In Advances in Neural Information Processing Systems, volume 36, pages 55734–55784. Curran Associates, Inc.
  25. Character-level convolutional networks for text classification.
  26. Judging llm-as-a-judge with mt-bench and chatbot arena.
  27. Can chatgpt reproduce human-generated labels. A Study of Social Computing Tasks.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Parth Patwa (28 papers)
  2. Simone Filice (9 papers)
  3. Zhiyu Chen (60 papers)
  4. Giuseppe Castellucci (10 papers)
  5. Oleg Rokhlenko (22 papers)
  6. Shervin Malmasi (40 papers)
Citations (7)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets