· Trending Papers
Emergent Mind

Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost

(2306.15766)
Published Jun 27, 2023 in cs.CL and cs.LG

Abstract

State-of-the-art supervised NLP models achieve high accuracy but are also susceptible to failures on inputs from low-data regimes, such as domains that are not represented in training data. As an approximation to collecting ground-truth labels for the specific domain, we study the use of LLMs for annotating inputs and improving the generalization of NLP models. Specifically, given a budget for LLM annotations, we present an algorithm for sampling the most informative inputs to annotate and retrain the NLP model. We find that popular active learning strategies such as uncertainty-based sampling do not work well. Instead, we propose a sampling strategy based on the difference in prediction scores between the base model and the finetuned NLP model, utilizing the fact that most NLP models are finetuned from a base model. Experiments with classification (semantic similarity) and ranking (semantic search) tasks show that our sampling strategy leads to significant gains in accuracy for both the training and target domains.

We're not able to analyze this paper right now due to high demand.

Please check back later (sorry!).

Sign up for a free account or log in to generate a summary of this paper:

Sign Up Now
or
Log In

We ran into a problem analyzing this paper.

Newsletter

Get summaries of trending comp sci papers delivered straight to your inbox:

Unsubscribe anytime.

References
  1. The extreme classification repository: Multi-label datasets and code
  2. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
  4. An empirical survey of data augmentation for limited data learning in nlp. Transactions of the Association for Computational Linguistics, 11:191–211.
  5. Improving Contrastive Learning of Sentence Embeddings from AI Feedback
  6. Chatgpt goes to law school. Available at SSRN.
  7. Ngame: Negative mining-aware mini-batching for extreme classification. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pages 258–266.
  8. Is GPT-3 a Good Data Annotator?
  9. To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering
  10. A survey on concept drift adaptation. ACM Comput. Surv., 46(4).
  11. ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
  12. Annotation Artifacts in Natural Language Inference Data
  13. Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507–517.
  14. AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
  15. Targeted Data Generation: Finding and Fixing Model Weaknesses
  16. Large Language Models are Zero-Shot Rankers for Recommender Systems
  17. David D Lewis. 1995. A sequential algorithm for training text classifiers: Corrigendum and additional data. In Acm Sigir Forum, volume 29, pages 13–19. ACM New York, NY, USA.
  18. Rethinking Distributional Matching Based Domain Adaptation
  19. UDApter -- Efficient Domain Adaptation Using Adapters
  20. Active Learning Principles for In-Context Learning with Large Language Models
  21. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  22. Domain divergences: A survey and empirical analysis. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1830–1849, Online. Association for Computational Linguistics.
  23. Neural Unsupervised Domain Adaptation in NLP---A Survey
  24. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
  25. Marco Tulio Ribeiro and Scott Lundberg. 2022. Adaptive testing and debugging of NLP models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3253–3267, Dublin, Ireland. Association for Computational Linguistics.
  26. UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
  27. Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers
  28. Burr Settles. 2009. Active learning literature survey.
  29. Mitigating Gender Bias in Natural Language Processing: Literature Review
  30. Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks
  31. Attention is all you need. Advances in neural information processing systems, 30.
  32. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  33. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering.
  34. Want To Reduce Labeling Cost? GPT-3 Can Help
  35. LLM-powered Data Augmentation for Enhanced Cross-lingual Performance
  36. A Survey of Active Learning for Natural Language Processing

Show All 36