Designing Informative Metrics for Few-Shot Example Selection (2403.03861v3)
Abstract: Pretrained LLMs (PLMs) have shown remarkable few-shot learning capabilities when provided with properly formatted examples. However, selecting the "best" examples remains an open challenge. We propose a complexity-based prompt selection approach for sequence tagging tasks. This approach avoids the training of a dedicated model for selection of examples, and instead uses certain metrics to align the syntactico-semantic complexity of test sentences and examples. We use both sentence- and word-level metrics to match the complexity of examples to the (test) sentence being considered. Our results demonstrate that our approach extracts greater performance from PLMs: it achieves state-of-the-art performance on few-shot NER, achieving a 5% absolute improvement in F1 score on the CoNLL2003 dataset for GPT-4. We also see large gains of upto 28.85 points (F1/Acc.) in smaller models like GPT-j-6B.
- Dhananjay Ashok and Zachary C. Lipton. 2023. Promptner: Prompting for named entity recognition.
- Prompting language models for linguistic structure.
- Language models are few-shot learners.
- How is chatgpt’s behavior changing over time?
- What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
- Rethinking the role of demonstrations: What makes in-context learning work?
- Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034–4043, Marseille, France. European Language Resources Association.
- Learning to retrieve prompts for in-context learning.
- Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations.
- Beyond the text: Analysis of privacy statements through syntactic and semantic role labeling. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 85–98, Singapore. Association for Computational Linguistics.
- Going against the (appropriate) flow: A contextual integrity approach to privacy policy analysis. In AAAI Conference on Human Computation & Crowdsourcing.
- An information-theoretic approach to prompt engineering without ground truth labels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
- Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task chunking. In Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop.
- Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147.
- Chain-of-thought prompting elicits reasoning in large language models.
- Amir Zeldes. 2017. The gum corpus: creating multilayer resources in the classroom. Lang. Resour. Eval., 51(3):581–612.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.