A Practical Survey on Zero-shot Prompt Design for In-context Learning (2309.13205v1)
Abstract: The remarkable advancements in LLMs have brought about significant improvements in Natural Language Processing(NLP) tasks. This paper presents a comprehensive review of in-context learning techniques, focusing on different types of prompts, including discrete, continuous, few-shot, and zero-shot, and their impact on LLM performance. We explore various approaches to prompt design, such as manual design, optimization algorithms, and evaluation methods, to optimize LLM performance across diverse tasks. Our review covers key research studies in prompt engineering, discussing their methodologies and contributions to the field. We also delve into the challenges faced in evaluating prompt performance, given the absence of a single "best" prompt and the importance of considering multiple metrics. In conclusion, the paper highlights the critical role of prompt design in harnessing the full potential of LLMs and provides insights into the combination of manual design, optimization techniques, and rigorous evaluation for more effective and efficient use of LLMs in various NLP tasks.
- Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
- Palm: Scaling language modeling with pathways.
- Rlprompt: Optimizing discrete text prompts with reinforcement learning. In Conference on Empirical Methods in Natural Language Processing.
- BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
- Making pre-trained language models better few-shot learners. ArXiv, abs/2012.15723.
- Thinking about gpt-3 in-context learning for biomedical ie? think again. In Conference on Empirical Methods in Natural Language Processing.
- Bertese: Learning to speak to bert. ArXiv, abs/2103.05327.
- Survey of hallucination in natural language generation. ACM Computing Surveys.
- How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
- Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.
- Sawan Kumar and Partha P. Talukdar. 2021. Reordering examples helps during priming-based few-shot learning. ArXiv, abs/2106.01751.
- BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. CoRR, abs/1910.13461.
- Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), abs/2101.00190.
- Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
- Holistic evaluation of language models.
- What makes good in-context examples for gpt-3? In Workshop on Knowledge Extraction and Integration for Deep Learning Architectures; Deep Learning Inside Out.
- P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61–68, Dublin, Ireland. Association for Computational Linguistics.
- GPT understands, too. CoRR, abs/2103.10385.
- Reframing instructional prompts to gptk’s language. In Findings.
- Grips: Gradient-free, edit-based instruction search for prompting large language models. ArXiv, abs/2203.07281.
- Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
- Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm.
- Learning to retrieve prompts for in-context learning. ArXiv, abs/2112.08633.
- Toward human readable prompt tuning: Kubrick’s the shining is a good movie, and a good prompt too? ArXiv, abs/2212.10539.
- Eliciting knowledge from language models using automatically generated prompts. ArXiv, abs/2010.15980.
- Chain-of-thought prompting elicits reasoning in large language models.
- David H. Wolpert and William G. Macready. 1995. No free lunch theorems for search.
- D.H. Wolpert and W.G. Macready. 1997. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1):67–82.
- Self-adaptive in-context learning. ArXiv, abs/2212.10375.
- Large language models are human-level prompt engineers. ArXiv, abs/2211.01910.
- Yinheng Li (14 papers)