Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

In-context Learning with Retrieved Demonstrations for Language Models: A Survey (2401.11624v5)

Published 21 Jan 2024 in cs.CL, cs.AI, and cs.IR
In-context Learning with Retrieved Demonstrations for Language Models: A Survey

Abstract: LLMs, especially pre-trained LLMs, have showcased remarkable abilities as few-shot in-context learners (ICL), adept at adapting to new tasks with just a few demonstrations in the input context. However, the model's ability to perform ICL is sensitive to the choice of the few-shot demonstrations. Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query. The implementation of demonstration retrieval is relatively straightforward, leveraging existing databases and retrieval systems. This not only improves the efficiency and scalability of the learning process but also has been shown to reduce biases inherent in manual example selection. In light of the encouraging results and growing research in ICL with retrieved demonstrations, we conduct an extensive review of studies in this area. In this survey, we discuss and compare different design choices for retrieval models, retrieval training procedures, and inference algorithms.

PDF HTML Abstract

Introduction

Few-shot in-context learning (ICL) is a capability of LLMs to adapt to new tasks using a limited number of demonstrations. This capability eludes the need for task-specific fine-tuning, presenting several advantages including resource efficiency and mitigating overfitting. Traditional ICL approaches use fixed demonstrations for all queries, leading to suboptimal use of the LLMs' potential. This survey provides an exhaustive review of a burgeoning variation: Retrieval-based ICL (RetICL), where tailored demonstrations are selected dynamically for each query to optimize model performance.

Few-shot In-context Learning for LLMs

LLMs have been pre-eminent in handling few-shot learning scenarios where models infer based on a handful of demonstrations without parameter updating. Despite significant strides, the success of ICL hinges on the quality, quantity and diversity of demonstrations. This calls for techniques that shift from static to dynamic, query-oriented demonstration selection. RetICL intends to maximize the relevance and usefulness of these demonstrations by considering key elements such as retrieval model complexity, diversity of the retrieval corpus and the retriever’s objectives during the selection process.

In-context Learning with Demonstration Retrieval

RetICL involves selecting demonstrations in alignment with the input query. Various strategies materialize this objective, with methods spanning one-hoc retrieval, clustering retrieval, and iterative retrieval. They differ in how demonstrations are selected—either independently, through clustering for diversity, or iteratively to build upon the context of previously chosen demonstrations. The retrieval corpus is equally vital and can range from in-domain, mix-domain, and cross-domain to raw text and unlabelled queries. Advanced RetICL techniques employ fine-tuning of retrieval models to curate training data, with the focus shifting towards training objectives that can range from InfoNCE loss to distillation by KL divergence, aiming for relevancy and diversity.

Applications and Future Directions

RetICL has established efficacy across several task categories including natural language understanding, reasoning, knowledge-based QA, and text generation. Challenges persist in corpus creation, retriever choice, training methods, and the need for active learning integration. Future research must resolve these challenges while also enhancing our theoretical comprehension of why similar demonstrations derived from retrieval methods outperform random selection, and how RetICL can be adapted to smaller models without sacrificing performance.

The insights gleaned from RetICL expose the remarkable yet untapped potential of LLMs when demonstrations are chosen with an informed, context-sensitive retrieval strategy. The exploration of this domain is poised to refine our understanding and utilization of LLMs, pushing the boundaries of AI and its application within complex, resource-constrained environments. This survey encapsulates the current state while guiding future developments in the evolving landscape of in-context learning.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (132)
  1. ComQA: A community-sourced dataset for complex factoid question answering with paraphrase clusters. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 307–317, Minneapolis, Minnesota. Association for Computational Linguistics.
  2. In-context examples selection for machine translation. arXiv preprint arXiv:2212.02437.
  3. Ben Taskar Alex Kulesz. 2012. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 5:123–286.
  4. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556–571.
  5. A neural probabilistic language model. Advances in neural information processing systems, 13.
  6. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544.
  7. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence.
  8. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642.
  9. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), page 632–642.
  10. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  11. Christopher J.C. Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Technical report, Microsoft Research Technical Report MSR-TR-2010-82.
  12. Semeval-2017 task 1: Semantic textual similarity - multilingual and cross-lingual focused evaluation. CoRR, abs/1708.00055.
  13. Improving in-context few-shot learning via self-supervised training. arXiv preprint arXiv:2205.01703.
  14. Uprise: Universal prompt retrieval for improving zero-shot evaluation. arXiv preprint arXiv:2303.08518.
  15. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.
  16. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
  17. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.
  18. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457.
  19. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  20. Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement. arXiv preprint arXiv:2204.13074.
  21. Finding contradictions in text. In Proceedings of acl-08: Hlt, pages 1039–1047.
  22. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314.
  23. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  24. Unsupervised construction of large paraphrase corpora: exploiting massively parallel news sources. In Proceedings of the 20th international conference on Computational Linguistics, pages 350–es.
  25. Semantic noise matters for neural natural language generation. In Proceedings of the 12th International Conference on Natural Language Generation, pages 421–426.
  26. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720.
  27. Ambiguity-aware in-context learning with large language models. arXiv preprint arXiv:2309.07900.
  28. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910.
  29. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583–30598.
  30. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009.
  31. Demix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5557–5576.
  32. Take one step at a time to know incremental utility of demonstration: An analysis on reranking for few-shot in-context learning. arXiv preprint arXiv:2311.09619.
  33. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  34. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
  35. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations.
  36. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  37. In-context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2627–2643.
  38. Can machines learn morality? the delphi experiment. arXiv preprint arXiv:2110.07574.
  39. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
  40. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
  41. James H. Jurafsky, Dan; Martin. 2021. N-gram language models. In Speech and Language Processing (3rd ed.).
  42. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.
  43. Understanding finetuning for factual knowledge extraction from language models. arXiv preprint arXiv:2301.11293.
  44. Lambada: Backward chaining for automated reasoning in natural language. In ACL.
  45. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL).
  46. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024.
  47. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. CoRR, abs/2004.12832.
  48. Large language models are zero-shot reasoners, 2022. URL https://arxiv. org/abs/2205.11916.
  49. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 5(2–3):123–286.
  50. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.
  51. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702.
  52. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059.
  53. Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2212.06800.
  54. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857.
  55. Mtop: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2950–2962.
  56. Unified demonstration retriever for in-context learning. arXiv preprint arXiv:2305.04320.
  57. Xiaonan Li and Xipeng Qiu. 2023a. Finding supporting examples for in-context learning. arXiv preprint arXiv:2302.13539.
  58. Xiaonan Li and Xipeng Qiu. 2023b. Mot: Pre-thinking and recalling enable chatgpt to self-improve with memory-of-thoughts. arXiv preprint arXiv:2305.05181.
  59. Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models. arXiv preprint arXiv:2005.00683.
  60. Nl2bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  61. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167.
  62. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114.
  63. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  64. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688.
  65. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In The Eleventh International Conference on Learning Representations.
  66. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098.
  67. Dr. icl: Demonstration-retrieved in-context learning. arXiv preprint arXiv:2305.14128.
  68. Z-icl: Zero-shot in-context learning with pseudo-demonstrations. arXiv preprint arXiv:2212.09865.
  69. Fairness-guided few-shot prompting for large language models. arXiv preprint arXiv:2303.13217.
  70. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.06009.
  71. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.
  72. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  73. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809.
  74. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837.
  75. Diversity of thought improves reasoning abilities of large language models. arXiv preprint arXiv:2310.07088.
  76. Dart: Open-domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 432–447.
  77. Ms marco: A human generated machine reading comprehension dataset. choice, 2640:660.
  78. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899.
  79. Cross-lingual retrieval augmented prompt for low-resource languages. ArXiv, abs/2212.09651.
  80. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  81. Totto: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173–1186.
  82. Language models as knowledge bases? arXiv preprint arXiv:1909.01066.
  83. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations.
  84. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  85. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.
  86. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
  87. Nils Reimers and Iryna Gurevych. 2019a. Sentence-bert: Sentence embeddings using siamese bert-networks. In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982––3992.
  88. Nils Reimers and Iryna Gurevych. 2019b. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
  89. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.
  90. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series.
  91. Ronald Rosenfeld. 2000. Two decades of statistical language modeling: Where do we go from here? Proceedings of the IEEE, 88(8):1270–1278.
  92. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671.
  93. Colbertv2: Effective and efficient retrieval via lightweight late interaction. CoRR, abs/2112.01488.
  94. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304:114135.
  95. Alexander Scarlatos and Andrew Lan. 2023. Reticl: Sequential retrieval of in-context examples with reinforcement learning. arXiv preprint arXiv:2305.14502.
  96. Xricl: Cross-lingual retrieval-augmented in-context learning for cross-lingual text-to-sql semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5248–5259.
  97. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980.
  98. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.
  99. Can language models be biomedical knowledge bases? arXiv preprint arXiv:2109.07154.
  100. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
  101. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158.
  102. Attention is all you need. Advances in neural information processing systems, 30.
  103. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151–35174. PMLR.
  104. Better zero-shot reasoning with self-adaptive prompting. arXiv preprint arXiv:2305.14106.
  105. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.
  106. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001.
  107. Text embeddings by weakly-supervised contrastive pre-training.
  108. Learning to retrieve in-context examples for large language models. arXiv preprint arXiv:2307.07164.
  109. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301.11916.
  110. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  111. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
  112. " according to…" prompting language models improves quoting from pre-training data. arXiv preprint arXiv:2305.13252.
  113. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122.
  114. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183–198.
  115. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations.
  116. mt5: A massively multilingual pre-trained text-to-text transformer. CoRR, abs/2010.11934.
  117. Large language models as optimizers. arXiv preprint arXiv:2309.03409.
  118. Multilingual universal sentence encoder for semantic retrieval. CoRR, abs/1907.04307.
  119. Compositional exemplars for in-context learning. arXiv preprint arXiv:2302.05698.
  120. In-context instruction learning. arXiv preprint arXiv:2302.14691.
  121. Xue Ying. 2019. An overview of overfitting and its solutions. In Journal of physics: Conference series, volume 1168, page 022022. IOP Publishing.
  122. Generate rather than retrieve: Large language models are strong context generators. In The Eleventh International Conference on Learning Representations.
  123. Tasklama: probing the complex task understanding of language models. arXiv preprint arXiv:2308.15299.
  124. John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pages 1050–1055.
  125. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800.
  126. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792.
  127. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
  128. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134–9148.
  129. PAWS: Paraphrase Adversaries from Word Scrambling. In Proc. of NAACL.
  130. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations.
  131. A survey of large language models. arXiv preprint arXiv:2303.18223.
  132. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697–12706. PMLR.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Xin Xu (187 papers)
  2. Yue Liu (256 papers)
  3. Panupong Pasupat (27 papers)
  4. Mehran Kazemi (26 papers)
  5. Man Luo (55 papers)
Citations (31)
View on Semantic Scholar