Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Understanding In-Context Learning via Supportive Pretraining Data (2306.15091v1)

Published 26 Jun 2023 in cs.CL

Abstract: In-context learning (ICL) improves LLMs' performance on a variety of NLP tasks by simply demonstrating a handful of examples at inference time. It is not well understood why ICL ability emerges, as the model has never been specifically trained on such demonstrations. Unlike prior work that explores implicit mechanisms behind ICL, we study ICL via investigating the pretraining data. Specifically, we first adapt an iterative, gradient-based approach to find a small subset of pretraining data that supports ICL. We observe that a continued pretraining on this small subset significantly improves the model's ICL ability, by up to 18%. We then compare the supportive subset constrastively with random subsets of pretraining data and discover: (1) The supportive pretraining data to ICL do not have a higher domain relevance to downstream tasks. (2) The supportive pretraining data have a higher mass of rarely occurring, long-tail tokens. (3) The supportive pretraining data are challenging examples where the information gain from long-range context is below average, indicating learning to incorporate difficult long-range context encourages ICL. Our work takes a first step towards understanding ICL via analyzing instance-level pretraining data. Our insights have a potential to enhance the ICL ability of LLMs by actively guiding the construction of pretraining data in the future.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661.
  2. Contributions to the study of sms spam filtering: new collection and results. In Proceedings of the 11th ACM symposium on Document engineering, pages 259–262.
  3. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  4. Data distributional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems.
  5. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  6. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559.
  7. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT.
  8. A survey for in-context learning. arXiv preprint arXiv:2301.00234.
  9. What can transformers learn in-context? a case study of simple function classes. arXiv preprint arXiv:2208.01066.
  10. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009.
  11. Pre-training to learn in context.
  12. Simfluence: Modeling the influence of individual training examples by simulating training runs. ArXiv, abs/2303.08114.
  13. Xiaochuang Han and Yulia Tsvetkov. 2022. Orca: Interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data. arXiv preprint arXiv:2205.12600.
  14. On the blind spots of model-based evaluation metrics for text generation. arXiv preprint arXiv:2212.10020.
  15. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.
  16. The curious case of neural text degeneration. In International Conference on Learning Representations.
  17. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  18. Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proc. ICML.
  19. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  20. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP.
  21. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46–51.
  22. In-context learning and induction heads. arXiv preprint arXiv:2209.11895.
  23. Fully sharded data parallel: faster ai training with fewer gpus. https://engineering.fb.com/2021/07/15/open-source/fsdp/.
  24. What in-context learning"learns"in-context: Disentangling task recognition and task learning.
  25. Language models as knowledge bases? In Proc. EMNLP.
  26. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In Proc. NeurIPS.
  27. Estimating training data influence by tracking gradient descent. In Proc. NeurIPS.
  28. Impact of pretraining term frequencies on few-shot reasoning. ArXiv, abs/2202.07206.
  29. Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proc. EACL.
  30. On the effect of pretraining corpora on in-context learning by a large-scale language model. arXiv preprint arXiv:2204.13509.
  31. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053.
  32. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.
  33. Attention is all you need. Advances in neural information processing systems, 30.
  34. Transformers learn in-context by gradient descent. arXiv preprint arXiv:2212.07677.
  35. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705.
  36. Larger language models do in-context learning differently. ArXiv, abs/2303.03846.
  37. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations.
  38. Tweetqa: A social media focused question answering dataset. arXiv preprint arXiv:1907.06292.
  39. Robustness of demonstration-based learning under limited data scenario. arXiv preprint arXiv:2210.10693.
  40. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
  41. Character-level convolutional networks for text classification. In Proc. NeurIPS.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Xiaochuang Han (23 papers)
  2. Daniel Simig (10 papers)
  3. Todor Mihaylov (23 papers)
  4. Yulia Tsvetkov (142 papers)
  5. Asli Celikyilmaz (80 papers)
  6. Tianlu Wang (33 papers)
Citations (26)
View on Semantic Scholar