2000 character limit reached
Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models (2305.16243v3)
Published 25 May 2023 in cs.CL
Abstract: Augmenting LLMs with a retrieval mechanism has been shown to significantly improve their performance while keeping the number of parameters low. Retrieval-augmented models commonly rely on a semantic retrieval mechanism based on the similarity between dense representations of the query chunk and potential neighbors. In this paper, we study the state-of-the-art Retro model and observe that its performance gain is better explained by surface-level similarities, such as token overlap. Inspired by this, we replace the semantic retrieval in Retro with a surface-level method based on BM25, obtaining a significant reduction in perplexity. As full BM25 retrieval can be computationally costly for large datasets, we also apply it in a re-ranking scenario, gaining part of the perplexity reduction with minimal computational overhead.
- Learning to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations.
- Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2206–2240. PMLR.
- Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
- BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
- The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027.
- Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437–450.
- Realm: Retrieval-augmented language model pre-training. In International Conference on Machine Learning, pages 3929–3938. PMLR.
- HuggingFace. Huggingface T5.
- Gautier Izacard and Edouard Grave. 2021a. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880, Online. Association for Computational Linguistics.
- Gautier Izacard and Edouard Grave. 2021b. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880, Online. Association for Computational Linguistics.
- Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299.
- Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547.
- Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.
- Nearest neighbor machine translation. In International Conference on Learning Representations.
- Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations (ICLR).
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster).
- A survey on retrieval-augmented text generation. arXiv preprint 2202.01110.
- Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 2356–2362, New York, NY, USA. Association for Computing Machinery.
- Apache Lucene. Lucene analyzer.
- On the generalization ability of retrieval-enhanced transformers. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1485–1493, Dubrovnik, Croatia. Association for Computational Linguistics.
- Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc.
- Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/2112.11446.
- Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
- In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083.
- Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
- Okapi at TREC-3. In Overview of the Third Text REtrieval Conference (TREC-3), pages 109–126. Gaithersburg, MD: NIST.
- Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
- Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
- Memorizing transformers. In International Conference on Learning Representations.
- Why do nearest neighbor language models work? arXiv preprint arXiv:2301.02828.
- Defending against neural fake news. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
- Ehsan Doostmohammadi (11 papers)
- Tobias Norlund (6 papers)
- Marco Kuhlmann (13 papers)
- Richard Johansson (18 papers)