Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
72 tokens/sec
GPT-4o
61 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (2407.16833v2)

Published 23 Jul 2024 in cs.CL, cs.AI, and cs.LG
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

Abstract: Retrieval Augmented Generation (RAG) has been a powerful tool for LLMs to efficiently process overly lengthy contexts. However, recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. We conduct a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both. We benchmark RAG and LC across various public datasets using three latest LLMs. Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG's significantly lower cost remains a distinct advantage. Based on this observation, we propose Self-Route, a simple yet effective method that routes queries to RAG or LC based on model self-reflection. Self-Route significantly reduces the computation cost while maintaining a comparable performance to LC. Our findings provide a guideline for long-context applications of LLMs using RAG and LC.

PDF HTML Abstract

Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

The paper "Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach," authored by Zhuowan Li and colleagues from Google DeepMind and the University of Michigan, presents a thorough comparison of Retrieval Augmented Generation (RAG) and Long-Context (LC) LLMs. This paper is performed using three state-of-the-art LLMs: Gemini-1.5, GPT-4O, and GPT-3.5-Turbo. The authors propose a hybrid approach named Self-Route, which seeks to optimize the trade-offs between performance and computational cost.

Comparison of RAG and LC Approaches

RAG has been an essential tool for extending the context window of LLMs by retrieving relevant information chunks and then using LLMs to generate responses. This process is computationally efficient and helps in scenarios where the model’s input context size is constrained. On the other hand, modern LLMs like Gemini-1.5 and GPT-4O can process extremely long contexts directly, leveraging their advanced architectural capabilities.

Benchmarking Analysis

The authors conducted extensive benchmarking across nine datasets extracted from LongBench and $ datasets, including NarrativeQA, Qasper, MultiFieldQA, and HotpotQA, among others. Metrics such as F1 scores, accuracy, and ROUGE were used for evaluation. Key findings reveal that LC methods outperform RAG consistently across most settings when sufficient computational resources are available:

  • Gemini-1.5-Pro: LC outperformed RAG by an average of 7.6%.
  • GPT-4O: LC outperformed RAG by an average of 13.1%.
  • GPT-3.5-Turbo: LC had a 3.6% average performance advantage over RAG.

Notably, the performance of retrieval-based approaches like RAG was particularly strong on datasets with extremely long contexts, where direct processing by LLMs like GPT-3.5-Turbo proved infeasible due to token limitations.

Self-Route Method

Motivated by a need to balance performance and computational cost, the Self-Route method is introduced. Self-Route targets queries to either RAG or LC based on their nature. Key advantages of the approach include:

  • Cost Efficiency: Reduces computational costs by leveraging RAG for queries flagged as answerable based on the chunked context.
  • Maintained Performance: Achieves performance levels similar to LC while requiring fewer computational resources. For instance, in the case of the Gemini-1.5-Pro, Self-Route reduces costs by 65% for Gemini-1.5-Pro and 39% for GPT-4O while maintaining comparable performance.

Numerical Insights

Interestingly, the analysis shows that RAG predictions matched LC predictions over 60% of the time. This overlap indicates an opportunity for cost-saving by dynamically switching between RAG and LC approaches based on initial predictions.

Implications and Future Directions

The paper provides a guideline for deploying LLMs in long-context applications, highlighting the feasibility of hybrid methods like Self-Route. From a practical standpoint, applications involving long document processing, retrieval-based QA systems, and real-time information synthesis stand to benefit significantly from these findings.

Theoretical implications include potential refinement of self-reflective mechanisms within models to optimize routing and enhanced failure analysis frameworks to further dissect retrieval shortcomings (e.g., ambiguous or multi-step queries).

Conclusion

By presenting a comprehensive analysis and an innovative hybrid approach, the paper elucidates the nuanced trade-offs between RAG and LC LLMs. As the capabilities of LLMs continue to evolve, hybrid methodologies like Self-Route may become integral to harnessing their full potential while managing computational resources efficiently. Future research might explore further tuning of routing algorithms, integration with advanced retrieval techniques, and application-specific adaptations of hybrid LLM models. This paper lays a robust foundation for these explorations, carrying significant implications for both theoretical developments and practical deployments in AI research.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088.
  3. Anthropic. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet/.
  4. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.
  5. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.
  6. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
  7. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR.
  8. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.00610.
  9. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.
  10. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595.
  11. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011.
  12. Google. 2024. Gemini pricing. https://ai.google.dev/pricing.
  13. Greg Kamradt. 2023. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/LLMTest_NeedleInAHaystack.
  14. Longt5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 724–736.
  15. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.
  16. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060.
  17. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654.
  18. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301.
  19. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.
  20. Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
  21. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839.
  22. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172.
  23. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328.
  24. In search of needles in a 10m haystack: Recurrent memory finds what llms miss. arXiv preprint arXiv:2402.10790.
  25. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848.
  26. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
  27. Llatrieval: Llm-verified retrieval for verifiable generation. arXiv preprint arXiv:2311.07838.
  28. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452.
  29. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  30. Yuanhua Lv and ChengXiang Zhai. 2009. Adaptive relevance feedback in information retrieval. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 255–264.
  31. Query rewriting for retrieval-augmented large language models. arXiv preprint arXiv:2305.14283.
  32. Evaluating very long-term conversational memory of llm agents. arXiv preprint arXiv:2402.17753.
  33. OpenAI. 2023. Gpt-3.5-turbo. https://platform.openai.com/docs/models/gpt-3-5-turbo.
  34. OpenAI. 2024a. Gpt-4o. https://openai.com/index/hello-gpt-4o/.
  35. OpenAI. 2024b. Openai-api pricing. https://platform.openai.com/docs/overview.
  36. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  37. Scrolls: Standardized comparison over long language sequences. arXiv preprint arXiv:2201.03533.
  38. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652.
  39. Counting-stars: A simple, efficient, and reasonable strategy for evaluating long-context large language models. arXiv preprint arXiv:2403.11802.
  40. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.
  41. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  42. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025.
  43. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884.
  44. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.
  45. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05136.
  46. Chengxiang Zhai and John Lafferty. 2001. Model-based feedback in the language modeling approach to information retrieval. In Proceedings of the tenth international conference on Information and knowledge management, pages 403–410.
  47. Infinity bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718.
  48. Qmsum: A new benchmark for query-based multi-domain meeting summarization. arXiv preprint arXiv:2104.05938.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Zhuowan Li (13 papers)
  2. Cheng Li (1094 papers)
  3. Mingyang Zhang (56 papers)
  4. Qiaozhu Mei (68 papers)
  5. Michael Bendersky (63 papers)
Citations (13)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube