ARAGOG: Advanced RAG Output Grading (2404.01037v1)
Abstract: Retrieval-Augmented Generation (RAG) is essential for integrating external knowledge into LLM outputs. While the literature on RAG is growing, it primarily focuses on systematic reviews and comparisons of new state-of-the-art (SoTA) techniques against their predecessors, with a gap in extensive experimental comparisons. This study begins to address this gap by assessing various RAG methods' impacts on retrieval precision and answer similarity. We found that Hypothetical Document Embedding (HyDE) and LLM reranking significantly enhance retrieval precision. However, Maximal Marginal Relevance (MMR) and Cohere rerank did not exhibit notable advantages over a baseline Naive RAG system, and Multi-query approaches underperformed. Sentence Window Retrieval emerged as the most effective for retrieval precision, despite its variable performance on answer similarity. The study confirms the potential of the Document Summary Index as a competent retrieval approach. All resources related to this research are publicly accessible for further investigation through our GitHub repository ARAGOG (https://github.com/predlico/ARAGOG). We welcome the community to further this exploratory study in RAG systems.
- Akash. Hybrid search: Optimizing rag implementation. https://medium.com/@csakash03/hybrid-search-is-a-method-to-optimize-rag-implementation-98d9d0911341, 2023. Accessed: 2024-04-01.
- T. Bratanic. Using a knowledge graph to implement a rag application. https://neo4j.com/developer-blog/knowledge-graph-rag-application/, 2023. Accessed: 2024-03-24.
- J. Carbonell and J. Goldstein. The use of mmr, diversity-based reranking for reordering documents and producing summaries. https://www.cs.cmu.edu/~jgc/publication/The_Use_MMR_Diversity_Based_LTMIR_1998.pdf, 1998. Accessed: 2024-03-24.
- Step back prompting: Enhancing llms with historical context retrieval. https://arxiv.org/abs/2310.06117, 2023.
- Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
- Precise zero-shot dense retrieval without relevance labels, 2022.
- Retrieval-augmented generation for large language models: A survey, 2024.
- James Calam. Ai arxiv dataset. https://huggingface.co/datasets/jamescalam/ai-arxiv, 2023. Accessed: 2024-03-24.
- Active retrieval augmented generation, 2023.
- D. Kiela. Stanford cs25: V3 i retrieval augmented language models. https://www.youtube.com/watch?v=mE7IDf2SmJg, 2024. Accessed: 2024-03-24.
- Langchain. Query transformations. https://blog.langchain.dev/query-transformations/, 2023. Accessed: 2024-03-23.
- J. Liu. A new document summary index for llm-powered qa systems. https://www.llamaindex.ai/blog/a-new-document-summary-index-for-llm-powered-qa-systems-9a32ece2f9ec, 2023a. Accessed: 2024-03-23.
- J. Liu. Using llms for retrieval and reranking. https://www.llamaindex.ai/blog/using-llms-for-retrieval-and-reranking-23cf2d3a14b6, 2023b. Accessed: 2024-03-24.
- Roberta: A robustly optimized bert pretraining approach, 2019.
- Markr.AI. Autorag: A framework for automated retrieval-augmented generation. https://github.com/Marker-Inc-Korea/AutoRAG, 2024. Accessed: 2024-03-24.
- K. Phaneendra. Deep dive into advanced rag applications in llm-based systems. https://phaneendrakn.medium.com/deep-dive-into-advanced-rag-applications-in-llm-based-systems-1ccee0473b3b, 2023. Accessed: 2024-04-01.
- Pinecone. Rerankers. https://www.pinecone.io/learn/series/rag/rerankers/, 2023. Accessed: 2024-03-24.
- Predlico. Aragog - advanced retrieval augmented generation output grading. https://github.com/predlico/ARAGOG, 2024. Accessed: 2024-03-24.
- RAGAS Documentation. Metrics. https://docs.ragas.io/en/v0.0.17/concepts/metrics/index.html, 2023. Accessed: 2024-03-24.
- Tonic AI. About rag metrics: Tonic validate rag metrics summary. https://docs.tonic.ai/validate/about-rag-metrics/tonic-validate-rag-metrics-summary, 2023. Accessed: 2024-03-24.
- S. Yang. Advanced rag 01: Small to big retrieval. https://towardsdatascience.com/advanced-rag-01-small-to-big-retrieval-172181b396d4, 2023. Accessed: 2024-03-23.