Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GRAG: Graph Retrieval-Augmented Generation (2405.16506v2)

Published 26 May 2024 in cs.LG
GRAG: Graph Retrieval-Augmented Generation

Abstract: Naive Retrieval-Augmented Generation (RAG) focuses on individual documents during retrieval and, as a result, falls short in handling networked documents which are very popular in many applications such as citation graphs, social media, and knowledge graphs. To overcome this limitation, we introduce Graph Retrieval-Augmented Generation (GRAG), which tackles the fundamental challenges in retrieving textual subgraphs and integrating the joint textual and topological information into LLMs to enhance its generation. To enable efficient textual subgraph retrieval, we propose a novel divide-and-conquer strategy that retrieves the optimal subgraph structure in linear time. To achieve graph context-aware generation, incorporate textual graphs into LLMs through two complementary views-the text view and the graph view-enabling LLMs to more effectively comprehend and utilize the graph context. Extensive experiments on graph reasoning benchmarks demonstrate that in scenarios requiring multi-hop reasoning on textual graphs, our GRAG approach significantly outperforms current state-of-the-art RAG methods.

PDF HTML Abstract

GRAG: Graph Retrieval-Augmented Generation

The paper presents "Graph Retrieval-Augmented Generation (GRAG)," a novel framework designed to improve the efficacy of Retrieval-Augmented Generation (RAG) methods when dealing with graph-based contexts. Traditional RAG approaches, while effective in enhancing the relevance and accuracy of generative LLMs, fall short in scenarios where both textual and topological information are critical. The GRAG framework addresses these limitations by introducing a mechanism to retrieve and utilize textual subgraphs, thus preserving the rich structural information essential for generating factually coherent responses.

Methodology

The GRAG framework consists of four main stages: textual subgraph retrieval, soft pruning, and generation with pruned textual subgraphs. Each of these components plays a critical role in ensuring that the generative process benefits from both the textual and topological information inherent in the graph data.

  1. Textual Subgraph Retrieval: GRAG initiates the retrieval process by indexing kk-hop ego-graphs. This indexing is accomplished using a pre-trained LLM (PLM) to convert textual attributes of nodes and edges into embeddings. The goal is to retrieve the most relevant kk-hop ego-graphs based on the similarity between the question embedding and graph embeddings. Rather than a naive entity retrieval, this method ensures that the structural context is preserved, providing a richer pool of information for generating responses.
  2. Soft Pruning: Once the relevant subgraphs are retrieved, a soft pruning mechanism is employed to reduce the impact of irrelevant entities. This is achieved by scaling node and edge attributes based on their relevance to the query. The soft pruning mechanism uses learned scaling factors to diminish the influence of non-contributing elements, thereby refining the subgraph to retain only the most pertinent information.
  3. Graph-Aware Generation: The final stage involves the generation of responses using both soft and hard prompts. GRAG introduces a novel dual prompting method where soft prompts retain topological information through graph embeddings, and hard prompts preserve textual nuances by converting the textual subgraph into hierarchical text descriptions. This structured approach ensures that the generation process benefits from the full spectrum of graph information, including both semantics and structure.

Results

Extensive experiments on multi-hop graph reasoning benchmarks such as the {\fontfamily{pcr}\selectfont WebQSP} and {\fontfamily{pcr}\selectfont Explagraphs} datasets demonstrate that GRAG significantly outperforms existing state-of-the-art RAG methods. Key performance metrics such as Hit@1, F1F_1 Score, Recall, and Accuracy were used to evaluate the efficacy of GRAG.

Key Highlights:

  • Hit@1 Performance: GRAG achieved a Hit@1 score of 0.7236 on the {\fontfamily{pcr}\selectfont WebQSP} dataset, outperforming the nearest competitor, G-Retriever, which scored 0.6808.
  • F1F_1 Score and Recall: On the same dataset, GRAG's F1F_1 Score and Recall stood at 0.5022 and 0.5099, respectively, illustrating its superior retrieval and generation capabilities.
  • Accuracy on {\fontfamily{pcr}\selectfont Explagraphs}: GRAG's accuracy was 0.9223, far surpassing the baseline figures, thus highlighting its effectiveness even in smaller, more reasoning-intensive datasets.

Implications and Future Developments

The implications of GRAG are far-reaching, both practically and theoretically. Practically, GRAG's ability to integrate both textual and topological information into the generative process significantly enhances the factual accuracy and contextual relevance of LLMs, making them more reliable for complex query-answering tasks. This can be particularly beneficial in domains such as scientific research, recommender systems, and expert systems where graph-based data structures are prevalent.

Theoretically, GRAG opens new avenues for research in the intersection of graph theory and natural language processing. Future developments could focus on further optimizing the retrieval and pruning stages, possibly through more advanced machine learning techniques or hybrid models. Additionally, exploring the scalability of GRAG to even larger graphs and its adaptability to other graph-based tasks, such as graph-based sentiment analysis or entity resolution in knowledge graphs, could lead to groundbreaking advancements in the field.

In summary, GRAG represents a significant step forward in retrieval-augmented generation, particularly for graph-based contexts. Its innovative approach to integrating textual and topological information addresses critical gaps in current RAG methods, setting a new benchmark for future research and applications in this domain.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Beyond text: A deep dive into large language models’ ability on understanding graph data. In NeurIPS 2023 Workshop: New Frontiers in Graph Learning, 2023a.
  2. Exploring the potential of large language models (llms) in learning on graphs. ACM SIGKDD Explorations Newsletter, 25(2):42–61, 2024.
  3. Talk like a graph: Encoding graphs for large language models. In The Twelfth International Conference on Learning Representations, 2023.
  4. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
  5. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, 2023.
  6. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.
  7. Realm: retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, pages 3929–3938, 2020.
  8. Yixuan Tang and Yi Yang. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. arXiv preprint arXiv:2401.15391, 2024.
  9. Harnessing explanations: Llm-to-lm interpreter for enhanced text-attributed graph representation learning. In The Twelfth International Conference on Learning Representations, 2023.
  10. Large language models on graphs: A comprehensive survey. arXiv preprint arXiv:2312.02783, 2023.
  11. A survey of graph meets large language model: Progress and future directions. arXiv preprint arXiv:2311.12399, 2023.
  12. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402.16837, 2024.
  13. Computers and intractability: A guide to the theory of NP-completeness. WH Freeman, 1979.
  14. Label-free node classification on graphs with large language models (llms). In The Twelfth International Conference on Learning Representations, 2023.
  15. Enhancing knowledge graph construction using large language models. arXiv preprint arXiv:2305.04676, 2023.
  16. Exploring large language models for knowledge graph completion. arXiv preprint arXiv:2308.13916, 2023.
  17. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36, 2024.
  18. Structgpt: A general framework for large language model to reason over structured data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9237–9251, 2023.
  19. Reasoning on graphs: Faithful and interpretable large language model reasoning. In The Twelfth International Conference on Learning Representations, 2023.
  20. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
  21. An empirical study of pre-trained language models in simple knowledge graph question answering. World Wide Web, 26(5):2855–2886, 2023b.
  22. Graph neural prompting with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19080–19088, 2024.
  23. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. arXiv preprint arXiv:2402.07630, 2024.
  24. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331, 2023.
  25. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023.
  26. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. In North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
  27. Knowledge graph-augmented language models for knowledge-grounded dialogue generation. arXiv preprint arXiv:2305.18846, 2023.
  28. Factkg: Fact verification via reasoning on knowledge graphs. In 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023, pages 16190–16206. Association for Computational Linguistics (ACL), 2023.
  29. Contextual path retrieval: A contextual entity relation embedding-based approach. ACM Transactions on Information Systems, 41(1):1–38, 2023.
  30. Complex logical reasoning over knowledge graphs using large language models. arXiv preprint arXiv:2305.01157, 2023.
  31. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130, 2024.
  32. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, 2019.
  33. Pdftriage: Question answering over long, structured documents. arXiv preprint arXiv:2309.08872, 2023.
  34. Graph attention networks. In International Conference on Learning Representations, 2018.
  35. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206, 2016.
  36. Explagraphs: An explanation graph generation task for structured commonsense reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7716–7740, 2021.
  37. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009.
  38. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, 2022.
  39. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021.
  40. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.
  41. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
  42. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  43. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022.
  44. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Yuntong Hu (12 papers)
  2. Zhihan Lei (2 papers)
  3. Zheng Zhang (486 papers)
  4. Bo Pan (31 papers)
  5. Chen Ling (65 papers)
  6. Liang Zhao (353 papers)
Citations (7)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube