Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation (2407.10805v6)

Abstract: Retrieval-augmented generation (RAG) has improved LLMs by using knowledge retrieval to overcome knowledge deficiencies. However, current RAG methods often fall short of ensuring the depth and completeness of retrieved information, which is necessary for complex reasoning tasks. In this work, we introduce Think-on-Graph 2.0 (ToG-2), a hybrid RAG framework that iteratively retrieves information from both unstructured and structured knowledge sources in a tight-coupling manner. Specifically, ToG-2 leverages knowledge graphs (KGs) to link documents via entities, facilitating deep and knowledge-guided context retrieval. Simultaneously, it utilizes documents as entity contexts to achieve precise and efficient graph retrieval. ToG-2 alternates between graph retrieval and context retrieval to search for in-depth clues relevant to the question, enabling LLMs to generate answers. We conduct a series of well-designed experiments to highlight the following advantages of ToG-2: 1) ToG-2 tightly couples the processes of context retrieval and graph retrieval, deepening context retrieval via the KG while enabling reliable graph retrieval based on contexts; 2) it achieves deep and faithful reasoning in LLMs through an iterative knowledge retrieval process of collaboration between contexts and the KG; and 3) ToG-2 is training-free and plug-and-play compatible with various LLMs. Extensive experiments demonstrate that ToG-2 achieves overall state-of-the-art (SOTA) performance on 6 out of 7 knowledge-intensive datasets with GPT-3.5, and can elevate the performance of smaller models (e.g., LLAMA-2-13B) to the level of GPT-3.5's direct reasoning. The source code is available on https://github.com/IDEA-FinAI/ToG-2.

Think-on-Graph 2.0: Deep and Interpretable LLM Reasoning with Knowledge Graph-guided Retrieval

The paper "Think-on-Graph 2.0: Deep and Interpretable LLM Reasoning with Knowledge Graph-guided Retrieval" introduces a sophisticated enhancement to the Retrieval-Augmented Generation (RAG) paradigm for LLMs. The proposed system, referred to as Think-on-Graph 2.0 (ToG2.0), integrates unstructured document context with structured knowledge graphs (KGs) to augment the accuracy and reliability of LLMs in reasoning tasks.

Problem Statement

While RAG systems significantly improve LLMs by dynamically retrieving pertinent information to address knowledge gaps and reduce hallucinations, they often struggle with complex reasoning and maintaining consistency across varied queries. Traditional RAG relies heavily on vector retrieval methods that, while useful for capturing semantic similarities, can be inefficient for intricate reasoning tasks. These methods often fail in tasks requiring long-range associations and logical coherence, resulting in issues like noise and low information density.

Think-on-Graph 2.0 Approach

ToG2.0 addresses these challenges by using KGs as navigational tools to introduce structure and depth in the retrieval process. The KG-guided navigation allows ToG2.0 to identify deep and long-range associations necessary for upholding logical coherence and enhancing retrieval precision. This hybrid approach overcomes the limitations of both pure semantic retrieval and KGs by combining their strengths.

The methodology involves several key steps:

1. Initialization: The system begins by performing Named Entity Recognition (NER) and Topic Pruning (TP) to identify suitable starting points for reasoning within the query, thereby avoiding broad or irrelevant explorations.
2. Iterative Process: Iteration includes three core components: Relation Prune (RP), Entity Prune (EP), and Examine and Reasoning (ER). Each iteration leverages both structured and unstructured data:
   • RP utilizes LLMs to evaluate and select important relations for effective exploration.
   • EP ranks and prunes entities using a two-stage search mechanism, integrating entity context from unstructured documents.
   • ER involves the LLM examining logical coherence and factual completeness of collected references. If the LLM deems it ready to answer, the iteration ends; otherwise, new clue-queries are generated for subsequent iterations.

Experimental Results

Extensive experiments were conducted on four public datasets: WebQSP, HotpotQA, QALD-10-en, and FEVER. ToG2.0 consistently outperformed baseline methods, including Vanilla RAG, Chain-of-Thought (CoT), Chain-of-Knowledge (CoK), and the initial Think-on-Graph (ToG). The performance improvements were particularly notable in complex multi-hop reasoning tasks like HotpotQA, where ToG2.0 surpassed the state-of-the-art by 5.51%.

Key Results:

• WebQSP: Improved Exact Match (EM) score by 6.58% over the baseline.
• HotpotQA: Achieved a 14.6% improvement in performance from the initial ToG.
• QALD-10-en: Enhanced EM score by over 3%.
• FEVER: Although marginally behind CoK in accuracy, ToG2.0 demonstrated substantial gains in other testbeds.

Contributions and Future Work

The integration of structured and unstructured knowledge sources in ToG2.0 demonstrates a significant leap in enhancing LLMs' reasoning capabilities. The hybrid retrieval model not only boosts the contextual depth of retrieved information but also maintains high relevance and logical coherence in responses.

Implications:

• Practical: ToG2.0 provides a robust framework for applications requiring deep reasoning, such as complex question answering, fact verification, and tasks involving multi-granularity associations.
• Theoretical: The research underscores the importance of combining KGs with unstructured data to bridge gaps in current LLM performance, highlighting potential areas for further exploration.

Future Directions may include:

• Scalability: Enhancing efficiency for real-time applications by optimizing retrieval algorithms.
• Knowledge Source Expansion: Integrating additional knowledge sources and improving the KG construction to mitigate issues of incompleteness and ambiguity.
• Adaptive Learning: Developing adaptive mechanisms to dynamically tune the depth and breadth of retrieval based on the complexity of queries.

In conclusion, Think-on-Graph 2.0 represents a significant advancement in the field of retrieval-augmented LLM reasoning, providing a promising pathway for more interpretable, accurate, and reliable AI systems.