KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation (2409.13731v3)

Abstract: The recently developed retrieval-augmented generation (RAG) technology has enabled the efficient construction of domain-specific applications. However, it also has limitations, including the gap between vector similarity and the relevance of knowledge reasoning, as well as insensitivity to knowledge logic, such as numerical values, temporal relations, expert rules, and others, which hinder the effectiveness of professional knowledge services. In this work, we introduce a professional domain knowledge service framework called Knowledge Augmented Generation (KAG). KAG is designed to address the aforementioned challenges with the motivation of making full use of the advantages of knowledge graph(KG) and vector retrieval, and to improve generation and reasoning performance by bidirectionally enhancing LLMs and KGs through five key aspects: (1) LLM-friendly knowledge representation, (2) mutual-indexing between knowledge graphs and original chunks, (3) logical-form-guided hybrid reasoning engine, (4) knowledge alignment with semantic reasoning, and (5) model capability enhancement for KAG. We compared KAG with existing RAG methods in multihop question answering and found that it significantly outperforms state-of-theart methods, achieving a relative improvement of 19.6% on 2wiki and 33.5% on hotpotQA in terms of F1 score. We have successfully applied KAG to two professional knowledge Q&A tasks of Ant Group, including E-Government Q&A and E-Health Q&A, achieving significant improvement in professionalism compared to RAG methods.

Knowledge Augmented Generation: Enhancing LLMs for Professional Domain Applications

The paper "KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation" introduces an innovative framework designed to address specific challenges associated with the integration of LLMs in domain-specific applications. This framework, referred to as Knowledge Augmented Generation (KAG), emphasizes the combined utilization of Knowledge Graphs (KGs) and vector retrieval techniques to enhance generation and reasoning tasks.

Recent advancements in Retrieval-Augmented Generation (RAG) have allowed LLMs to access domain-specific knowledge via external systems, thereby reducing the likelihood of generating inaccurate or irrelevant answers. However, RAG systems face limitations regarding coherent and logical content generation, particularly in fields requiring rigorous analytical reasoning, such as law and medicine. The paper identifies the primary reasons for these shortcomings, including the reliance on vector similarity for retrievals and a general insensitivity to logical reasoning and knowledge structure, which KAG aims to improve upon.

The KAG framework addresses these limitations through a series of innovations aimed at enhancing the symbiotic relationship between LLMs and KGs:

1. LLM-Friendly Knowledge Representation: KAG introduces LLMFriSPG, a hierarchical data representation model inspired by the DIKW pyramid. It facilitates schema-free information extraction while supporting schema-constrained expert knowledge construction, thus improving the symbiosis between structured knowledge and unstructured data.
2. Mutual Indexing: By establishing a dual index that bridges knowledge graph structures and original text chunks, KAG enables a comprehensive information retrieval process that supports both structured and unstructured data queries.
3. Logical-Form-Guided Hybrid Reasoning Engine: The framework combines various operators, such as planning, reasoning, and retrieval, to deconstruct natural language queries into problem-solving sequences. This approach allows for multimodal problem-solving, encompassing retrieval-based, KG-based, language-based, and numerical reasoning techniques.
4. Knowledge Alignment through Semantic Reasoning: By defining and leveraging semantic relationships like synonyms and hyponyms, KAG enhances the standardization and connectivity of various knowledge components, resulting in more accurate and logical KGs.
5. Model Capability Enhancement: To support multi-faceted tasks such as indexing, retrieval, and reasoning, the KAG framework builds on existing LLM capabilities, enhancing Natural Language Understanding (NLU), Natural Language Inference (NLI), and Natural Language Generation (NLG).

The empirical evaluation of KAG utilized three complex Q&A datasets: HotpotQA, 2WikiMultiHopQA, and MuSiQue. The framework demonstrated significant enhancement in performance over existing RAG methods, notably achieving F1 score improvements of 19.6%, 33.5%, and notable gains in retrieval accuracy metrics. Furthermore, KAG's application in Ant Group's E-Government and E-Health Q&A systems has shown a marked increase in accuracy over traditional RAG methods, signifying its potential to advance professional applications in a variety of critical domains.

A noteworthy implication of KAG is its provision of an architecture that not only addresses LLMs' limitations in domain-specific contexts but also facilitates the efficient development of localized knowledge services. This integration of KGs with enhanced LLMs paves the way for future developments in AI, particularly in crafting domain-specialized intelligence systems that require both expansive knowledge retrieval and precise reasoning capabilities. While promising, the framework also highlights areas for continued research, such as the optimization of multi-step problem-solving and the alignment of knowledge extraction with professional standards. These areas hold potential pathways for further enhancing the precision and efficiency of AI systems in domain-specific applications.