Filter-then-Generate: Large Language Models with Structure-Text Adapter for Knowledge Graph Completion (2412.09094v3)

Abstract: LLMs present massive inherent knowledge and superior semantic comprehension capability, which have revolutionized various tasks in natural language processing. Despite their success, a critical gap remains in enabling LLMs to perform knowledge graph completion (KGC). Empirical evidence suggests that LLMs consistently perform worse than conventional KGC approaches, even through sophisticated prompt design or tailored instruction-tuning. Fundamentally, applying LLMs on KGC introduces several critical challenges, including a vast set of entity candidates, hallucination issue of LLMs, and under-exploitation of the graph structure. To address these challenges, we propose a novel instruction-tuning-based method, namely FtG. Specifically, we present a filter-then-generate paradigm and formulate the KGC task into a multiple-choice question format. In this way, we can harness the capability of LLMs while mitigating the issue casused by hallucinations. Moreover, we devise a flexible ego-graph serialization prompt and employ a structure-text adapter to couple structure and text information in a contextualized manner. Experimental results demonstrate that FtG achieves substantial performance gain compared to existing state-of-the-art methods. The instruction dataset and code are available at https://github.com/LB0828/FtG.

• The paper introduces FtG, a filter-then-generate approach that enhances knowledge graph completion by mitigating LLM hallucinations.
• It employs an ego-graph serialization prompt and a structure-text adapter to effectively inject graph structural cues into LLM inputs.
• Experimental results show significant improvements, including a 33.2% increase in Hits@1 on benchmark datasets like FB15k-237.

Filter-then-Generate: LLMs with Structure-Text Adapter for Knowledge Graph Completion

In the paper titled "Filter-then-Generate: LLMs with Structure-Text Adapter for Knowledge Graph Completion," the authors address the limitations of LLMs in performing knowledge graph completion (KGC). Despite the advances in natural language processing facilitated by LLMs, their application in KGC has been constrained by challenges such as the vast array of entity candidates, hallucination phenomena in LLMs, and the underutilization of the graph structure inherent in knowledge graphs (KGs).

The central contribution of the paper is the introduction of a novel instruction-tuning-based method called FtG. This method includes a filter-then-generate framework which reformulates the KGC task into a multiple-choice question. In this paradigm, a conventional KGC method is first employed as a "filter" to narrow down the set of likely candidate entities. Following this, an LLM is used to "generate" the target entity from the top candidates. This approach effectively leverages the reasoning capabilities of LLMs while mitigating their tendency towards hallucination.

To further enhance the integration of LLMs with KG structures, the authors propose a unique ego-graph serialization prompt along with a structure-text adapter. The former involves the textual serialization of the 1-hop ego-graph of the query entity, aiming to better convey relational structures to the LLMs. The latter, a light-weight adapter, translates structured graph information into the text domain, thereby enriching the textual inputs with relevant structural cues.

Experimental validation across three benchmark datasets (FB15k-237, CoDEx-M, and NELL-995) demonstrates the substantial performance gains of FtG over existing methods, both structure-based and pre-trained LLM-based. Notably, FtG achieves improvements of up to 33.2% in the Hits@1 metric on the FB15k-237 dataset when compared to RotatE, the KGC method employed as the filter.

The implications of these findings are significant both practically and theoretically. Practically, FtG illustrates an effective framework for enhancing KGC by leveraging the inherent understanding and reasoning capabilities of LLMs, which is crucial given the evolving nature of real-world knowledge graphs. Theoretically, the research advances our understanding of how structured graph information can be effectively integrated into the predominantly text-based processing of LLMs, suggesting potential pathways for the development of hybrid systems that marry symbolic and sub-symbolic AI.

Looking forward, this work opens up avenues for extending the FtG paradigm to other domains that require structurally aware reasoning, such as recommendation systems and open-domain question answering. Furthermore, the structure-text adapter presents possibilities for future exploration in encoding complex graph structures in a manner interpretable by LLMs, thus broadening the application scope of these models in AI. The success of the FtG initiative signifies a pertinent shift in how LLMs can be utilized beyond traditional NLP tasks, driving the frontier of knowledge-based intelligent systems.