Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning (1711.05851v2)

Abstract: Knowledge bases (KB), both automatically and manually constructed, are often incomplete --- many valid facts can be inferred from the KB by synthesizing existing information. A popular approach to KB completion is to infer new relations by combinatory reasoning over the information found along other paths connecting a pair of entities. Given the enormous size of KBs and the exponential number of paths, previous path-based models have considered only the problem of predicting a missing relation given two entities or evaluating the truth of a proposed triple. Additionally, these methods have traditionally used random paths between fixed entity pairs or more recently learned to pick paths between them. We propose a new algorithm MINERVA, which addresses the much more difficult and practical task of answering questions where the relation is known, but only one entity. Since random walks are impractical in a setting with combinatorially many destinations from a start node, we present a neural reinforcement learning approach which learns how to navigate the graph conditioned on the input query to find predictive paths. Empirically, this approach obtains state-of-the-art results on several datasets, significantly outperforming prior methods.

Efficient Path-Based Query Answering in Knowledge Graphs Using Reinforcement Learning

The paper, "Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning," introduces an innovative approach to query answering in knowledge bases (KBs), focusing on the task of inferring unknown entities in a query where the relation and one entity are provided. This approach leverages reinforcement learning (RL) to navigate the knowledge graph efficiently, bypassing the need for precomputed paths which has been a limitation in many prior methods.

Overview

In the landscape of knowledge base completion, traditional methods often rely on either embedding-based techniques, which excel at capturing relational semantics but struggle with longer reasoning chains, or path-based methods, which can address multi-hop reasoning but require fixed entity pairs for path computation. The authors address these challenges by proposing Minerva, a path-based RL approach that operates purely based on observed evidence in the graph to answer queries of the form (entity_1, relation, ?).

Key Contributions

1. Reinforcement Learning Framework: The paper presents a novel MDP for knowledge graph traversal where the state comprises the current entity, the starting entity, the query relation, and the hidden target entity. Actions correspond to selecting an outgoing edge at each entity node, and the agent is trained using policy gradients to maximize the likelihood of reaching the correct answer entity.
2. Query Sensitivity and Path Flexibility: The model is notable for its ability to condition path selection on the query relation, dynamically discovering the necessary path lengths. This capability allows Minerva to outperform previous methods that rely on precomputed paths or heuristic-driven searches.
3. Competency on Diverse Datasets: Empirical evaluations across multiple standard datasets demonstrate Minerva's competitive performance against state-of-the-art methods like Neural Theorem Provers, ComplEx, and Neural LP, particularly on challenging query answering tasks that require complex multi-hop reasoning.

Numerical Results and Implications

The paper presents extensive experimental results on small and large KB datasets, including countries, umls, kinship, wn18rr, nell-995, and fb15k-237. Minerva consistently shows competitive accuracy, managing complex inference sequences more robustly than both embedding-based and multi-hop models. One noteworthy observation is its superior performance in tasks requiring longer reasoning paths, highlighting its strength in capturing intricate relational dependencies within KBs.

Practical and Theoretical Implications

Practically, the adaptive path exploration empowered by RL in Minerva can lead to significant efficiency improvements, particularly in large-scale KBs where exhaustive ranking of all entities by traditional embeddings is computationally prohibitive. Theoretically, this method paves the way towards more generalizable reasoning strategies in AI, where discrete decision-making processes emulate human-like path reasoning across vast knowledge spaces.

Future Directions

The promising results invoke several intriguing avenues for future exploration, such as extending RL-based path reasoning to handle unstructured natural language queries more seamlessly, thereby narrowing the gap between human question-answering and automated reasoning in KBs. The integration with pretrained LLMs could further enhance its applicability to domains with sparse, incomplete, and unstructured data.

In summary, the paper presents a substantial forward step in knowledge base completion, offering a robust framework for query-answering tasks centered around graph-based reasoning. By effectively utilizing the neural reinforcement learning paradigm, it demonstrates a powerful mechanism for uncovering latent paths and reaching accurate predictions, thereby enriching the potential capabilities of AI systems navigating complex informational domains.