A Review of Relational Machine Learning for Knowledge Graphs (1503.00759v3)

Abstract: Relational machine learning studies methods for the statistical analysis of relational, or graph-structured, data. In this paper, we provide a review of how such statistical models can be "trained" on large knowledge graphs, and then used to predict new facts about the world (which is equivalent to predicting new edges in the graph). In particular, we discuss two fundamentally different kinds of statistical relational models, both of which can scale to massive datasets. The first is based on latent feature models such as tensor factorization and multiway neural networks. The second is based on mining observable patterns in the graph. We also show how to combine these latent and observable models to get improved modeling power at decreased computational cost. Finally, we discuss how such statistical models of graphs can be combined with text-based information extraction methods for automatically constructing knowledge graphs from the Web. To this end, we also discuss Google's Knowledge Vault project as an example of such combination.

• The paper shows that combining latent feature models with graph feature models enhances edge prediction accuracy in large-scale knowledge graphs.
• The paper details that latent feature models such as RESCAL, ER-MLP, and NTN capture complex relational patterns through semantic embeddings.
• The paper highlights that hybrid approaches like stacking and additive relational effects effectively balance scalability and interpretability for practical KG applications.

Essay on "A Review of Relational Machine Learning for Knowledge Graphs" by Nickel, Murphy, Tresp, and Gabrilovich

This paper provides a comprehensive review of methodologies in relational machine learning (RML) as applied to large-scale knowledge graphs (KGs). The authors elucidate the principal techniques for training statistical models on extensive KGs to facilitate the prediction of new facts, essentially addressing the task of edge prediction in graph-structured data. Two main categories of statistical relational models are discussed: latent feature models and graph feature models, with a particular focus on the combination of these approaches for enhanced scalability and modeling power.

Types of Relational Learning Models

The review primarily categorizes the methodologies into three model classes:

1. Latent Feature Models (LFM): These models, including those based on tensor factorization and multiway neural networks (e.g., RESCAL, ER-MLP, NTN), leverage latent variables to capture correlations in the data.
2. Graph Feature Models: These models rely on observed graph features, implementing methods like the Path Ranking Algorithm (PRA) to extract meaningful patterns within KGs.
3. Markov Random Fields (MRF): This category represents relationships as local interactions within a probabilistic graphical model framework like Markov Logic Networks (MLNs) and PSL.

Latent Feature Models

These models assume conditional independence of the random variables representing possible edges in the KG, given certain latent features of entities and relations. The authors detail several prominent methodologies under this category, such as:

• RESCAL: This bilinear model captures pairwise interactions through matrix factorization, encoding patterns such as block structure and homophily efficiently.
• E-MLP and ER-MLP: These neural network-based models represent multiway interactions through hidden layers and feedforward architectures, thus capturing non-linearity in relational data.
• NTN (Neural Tensor Networks): They extend the bilinear models by integrating multiway neural network structures, facilitating complex interaction modeling.

The efficacy of LFMs is underscored by their semantic embedding properties and the ability to capture relational similarities within the KG through shared representations.

Graph Feature Models

Graph feature models base their predictions on observed graph patterns:

• Path Ranking Algorithm (PRA): This algorithm ranks paths within the KG and converts these paths into feature vectors, using logistic regression to predict the existence of new edges.
• Rule Mining: Utilizing ILP techniques, rules and patterns are mined from the KG, enhancing the interpretability of the relational data and supporting the reasoning behind edge predictions.

The paper highlights the interpretability benefit of PRA through its ability to present learned relation paths as weighted rules.

Combining Models

Acknowledging the complementary strengths of latent and graph feature models, the authors discuss several hybrid approaches:

• Additive Relational Effects (ARE): By combining LFMs like RESCAL with observable models such as PRA, ARE captures complex graph patterns while ensuring scalability and efficiency.
• Stacking: This method involves training separate models and combining their outputs at a fusion layer, as demonstrated in the Knowledge Vault (KV) project.

Knowledge Vault Project

The KV project exemplifies the application of SRL techniques in constructing a substantial KG by integrating information extracted from diverse web sources with prior KG knowledge. It employs both ER-MLP and PRA models for edge prediction and combines multiple sources' outputs using a fusion layer to improve the confidence in automatically extracted triples.

Implications and Future Work

This review elucidates the practical and theoretical advancements in the field of RML for KGs. The methodologies enable the efficient and accurate prediction of new facts, tackling the challenges posed by the vastness and sparsity of KGs. Practical implications include enhanced question answering systems, structured search functionalities, and digital assistants.

Looking forward, future developments in AI may involve addressing the complexity of representing and reasoning with common-sense knowledge and procedural information. Extending methodologies to handle more intricate relationships and temporal dynamics within KGs will be pivotal in advancing the semantic understanding and capabilities of AI systems.

The paper contributes significantly to the understanding of relational machine learning and its application to KGs, providing a foundational framework that can be leveraged and expanded upon in the field of AI and machine learning.