MMKG: Multi-Modal Knowledge Graphs (1903.05485v1)

Abstract: We present MMKG, a collection of three knowledge graphs that contain both numerical features and (links to) images for all entities as well as entity alignments between pairs of KGs. Therefore, multi-relational link prediction and entity matching communities can benefit from this resource. We believe this data set has the potential to facilitate the development of novel multi-modal learning approaches for knowledge graphs.We validate the utility ofMMKG in the sameAs link prediction task with an extensive set of experiments. These experiments show that the task at hand benefits from learning of multiple feature types.

Insights into Multi-Modal Knowledge Graphs for Link Prediction and Entity Matching

The paper "MMKG: Multi-Modal Knowledge Graphs" outlines a notable contribution to the field of knowledge graph completion by introducing a dataset called MMKG, designed specifically to foster advances in link prediction and entity matching through the incorporation of multi-modal data. This work leverages the inherent relational structure of knowledge graphs while enriching them with numerical features and visual imagery.

Dataset Composition and Implementation

MMKG comprises three distinct knowledge graphs: FB15k, DB15k, and Yago15k, each offering numerical features and images linked to their respective entities, along with interconnected entity alignments. These knowledge graphs are derived and aligned primarily using existing benchmarks such as Freebase15k, serving as a foundational template from which entities and alignments are established. The creation of DB15k and Yago15k involves intricate techniques to ensure entity and relational alignments, followed by population with additional entities to maintain equivalency in entity counts relative to Freebase15k.

Various data modalities are embedded into MMKG to heighten the robustness of machine learning tasks. Images associated with entities are encoded via the VGG16 neural network architecture, providing a consistent and rich visual feature space conducive to learning. Numerical literals extracted and curated enhance the possibility of inferring meaningful relationships through quantitative attributes.

Relevance and Use Cases

MMKG emerges as a crucial component for evaluating link prediction models and entity matching algorithms. Traditional tasks such as Named-entity Linking (NEL) and relationship extraction are often bottlenecked by the limitations posed by single-modal data sources. MMKG's multi-modal nature allows for a more extensive exploration of relationships and representations via complementary data types. The extensive labeling and alignment between knowledge graphs proffer a rich dataset for training models capable of predicting $\mathtt{sameAs}$ links, reflecting real-world applicability in semantic alignment across disparate datasets.

Experimental Evaluation

The empirical evaluation in this paper underscores MMKG's potential to improve link prediction via multi-modal integration. The paper posits that different modalities provide complementary information crucial for predicting $\mathtt{sameAs}$ relations. This hypothesis is validated using a tested product of experts (PoE) approach combined with several learning strategies like concatenation and ensembling. Results indicate a significant increase in performance by involving numerical and visual data alongside traditional relational and latent features.

The experiments show varied performance based on the alignment percentage of entities between knowledge graphs, with significant improvements when higher alignment exists, reinforcing the dataset’s utility under varied conditions. Despite challenges posed by image dissimilarity and numerical data sparsity, the MMKG dataset offers substantial improvements in knowledge graph completion tasks, as demonstrated in the experiments utilizing FB15k, DB15k, and Yago15k.

Conclusion and Future Directions

The introduction of MMKG sets a new benchmark for multi-modal knowledge graph datasets, enabling the exploration of novel link prediction and entity matching algorithms. Future research may focus on optimizing the integration of different data modalities and exploring additional application areas, such as domain-specific knowledge extraction. As methodologies evolve, MMKG will likely inspire adaptations and extensions that incorporate even more sophisticated features, including textual data or advanced image encodings, further bridging the heterogeneity inherent in large-scale knowledge graphs.