Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Survey on Embedding Models for Knowledge Graph and its Applications (2404.09167v1)

Published 14 Apr 2024 in cs.SI and cs.AI

Abstract: Knowledge Graph (KG) is a graph based data structure to represent facts of the world where nodes represent real world entities or abstract concept and edges represent relation between the entities. Graph as representation for knowledge has several drawbacks like data sparsity, computational complexity and manual feature engineering. Knowledge Graph embedding tackles the drawback by representing entities and relation in low dimensional vector space by capturing the semantic relation between them. There are different KG embedding models. Here, we discuss translation based and neural network based embedding models which differ based on semantic property, scoring function and architecture they use. Further, we discuss application of KG in some domains that use deep learning models and leverage social media data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor, “Freebase: A collaboratively created graph database for structuring human knowledge,” in Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, (New York, NY, USA), p. 1247–1250, Association for Computing Machinery, 2008.
  2. C. Bizer, J. Lehmann, G. Kobilarov, S. Auer, C. Becker, R. Cyganiak, and S. Hellmann, “Dbpedia-a crystallization point for the web of data,” Journal of web semantics, vol. 7, no. 3, pp. 154–165, 2009.
  3. “Wikidata:introduction.” https://www.wikidata.org/wiki/Wikidata:Introduction. Accessed: 2022-07-21.
  4. F. M. Suchanek, G. Kasneci, and G. Weikum, “Yago: A large ontology from wikipedia and wordnet,” Journal of Web Semantics, vol. 6, no. 3, pp. 203–217, 2008.
  5. G. E. H. David E. Rumelhart and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, pp. 533–536, 10 1986.
  6. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, p. 1735–1780, nov 1997.
  7. K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder–decoder approaches,” in Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, (Doha, Qatar), pp. 103–111, Association for Computational Linguistics, Oct. 2014.
  8. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural computation, vol. 1, no. 4, pp. 541–551, 1989.
  9. MIT press, 2016.
  10. Y. Dai, S. Wang, N. N. Xiong, and W. Guo, “A survey on knowledge graph embedding: Approaches, applications and benchmarks,” Electronics, vol. 9, no. 5, p. 750, 2020.
  11. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in neural information processing systems, vol. 26, 2013.
  12. A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko, “Translating embeddings for modeling multi-relational data,” Advances in neural information processing systems, vol. 26, 2013.
  13. Q. Wang, Z. Mao, B. Wang, and L. Guo, “Knowledge graph embedding: A survey of approaches and applications,” IEEE Transactions on Knowledge and Data Engineering, vol. 29, no. 12, pp. 2724–2743, 2017.
  14. J. Leskovec, “Heterogeneous graphs and knowledge graph embeddings.” 2022.
  15. Y. Lin, Z. Liu, M. Sun, Y. Liu, and X. Zhu, “Learning entity and relation embeddings for knowledge graph completion,” in Twenty-ninth AAAI conference on artificial intelligence, 2015.
  16. B. Yang, S. W.-t. Yih, X. He, J. Gao, and L. Deng, “Embedding entities and relations for learning and inference in knowledge bases,” in Proceedings of the International Conference on Learning Representations (ICLR) 2015, 2015.
  17. T. Trouillon, J. Welbl, S. Riedel, E. Gaussier, and G. Bouchard, “Complex embeddings for simple link prediction,” in Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, p. 2071–2080, JMLR.org, 2016.
  18. A. Bordes, X. Glorot, J. Weston, and Y. Bengio, “A semantic matching energy function for learning with multi-relational data,” Mach. Learn., vol. 94, pp. 233–259, Feb. 2014.
  19. X. Dong, E. Gabrilovich, G. Heitz, W. Horn, N. Lao, K. Murphy, T. Strohmann, S. Sun, and W. Zhang, “Knowledge vault: A web-scale approach to probabilistic knowledge fusion,” in Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, (New York, NY, USA), p. 601–610, Association for Computing Machinery, 2014.
  20. T. D. N. Dai Quoc Nguyen, D. Q. Nguyen, and D. Phung, “A novel embedding model for knowledge base completion based on convolutional neural network,” in Proceedings of NAACL-HLT, pp. 327–333, 2018.
  21. R. Socher, D. Chen, C. D. Manning, and A. Y. Ng, “Reasoning with neural tensor networks for knowledge base completion,” in Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1, NIPS’13, (Red Hook, NY, USA), p. 926–934, Curran Associates Inc., 2013.
  22. Q. Liu, H. Jiang, A. Evdokimov, Z.-H. Ling, X. Zhu, S. Wei, and Y. Hu, “Probabilistic reasoning via deep learning: Neural association models,” arXiv preprint arXiv:1603.07704, 2016.
  23. L. Cai and W. Y. Wang, “KBGAN: Adversarial learning for knowledge graph embeddings,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), (New Orleans, Louisiana), pp. 1470–1480, Association for Computational Linguistics, June 2018.
  24. M. Schlichtkrull, T. N. Kipf, P. Bloem, R. v. d. Berg, I. Titov, and M. Welling, “Modeling relational data with graph convolutional networks,” in European semantic web conference, pp. 593–607, Springer, 2018.
  25. H. Allcott and M. Gentzkow, “Social media and fake news in the 2016 election,” Journal of Economic Perspectives, vol. 31, pp. 211–36, May 2017.
  26. J. Z. Pan, S. Pavlova, C. Li, N. Li, Y. Li, and J. Liu, “Content Based Fake News Detection Using Knowledge Graphs,” in The Semantic Web – ISWC 2018, vol. 11136, pp. 669–683, Cham: Springer International Publishing, 2018. Series Title: Lecture Notes in Computer Science.
  27. P. Shiralkar, A. Flammini, F. Menczer, and G. L. Ciampaglia, “Finding streams in knowledge graphs to support fact checking,” in 2017 IEEE International Conference on Data Mining (ICDM), pp. 859–864, 2017.
  28. J. Ma, W. Gao, P. Mitra, S. Kwon, B. Jansen, K. Wong, and M. Cha, “Detecting rumors from microblogs with recurrent neural networks,” IJCAI International Joint Conference on Artificial Intelligence, vol. 2016-January, pp. 3818–3824, 2016. 25th International Joint Conference on Artificial Intelligence, IJCAI 2016 ; Conference date: 09-07-2016 Through 15-07-2016.
  29. J. Tassone, P. Yan, M. Simpson, C. Mendhe, V. Mago, and S. Choudhury, “Utilizing deep learning and graph mining to identify drug use on twitter data,” BMC Medical Informatics and Decision Making, vol. 20, no. 11, pp. 1–15, 2020.
  30. Y. Qian, Y. Zhang, Y. Ye, and C. Zhang, “Distilling meta knowledge on heterogeneous graph for illicit drug trafficker detection on social media,” Advances in Neural Information Processing Systems, vol. 34, pp. 26911–26923, 2021.
  31. M. Gaur, A. Alambo, J. P. Sain, U. Kursuncu, K. Thirunarayan, R. Kavuluru, A. Sheth, R. Welton, and J. Pathak, “Knowledge-aware assessment of severity of suicide risk for early intervention,” in The World Wide Web Conference, WWW ’19, (New York, NY, USA), p. 514–525, Association for Computing Machinery, 2019.
  32. L. Cao, H. Zhang, and L. Feng, “Building and using personal knowledge graph to improve suicidal ideation detection on social media,” IEEE Transactions on Multimedia, vol. 24, pp. 87–102, 2022.
  33. H. Hazimeh, E. Mugellini, S. Ruffieux, O. A. Khaled, and P. Cudré-Mauroux, “Automatic embedding of social network profile links into knowledge graphs,” in Proceedings of the Ninth International Symposium on Information and Communication Technology, SoICT 2018, (New York, NY, USA), p. 16–23, Association for Computing Machinery, 2018.

Summary

  • The paper presents a comprehensive review of neural embedding models for knowledge graphs, highlighting translation-based and layered neural network approaches.
  • It details how deep learning techniques like RNNs, LSTMs, and CNNs enhance feature extraction and address scalability challenges in KG systems.
  • The study demonstrates practical applications of these techniques in areas such as fake news detection, drug discovery, and mental health monitoring.

Exploring Knowledge Graph Embeddings: Applications and Innovations in Neural Approaches

Introduction

The paper explores the fundamental structures and applications of Knowledge Graphs (KGs) and Knowledge Bases (KBs), defining them as structured representations of real-world facts. KGs in particular are highlighted as heterogeneous directed graphs consisting of nodes (entities) and edges (relationships), amenable to dynamic schema extensions and various graph query operations. The major focus is on addressing the classic challenges of KGs, including computational inefficiencies, data sparsity, and intricate feature engineering demands, through innovative Knowledge Graph Embedding (KGE) techniques.

Large-Scale Knowledge Graphs

The document outlines several influential KGs such as Freebase, DBpedia, Wikidata, and YAGO, each serving as a foundational structure for various applications across semantic web, data integration, and AI reasoning platforms. It points out their specific architectural constructs and the scope of their informational content, emphasizing their particular benefits and ubiquity in research and practical applications.

Deep Learning Models for KG

An extensive discussion is provided on various deep learning models that pertain to handling structured graph data. These include:

  • Recurrent Neural Networks (RNNs), providing sequential data processing.
  • Long Short-Term Memory networks (LSTMs) and Gated Recurrent Units (GRUs), with mechanisms to address the vanishing gradient problem typical of standard RNNs.
  • Convolutional Neural Networks (CNNs), primarily utilized for grid-like data structures (e.g., images) and distinguished by their use of convolution operations for feature extraction.

Knowledge Graph Embedding Techniques

The paper explores the transition from traditional KG representations to embedding techniques that place entities and relationships within a low-dimensional, continuous vector space. This method significantly alleviates issues of scalability, sparsity, and manual feature engineering. It details various embedding models, including:

  • Translation-based models like TranE, which conceptualizes relationships as translations in vector space.
  • Neural Network-based models such as SME and MLP, which utilize layered architectures to derive embeddings.

These models are described in terms of their architecture, operational mechanisms, and the specific types of relational data they best accommodate.

Practical Applications of KG Embeddings

Giving a clear perspective on the utilization of KG embeddings, the paper outlines several applications:

  • Fake News Detection: Leveraging KGs to assess the veracity of news by fact-checking against established KGs.
  • Drug Discovery and Social Media Monitoring: Utilizing embeddings to detect social media content related to illegal drug activity.
  • Mental Health Applications: Deploying KGs to identify and support mental health issues through social media data analysis.

Conclusion and Future Directions

The summary underscores the enhancement of KG and KB systems through embeddings, which render these systems more efficient and capable of managing the complex, real-world data. It suggests that future explorations could consider extending these embedding techniques to broader domains such as propaganda detection, an understanding of misinformation spread, and other socially impactful applications.

The results and frameworks discussed provide a comprehensive insight into modern KG handling approaches, bridging traditional graph structures with advanced machine learning techniques to facilitate better decision-making and knowledge discovery in various domains.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets