Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 153 tok/s
Gemini 2.5 Pro 50 tok/s Pro
GPT-5 Medium 20 tok/s Pro
GPT-5 High 28 tok/s Pro
GPT-4o 79 tok/s Pro
Kimi K2 198 tok/s Pro
GPT OSS 120B 428 tok/s Pro
Claude Sonnet 4.5 38 tok/s Pro
2000 character limit reached

Towards Foundation Models for Knowledge Graph Reasoning (2310.04562v2)

Published 6 Oct 2023 in cs.CL and cs.AI

Abstract: Foundation models in language and vision have the ability to run inference on any textual and visual inputs thanks to the transferable representations such as a vocabulary of tokens in language. Knowledge graphs (KGs) have different entity and relation vocabularies that generally do not overlap. The key challenge of designing foundation models on KGs is to learn such transferable representations that enable inference on any graph with arbitrary entity and relation vocabularies. In this work, we make a step towards such foundation models and present ULTRA, an approach for learning universal and transferable graph representations. ULTRA builds relational representations as a function conditioned on their interactions. Such a conditioning strategy allows a pre-trained ULTRA model to inductively generalize to any unseen KG with any relation vocabulary and to be fine-tuned on any graph. Conducting link prediction experiments on 57 different KGs, we find that the zero-shot inductive inference performance of a single pre-trained ULTRA model on unseen graphs of various sizes is often on par or better than strong baselines trained on specific graphs. Fine-tuning further boosts the performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Bringing light into the dark: A large-scale evaluation of knowledge graph embedding models under a unified framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. doi: 10.1109/TPAMI.2021.3124805.
  2. On the opportunities and risks of foundation models. ArXiv, 2021. URL https://crfm.stanford.edu/assets/report.pdf.
  3. Graph neural networks for link prediction with subgraph sketching. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=m1oqEOAozQU.
  4. Building a knowledge graph to enable precision medicine. Nature Scientific Data, 2023. doi: https://doi.org/10.1038/s41597-023-01960-3. URL https://www.nature.com/articles/s41597-023-01960-3.
  5. Meta relational learning for few-shot link prediction in knowledge graphs. In EMNLP, pp.  4217–4226, 2019.
  6. Generalizing to unseen elements: A survey on knowledge extrapolation for knowledge graphs. arXiv preprint arXiv:2302.01859, 2023.
  7. Relation prediction as an auxiliary training objective for improving multi-relational graph representations. In 3rd Conference on Automated Knowledge Base Construction, 2021. URL https://openreview.net/forum?id=Qa3uS3H7-Le.
  8. Inductive entity representations from text via link prediction. In Proceedings of the Web Conference 2021, pp.  798–808, 2021.
  9. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  10. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.  4171–4186. Association for Computational Linguistics, 2019. URL https://aclanthology.org/N19-1423.
  11. Improving knowledge graph embedding using simple constraints. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  110–121. Association for Computational Linguistics, 2018. doi: 10.18653/v1/P18-1011. URL https://aclanthology.org/P18-1011.
  12. Xin Luna Dong. Challenges and innovations in building a product knowledge graph. In Proceedings of the 24th ACM SIGKDD International conference on knowledge discovery & data mining, pp.  2869–2869, 2018.
  13. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
  14. An Open Challenge for Inductive Link Prediction on Knowledge Graphs. arXiv preprint arXiv:2203.01520, 2022a. URL http://arxiv.org/abs/2203.01520.
  15. Nodepiece: Compositional and parameter-efficient representations of large knowledge graphs. In International Conference on Learning Representations, 2022b. URL https://openreview.net/forum?id=xMJWUKJnFSw.
  16. Inductive logical query answering in knowledge graphs. In Advances in Neural Information Processing Systems, 2022c. URL https://openreview.net/forum?id=-vXEN5rIABY.
  17. Double permutation equivariance for knowledge graph completion. arXiv preprint arXiv:2302.01313, 2023.
  18. Relational message passing for fully inductive knowledge graph completion. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp.  1221–1233. IEEE, 2023.
  19. Raild: Towards leveraging relation features for inductive link prediction in knowledge graphs. In Proceedings of the 11th International Joint Conference on Knowledge Graphs, pp.  82–90, 2022.
  20. BiQUE: Biquaternionic embeddings of knowledge graphs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.  8338–8351. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.emnlp-main.657. URL https://aclanthology.org/2021.emnlp-main.657.
  21. Knowledge transfer for out-of-knowledge-base entities : A graph neural network approach. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp.  1802–1808, 2017. doi: 10.24963/ijcai.2017/250. URL https://doi.org/10.24963/ijcai.2017/250.
  22. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  770–778, 2016.
  23. Exploring & exploiting high-order graph structure for sparse knowledge graph completion. arXiv preprint arXiv:2306.17034, 2023.
  24. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. Elife, 6:e26726, 2017.
  25. Few-shot relational reasoning via connection subgraph pretraining. In Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=LvW71lgly25.
  26. A theory of link prediction via relational weisfeiler-leman. arXiv preprint arXiv:2302.02209, 2023.
  27. Saga: A platform for continuous construction and serving of knowledge at scale. In Proceedings of the 2022 International Conference on Management of Data, pp.  2259–2272, 2022.
  28. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=SJU4ayYgl.
  29. InGram: Inductive knowledge graph embedding via relation graphs. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp.  18796–18809. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/lee23c.html.
  30. Distance encoding: Design provably more powerful neural networks for graph representation learning. Advances in Neural Information Processing Systems, 33:4465–4478, 2020.
  31. Indigo: Gnn-based inductive knowledge graph completion using pair-wise encoding. In Advances in Neural Information Processing Systems, volume 34, pp.  2034–2045. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/0fd600c953cde8121262e322ef09f70e-Paper.pdf.
  32. Dynamic anticipation and completion for multi-hop reasoning over sparse knowledge graph. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.  5694–5703. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-main.459. URL https://aclanthology.org/2020.emnlp-main.459.
  33. Yago3: A knowledge base from multilingual wikipedias. In 7th biennial conference on innovative data systems research. CIDR Conference, 2014.
  34. Commonsense knowledge base completion with structural and semantic context. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.  2925–2933, 2020.
  35. StATIK: Structure and text for inductive knowledge graph completion. In Findings of the Association for Computational Linguistics: NAACL 2022, pp.  604–615, 2022. doi: 10.18653/v1/2022.findings-naacl.46. URL https://aclanthology.org/2022.findings-naacl.46.
  36. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  37. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, pp.  8748–8763. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/radford21a.html.
  38. CoDEx: A Comprehensive Knowledge Graph Completion Benchmark. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.  8328–8350, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.669. URL https://www.aclweb.org/anthology/2020.emnlp-main.669.
  39. The materials experiment knowledge graph. Digital Discovery, 2:909–914, 2023. doi: 10.1039/D3DD00067B. URL http://dx.doi.org/10.1039/D3DD00067B.
  40. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HkgEQnRqYQ.
  41. Inductive relation prediction by subgraph reasoning. In International Conference on Machine Learning, pp.  9448–9457. PMLR, 2020.
  42. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality, pp.  57–66, 2015.
  43. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  44. Composition-based multi-relational graph convolutional networks. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BylA_C4tPr.
  45. Graph attention networks. In International Conference on Learning Representations, 2018.
  46. Matkg: The largest knowledge graph in materials science – entities, relations, and link prediction through graph representation learning, 2022.
  47. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176–194, 2021. doi: 10.1162/tacl_a_00360.
  48. DeepPath: A reinforcement learning method for knowledge graph reasoning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp.  564–573. Association for Computational Linguistics, 2017. doi: 10.18653/v1/D17-1060. URL https://aclanthology.org/D17-1060.
  49. Embedding entities and relations for learning and inference in knowledge bases. International Conference on Learning Representations, 2015.
  50. Do transformers really perform badly for graph representation? In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=OeWooOxFwDa.
  51. Few-shot knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, pp.  3041–3048, 2020.
  52. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
  53. Labeling trick: A theory of using graph neural networks for multi-node representation learning. In Advances in Neural Information Processing Systems, volume 34, pp.  9061–9073. Curran Associates, Inc., 2021.
  54. Knowledge graph reasoning with relational digraph. In Proceedings of the ACM Web Conference 2022, pp.  912–924, 2022.
  55. Adaprop: Learning adaptive propagation for graph neural network based knowledge graph reasoning. In KDD, 2023.
  56. Towards predicting equilibrium distributions for molecular systems with deep learning. arXiv preprint arXiv:2306.05445, 2023.
  57. An ood multi-task perspective for link prediction with new relation types and nodes. arXiv preprint arXiv:2307.06046, 2023.
  58. Neural bellman-ford networks: A general graph neural network framework for link prediction. Advances in Neural Information Processing Systems, 34:29476–29490, 2021.
  59. Neural-symbolic models for logical queries on knowledge graphs. In International Conference on Machine Learning, ICML 2022, Proceedings of Machine Learning Research. PMLR, 2022a.
  60. Torchdrug: A powerful and flexible machine learning platform for drug discovery. arXiv preprint arXiv:2202.08320, 2022b.
  61. A*net: A scalable path-based reasoning approach for knowledge graphs. arXiv preprint arXiv:2206.04798, 2022c.
Citations (29)
View on Semantic Scholar

Summary

  • The paper presents Ultra, a novel method that learns transferable graph representations to enable zero-shot inference across diverse knowledge graphs.
  • Ultra employs graph neural networks to capture relational patterns, significantly enhancing link prediction performance on 57 varied KGs.
  • Experimental results demonstrate Ultra's scalability and superiority over baseline models, paving the way for adaptable KG reasoning systems.

Towards Foundation Models for Knowledge Graph Reasoning: Introducing Ultra

Introduction

The pre-training and fine-tuning paradigm has become a cornerstone in the field of machine learning, particularly with the advent of foundation models (FMs) that have shown remarkable transferability across a wide range of tasks in natural language processing, computer vision, and more. However, the application of this paradigm to knowledge graphs (KGs) has been limited. Knowledge graphs, with their distinct entity and relation vocabularies, pose a unique challenge for achieving transfer learning due to the lack of generalizable representations that can easily adapt to unseen graphs. Addressing this gap, we introduce Ultra, a novel approach designed to learn universal and transferable graph representations, aiming to facilitate reasoning over any KG regardless of its entity or relation vocabularies.

Approach

Ultra is built upon two pivotal observations: the potential for relational interactions to exhibit transferable patterns across different knowledge graphs, and the feasibility of conditioning initial relation representations on these interactions. This strategy allows Ultra to generalize inductively to unseen KGs, enabling both zero-shot inference and fine-tuning capabilities on new graphs. In essence, Ultra generates a relation graph capturing the interactions between relations in the original KG, leverages a graph neural network (GNN) to attain unique relative representations for each relation, and employs these representations for downstream tasks like KG completion.

Experimental Evaluation

Our empirical evaluation spans 57 diverse KGs, demonstrating that Ultra, even in a zero-shot setting, can achieve competitive or superior performance against strong baselines specifically trained on those KGs. Particularly noteworthy is Ultra's performance on link prediction tasks across various graph sizes, where it frequently outperforms baselines by significant margins. These findings underscore the efficacy of Ultra’s transfer learning capabilities, showcasing its potential as a foundation model for KG reasoning.

Implications and Future Work

The results presented carry profound implications for the future of KG inference systems. By establishing a method for learning transferable representations across KGs, we pave the way for more efficient and scalable models capable of adapting to a vast array of graphs without the need for graph-specific training. This advancement could significantly impact fields relying on KGs, such as precision medicine, materials science, and e-commerce, by streamlining the development of AI systems tailored to these domains.

Future research will explore enhancing the scalability and performance of Ultra, particularly in handling larger and more complex graphs. Exploring the integration of other forms of knowledge, such as textual or visual data, into Ultra's framework may also yield further improvements in its generalizability and efficacy across a broader spectrum of KGs.

In conclusion, Ultra represents a significant step toward the realization of foundation models for KG reasoning, inaugurating a new chapter in the pursuit of transferable and scalable AI systems capable of navigating the intricate landscape of knowledge graphs.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 1 tweet and received 111 likes.

Upgrade to Pro to view all of the tweets about this paper:

Start a free 7-day Pro trial