Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GenTKG: Generative Forecasting on Temporal Knowledge Graph with Large Language Models (2310.07793v5)

Published 11 Oct 2023 in cs.CL, cs.AI, and cs.LG

Abstract: The rapid advancements in LLMs have ignited interest in the temporal knowledge graph (tKG) domain, where conventional embedding-based and rule-based methods dominate. The question remains open of whether pre-trained LLMs can understand structured temporal relational data and replace them as the foundation model for temporal relational forecasting. Therefore, we bring temporal knowledge forecasting into the generative setting. However, challenges occur in the huge chasms between complex temporal graph data structure and sequential natural expressions LLMs can handle, and between the enormous data sizes of tKGs and heavy computation costs of finetuning LLMs. To address these challenges, we propose a novel retrieval-augmented generation framework named GenTKG combining a temporal logical rule-based retrieval strategy and few-shot parameter-efficient instruction tuning to solve the above challenges, respectively. Extensive experiments have shown that GenTKG outperforms conventional methods of temporal relational forecasting with low computation resources using extremely limited training data as few as 16 samples. GenTKG also highlights remarkable cross-domain generalizability with outperforming performance on unseen datasets without re-training, and in-domain generalizability regardless of time split in the same dataset. Our work reveals the huge potential of LLMs in the tKG domain and opens a new frontier for generative forecasting on tKGs. Code and data are released here: https://github.com/mayhugotong/GenTKG.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 78–106, Toronto, Canada. Association for Computational Linguistics.
  2. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models.
  3. Icews coded event data. Harvard Dataverse, 12.
  4. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  5. Few-shot inductive learning on temporal knowledge graphs using concept-aware information. arXiv preprint arXiv:2211.08169.
  6. A survey for in-context learning. arXiv preprint arXiv:2301.00234.
  7. Towards foundation models for knowledge graph reasoning. arXiv preprint arXiv:2310.04562.
  8. Comparing apples and oranges? on the evaluation of methods for temporal knowledge graph forecasting. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 533–549. Springer.
  9. Diachronic embedding for temporal knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3988–3995.
  10. Explainable subgraph reasoning for forecasting on temporal knowledge graphs. In International Conference on Learning Representations.
  11. Learning neural ordinary equations for forecasting future links on temporal knowledge graphs. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 8352–8364.
  12. Enhanced temporal knowledge embeddings with contextualized language representations.
  13. Dyernie: Dynamic evolution of riemannian manifold embeddings for temporal knowledge graph completion. arXiv preprint arXiv:2011.03984.
  14. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.
  15. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  16. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
  17. Temporal knowledge graph forecasting without knowledge using in-context learning. arXiv preprint arXiv:2305.10613.
  18. Kalev Leetaru and Philip A Schrodt. 2013. Gdelt: Global data on events, location, and tone, 1979–2012. In ISA annual convention, volume 2, pages 1–49. Citeseer.
  19. A condensed transition graph framework for zero-shot link prediction with large language models. arXiv preprint arXiv:2402.10779.
  20. Temporal knowledge graph reasoning based on evolutional representation learning.
  21. Tlogic: Temporal logical rules for explainable link forecasting on temporal knowledge graphs. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 4120–4127.
  22. Yago3: A knowledge base from multilingual wikipedias. In CIDR.
  23. Deep learning, reinforcement learning, and world models. Neural Networks, 152:267–275.
  24. Augmented language models: a survey. arXiv preprint arXiv:2302.07842.
  25. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11–33.
  26. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  27. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.
  28. Dialokg: Knowledge-structure aware task-oriented dialogue generation. arXiv preprint arXiv:2204.09149.
  29. Timetraveler: Reinforcement learning for temporal knowledge graph forecasting. arXiv preprint arXiv:2109.04101.
  30. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.07697.
  31. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  32. Learning with memory embeddings. arXiv preprint arXiv:1511.07972.
  33. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  34. Re-gcn: relation enhanced graph convolutional network for entity alignment in heterogeneous knowledge graphs. In Database Systems for Advanced Applications: 25th International Conference, DASFAA 2020, Jeju, South Korea, September 24–27, 2020, Proceedings, Part II 25, pages 432–447. Springer.
  35. Drlk: dynamic hierarchical reasoning with language model and knowledge graph for question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5123–5133.
  36. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Ruotong Liao (8 papers)
  2. Xu Jia (57 papers)
  3. Yunpu Ma (57 papers)
  4. Yangzhe Li (1 paper)
  5. Volker Tresp (158 papers)
Citations (16)
View on Semantic Scholar