Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking (2305.15066v2)

Published 24 May 2023 in cs.AI and cs.CL

Abstract: LLMs~(LLM) like ChatGPT have become indispensable to artificial general intelligence~(AGI), demonstrating excellent performance in various natural language processing tasks. In the real world, graph data is ubiquitous and an essential part of AGI and prevails in domains like social network analysis, bioinformatics and recommender systems. The training corpus of LLMs often includes some algorithmic components, which allows them to achieve certain effects on some graph data-related problems. However, there is still little research on their performance on a broader range of graph-structured data. In this study, we conduct an extensive investigation to assess the proficiency of LLMs in comprehending graph data, employing a diverse range of structural and semantic-related tasks. Our analysis encompasses 10 distinct tasks that evaluate the LLMs' capabilities in graph understanding. Through our study, we not only uncover the current limitations of LLMs in comprehending graph structures and performing associated reasoning tasks but also emphasize the necessity for further advancements and novel approaches to enhance their graph processing capabilities. Our findings contribute valuable insights towards bridging the gap between LLMs and graph understanding, paving the way for more effective graph mining and knowledge extraction.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Marc Barthelemy. 2004. Betweenness centrality in large complex networks. The European physical journal B, 38(2):163–168.
  2. Bioinformatics. John Wiley & Sons.
  3. Node classification in social networks. arXiv preprint arXiv:1101.3291.
  4. Richard J Bolton and David J Hand. 2002. Statistical fraud detection: A review. Statistical science, 17(3):235–255.
  5. Ulrik Brandes. 2001. A faster algorithm for betweenness centrality. Journal of mathematical sociology, 25(2):163–177.
  6. Graph markup language (graphml).
  7. Towards table-to-text generation with pretrained language model: A table structure understanding and text deliberating approach. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8199–8210, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  8. An upper bound on the diameter of a graph from eigenvalues associated with its laplacian. SIAM Journal on Discrete Mathematics, 7(3):443–457.
  9. Mate: multi-view attention for table transformer efficiency. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing".
  10. A fair comparison of graph neural networks for graph classification. arXiv preprint arXiv:1912.09893.
  11. Tablegpt: Few-shot table-to-text generation with table structure reconstruction and content matching. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1978–1988.
  12. Michael Himsolt. 1997. Gml: Graph modelling language. University of Passau.
  13. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118–22133.
  14. Knowledge graph embedding based question answering. In Proceedings of the twelfth ACM international conference on web search and data mining, pages 105–113.
  15. Recommendation systems: Principles, methods and evaluation. Egyptian informatics journal, 16(3):261–273.
  16. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494–514.
  17. Structgpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645.
  18. Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.
  19. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
  20. PLOG: Table-to-logic pretraining for logical table-to-text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5531–5546, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  21. Mark EJ Newman. 2005. A measure of betweenness centrality based on random walks. Social networks, 27(1):39–54.
  22. Ranking of closeness centrality for large-scale social networks. Lecture Notes in Computer Science, 5059:186–195.
  23. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  24. Rethinking table recognition using graph neural networks. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 142–147. IEEE.
  25. Dropedge: Towards deep graph convolutional networks on node classification. arXiv preprint arXiv:1907.10903.
  26. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761.
  27. Evaluating and enhancing structural understanding capabilities of large language models on tables via input designs. arXiv preprint arXiv:2305.13062.
  28. Arnetminer: extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 990–998.
  29. Graph attention networks. stat, 1050(20):10–48550.
  30. W Patrick Walters and Regina Barzilay. 2020. Applications of deep learning in molecule generation and molecular property prediction. Accounts of chemical research, 54(2):263–270.
  31. Stanley Wasserman and Katherine Faust. 1994. Social network analysis: Methods and applications.
  32. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
  33. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601.
  34. Jiawei Zhang. 2023. Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt. arXiv preprint arXiv:2304.11116.
  35. Degree centrality, betweenness centrality, and closeness centrality in social network. In 2017 2nd international conference on modelling, simulation and applied mathematics (MSAM2017), pages 300–303. Atlantis press.
  36. Variational reasoning for question answering with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  37. Data augmentation for graph neural networks. In Proceedings of the aaai conference on artificial intelligence, volume 35, pages 11015–11023.
  38. A survey of large language models. arXiv preprint arXiv:2303.18223.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Jiayan Guo (15 papers)
  2. Lun Du (50 papers)
  3. Hengyu Liu (30 papers)
  4. Mengyu Zhou (24 papers)
  5. Xinyi He (19 papers)
  6. Shi Han (74 papers)
Citations (117)
View on Semantic Scholar