· Trending Papers
Emergent Mind

GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking

(2305.15066)
Published May 24, 2023 in cs.AI and cs.CL

Abstract

Large language models~(LLM) like ChatGPT have become indispensable to artificial general intelligence~(AGI), demonstrating excellent performance in various natural language processing tasks. In the real world, graph data is ubiquitous and an essential part of AGI and prevails in domains like social network analysis, bioinformatics and recommender systems. The training corpus of LLMs often includes some algorithmic components, which allows them to achieve certain effects on some graph data-related problems. However, there is still little research on their performance on a broader range of graph-structured data. In this study, we conduct an extensive investigation to assess the proficiency of LLMs in comprehending graph data, employing a diverse range of structural and semantic-related tasks. Our analysis encompasses 10 distinct tasks that evaluate the LLMs' capabilities in graph understanding. Through our study, we not only uncover the current limitations of language models in comprehending graph structures and performing associated reasoning tasks but also emphasize the necessity for further advancements and novel approaches to enhance their graph processing capabilities. Our findings contribute valuable insights towards bridging the gap between language models and graph understanding, paving the way for more effective graph mining and knowledge extraction.

We're not able to analyze this paper right now due to high demand.

Please check back later (sorry!).

Sign up for a free account or log in to generate a summary of this paper:

Sign Up Now
or
Log In

We ran into a problem analyzing this paper.

Newsletter

Get summaries of trending comp sci papers delivered straight to your inbox:

Unsubscribe anytime.

References
  1. Marc Barthelemy. 2004. Betweenness centrality in large complex networks. The European physical journal B, 38(2):163–168.
  2. Bioinformatics. John Wiley & Sons.
  3. Node Classification in Social Networks
  4. Richard J Bolton and David J Hand. 2002. Statistical fraud detection: A review. Statistical science, 17(3):235–255.
  5. Ulrik Brandes. 2001. A faster algorithm for betweenness centrality. Journal of mathematical sociology, 25(2):163–177.
  6. Graph markup language (graphml)
  7. Towards table-to-text generation with pretrained language model: A table structure understanding and text deliberating approach. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8199–8210, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  8. An upper bound on the diameter of a graph from eigenvalues associated with its laplacian. SIAM Journal on Discrete Mathematics, 7(3):443–457.
  9. Mate: multi-view attention for table transformer efficiency. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing".
  10. A Fair Comparison of Graph Neural Networks for Graph Classification
  11. Tablegpt: Few-shot table-to-text generation with table structure reconstruction and content matching. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1978–1988.
  12. Michael Himsolt. 1997. Gml: Graph modelling language. University of Passau.
  13. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118–22133.
  14. Knowledge graph embedding based question answering. In Proceedings of the twelfth ACM international conference on web search and data mining, pages 105–113.
  15. Recommendation systems: Principles, methods and evaluation. Egyptian informatics journal, 16(3):261–273.
  16. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494–514.
  17. StructGPT: A General Framework for Large Language Model to Reason over Structured Data
  18. Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.
  19. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
  20. PLOG: Table-to-logic pretraining for logical table-to-text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5531–5546, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  21. Mark EJ Newman. 2005. A measure of betweenness centrality based on random walks. Social networks, 27(1):39–54.
  22. Ranking of closeness centrality for large-scale social networks. Lecture Notes in Computer Science, 5059:186–195.
  23. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  24. Rethinking table recognition using graph neural networks. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 142–147. IEEE.
  25. DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
  26. Toolformer: Language Models Can Teach Themselves to Use Tools
  27. Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study
  28. Arnetminer: extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 990–998.
  29. Graph attention networks. stat, 1050(20):10–48550.
  30. W Patrick Walters and Regina Barzilay. 2020. Applications of deep learning in molecule generation and molecular property prediction. Accounts of chemical research, 54(2):263–270.
  31. Stanley Wasserman and Katherine Faust. 1994. Social network analysis: Methods and applications.
  32. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
  33. Tree of Thoughts: Deliberate Problem Solving with Large Language Models
  34. Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT
  35. Degree centrality, betweenness centrality, and closeness centrality in social network. In 2017 2nd international conference on modelling, simulation and applied mathematics (MSAM2017), pages 300–303. Atlantis press.
  36. Variational reasoning for question answering with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  37. Data augmentation for graph neural networks. In Proceedings of the aaai conference on artificial intelligence, volume 35, pages 11015–11023.
  38. A Survey of Large Language Models

Show All 38