Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Talk like a Graph: Encoding Graphs for Large Language Models (2310.04560v1)

Published 6 Oct 2023 in cs.LG

Abstract: Graphs are a powerful tool for representing and analyzing complex relationships in real-world applications such as social networks, recommender systems, and computational finance. Reasoning on graphs is essential for drawing inferences about the relationships between entities in a complex system, and to identify hidden patterns and trends. Despite the remarkable progress in automated reasoning with natural text, reasoning on graphs with LLMs remains an understudied problem. In this work, we perform the first comprehensive study of encoding graph-structured data as text for consumption by LLMs. We show that LLM performance on graph reasoning tasks varies on three fundamental levels: (1) the graph encoding method, (2) the nature of the graph task itself, and (3) interestingly, the very structure of the graph considered. These novel results provide valuable insight on strategies for encoding graphs as text. Using these insights we illustrate how the correct choice of encoders can boost performance on graph reasoning tasks inside LLMs by 4.8% to 61.8%, depending on the task.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training, 2021.
  2. Statistical mechanics of complex networks. Reviews of modern physics, 74(1):47, 2002.
  3. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  4. Emergence of scaling in random networks. science, 286(5439):509–512, 1999.
  5. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020a.
  6. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020b.
  7. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  8. Exploring the potential of large language models (llms) in learning on graphs. arXiv preprint arXiv:2307.03393, 2023.
  9. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  10. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
  11. Transformers as soft reasoners over language. arXiv preprint arXiv:2002.05867, 2020.
  12. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  13. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.
  14. On random graphs. Publicationes Mathematicae Debrecen, 6:290–297, 1959.
  15. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929–3938. PMLR, 2020.
  16. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States), 2008.
  17. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
  18. Stochastic blockmodels: First steps. Social networks, 5(2):109–137, 1983.
  19. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  20. Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403, 2022.
  21. Patton: Language model pretraining on text-rich networks. arXiv preprint arXiv:2305.12268, 2023.
  22. Lambada: Backward chaining for automated reasoning in natural language. arXiv preprint arXiv:2212.13894, 2022.
  23. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022.
  24. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.
  25. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
  26. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.
  27. What makes good in-context examples for gpt-3333? arXiv preprint arXiv:2101.06804, 2021.
  28. Attending to graph transformers. arXiv preprint arXiv:2302.04181, 2023.
  29. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  30. Graphworld: Fake graphs bring real insights for gnns. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.  3691–3701, 2022.
  31. Unifying large language models and knowledge graphs: A roadmap, 2023.
  32. Automatic prompt optimization with” gradient descent” and beam search. arXiv preprint arXiv:2305.03495, 2023.
  33. A decade of knowledge graphs in natural language processing: A survey. arXiv preprint arXiv:2210.00105, 2022.
  34. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034, 2021.
  35. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  36. Can language models solve graph problems in natural language? arXiv preprint arXiv:2305.10037, 2023.
  37. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  38. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023.
  39. Examining the effects of degree distribution and homophily in graph learning models, 2023.
  40. Deep bidirectional language-knowledge graph pretraining, 2022.
  41. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134, 2023.
  42. Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.05140, 2020.
  43. Exploring the mit mathematics and eecs curriculum using large language models. arXiv preprint arXiv:2306.08997, 2023a.
  44. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023b.
  45. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
  46. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022a.
  47. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022b.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Bahare Fatemi (22 papers)
  2. Jonathan Halcrow (10 papers)
  3. Bryan Perozzi (58 papers)
Citations (76)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube