· Trending Papers
Emergent Mind

Talk like a Graph: Encoding Graphs for Large Language Models

(2310.04560)
Published Oct 6, 2023 in cs.LG

Abstract

Graphs are a powerful tool for representing and analyzing complex relationships in real-world applications such as social networks, recommender systems, and computational finance. Reasoning on graphs is essential for drawing inferences about the relationships between entities in a complex system, and to identify hidden patterns and trends. Despite the remarkable progress in automated reasoning with natural text, reasoning on graphs with LLMs remains an understudied problem. In this work, we perform the first comprehensive study of encoding graph-structured data as text for consumption by LLMs. We show that LLM performance on graph reasoning tasks varies on three fundamental levels: (1) the graph encoding method, (2) the nature of the graph task itself, and (3) interestingly, the very structure of the graph considered. These novel results provide valuable insight on strategies for encoding graphs as text. Using these insights we illustrate how the correct choice of encoders can boost performance on graph reasoning tasks inside LLMs by 4.8% to 61.8%, depending on the task.

We're not able to analyze this paper right now due to high demand.

Please check back later (sorry!).

Sign up for a free account or log in to generate a summary of this paper:

Sign Up Now
or
Log In

We ran into a problem analyzing this paper.

Newsletter

Get summaries of trending comp sci papers delivered straight to your inbox:

Unsubscribe anytime.

References
  1. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training
  2. Statistical mechanics of complex networks. Reviews of modern physics, 74(1):47
  3. PaLM 2 Technical Report
  4. Emergence of scaling in random networks. science, 286(5439):509–512
  5. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020a.
  6. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020b.
  7. Sparks of Artificial General Intelligence: Early experiments with GPT-4
  8. Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs
  9. PaLM: Scaling Language Modeling with Pathways
  10. Scaling Instruction-Finetuned Language Models
  11. Transformers as Soft Reasoners over Language
  12. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  13. A Generalization of Transformer Networks to Graphs
  14. On random graphs. Publicationes Mathematicae Debrecen, 6:290–297
  15. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929–3938. PMLR
  16. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States)
  17. Measuring Mathematical Problem Solving With the MATH Dataset
  18. Stochastic blockmodels: First steps. Social networks, 5(2):109–137
  19. LoRA: Low-Rank Adaptation of Large Language Models
  20. Towards Reasoning in Large Language Models: A Survey
  21. Patton: Language Model Pretraining on Text-Rich Networks
  22. LAMBADA: Backward Chaining for Automated Reasoning in Natural Language
  23. Decomposed Prompting: A Modular Approach for Solving Complex Tasks
  24. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213
  25. The Power of Scale for Parameter-Efficient Prompt Tuning
  26. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474
  27. What Makes Good In-Context Examples for GPT-$3$?
  28. Attending to Graph Transformers
  29. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744
  30. Graphworld: Fake graphs bring real insights for gnns. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.  3691–3701
  31. Unifying large language models and knowledge graphs: A roadmap
  32. Automatic Prompt Optimization with "Gradient Descent" and Beam Search
  33. A Decade of Knowledge Graphs in Natural Language Processing: A Survey
  34. Generate & Rank: A Multi-task Framework for Math Word Problems
  35. Attention is all you need. Advances in neural information processing systems, 30
  36. Can Language Models Solve Graph Problems in Natural Language?
  37. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837
  38. Large Language Models as Optimizers
  39. Examining the effects of degree distribution and homophily in graph learning models
  40. Deep bidirectional language-knowledge graph pretraining
  41. Language is All a Graph Needs
  42. Graph-Bert: Only Attention is Needed for Learning Graph Representations
  43. Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models
  44. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
  45. A Survey of Large Language Models
  46. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
  47. Large Language Models Are Human-Level Prompt Engineers

Show All 47