Talk like a Graph: Encoding Graphs for Large Language Models (2310.04560v1)

Abstract: Graphs are a powerful tool for representing and analyzing complex relationships in real-world applications such as social networks, recommender systems, and computational finance. Reasoning on graphs is essential for drawing inferences about the relationships between entities in a complex system, and to identify hidden patterns and trends. Despite the remarkable progress in automated reasoning with natural text, reasoning on graphs with LLMs remains an understudied problem. In this work, we perform the first comprehensive study of encoding graph-structured data as text for consumption by LLMs. We show that LLM performance on graph reasoning tasks varies on three fundamental levels: (1) the graph encoding method, (2) the nature of the graph task itself, and (3) interestingly, the very structure of the graph considered. These novel results provide valuable insight on strategies for encoding graphs as text. Using these insights we illustrate how the correct choice of encoders can boost performance on graph reasoning tasks inside LLMs by 4.8% to 61.8%, depending on the task.

• The paper introduces a novel framework that converts graph structures into text, facilitating LLMs to perform graph-based tasks.
• It demonstrates that incident encoding and tailored prompt engineering significantly enhance performance on graph reasoning tasks.
• Results reveal that LLMs struggle with global graph properties, underscoring limitations and pointing to the need for hybrid model approaches.

Encoding Graphs for LLMs: A Comprehensive Study

Introduction

The paper "Talk like a Graph: Encoding Graphs for LLMs" (2310.04560) presents a systematic investigation into the problem of representing graph-structured data as text for consumption by LLMs. The paper addresses a critical gap: while LLMs have demonstrated strong performance on a variety of text-based reasoning tasks, their ability to reason over graph-structured data—ubiquitous in domains such as social networks, recommender systems, and knowledge graphs—remains underexplored. The authors introduce a new benchmark, GraphQA, and conduct extensive experiments to analyze how graph encoding, prompt engineering, and graph structure affect LLM performance on fundamental graph reasoning tasks.

Figure 1: Overview of the framework for reasoning with graphs using LLMs, highlighting the modularity of graph encoding and prompt engineering.

Graph Encoding as Text: Methodological Framework

The core technical challenge addressed is the transformation of arbitrary graphs $G = (V, E)$ into textual sequences $W$ suitable for LLM input. The authors formalize this as the design of a graph encoding function $g: G \mapsto W$ and a question rephrasing function $q: W \mapsto W$, such that the LLM $f$ can be queried as $A = f(g(G), q(Q))$ for a question $Q$ about the graph.

Figure 2: Overview of the framework for encoding graphs via text, illustrating the mapping from graph structure to natural language representations.

The paper systematically explores a taxonomy of graph encoding strategies, varying both node and edge representations. Node encodings include integer indices, English names, character names from popular media, and alphabetic labels. Edge encodings range from explicit adjacency lists to natural language statements of relationships (e.g., "A and B are friends"). The authors also experiment with different prompt engineering heuristics, including zero-shot, few-shot, chain-of-thought (CoT), and bag prompting.

Empirical Evaluation: GraphQA Benchmark

The GraphQA benchmark comprises a suite of basic graph tasks: edge existence, node degree, node count, edge count, connected nodes, cycle check, and disconnected nodes. These tasks are designed to probe both local and global graph reasoning capabilities of LLMs.

Key Findings

1. LLMs Underperform on Basic Graph Tasks

Across all evaluated models and tasks, LLMs exhibit poor performance on basic graph reasoning, often failing to surpass simple majority baselines, especially for tasks such as edge existence and cycle detection. This highlights a fundamental limitation in the ability of LLMs to perform even elementary graph computations when provided with naïve textual encodings.

2. Graph Encoding Function Critically Impacts Performance

The choice of graph encoding function $g(.)$ has a substantial effect on LLM accuracy. For example, incident encoding (where each node lists its neighbors) outperforms adjacency encoding for tasks like node degree and connected nodes, as it places relevant information in closer textual proximity. Integer node encodings improve arithmetic tasks, while named node encodings are advantageous for tasks with non-integer outputs.

3. Prompt Engineering and Question Framing Matter

Prompting strategies significantly influence outcomes. Zero-shot prompting suffices for simple tasks, but few-shot and CoT prompting yield improvements for more complex queries. Notably, rephrasing questions in application-specific language (e.g., "How many friends does Alice have?") consistently outperforms abstract graph-theoretic formulations.

4. Model Capacity Correlates with Graph Reasoning Ability

Larger LLMs (e.g., PaLM 62B) demonstrate improved performance on graph tasks compared to smaller variants, but even the largest models do not consistently outperform majority baselines on all tasks. The effect of scale is more pronounced for tasks requiring aggregation or multi-hop reasoning.

5. Graph Structure and Generator Influence LLM Performance

The structure of the input graph—determined by the graph generator (Erdős–Rényi, Barabási–Albert, SBM, star, path, complete)—has a marked impact on LLM accuracy. For instance, cycle detection is trivial for complete graphs but challenging for path graphs, reflecting LLMs' strong priors and susceptibility to distractors in the encoding.

Figure 3: Samples of graphs generated with different graph generators, illustrating the diversity of structures in the GraphQA benchmark.

6. LLMs Lack a Global Model of the Graph

Tasks requiring reasoning about the absence of edges (e.g., disconnected nodes) expose a critical weakness: LLMs are unable to infer global properties not explicitly encoded in the text, achieving near-zero accuracy.

Analysis of Graph Encoding Strategies

The authors provide a detailed ranking of encoding functions across tasks and prompting methods. Incident encoding is generally optimal for most prompting strategies, except in zero-shot settings where encodings with familiar names (e.g., politicians, fictional characters) perform better. The paper also finds that distractive statements in the encoding degrade performance, especially in dense graphs.

Figure 4: Example graph used to illustrate the output of different graph encoding functions.

Implications and Future Directions

Practical Implications

• Black-box LLMs: The paper focuses on scenarios where LLM weights are inaccessible, emphasizing the importance of prompt and encoding design for practical deployment.
• Task-Specific Encoding: Careful selection of encoding and question phrasing can yield performance gains of 4.8% to 61.8% on graph reasoning tasks, underscoring the need for task-aware prompt engineering.
• Benchmarking: The GraphQA benchmark provides a valuable resource for evaluating and comparing LLMs on structured reasoning tasks.

Theoretical Implications

• Limitations of Textual Encodings: The inability of LLMs to construct a global model of the graph from text suggests fundamental representational bottlenecks.
• Inductive Biases: LLMs exhibit strong priors based on training data distributions, which can be maladaptive for synthetic or out-of-distribution graph structures.

Future Research Directions

• Hybrid Architectures: Integrating explicit graph neural modules or external memory with LLMs may address the observed limitations in global reasoning.
• Automated Encoding Search: Meta-learning or reinforcement learning approaches to discover optimal graph-to-text encodings could further improve performance.
• Instruction Tuning: Fine-tuning LLMs on graph-structured data or augmenting pretraining corpora with synthetic graph-text pairs may enhance inductive biases for structured reasoning.

Conclusion

This paper provides a rigorous, empirical foundation for understanding how LLMs process graph-structured data when presented as text. The results demonstrate that LLM performance on graph reasoning is highly sensitive to encoding choices, prompt engineering, and graph structure. While current LLMs are not yet reliable for general graph reasoning in a black-box setting, the insights and benchmarks introduced here lay the groundwork for future advances in structured reasoning with LLMs.