Graph4real: Real-World Graph Benchmark

• Graph4real is a comprehensive real-world graph benchmark evaluating LLMs on complex topologies, diverse text encodings, and multi-step reasoning.
• It covers 21 tasks across four domains—Transportation, Social, Web, and Citation—with graph scales up to 10,000 nodes.
• The benchmark stresses LLMs by testing realistic graph representations, working memory capacity, and algorithmic control flow under practical conditions.

Searching arXiv for the primary source on Graph4real. Graph4real is a large-scale, real-world benchmark specifically designed to test and stress LLMs on genuine graph reasoning, beyond the toy settings and synthetic graphs used by prior work. It was introduced together with GraphCogent as a comprehensive evaluation suite for LLM-based graph reasoning, but it can be viewed independently as a benchmark of real-world graphs and graph reasoning tasks built to expose and measure the limitations of LLMs on complex, large-scale graph problems (Wang et al., 17 Aug 2025). Its design centers on four real-world domains, 21 distinct tasks, graph scales that are 10 times larger than existing LLM graph benchmarks, and multiple text representations, with the stated objective of evaluating whether LLMs can handle realistic graph topology, heterogeneous graph encodings, and multi-step reasoning under working memory constraints (Wang et al., 17 Aug 2025).

1. Definition and motivation

Graph4real is defined as a benchmark of real-world graphs and graph reasoning tasks built to expose and measure the limitations of LLMs on complex, large-scale graph problems (Wang et al., 17 Aug 2025). The benchmark was motivated by three shortcomings in existing benchmarks. First, scale: most benchmarks such as NLGraph and GraphWiz use small synthetic graphs, typically less than or equal to 40–100 nodes and often randomly generated. The benchmark description states that such settings lack real-world topology, including skewed degree distributions and communities, and do not pressure LLM working memory (Wang et al., 17 Aug 2025). Second, representation: prior work usually assumes clean adjacency lists or preprocessed files, whereas graph information in realistic settings appears as edge lists in varied formats, symbolic forms such as 0→1, and natural language descriptions such as “user A follows user B” and “paper X cites paper Y” (Wang et al., 17 Aug 2025). Third, task realism: many earlier tasks are described as artificial because the query directly specifies an algorithm, often as a single sentence, whereas realistic graph reasoning involves vague intent and requires both interpretation and graph computation (Wang et al., 17 Aug 2025).

The benchmark therefore aims to provide a setting in which graph sizes approach those encountered in practice, tasks are grounded in realistic scenarios and domain semantics, and LLMs must handle diverse textual encodings of graphs and multi-step reasoning under working memory constraints (Wang et al., 17 Aug 2025). The description explicitly states that Graph4real uses graphs up to 1,000 nodes in the main experiments and up to 10,000 nodes in scalability tests (Wang et al., 17 Aug 2025).

A plausible implication is that Graph4real is intended not merely as a task collection but as a stress test for the interaction between graph parsing, memory retention, and algorithmic control flow in LLM systems. That interpretation is consistent with its integration into the GraphCogent framework, whose motivating claim is that current LLMs fail when handling real-world graphs with complex queries because they cannot effectively process complex graph topology and perform multi-step reasoning simultaneously (Wang et al., 17 Aug 2025).

2. Benchmark structure and graph domains

Graph4real covers four domains drawn from established graph datasets: Transportation, Social, Web, and Citation (Wang et al., 17 Aug 2025). In Transportation, the source is PeMS traffic networks, with nodes corresponding to traffic sensors or road segments and edges corresponding to road connections; raw PeMS data is temporal, but for most structural and algorithmic tasks the graphs are treated as static, while traffic prediction uses time series (Wang et al., 17 Aug 2025). In Social, the source is SNAP Social Circles, with users as nodes and social relationships as edges; these are described as typically sparse graphs with heterogeneous degrees (Wang et al., 17 Aug 2025). In Web, the source is the Google Web Graph, with web pages as nodes and hyperlinks as directed edges, producing graphs with strongly non-uniform degree distribution and PageRank-like dynamics (Wang et al., 17 Aug 2025). In Citation, the source is the Cora citation network, with papers as nodes, citation links as edges, and node features available for predictive tasks (Wang et al., 17 Aug 2025).

For all domains, the authors apply biased random walk sampling to extract graphs at three main scales: 40 nodes, 100 nodes, and 1,000 nodes (Wang et al., 17 Aug 2025). They also report performance on 2,000, 5,000, and 10,000 nodes. The formal specification given for each sampled graph is $G = (V, E)$, with $|V|$ in $\{40, 100, 1000, 2000, 5000, 10000\}$, and edges $E$ drawn from the domain graph via random walks, preserving realistic degree and connectivity patterns (Wang et al., 17 Aug 2025).

The domain-specific challenge profile is part of the benchmark design. Transportation includes shortest path, maximum flow, diameter, and traffic prediction. Social includes community-related quantities such as cores and components, link prediction, and influence patterns via neighborhood and degree. Web includes PageRank estimation, common neighbors, and link prediction. Citation includes reference overlap, clustering coefficients, and node classification by research topic (Wang et al., 17 Aug 2025).

Domain	Source	Domain challenges
Transportation	PeMS traffic networks	Shortest path, maximum flow, diameter, traffic prediction
Social	SNAP Social Circles	Cores, components, link prediction, influence patterns
Web	Google Web Graph	PageRank, common neighbors, link prediction
Citation	Cora citation network	Reference overlap, clustering coefficient, node classification

This domain construction is significant because it couples graph-theoretic heterogeneity with application semantics. The benchmark description repeatedly emphasizes that the tasks are grounded in realistic scenarios and domain semantics rather than abstract graph operations alone (Wang et al., 17 Aug 2025).

3. Task taxonomy and formulations

Graph4real defines 21 tasks organized into three categories: Graph Structural Querying, Graph Algorithmic Reasoning, and Graph Predictive Modeling (Wang et al., 17 Aug 2025). The structural querying category contains five tasks: Edge Count, Node Count, Degree Count, Edge Existence, and Node Existence. These are implemented as simple direct lookups in tools with time complexity $O(1)$, and they are described as testing whether the LLM preserves and can query structural information (Wang et al., 17 Aug 2025).

Graph Algorithmic Reasoning contains 12 tasks. Four are designated as in-toolset algorithmic tasks: Cycle Detection, Triangle Count, Path Existence, and Shortest Path. The benchmark description specifies the associated algorithms and stated complexities: Depth-First Search for cycle detection with $O(|V| + |E|)$, neighborhood-based counting for triangle count with roughly $O(|V| \cdot d_{\max}^2)$, DFS or BFS for path existence with $O(|V| + |E|)$, and Dijkstra for shortest path with $O(|V|^2 + |E|)$ in their setup (Wang et al., 17 Aug 2025). Eight are designated as out-toolset algorithmic tasks: Maximum Flow, Diameter Calculation, Max Core Calculation, Connected Components, Common Neighbors, PageRank Calculation, Reference Match, and Clustering Coefficient, with the benchmark specifying the associated algorithms and complexities case by case (Wang et al., 17 Aug 2025).

The predictive modeling category contains four tasks. Transportation uses Traffic Prediction with time series from PeMS stored as NumPy arrays and an LSTM model, evaluated with Mean Absolute Error. Social uses Link Prediction with social graphs processed as PyG tensors and a GCN model. Web uses Link Prediction with GraphSAGE. Citation uses Node Classification on Cora with node features and a GAT model (Wang et al., 17 Aug 2025).

Task category	Number of tasks	Included tasks
Graph Structural Querying	5	Edge Count, Node Count, Degree Count, Edge Existence, Node Existence
Graph Algorithmic Reasoning	12	Cycle Detection, Triangle Count, Path Existence, Shortest Path, Maximum Flow, Diameter Calculation, Max Core Calculation, Connected Components, Common Neighbors, PageRank Calculation, Reference Match, Clustering Coefficient
Graph Predictive Modeling	4	Traffic Prediction, Link Prediction (Social), Link Prediction (Web), Node Classification

Tasks are presented as contextually grounded questions rather than bare algorithm calls (Wang et al., 17 Aug 2025). The dataset construction uses, for each domain, 5 scenarios, 20 prompt templates, and natural language questions generated by DeepSeek-R1 from templates and specific graphs (Wang et al., 17 Aug 2025). Each question embeds the graph in textual format, a scenario description, and a query. The output requirements are numeric or boolean answers for structural and algorithmic tasks, and predicted labels or probabilities for predictive tasks (Wang et al., 17 Aug 2025).

This design indicates that Graph4real evaluates not only execution of graph procedures but also query interpretation. The benchmark description makes that distinction explicit by contrasting its contextual questions with earlier benchmarks that directly specify the target algorithm (Wang et al., 17 Aug 2025).

4. Graph scale, representations, and evaluation protocol

A central property of Graph4real is scale. Compared to prior benchmarks, NLGraph uses graphs less than 40 nodes and GraphWiz uses up to about 100 nodes, whereas Graph4real uses 40-node, 100-node, and 1,000-node graphs in the main setup and tests up to 10,000 nodes in scalability experiments (Wang et al., 17 Aug 2025). The benchmark description states that typical Graph4real graphs are approximately 10 times larger than those in NLGraph and GraphWiz in terms of node counts (Wang et al., 17 Aug 2025).

Graph4real deliberately uses three text representation formats to test LLMs’ “sensory” capability: adjacency lists, symbolic notation, and linguistic descriptions (Wang et al., 17 Aug 2025). Adjacency lists are the mainstream representation for the main experiments. Symbolic notation includes forms such as 0→1. Linguistic descriptions use domain-specific predicates such as “A is linked to B,” “user U follows V,” “road A is connected to B,” and “paper X cites Y” (Wang et al., 17 Aug 2025).

The task evaluation protocol includes 4200 questions total, distributed across 21 tasks (Wang et al., 17 Aug 2025). For evaluation, the description states that for each task and each scale—40, 100, and 1000 nodes—across four domains, there are 200 questions per domain, combined into 500 test instances per task and scale (Wang et al., 17 Aug 2025). For training GraphCogent, 4 of 20 prompt templates are reserved for training, and their corresponding tasks are excluded from test sets to enforce generalization across templates (Wang et al., 17 Aug 2025).

Accuracy is used for all tasks except traffic prediction, for which Mean Absolute Error is used (Wang et al., 17 Aug 2025). The formulas provided are

$$\text{Accuracy} = \frac{\text{\# correct predictions}}{\text{\# total instances}}$$

and

$|V|$0

Token usage and reasoning time per task are also reported to measure efficiency (Wang et al., 17 Aug 2025).

The benchmark description also introduces the Graph Efficiency Coefficient in the Sensory Module: $|V|$1 where $|V|$2 is the original graph, $|V|$3 is the transformed graph, $|V|$4 measures structural deviation in edges, and the token term captures total token cost across $|V|$5 LLM interactions (Wang et al., 17 Aug 2025). The benchmark description states that this metric balances structural fidelity against computational cost and that for Llama3.1‑8B, subgraphs of roughly 50 edges achieve the best GEC (Wang et al., 17 Aug 2025).

A plausible implication is that Graph4real operationalizes graph reasoning as a joint problem of parsing cost, memory retention, and structural preservation, rather than as isolated answer generation. That interpretation follows directly from the role of multiple encodings, token accounting, and edit-distance-based transformation quality in the benchmark design (Wang et al., 17 Aug 2025).

5. Integration with GraphCogent

Graph4real is the primary testbed for GraphCogent, a collaborative agent framework inspired by the human Working Memory Model and organized into Sensory Module, Buffer Module, and Execution Module (Wang et al., 17 Aug 2025). Within that system, Graph4real provides the graphs, task formats, and evaluation environment against which the framework is trained and tested.

The Sensory Module takes Graph4real’s graph in any text format, samples subgraphs guided by GEC, converts the input to standardized adjacency lists, and verifies the result with a Graph Verifier that checks edge count consistency and format compliance (Wang et al., 17 Aug 2025). The Buffer Module integrates adjacency lists to reconstruct $|V|$6, produces multiple formats including NetworkX graph objects, NumPy arrays, and PyG tensors, and builds indices describing dimensionality, schema, and metadata (Wang et al., 17 Aug 2025). The Execution Module contains a Reasoning Agent and a Model Agent. The Reasoning Agent decides whether a task is within toolset coverage and calls relevant tools on Buffer data for in-toolset tasks. The Model Agent handles out-toolset tasks by generating task-specific models that consume Buffer formats (Wang et al., 17 Aug 2025).

Experimental settings on Graph4real use Llama3.1‑8B-Instruct as the backbone model, with LoRA for both Reasoning Agent and Model Agent, SFT + DPO for the Reasoning Agent, and Model-Enhanced SFT for the Model Agent (Wang et al., 17 Aug 2025). Baselines include text-based systems such as GPT‑4o, Claude‑3.7, and DeepSeek‑R1 with 2-shot CoT; tool-based systems such as Graph-Toolformer, GraphTool-Instruction, and GPT‑4o Function Calling; and the agent-based baseline GraphTeam (Wang et al., 17 Aug 2025).

This integration matters because Graph4real is not defined merely as a static benchmark. It is tightly integrated with a system architecture that treats graph reasoning as decomposition into sensory normalization, buffered representation management, and execution-time tool or model selection (Wang et al., 17 Aug 2025). At the same time, the benchmark description explicitly states that it can be viewed independently as a comprehensive evaluation suite for LLM-based graph reasoning (Wang et al., 17 Aug 2025).

6. Empirical findings, limitations, and significance

The reported experiments position Graph4real as a benchmark that exposes substantial weaknesses in current LLMs on real-world graphs. In Graph N-back tests with 100-node, 300-edge graphs, Llama3.1‑8B’s accuracy drops sharply at $|V|$7 across adjacency, symbolic, and linguistic representations, indicating that after 3 turns it “forgets” edges (Wang et al., 17 Aug 2025). The benchmark analysis also states that text-based baselines perform reasonably on small structural tasks but fail on multi-step algorithmic tasks at larger scales, with shortest path accuracy dropping to 7–8% on 100-node graphs for such models (Wang et al., 17 Aug 2025).

The GraphCogent experiments on Graph4real report high performance on in-toolset tasks. On middle-scale 100-node graphs, GraphCogent achieves average accuracy around 98.5% across structural and common algorithmic tasks, compared with about 90.1% for GPT‑4o Function Calling, about 88.8% for GraphTool-Instruction, about 80.4% for GraphTeam, and about 47.2% for DeepSeek‑R1 on these tasks (Wang et al., 17 Aug 2025). On small-scale 40-node graphs, GraphCogent reaches about 99.1% average, slightly above GPT‑4o-FC at about 98.9%. For 1,000-node graphs, only GraphCogent reports results, maintaining about 97.6% average accuracy (Wang et al., 17 Aug 2025).

On out-toolset tasks at the middle scale, GraphCogent is reported to outperform GraphTeam across transportation, social, web, and citation tasks. Examples given include Transportation Max Flow and Diameter at 89.6% and 92.4% for GraphCogent versus 72.4% and 61.6% for GraphTeam; Web Common Neighbors, PageRank, and Link Prediction at about 86.4–92.6% for GraphCogent versus about 51.2–76.8% for GraphTeam; and Citation Reference Match, Clustering Coefficient, and Node Classification at about 85.2–90.0% for GraphCogent versus about 72.0–78.8% for GraphTeam (Wang et al., 17 Aug 2025).

Efficiency measurements are also emphasized. For middle-scale 100-node Shortest Path, GraphTeam uses 4248 input tokens and 3285 output tokens with 65.8% accuracy, whereas GraphCogent uses 2794 input tokens and 615 output tokens with 98.2% accuracy, corresponding to about an 80% reduction in output tokens (Wang et al., 17 Aug 2025). For middle-scale Max Flow, GraphTeam uses 4122 input tokens and 3023 output tokens with 72.4% accuracy, whereas GraphCogent uses 3009 input tokens and 1044 output tokens with 89.6% accuracy, corresponding to about a 30% reduction in output tokens (Wang et al., 17 Aug 2025). The benchmark description further states that compared to a state-of-the-art agent-based baseline, the framework outperforms by 20% in accuracy while reducing token usage by 80% for in-toolset tasks and 30% for out-toolset tasks (Wang et al., 17 Aug 2025).

Reasoning-time measurements show a trade-off. On 100-node Shortest Path, GPT‑4o Function Calling records 12.2 s with 89.0% accuracy, GraphCogent records 22.8 s with 98.2% accuracy, and GraphTeam records 58.1 s with 65.8% accuracy (Wang et al., 17 Aug 2025). The benchmark analysis states that GraphCogent is slower than GPT‑4o-FC due to additional sensory sampling, but far faster than GraphTeam while achieving higher accuracy (Wang et al., 17 Aug 2025).

Scalability results on selected tasks from 2,000 to 10,000 nodes report 97.0–98.2% accuracy for Shortest Path, about 96.8–97.0% for Cycle Detection, about 98.6–99.0% for Triangle Count, and about 97.0–98.0% for Path Existence (Wang et al., 17 Aug 2025). These results support the benchmark’s stated purpose as an evaluation suite for graph reasoning beyond the 1,000-node scale central to the main experiments (Wang et al., 17 Aug 2025).

The benchmark analysis identifies three failure modes: working memory constraints, global structure loss, and tool and code fragility (Wang et al., 17 Aug 2025). Working memory constraints appear in the N-back results. Global structure loss appears when sequential text presentation causes models to miscount edges, mis-evaluate path existence, and mishandle shortest paths as graphs become larger. Tool and code fragility is attributed to agent-based baselines that rely on full code generation from LLMs (Wang et al., 17 Aug 2025). This suggests that Graph4real serves not only as an accuracy benchmark but also as an instrument for diagnosing where graph reasoning systems break under realistic graph, query, and representation conditions.

In that sense, Graph4real occupies a specific place in the emerging literature on graph reasoning for LLMs: it is designed to move evaluation away from small synthetic graphs and toward genuine graph reasoning on real-world domains, larger scales, heterogeneous textual encodings, and realistic query formulations (Wang et al., 17 Aug 2025).