GraphPile: Large-Scale Graph Reasoning Corpus

• GraphPile is a large-scale corpus of 2.6M+ examples across 23 graph tasks designed to enhance LLM reasoning via graph problem solving.
• It integrates four distinct annotation strategies—Chain-of-Thought, Program-of-Thought, Trace-of-Execution, and real-world data—to support diverse reasoning methods.
• Empirical evaluations demonstrate significant improvements in logical, code, and graph reasoning tasks when models are pretrained on GraphPile.

GraphPile is a large-scale corpus specifically designed for continued pretraining (CPT) of LLMs using graph problem reasoning (GPR) data. Uniquely targeting the development of generalized reasoning capabilities, GraphPile consists of 10,930,682,053 tokens across 2,684,675 examples, organized into 23 distinct graph tasks. The dataset was central to the development and evaluation of the GraphMind model family, and its structure, coverage, construction methodology, and empirical impact on reasoning performance collectively advance the intersection of graph algorithms and language modeling (Zhang et al., 23 Jul 2025).

1. Corpus Composition and Task Spectrum

GraphPile incorporates four major data components representing complementary annotation paradigms: Chain-of-Thought (CoT), Program-of-Thought (PoT), Trace-of-Execution (ToE), and real-world graph data. Each component contributes a substantial share of the total data, as shown below:

Component	Examples	Tokens	Token Share (%)
Chain-of-Thought (CoT)	848,965	2,809,225,185	25.7
Real-world Graph Data	743,465	3,203,590,685	29.3
Program-of-Thought	759,851	2,190,746,959	20.0
Trace-of-Execution	332,394	2,727,119,224	25.0

Tasks span logical reasoning, topological reasoning, numerical computation, combinatorial enumeration, division, and spatial reasoning. Explicitly covered problems include cycle detection, bipartite checking, topological sorting, shortest path, maximum flow, Hamilton path, maximum clique, connectivity, PageRank, degree counting, Jaccard coefficient, edge/neighbor/predecessor queries, diameter, minimum spanning tree, clustering coefficient, Euler path, and planarity testing. Task difficulty ranges from tractable polynomial-time problems to NP-complete challenges, with input graphs of 6–40 nodes to balance reasoning complexity and reasoning-path tractability (Zhang et al., 23 Jul 2025).

2. Data Annotation Paradigms and Formats

GraphPile encompasses four annotation strategies:

Chain-of-Thought (CoT): Each instance comprises a natural-language problem, a graph structure serialized in text (adjacency matrix, list, or edge list), and a multi-step reasoning trace culminating in a final answer. CoT construction utilizes program-guided solvers, traces rephrased to natural language via GPT-4o, and further GPT-4o-based correctness verification. Up to three distinct valid solutions are provided for problems admitting non-unique solutions.

Program-of-Thought (PoT): Examples map graph reasoning tasks to executable Python code, often leveraging NetworkX routines such as nx.Graph(), nx.bipartite.maximum_matching, and nx.pagerank. Code examples are retrieved, rewritten for diversity, and filtered for correctness and execution.

Trace-of-Execution (ToE): This novel task type provides a graph problem, code snippet, variable initializations, and an explicit execution trace logging function calls, iteration, variable state changes, and recursion. Additional queries target intermediate variable states (e.g., max_matching, stack contents), offering process-level supervision.

Real-world Graph Data: Problems are derived by domain-adaptively replacing synthetic node identifiers with real-world entities (e.g., author names, airport codes, molecular structures) from sources such as DBLP, OpenFlights, PubChemQC, social network datasets, and DBpedia. Post-rephrasing, automated filtering enforces structure-preserving node and edge correspondence (Zhang et al., 23 Jul 2025).

3. Dataset Construction and Quality Controls

GraphPile construction applies a multi-stage data curation protocol to ensure structural, solution, and annotation diversity, as well as data integrity. Synthetic graphs are generated following the Erdős–Rényi (ER) model (parameters not detailed), supporting both directed and undirected, weighted and unweighted forms, and multiple serialization formats. Real-world graphs are sourced and transformed as described above. Diversity controls address task coverage, graph structure, supervision type, and solution multiplicity (up to three valid solutions per instance).

Quality validation involves a PhD-level graph algorithms expert who authors and verifies algorithmic solvers, checks efficiency, and oversees annotation processes. CoT traces are checked for solution alignment post-GPT-4o rephrasing. PoT code must execute and match expected results. Real-world mappings are filtered for exact node/edge correspondence. ToE traces are deterministically generated from high-quality instrumented code. Errors are filtered at each stage to maintain high annotation fidelity (Zhang et al., 23 Jul 2025).

4. Pretraining, Model Integration, and Evaluation

Continued pretraining uses an autoregressive loss:

$$\mathcal{L}_\mathrm{CPT} = -\sum_{i=1}^N \log p_\theta(x_i \mid x_{<i})$$

Standard hyperparameters are used (learning rate $3 \times 10^{-5}$, 3 epochs, sequence length 8192, batch size 1024). The CPT process is performed on 32 H100 GPUs; downstream tuning uses 4 A800 GPUs. No architectural modifications are introduced to the base models—Llama-3-8B, Llama-3.1-8B, and Gemma-2-2B—but all undergo CPT solely on GraphPile (Zhang et al., 23 Jul 2025).

Evaluation is comprehensive, spanning 22 benchmarks across mathematics (GSM8K, MATH, SVAMP, etc.), logical reasoning (Zebra Puzzle, Ruletaker), commonsense (StrategyQA, Hellaswag), code reasoning (LiveCodeBench, CLRS), multi-hop QA (HotpotQA, PopQA), and in-domain graph reasoning (GraphWiz, GraphInstruct). Accuracy improvement is measured as:

$$\Delta \mathrm{Acc} = \mathrm{Acc}_\mathrm{GraphMind} - \mathrm{Acc}_\mathrm{Base}$$

5. Empirical Performance, Ablations, and Analysis

GraphMind models achieve up to +4.9% average accuracy gain on mathematical reasoning (Llama-3-8B: 48.9→53.8) and up to +21.2% improvement on non-mathematical tasks. Best-case, logical reasoning, code reasoning, and graph reasoning see relative gains of +33.4%, +46.3%, and +53.1%, respectively. Notably, graph-specific evaluations show base Llama-3.1-8B accuracy on GraphWiz improving from 1.9 to 52.2 (Δ=+50.3) and on GraphInstruct from 31.97 to 72.88 (Δ=+40.91).

Ablation studies quantify the impact of each annotation category: removing CoT or ToE induces the largest drops in graph/general reasoning, with ToE absence producing strong degradations in logical and code reasoning (logical: 9.0→4.8, code: 19.9→9.7). Data scaling analysis shows continued improvement as more GraphPile data is used, confirming scalability. Additional CPT on code-specialized models (Qwen-2.5-Coder-1.5B/7B) delivers further, albeit diminishing, gains across tasks (Zhang et al., 23 Jul 2025).

6. Comparison to Prior Graph Reasoning Datasets

GraphPile substantially exceeds prior efforts in scale, domain/generalization breadth, and supervision style. While datasets like GraphWiz and GraphInstruct offer only tens of thousands of synthetic CoT problems over 6–21 tasks and lack CPT compatibility, GraphPile delivers 2.6M examples across 23 tasks, incorporating real-world data and process-level traces. No other dataset supports all four annotation types or targets CPT at this scale.

Dataset	Graph Category	Paradigm	Tasks	Samples	CPT-Compatible
GraphWiz	Synthetic	CoT	9	17,158	No
GraphInstruct	Synthetic	CoT	21	16,800	No
GraphArena	Real-World	Simple Answer	10	10,000	No
GraphPile	Synthetic + Real-World	CoT, PoT, ToE	23	2,684,675	Yes

7. Significance, Limitations, and Impact

GraphPile operationalizes the hypothesis that graph problem reasoning encodes a versatile and transferable repertoire of reasoning strategies—logical, topological, symbolic, and procedural—that can enhance an LLM's performance beyond mathematical reasoning tasks. The resulting GraphMind models demonstrate robust, transferable improvements across mathematical, logical, code-based, commonsense, and multi-hop QA benchmarks. The ablation and error analysis underscore the unique contributions of multi-format supervision—especially explicit execution tracing.

Reported limitations include the absence of explicit mixing ratios for original pretraining corpora, no wall-clock training times, and observed minor declines in non-reasoning capabilities (e.g., translation, summarization). These are presented as consistent with the tradeoffs inherent to reasoning-oriented CPT (Zhang et al., 23 Jul 2025). Statistical significance testing confirms improvements in select benchmarks (Ruletaker, GraphInstruct) but not universally, attributed to variance across temperature settings.

In summary, GraphPile is a foundational, high-fidelity resource enabling models to acquire general reasoning capabilities from graph-centric problem solving, and is a substantial advancement in the design of CPT corpora for broad, cross-domain reasoning robustness (Zhang et al., 23 Jul 2025).