Graph-ToolFormer Framework

• Graph-ToolFormer is a framework that enables LLMs to accurately analyze graph structures via external API calls.
• It augments prompt engineering with ChatGPT to generate and validate diverse API call examples for complex graph tasks.
• Fine-tuning on extensive datasets enhances multi-step reasoning and precise graph property extraction, overcoming standard LLM limitations.

Graph-ToolFormer is a framework designed to enable LLMs to reason over complex graph-structured data by learning to call external graph reasoning APIs. LLMs such as GPT-3, GPT-4, and LLaMA excel at natural language understanding and generation but exhibit substantive deficiencies in multi-step logic reasoning, exact mathematical calculation, and perception of spatial, topological, and temporal structures—all foundational to graph learning tasks. Building on principles from Toolformer, which taught LLMs to insert API calls for general utilities, Graph-ToolFormer augments prompt engineering using ChatGPT to teach LLMs to interface with rich, domain-specific graph analytics toolkits, including operations for graph property extraction, node and link prediction, and advanced analytics over scientific and social network datasets (Zhang, 2023).

1. Motivation and Limitations of Current LLMs

LLMs perform robustly on text-based and multimodal problems but systematically fail in tasks requiring:

• Multi-step logical reasoning: Traversing graphs or chaining transformations.
• Precise mathematical computation: Calculating properties such as diameter, periphery, and shortest paths.
• Spatial, topological, and temporal perception: Understanding network layouts, dynamics, and structural invariants.

Standard LLMs tend to hallucinate or guess imprecise answers when challenged by bibliometric networks, molecular graphs, social networks, recommender-system bipartite graphs, or knowledge graphs. Prior works such as Toolformer (Schick et al., 2023) enabled general API usage but required manual prompt engineering and lacked support for domain-specific APIs for graph analytics.

2. Architecture and Workflow of Graph-ToolFormer

The Graph-ToolFormer pipeline consists of the following structured stages:

1. Instruction and Example Prompt Crafting: Manually curate instructions and seed prompt pairs for each distinct graph reasoning task.
2. Prompt Augmentation with ChatGPT: Use an in-context learning setup with ChatGPT (gpt-3.5-turbo) to generate thousands of variants from a small set of initial examples, each integrating explicit API calls.
3. Dataset Validation: Run all API calls embedded in ChatGPT outputs using real graph toolkits (e.g., NetworkX-based "toolx", pretrained GNNs) and retain only valid, ground-truth-matching pairs.
4. Fine-tuning a Causal LLM: Adapt a pre-trained LLM (GPT-J 6B-8bit or LLaMA) using the dataset so that the model learns to insert special <API> ... </API> tokens and determine which API (function/domain) and arguments to use at generation.
5. Inference and Execution Loop: During inference, mixed natural language and API calls are generated. A parser extracts these calls, executes them, caches/interpolates results, and completes the final output.

Key formulas formalize basic graph concepts:

• Graph $G=(V,E)$, order $n=|V|$, size $m=|E|$
• Distance $d(u,v)$ (shortest path),
• Diameter $D = \max_{u,v\in V} d(u,v)$,
• Radius $r = \min_{u\in V}\max_{v\in V} d(u,v)$,
• Periphery $P = \{w\in V | \max_{v\in V} d(w,v) = D\}$.

API calls utilize custom syntax; e.g., <API> GR(G, "toolx:diameter") -> r </API> for diameter, <API> GL("dataset-name", node_subset, link_subset) -> G </API> for graph loading.

Pseudocode for the inference loop:

prompt = user_input output = LLM.generate(prompt) for each <API>…</API> span in output: call, wants_result = parse_api_span(span) result = execute(call) # loads graph / calls GNN if wants_result: output = substitute(output, span, result) return output

3. Prompt Augmentation via ChatGPT

Prompt augmentation methodology employs ChatGPT to synthetically expand diverse prompt formats:

• Provide system-level instructions and 3–5 raw example pairs ("before/after") for each graph task.
• Instruct ChatGPT to generate thousands of new variants incorporating API calls.
• Examples include:
  • Graph loading: Transforming “The structure of the molecular graph of benzene ring contains a hexagon.” into “The structure of the [GL("benzene-ring")] molecular graph of benzene ring contains a hexagon.”
  • Graph properties: “The diameter of the lollipop graph is [GR(GL("lollipop"), "toolx:diameter") -> r].”
  • Node classification: “Paper #10 in Cora focuses on the topic of [GR(GL("cora"), "graph-bert:topic", paper#10) -> r].”

ChatGPT also increases prompt diversity by rephrasing inputs. Outputs are validated by executing API calls and discarding incorrect generations. A dataset of ≈2,803 valid graph-load prompts exemplifies this filtering approach.

4. Model Training and Optimization

Training is conducted over a large-scale dataset composed of:

• ≈2,800 graph-loading examples
• ≈13,000 property reasoning prompts
• ≈224,000 advanced task prompts
• 15 total datasets, for a cumulative ≈450,000 examples

Data is split such that 160 per task pool are reserved for testing; training exploits up to 1,600 per pool. The base model is GPT-J 6B, quantized to 8-bit using bitsandbytes, and adapted with LoRA (Low-Rank Adaptation) adapters. Optimization uses 8-bit AdamW (lr = $1\times10^{-5}$, weight_decay = 0.01) for 3 epochs (batch_size = 32). Generation applies beam search ($\text{beam}=5$), top-k (5), top-p (0.95), high temperature (1.9), and bounded output length (128 tokens).

Token-level cross-entropy loss guides training, including penalties for incorrect/missing API insertions and arguments. At inference, special <API> tokens trigger the API-insertion module, and arguments are defaulted or penalized if omitted.

5. Empirical Results and Benchmarking

Graph-ToolFormer is evaluated on annotated prompts (reproduction accuracy) and end-to-end execution. Metrics include ROUGE-{1,2,L,Lsum}, BLEU, brevity penalty, and API-generation accuracy (exact API call extraction).

Relevant benchmarks and results:

Task	ROUGE-1	BLEU	API Accuracy
Graph Loading	82.3	63.5	4.4%
Property Reason	94.6	91.5	80.0%
Paper Topic	≈100	≈100	≈100
Molecular Func.	>99.6	>99.6	100.0
Recommendation	97.5–99.9	93.1–100	85.6–100
Social Comm.	~99	~99	>95
Knowledge Graph	~92–98.7	~92–98	54–96.9

Failure analysis reveals error sources including token duplication, missing arguments, and absent API insertions. Ablation studies indicate moderate zero-shot generalization to unseen graphs but poor performance on completely novel reasoning types outside the training set. Fine-tuning for graph reasoning impairs generic LLM fluency on unrelated corpora.

6. Achievements, Challenges, and Future Directions

Graph-ToolFormer demonstrates that causal LLMs can be reliably fine-tuned to use domain-specific graph APIs when trained on appropriately augmented prompts. Released resources include prompt datasets from 15 benchmarks, pretrained graph models (toolx, Graph-Bert, SEG-Bert, KMeans, BPR, TransE), checkpoints, and an interface demo.

Outstanding challenges include:

• Transferability of GNNs: Every new (dataset, task) pair requires a distinct pretrained model—a unified pretraining strategy is an open area.
• Deeper integration: Current method treats APIs as black boxes; joint LLM–GNN training for access to GNN representations could enhance explainability.
• Catastrophic forgetting: As LLMs specialize in graph reasoning, they lose general text fluency; modular adapters or continual learning may mitigate this.
• Scalability: Efficient execution remains demanding; streaming subgraphs or sparse/approximate algorithms is a promising direction.
• Extensibility: Incorporating new API domains (e.g., physics simulators, GIS tools) ideally avoids re-tuning; possible solutions may include meta-learning the API-insertion logic.
• Graph scale challenges: Billion-edge graphs require scalable API designs (subgraph fetch, summaries).

Graph-ToolFormer thus closes a critical gap by enabling LLMs to perform precise, explainable reasoning over graph data through prompt-based, API-augmented interaction, supporting diverse scientific and industrial applications (Zhang, 2023).