GAP: Graph-Based Agent Planning with Parallel Tool Use and Reinforcement Learning

Abstract: Autonomous agents powered by LLMs have shown impressive capabilities in tool manipulation for complex task-solving. However, existing paradigms such as ReAct rely on sequential reasoning and execution, failing to exploit the inherent parallelism among independent sub-tasks. This sequential bottleneck leads to inefficient tool utilization and suboptimal performance in multi-step reasoning scenarios. We introduce Graph-based Agent Planning (GAP), a novel framework that explicitly models inter-task dependencies through graph-based planning to enable adaptive parallel and serial tool execution. Our approach trains agent foundation models to decompose complex tasks into dependency-aware sub-task graphs, autonomously determining which tools can be executed in parallel and which must follow sequential dependencies. This dependency-aware orchestration achieves substantial improvements in both execution efficiency and task accuracy. To train GAP, we construct a high-quality dataset of graph-based planning traces derived from the Multi-Hop Question Answering (MHQA) benchmark. We employ a two-stage training strategy: supervised fine-tuning (SFT) on the curated dataset, followed by reinforcement learning (RL) with a correctness-based reward function on strategically sampled queries where tool-based reasoning provides maximum value. Experimental results on MHQA datasets demonstrate that GAP significantly outperforms traditional ReAct baselines, particularly on multi-step retrieval tasks, while achieving dramatic improvements in tool invocation efficiency through intelligent parallelization. The project page is available at: https://github.com/WJQ7777/Graph-Agent-Planning.

• The paper presents a graph-based planning framework that decomposes queries into dependency-aware subtasks executed in parallel, improving both speed and accuracy.
• It leverages supervised fine-tuning and reinforcement learning to optimize tool use, reducing interaction turns and response lengths in multi-hop QA tasks.
• Experimental results show up to a 33.4% reduction in interaction turns and a 0.9% accuracy boost, highlighting its practical benefits for scalable autonomous agents.

Graph-Based Agent Planning with Parallel Tool Use and Reinforcement Learning

Introduction

The paper presents Graph-based Agent Planning (GAP), a paradigm for training LLM-based agents to perform dependency-aware reasoning and adaptive tool execution. GAP addresses the inefficiencies of sequential tool invocation in existing frameworks such as ReAct by enabling agents to decompose complex tasks into dependency graphs and execute independent sub-tasks in parallel. This approach leverages both supervised fine-tuning and reinforcement learning to optimize for correctness and computational efficiency, with empirical validation on multi-hop question answering (MHQA) benchmarks.

Figure 1: GAP decomposes tasks into dependency-aware subtasks, enabling parallel tool operations and efficient agent orchestration.

Methodology

Problem Formulation and Graph Construction

GAP formalizes task-solving as a graph-based planning problem. Given a query $q$ and a set of tools $\mathcal{T}$, the agent decomposes $q$ into atomic sub-tasks $S = \{s_1, ..., s_m\}$, each mapped to a node in a directed acyclic graph (DAG) $G = (V, E)$. Edges encode dependencies: $(v_i, v_j) \in E$ iff $s_j$ requires the output of $s_i$. Absence of an edge denotes independence, allowing parallel execution.

Sub-task identification and dependency analysis are performed via structured prompting and model inference. The agent outputs a graph representation, e.g.:

<graph> <node id="s1">search("capital of France")</node> <node id="s2">search("capital of Germany")</node> <node id="s3" depends="s1">search("population of {s1}")</node> <node id="s4" depends="s2">search("population of {s2}")</node> </graph>

Dependency-Aware Execution

GAP partitions the dependency graph into execution levels via topological sorting. All nodes in a level are executed in parallel, subject to dependency constraints. The agent generates tool call batches for each level, waits for all results, and proceeds to subsequent levels. This scheduling is formalized in the provided algorithm, which integrates planning, parallel tool invocation, and synthesis phases.

Training Pipeline

• Data Synthesis: GAP training data consists of 7,000 graph-based planning traces synthesized from MHQA datasets using GPT-4o, filtered for complexity, diversity, and length.
• Supervised Fine-Tuning (SFT): The Qwen2.5-3B-Instruct model is fine-tuned to internalize graph-based planning, minimizing a masked cross-entropy loss to prevent environmental noise propagation.
• Reinforcement Learning (RL): End-to-end RL further optimizes the agent for efficiency and correctness using DAPO and VeRL frameworks. The reward function is binary, based on answer correctness.

Experimental Results

Accuracy and Efficiency

GAP is evaluated on seven QA benchmarks, including HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle. The model demonstrates a 0.9% average improvement in multi-hop accuracy over state-of-the-art baselines. Notably, GAP achieves substantial reductions in interaction turns (up to 33.4%) and response length (up to 24.9%), with robust generalization to out-of-domain datasets.

Figure 2: GAP-3B achieves the best performance-cost trade-off on HotpotQA, combining high accuracy with low computational cost.

GAP's parallel execution strategy yields lower deployment costs, as measured by the cost-of-pass metric, and faster inference times. For example, on HotpotQA, GAP reduces the average number of turns from 2.27 (AFM-RL-3B) to 1.78 and response length from 554 to 416 tokens. Similar improvements are observed on 2WikiMultiHopQA.

Figure 3: GAP reduces response length and the number of turns required to solve problems on HotpotQA and 2WikiMultiHopQA.

Case Study

A representative case demonstrates GAP's ability to decompose a multi-hop query into independent sub-tasks, execute parallel searches, and synthesize results efficiently. The agent identifies shared occupations between two individuals by parallelizing career information retrieval and then comparing results, exemplifying the practical utility of graph-based planning.

Implications and Future Directions

GAP establishes graph-based dependency modeling as a critical direction for agentic reasoning. By bridging sequential TIR models and multi-agent coordination, GAP enables efficient, scalable, and generalizable tool-augmented intelligence. The demonstrated improvements in accuracy, efficiency, and cost-effectiveness have direct implications for real-world deployment of autonomous agents in domains requiring complex multi-step reasoning.

Theoretically, GAP's explicit modeling of dependencies and parallelism provides a foundation for further research in agentic RL, multi-agent simulation, and compositional reasoning. Future work may explore multi-objective reward functions, more expressive graph representations, and integration with larger-scale AFMs.

Conclusion

GAP introduces a principled framework for dependency-aware planning and parallel tool use in LLM-based agents. Through graph-based reasoning, supervised and reinforcement learning, and empirical validation, GAP achieves superior performance and efficiency on multi-hop QA tasks. The paradigm advances the state-of-the-art in agentic intelligence, with significant implications for both practical deployment and theoretical research in AI.