Parallel Tool Calling Generation

• Parallel tool calling generation is a technique for efficiently producing, scheduling, and executing multiple structured tool invocations concurrently in language model workflows.
• It unifies compiler-style planning, schema-aware decoding, and protocol-agnostic execution to optimize dependency-aware multitool interactions.
• Recent studies demonstrate up to 4× decoding speedup and significant latency reductions by parallelizing both token-level generation and external tool execution.

Searching arXiv for papers on parallel tool calling generation and closely related tool-calling acceleration/orchestration.

Parallel tool calling generation is the study and practice of producing multiple structured tool invocations efficiently within LLM workflows, including both the generation of tool-call representations themselves and the orchestration patterns that allow independent calls to be issued, verified, or executed concurrently. In the recent literature, the topic spans several distinct but increasingly connected problem formulations: compiler-style planning over tool-call DAGs, generation-time acceleration of JSON or function-call arguments, factorized decoding of function names and arguments, generative retrieval of tool identities, synthetic-data construction for multi-tool traces, and protocol-agnostic execution layers that normalize heterogeneous tool interfaces. A central distinction is between methods that parallelize tool execution and methods that parallelize tool-call generation; recent work argues that both matter, because once tool use becomes multi-step and multi-turn, the cost of generating structured calls token by token can itself dominate end-to-end latency (Xia et al., 15 Apr 2026).

1. Parallel tool calling generation as a systems problem

In the literature, parallel tool calling generation is not a single algorithmic primitive but a layered systems problem. One line of work treats tool calls as a dependency graph and seeks to schedule independent nodes concurrently. Another line treats the structured text of tool calls—JSON objects, function signatures, argument lists—as a decoding problem whose sequential token dependence can be reduced. A third line reframes tool retrieval and invocation as next-token generation, making tool selection itself part of the language-model distribution rather than a separate retrieval stage (Kim et al., 2023).

Compiler-style formulations are prominent. LLMCompiler introduces a planner, task fetching unit, and executor, representing tool calls as tasks with placeholder variables such as \$1 and dependencies inferred from variable usage, then executing dependency-free tasks in parallel (Kim et al., 2023). DTA-Llama similarly converts successful tool-search paths into DAGs and trains a model to iteratively divide the current task into several parallel tool invocation sub-tasks and aggregate the invocation results to decide the next actions (Zhu et al., 21 Jan 2025). W&D extends the perspective to deep research agents, contrasting scaling depth by increasing sequential steps with scaling width by increasing the number of tool calls per reasoning step (Lin et al., 7 Feb 2026).

A separate but related distinction concerns whether the bottleneck lies in the runtime of external tools or in the model’s own structured generation. ToolSpec argues that prior work on parallel execution and overlap leaves untouched the part of the pipeline where the model must still generate JSON or function-call arguments token by token, and reports that on ToolBench with Qwen2.5-14B–Instruct about 80% of end-to-end latency is spent on tool-call generation, while for Qwen2.5-72B–Instruct the share reaches 96% (Xia et al., 15 Apr 2026). This makes “parallel tool calling generation” broader than concurrent API dispatch: it includes token-level parallelism inside the decoder.

2. Architectural patterns: DAGs, planners, schedulers, and fused tools

A canonical architectural pattern represents tool use as a directed acyclic graph. In LLMCompiler, the planner emits a textual plan whose lines are tasks with tool names and arguments, and dependencies are encoded by placeholder references such as "$i&quot;</code> inside downstream arguments. The task fetching unit greedily dispatches all ready tasks whose dependencies have been satisfied, while the executor runs those tasks concurrently. The resulting latency model contrasts ReAct-style sequential execution,</p> <p>$T^R = \sum_{i=1}^{N} \big( T_P^R(P_i) + T_E(E_i) \big),$$</p> <p>with a compiler-style formulation,</p> <p>$$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$</p> <p>and a streaming formulation,</p> <p>$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$</p> <p>The paper reports latency speedup of up to$$3.7\times$, cost savings of up to$6.7\times$, and accuracy improvement of up to$\sim 9\%$ compared to <a href="https://www.emergentmind.com/topics/react" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">ReAct</a> (<a href="/papers/2312.04511" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Kim et al., 2023</a>).</p> <p>A related but distinct pattern is runtime tool fusion. “An LLM-Tool Compiler for Fused Parallel Function Calling” dynamically groups similar tools into fused composite tools before the main agent runs, so that one call from the model can expand into several primitive calls at execution time. On a large-scale Copilot platform, this approach achieves up to four times more parallel calls than existing methods, reducing token costs and latency by up to <strong>40%</strong> and <strong>12%</strong>, respectively (<a href="/papers/2405.17438" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Singh et al., 2024</a>). The same paper reports that filter-operation parallelization can rise from values such as <strong>17.04±1.79%</strong> or <strong>36.01±0.45%</strong> under baseline settings to values such as <strong>97.02±0.22%</strong> or <strong>99.41±0.04%</strong> with the compiler, depending on model and prompting configuration (<a href="/papers/2405.17438" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Singh et al., 2024</a>).</p> <p>W&amp;D introduces a scheduler-centric view. It formalizes a sequential trace</p> <p>$\tau_{\text{seq}} = \left\langle X, (R_1, A_1, O_1), \dots, (R_{T-1}, A_{T-1}, O_{T-1}), (R_T, \hat{Y}) \right\rangle$</p> <p>and a parallel trace</p> <p>$\tau_{\text{par}} = \left\langle X, (R_1, \mathcal{A}^{\text{par}}_1, \mathcal{O}^{\text{par}}_1), \dots, (R_{T-1}, \mathcal{A}^{\text{par}}_{T-1}, \mathcal{O}^{\text{par}}_{T-1}), (R_T, \hat{Y}) \right\rangle.$$</p> <p>The framework studies constant, ascending, descending, and automatic schedules for the number of calls per step$$m_t$, and finds on BrowseComp with GPT-5-Medium that a Descending scheduler yields 74 accuracy with 23.5 average turns, compared with 66 and 45.7 for Constant 1 Tool, and 68 and 23.8 for Constant 3 Tools (Lin et al., 7 Feb 2026). This suggests that, for deep research agents, the parallelism problem is not merely “execute more calls” but “allocate width across the trajectory.”

3. Generation-side acceleration: schema-aware, factorized, and speculative decoding

The strongest recent shift in the literature is the claim that tool-call generation itself is highly structured and therefore amenable to specialized parallel decoding. ToolSpec formalizes tool-call generation as constrained decoding under a known schema. It analyzes API-Bank, ToolAlpaca, BFCLv2, and ToolBench and divides outputs into schema tokens, tool and parameter names from finite sets, and unconstrained parameter values. Because format adherence exceeds 99% for models such as LLaMA-3.1 and Qwen2.5 on API-Bank and ToolAlpaca, ToolSpec models the process with a finite-state machine

$$\mathcal{F} = (\mathcal{Q}, \Sigma_{\text{schema}}, \delta, q_0),$$

with states $$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$$0 for tool name, $$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$$1 for parameter name, $$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$$2 for parameter value, and $$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$$3 for other text. Deterministic schema segments are drafted without LLM calls, while unconstrained spans use Token Recycling; multiple draft paths are verified in parallel through tree-structured attention (Xia et al., 15 Apr 2026).

The method is retrieval-augmented as well. ToolSpec stores historical tool invocations

$$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$$4

retrieves top-$$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$$5 similar histories using cosine similarity, and attempts suffix matching

$$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$$6

to reuse the next $$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$$7 tokens of a previous invocation as a speculative continuation. The reported parameters are $$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$$8, $$T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),$$9, and continuation lengths $$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$0 (Xia et al., 15 Apr 2026). Across Qwen2.5, LLaMA-3.1/3.2, ToolLLaMA, and multiple datasets, ToolSpec achieves roughly 3.5×–4.2× speedup over vanilla autoregressive decoding; for LLaMA-3.1-8B on API-Bank it reports #MAT $$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$1, 125.59 tokens/s, and 4.45× speedup versus vanilla #MAT $$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$2 and 28.21 tokens/s (Xia et al., 15 Apr 2026).

SimpleTool attacks the same generation bottleneck with a different factorization. It introduces 17 special tokens, including <function>, </function>, <argk>, </argk>, and <|null|>, so that structured JSON scaffolding is compressed and function name plus arguments can be decoded in independent streams. It approximates

$$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$3

and replaces baseline latency

$$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$4

with

$$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$5

The paper reports a 4–6x reduction in low-entropy tokens, 3–6x end-to-end speedup (up to 9.6x), and only +8.2% parallelization overhead. With quantization on consumer-grade GPU, ST-Qwen3-4B reaches 61.2ms P50 latency, enabling 16 Hz real-time control at 4B model scale (Shi et al., 4 Feb 2026).

These two approaches define different forms of generation-side parallelism. ToolSpec increases the mean accepted tokens per step by speculative verification over schema-aware and retrieved drafts (Xia et al., 15 Apr 2026). SimpleTool changes the representation so that multiple heads share the same prefix KV cache and decode independent value spans concurrently (Shi et al., 4 Feb 2026). A plausible implication is that the field now distinguishes at least three granularities of parallelism: tool-execution parallelism, call-level planning parallelism, and token-level generation parallelism.

4. Generative tool selection and unified tool abstractions

Parallel tool calling generation also depends on how tools are represented before execution. ToolGen turns both tool retrieval and tool calling into next-token generation by extending the vocabulary with one special token per tool. Given a large tool set $$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$6, each tool is mapped through atomic indexing, yielding

$$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$7

In the implementation described, a Llama-3-8B base vocabulary of 128,256 tokens is extended with 46,985 tool tokens to a final vocabulary size of 175,241, and the system is evaluated with over 47,000 tools (Wang et al., 2024). Tool memorization, retrieval training, and agent tuning are all expressed as standard language-model objectives, with tool tokens emitted as ordinary next-token predictions. During action generation, ToolGen uses constrained beam search over a disjunctive trie so that the action token is guaranteed to be a valid tool token (Wang et al., 2024).

The generative design is sequential in the paper’s experiments, but it already supports multiple tools over multiple turns and provides a direct mechanism for top-$$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$8 candidate tool generation. The paper explicitly notes that one could retain the top-$$T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.$$9 distinct tool tokens from beam search and interpret them as a bundle of parallel calls, or train an action format that emits a list of tool tokens such as [<TOOL_A>, <TOOL_B>, <TOOL_C>] (Wang et al., 2024). This suggests that “parallel tool calling generation” can arise not only from faster decoding, but from representing tool identity itself as a discrete object over which the model can branch.

At the systems level, “Unified Tool Integration for LLMs: A Protocol-Agnostic Approach to Function Calling” provides a complementary abstraction. It defines a Tool object with name, description, parameters as JSON Schema, callable, is_async, and parameters_model, and a [ToolRegistry](https://www.emergentmind.com/topics/toolregistry) that normalizes Python functions, OpenAPI services, MCP servers, and LangChain tools into one protocol-agnostic interface (Ding et al., 5 Aug 2025). The associated executor supports two concurrency modes—thread and process—and thereby separates the LLM’s generation problem from the heterogeneity of underlying protocols.

The measured throughput differences are substantial. For 100 concurrent tool calls, native functions achieve 3,060 calls/s with threads versus 1,287 with processes, while OpenAPI tools achieve 204 with threads versus 373 with processes, and MCP SSE tools 41 with threads versus 128 with processes (Ding et al., 5 Aug 2025). This is execution-side rather than generation-side parallelism, but it conditions what kinds of generation are useful. A plausible implication is that unified schemas and validated parameter models make it easier for the model to emit multi-call structures that can later be dispatched concurrently across heterogeneous backends.

5. Data generation and supervision for parallel or multi-tool behavior

Parallel tool calling generation depends heavily on training traces. ToolFlow addresses the data-synthesis problem by building a graph $3.7\times$0 over tools, with edges induced by parameter–parameter and return–parameter similarity above a threshold $3.7\times$1. Parameter and return descriptions are embedded with Sentence-BERT, and the graph is used to sample connected subsets of tools through a random-walk procedure (Wang et al., 2024). The intent is to avoid random, weakly related tool combinations and instead synthesize coherent multi-turn dialogues in which tools naturally compose.

This matters directly for parallel calling because BFCL evaluation includes Simple, Multiple, Parallel, and Parallel Multiple categories, where the latter two require several tools to be called in one turn (Wang et al., 2024). ToolFlow’s best configuration, combining graph-based sampling and planned generation, reaches 90.0% on Parallel and 85.0% on Parallel Multiple, which is reported as a +11.5% absolute gain versus base LLaMA-3.1-8B on the latter category (Wang et al., 2024). The paper does not say that it explicitly annotates concurrency, but it does synthesize tool sets with shared-input and chained structures that align with parallel and sequential orchestration.

ITC makes the multi-tool typology explicit. It defines Single Tool Calling and Multiple Tools Calling, with the latter split into Repeated, Parallel, and Nested tasks. The dataset contains 3,571 real APIs and 17,540 tool calling tasks, of which 3,245 are multiple-tool tasks (Zhang et al., 21 Jan 2026). Model outputs must include a Thought field and an Action list of calls, for example $3.7\times$4 and the canonical parallel example is "[latest(from='EUR', to='USD,AUD'), currencies()]" (Zhang et al., 21 Jan 2026). The paper evaluates Language Matching, Format Matching, Tool Selection precision/recall/F1, and Tool Invocation precision/recall/F1, thereby providing metrics that naturally penalize omitted or extra calls in parallel settings.

DiGiT-TC approaches the data problem from the opposite direction: stateless generation of complex multi-turn tool-calling interactions. It represents each assistant tool turn as a list of tool calls with id fields, builds a DAG over those calls, keeps the largest connected component,

$3.7\times$2

and uses a novel implicit/explicit split in which some tool calls are hidden from the user request while others are surfaced. It further filters examples with Kendall

$3.7\times$3

to avoid trivial mappings between request text and plan structure (Crouse et al., 6 Jan 2026). Although it does not execute calls in parallel, its message format already allows each assistant tool turn to contain a sequence of calls that forms a DAG. This suggests a natural route for generating explicit parallel-call supervision in stateless settings: compute topological levels within each turn and annotate those levels as parallel batches.

6. Empirical patterns, misconceptions, and limitations

A recurring empirical pattern is that parallelism improves both efficiency and, in some cases, accuracy. On BrowseComp, W&D reports that with GPT-5-Medium and 3 tools/turn, the system reaches 68% accuracy with 23.8 average iterations, versus 66% and 45.7 for 1 tool/turn at the 100-iteration setting; on the full benchmark it reports 62.2% accuracy with GPT-5-Medium, exceeding the 54.9% reported for GPT-5-High in the cited comparison (Lin et al., 7 Feb 2026). DTA-Llama reports average SoPR 66.1% and SoWR 59.1%, compared with ToolLLaMA ReAct at 37.9% and 39.3%, and ToolLLaMA DFSDT at 54.2% and 47.1%, while also using fewer inference steps (Zhu et al., 21 Jan 2025). These gains are not purely a matter of overlapping I/O; they arise from better global planning, broader perceptual scope, and reduced repeated LLM turns.

A common misconception is that parallel tool calling is equivalent to “calling many tools at once.” The literature is more restrictive. In LLMCompiler and DTA-Llama, parallelism is dependency-aware: only tasks whose prerequisites are satisfied are dispatched (Kim et al., 2023, Zhu et al., 21 Jan 2025). In ToolSpec and SimpleTool, the term refers to parallel verification or parallel decoding streams even when only a single structured call is being produced (Xia et al., 15 Apr 2026, Shi et al., 4 Feb 2026). In ITC, Parallel tasks are explicitly distinguished from Nested tasks, where outputs of earlier calls feed later calls and therefore cannot simply be executed simultaneously (Zhang et al., 21 Jan 2026).

Another misconception is that multi-tool ability automatically implies parallel-tool ability. EKTC/TOOL-ED demonstrates the opposite boundary condition. It uses a single EmotionKnowledgebase tool, learns when to invoke it, and shows that sparse, selective tool use improves empathetic response generation; but the paper explicitly states that its definition of tools “does not cover the process of using multiple tools” (Cao et al., 2024). Likewise, ToolGen is trained on multi-tool trajectories yet executes them sequentially in the current design (Wang et al., 2024). These cases show that parallel tool calling generation requires explicit representational and orchestration choices beyond generic tool-use fine-tuning.

Limitations recur across the literature. ToolSpec assumes strong schema adherence and a parseable schema; cold-start scenarios weaken retrieval-augmented speculation (Xia et al., 15 Apr 2026). SimpleTool assumes a fixed maximum number of argument heads—6 in the default configuration, covering 95.2% of real APIs in its benchmarks—and may need overflow handling for larger signatures (Shi et al., 4 Feb 2026). ToolGen underperforms on unseen tools and thereby exposes a limitation of generative retrieval when tools are out of vocabulary (Wang et al., 2024). ToolRegistry notes serialization constraints in process mode and only partial coverage of provider-specific features (Ding et al., 5 Aug 2025). W&D reports that open-source models show smaller gains from width scaling, which suggests that effective parallel tool use is not yet a universal emergent capability (Lin et al., 7 Feb 2026).

7. Research directions and synthesis

Recent work converges on the view that parallel tool calling generation has at least four interlocking abstractions. The first is the graph abstraction, where tool calls are nodes with typed dependencies and parallelism emerges from level sets or ready sets (Kim et al., 2023, Zhu et al., 21 Jan 2025). The second is the schema abstraction, where much of a tool call is deterministic structure that can be drafted, skipped, or compressed (Xia et al., 15 Apr 2026, Shi et al., 4 Feb 2026). The third is the vocabulary abstraction, where tools themselves become atomic tokens or constrained outputs over a finite action space (Wang et al., 2024). The fourth is the registry abstraction, where heterogeneous protocols are normalized into a common schema and execution interface (Ding et al., 5 Aug 2025).

From these abstractions, several design principles recur. Explicit schemas, whether JSON Schema, function signatures, or normalized parameter models, make both decoding and validation easier (Xia et al., 15 Apr 2026, Ding et al., 5 Aug 2025). DAG or levelized traces are a strong supervision format for training parallel planners (Zhu et al., 21 Jan 2025, Crouse et al., 6 Jan 2026). Multi-tool data must balance diversity with coherence; ToolFlow shows that graph-based sampling improves diversity metrics while planned generation improves coherence metrics, and both correlate positively with BFCL performance (Wang et al., 2024). Width should be scheduled, not merely maximized; W&D’s descending policy indicates that early wide exploration followed by narrower exploitation can outperform constant-width and ascending strategies (Lin et al., 7 Feb 2026).

A plausible implication is that future systems will combine these strands rather than choosing among them. One could imagine a pipeline in which a generative retriever emits candidate tool tokens, a planner organizes them into a DAG, a schema-aware decoder or factorized representation emits the structured calls with high #MAT or compressed heads, and a protocol-agnostic registry executes the resulting independent calls in a mode-aware concurrent executor. The existing literature already contains each component in isolation. Parallel tool calling generation, in this broader sense, is therefore becoming a unifying problem statement for LLM agents: not merely how to choose tools, and not merely how to run them concurrently, but how to represent, generate, verify, and schedule structured tool interactions so that multi-step systems are both faster and more reliable (Xia et al., 15 Apr 2026).