---
title: Parallel Tool Calling Generation
url: https://www.emergentmind.com/topics/parallel-tool-calling-generation
type: topic
---

# Parallel Tool Calling Generation

Searching arXiv for recent papers on parallel tool calling generation and closely related tool-calling acceleration/orchestration.

Parallel tool calling generation is the study and practice of producing multiple structured tool invocations efficiently within large language model workflows, including both the generation of tool-call representations themselves and the orchestration patterns that allow independent calls to be issued, verified, or executed concurrently. In the recent literature, the topic spans several distinct but increasingly connected problem formulations: compiler-style planning over tool-call DAGs, generation-time acceleration of JSON or function-call arguments, factorized decoding of function names and arguments, generative retrieval of tool identities, synthetic-data construction for multi-tool traces, and protocol-agnostic execution layers that normalize heterogeneous tool interfaces. A central distinction is between methods that parallelize **tool execution** and methods that parallelize **tool-call generation**; recent work argues that both matter, because once tool use becomes multi-step and multi-turn, the cost of generating structured calls token by token can itself dominate end-to-end latency [2604.13519].

## 1. Parallel tool calling generation as a systems problem

In the literature, parallel tool calling generation is not a single algorithmic primitive but a layered systems problem. One line of work treats tool calls as a dependency graph and seeks to schedule independent nodes concurrently. Another line treats the structured text of tool calls—JSON objects, function signatures, argument lists—as a decoding problem whose sequential token dependence can be reduced. A third line reframes tool retrieval and invocation as next-token generation, making tool selection itself part of the language-model distribution rather than a separate retrieval stage [2312.04511].

Compiler-style formulations are prominent. LLMCompiler introduces a planner, task fetching unit, and executor, representing tool calls as tasks with placeholder variables such as `\$1` and dependencies inferred from variable usage, then executing dependency-free tasks in parallel [2312.04511]. DTA-Llama similarly converts successful tool-search paths into DAGs and trains a model to iteratively divide the current task into several parallel tool invocation sub-tasks and aggregate the invocation results to decide the next actions [2501.12432]. W&D extends the perspective to deep research agents, contrasting scaling **depth** by increasing sequential steps with scaling **width** by increasing the number of tool calls per reasoning step [2602.07359].

A separate but related distinction concerns whether the bottleneck lies in the runtime of external tools or in the model’s own structured generation. ToolSpec argues that prior work on parallel execution and overlap leaves untouched the part of the pipeline where the model must still generate JSON or function-call arguments token by token, and reports that on ToolBench with Qwen2.5-14B–Instruct about **80%** of end-to-end latency is spent on tool-call generation, while for Qwen2.5-72B–Instruct the share reaches **96%** [2604.13519]. This makes “parallel tool calling generation” broader than concurrent API dispatch: it includes token-level parallelism inside the decoder.

## 2. Architectural patterns: DAGs, planners, schedulers, and fused tools

A canonical architectural pattern represents tool use as a directed acyclic graph. In LLMCompiler, the planner emits a textual plan whose lines are tasks with tool names and arguments, and dependencies are encoded by placeholder references such as `"$i"` inside downstream arguments. The task fetching unit greedily dispatches all ready tasks whose dependencies have been satisfied, while the executor runs those tasks concurrently. The resulting latency model contrasts ReAct-style sequential execution,
\[
T^R = \sum_{i=1}^{N} \big( T_P^R(P_i) + T_E(E_i) \big),
\]
with a compiler-style formulation,
\[
T^C = \sum_{i=1}^{N} T_P^C(P_i) + \max_{k \in \{1,\dots,N\}} T_E(E_k),
\]
and a streaming formulation,
\[
T^{SC} = \sum_{i=1}^{N} T_P^C(P_i) + T_E(E_N), \quad T^{SC} \le T^C.
\]
The paper reports latency speedup of up to \(3.7\times\), cost savings of up to \(6.7\times\), and accuracy improvement of up to \(\sim 9\%\) compared to ReAct [2312.04511].

A related but distinct pattern is runtime tool fusion. “An LLM-Tool Compiler for Fused Parallel Function Calling” dynamically groups similar tools into fused composite tools before the main agent runs, so that one call from the model can expand into several primitive calls at execution time. On a large-scale Copilot platform, this approach achieves up to four times more parallel calls than existing methods, reducing token costs and latency by up to **40%** and **12%**, respectively [2405.17438]. The same paper reports that filter-operation parallelization can rise from values such as **17.04±1.79%** or **36.01±0.45%** under baseline settings to values such as **97.02±0.22%** or **99.41±0.04%** with the compiler, depending on model and prompting configuration [2405.17438].

W&D introduces a scheduler-centric view. It formalizes a sequential trace
\[
\tau_{\text{seq}} = \left\langle X, (R_1, A_1, O_1), \dots, (R_{T-1}, A_{T-1}, O_{T-1}), (R_T, \hat{Y}) \right\rangle
\]
and a parallel trace
\[
\tau_{\text{par}} = \left\langle X, (R_1, \mathcal{A}^{\text{par}}_1, \mathcal{O}^{\text{par}}_1), \dots, (R_{T-1}, \mathcal{A}^{\text{par}}_{T-1}, \mathcal{O}^{\text{par}}_{T-1}), (R_T, \hat{Y}) \right\rangle.
\]
The framework studies constant, ascending, descending, and automatic schedules for the number of calls per step \(m_t\), and finds on BrowseComp with GPT-5-Medium that a **Descending** scheduler yields **74** accuracy with **23.5** average turns, compared with **66** and **45.7** for **Constant 1 Tool**, and **68** and **23.8** for **Constant 3 Tools** [2602.07359]. This suggests that, for deep research agents, the parallelism problem is not merely “execute more calls” but “allocate width across the trajectory.”

## 3. Generation-side acceleration: schema-aware, factorized, and speculative decoding

The strongest recent shift in the literature is the claim that tool-call generation itself is highly structured and therefore amenable to specialized parallel decoding. ToolSpec formalizes tool-call generation as constrained decoding under a known schema. It analyzes API-Bank, ToolAlpaca, BFCLv2, and ToolBench and divides outputs into schema tokens, tool and parameter names from finite sets, and unconstrained parameter values. Because format adherence exceeds **99%** for models such as LLaMA-3.1 and Qwen2.5 on API-Bank and ToolAlpaca, ToolSpec models the process with a finite-state machine
\[
\mathcal{F} = (\mathcal{Q}, \Sigma_{\text{schema}}, \delta, q_0),
\]
with states \(q_t\) for tool name, \(q_p\) for parameter name, \(q_v\) for parameter value, and \(q_o\) for other text. Deterministic schema segments are drafted without LLM calls, while unconstrained spans use Token Recycling; multiple draft paths are verified in parallel through tree-structured attention [2604.13519].

The method is retrieval-augmented as well. ToolSpec stores historical tool invocations
\[
\mathcal{H} = \{(\boldsymbol{h}_j, \boldsymbol{y}_j)\}_{j=1}^{|\mathcal{H}|},
\]
retrieves top-\(k\) similar histories using cosine similarity, and attempts suffix matching
\[
\boldsymbol{y}_{t-L:t-1} = \boldsymbol{y}'_{m-L:m-1}
\]
to reuse the next \(n\) tokens of a previous invocation as a speculative continuation. The reported parameters are \(k=3\), \(L \in \{5,6,7\}\), and continuation lengths \(n \in \{32,16,8,8\}\) [2604.13519]. Across Qwen2.5, LLaMA-3.1/3.2, ToolLLaMA, and multiple datasets, ToolSpec achieves roughly **3.5×–4.2× speedup** over vanilla autoregressive decoding; for LLaMA-3.1-8B on API-Bank it reports \#MAT \(= 5.80\), **125.59 tokens/s**, and **4.45×** speedup versus vanilla \#MAT \(=1.00\) and **28.21 tokens/s** [2604.13519].

SimpleTool attacks the same generation bottleneck with a different factorization. It introduces **17 special tokens**, including `<function>`, `</function>`, `<argk>`, `</argk>`, and `<|null|>`, so that structured JSON scaffolding is compressed and function name plus arguments can be decoded in independent streams. It approximates
\[
p(a_1,\dots,a_m \mid f,\text{context}) \approx \prod_{i=1}^m p(a_i \mid f,\text{context}),
\]
and replaces baseline latency
\[
T_{\text{baseline}} = T_p + N \cdot T_d
\]
with
\[
T_{\text{ours}} \approx T_p + \max_{i \in \{1,\dots,H\}}(N_i)\cdot T_d.
\]
The paper reports a **4–6x reduction** in low-entropy tokens, **3–6x end-to-end speedup (up to 9.6x)**, and only **+8.2% parallelization overhead**. With quantization on consumer-grade GPU, ST-Qwen3-4B reaches **61.2ms P50 latency**, enabling **16 Hz** real-time control at 4B model scale [2603.00030].

These two approaches define different forms of generation-side parallelism. ToolSpec increases the **mean accepted tokens per step** by speculative verification over schema-aware and retrieved drafts [2604.13519]. SimpleTool changes the representation so that multiple heads share the same prefix KV cache and decode independent value spans concurrently [2603.00030]. A plausible implication is that the field now distinguishes at least three granularities of parallelism: tool-execution parallelism, call-level planning parallelism, and token-level generation parallelism.

## 4. Generative tool selection and unified tool abstractions

Parallel tool calling generation also depends on how tools are represented before execution. ToolGen turns both tool retrieval and tool calling into next-token generation by extending the vocabulary with one special token per tool. Given a large tool set \(D=\{d_1,\dots,d_N\}\), each tool is mapped through atomic indexing, yielding
\[
T = \{ \mathrm{Index}(d) \mid \forall d \in D \}.
\]
In the implementation described, a Llama-3-8B base vocabulary of **128,256** tokens is extended with **46,985** tool tokens to a final vocabulary size of **175,241**, and the system is evaluated with over **47,000 tools** [2410.03439]. Tool memorization, retrieval training, and agent tuning are all expressed as standard language-model objectives, with tool tokens emitted as ordinary next-token predictions. During action generation, ToolGen uses constrained beam search over a disjunctive trie so that the action token is guaranteed to be a valid tool token [2410.03439].

The generative design is sequential in the paper’s experiments, but it already supports multiple tools over multiple turns and provides a direct mechanism for top-\(k\) candidate tool generation. The paper explicitly notes that one could retain the top-\(k\) distinct tool tokens from beam search and interpret them as a bundle of parallel calls, or train an action format that emits a list of tool tokens such as `[<TOOL_A>, <TOOL_B>, <TOOL_C>]` [2410.03439]. This suggests that “parallel tool calling generation” can arise not only from faster decoding, but from representing tool identity itself as a discrete object over which the model can branch.

At the systems level, “Unified Tool Integration for LLMs: A Protocol-Agnostic Approach to Function Calling” provides a complementary abstraction. It defines a `Tool` object with `name`, `description`, `parameters` as JSON Schema, `callable`, `is_async`, and `parameters_model`, and a `ToolRegistry` that normalizes Python functions, OpenAPI services, MCP servers, and LangChain tools into one protocol-agnostic interface [2508.02979]. The associated executor supports two concurrency modes—thread and process—and thereby separates the LLM’s generation problem from the heterogeneity of underlying protocols.

The measured throughput differences are substantial. For **100 concurrent tool calls**, native functions achieve **3,060** calls/s with threads versus **1,287** with processes, while OpenAPI tools achieve **204** with threads versus **373** with processes, and MCP SSE tools **41** with threads versus **128** with processes [2508.02979]. This is execution-side rather than generation-side parallelism, but it conditions what kinds of generation are useful. A plausible implication is that unified schemas and validated parameter models make it easier for the model to emit multi-call structures that can later be dispatched concurrently across heterogeneous backends.

## 5. Data generation and supervision for parallel or multi-tool behavior

Parallel tool calling generation depends heavily on training traces. ToolFlow addresses the data-synthesis problem by building a graph \(G=(V,E)\) over tools, with edges induced by parameter–parameter and return–parameter similarity above a threshold \(\tau = 0.82\). Parameter and return descriptions are embedded with Sentence-BERT, and the graph is used to sample connected subsets of tools through a random-walk procedure [2410.18447]. The intent is to avoid random, weakly related tool combinations and instead synthesize coherent multi-turn dialogues in which tools naturally compose.

This matters directly for parallel calling because BFCL evaluation includes **Simple**, **Multiple**, **Parallel**, and **Parallel Multiple** categories, where the latter two require several tools to be called in one turn [2410.18447]. ToolFlow’s best configuration, combining graph-based sampling and planned generation, reaches **90.0%** on **Parallel** and **85.0%** on **Parallel Multiple**, which is reported as a **+11.5%** absolute gain versus base LLaMA-3.1-8B on the latter category [2410.18447]. The paper does not say that it explicitly annotates concurrency, but it does synthesize tool sets with shared-input and chained structures that align with parallel and sequential orchestration.

ITC makes the multi-tool typology explicit. It defines **Single Tool Calling** and **Multiple Tools Calling**, with the latter split into **Repeated**, **Parallel**, and **Nested** tasks. The dataset contains **3,571 real APIs** and **17,540 tool calling tasks**, of which **3,245** are multiple-tool tasks [2603.05515]. Model outputs must include a `Thought` field and an `Action` list of calls, for example
```json
{
  "Thought": "Your thought process when facing a problem.",
  "Action": "[api_name(parameter_1=\"\",parameter_2=\"\"),api_name(parameter_1=\"\",parameter_2=\"\")]"
}
```
and the canonical parallel example is `"[latest(from='EUR', to='USD,AUD'), currencies()]"` [2603.05515]. The paper evaluates Language Matching, Format Matching, Tool Selection precision/recall/F1, and Tool Invocation precision/recall/F1, thereby providing metrics that naturally penalize omitted or extra calls in parallel settings.

DiGiT-TC approaches the data problem from the opposite direction: stateless generation of complex multi-turn tool-calling interactions. It represents each assistant tool turn as a list of tool calls with `id` fields, builds a DAG over those calls, keeps the largest connected component,
\[
S = \argmax_{C_i \in C} |C_i|,
\]
and uses a novel implicit/explicit split in which some tool calls are hidden from the user request while others are surfaced. It further filters examples with Kendall
\[
\tau_b = \dfrac{C - D}{\sqrt{(C + D + X_0)(C + D + Y_0)}}
\]
to avoid trivial mappings between request text and plan structure [2601.19914]. Although it does not execute calls in parallel, its message format already allows each assistant tool turn to contain a sequence of calls that forms a DAG. This suggests a natural route for generating explicit parallel-call supervision in stateless settings: compute topological levels within each turn and annotate those levels as parallel batches.

## 6. Empirical patterns, misconceptions, and limitations

A recurring empirical pattern is that parallelism improves both efficiency and, in some cases, accuracy. On BrowseComp, W&D reports that with GPT-5-Medium and **3 tools/turn**, the system reaches **68%** accuracy with **23.8** average iterations, versus **66%** and **45.7** for **1 tool/turn** at the 100-iteration setting; on the full benchmark it reports **62.2%** accuracy with GPT-5-Medium, exceeding the **54.9%** reported for GPT-5-High in the cited comparison [2602.07359]. DTA-Llama reports average SoPR **66.1%** and SoWR **59.1%**, compared with ToolLLaMA ReAct at **37.9%** and **39.3%**, and ToolLLaMA DFSDT at **54.2%** and **47.1%**, while also using fewer inference steps [2501.12432]. These gains are not purely a matter of overlapping I/O; they arise from better global planning, broader perceptual scope, and reduced repeated LLM turns.

A common misconception is that parallel tool calling is equivalent to “calling many tools at once.” The literature is more restrictive. In LLMCompiler and DTA-Llama, parallelism is dependency-aware: only tasks whose prerequisites are satisfied are dispatched [2312.04511; 2501.12432]. In ToolSpec and SimpleTool, the term refers to parallel verification or parallel decoding streams even when only a single structured call is being produced [2604.13519; 2603.00030]. In ITC, **Parallel** tasks are explicitly distinguished from **Nested** tasks, where outputs of earlier calls feed later calls and therefore cannot simply be executed simultaneously [2603.05515].

Another misconception is that multi-tool ability automatically implies parallel-tool ability. EKTC/TOOL-ED demonstrates the opposite boundary condition. It uses a single `EmotionKnowledgebase` tool, learns when to invoke it, and shows that sparse, selective tool use improves empathetic response generation; but the paper explicitly states that its definition of tools “does not cover the process of using multiple tools” [2412.03096]. Likewise, ToolGen is trained on multi-tool trajectories yet executes them sequentially in the current design [2410.03439]. These cases show that parallel tool calling generation requires explicit representational and orchestration choices beyond generic tool-use fine-tuning.

Limitations recur across the literature. ToolSpec assumes strong schema adherence and a parseable schema; cold-start scenarios weaken retrieval-augmented speculation [2604.13519]. SimpleTool assumes a fixed maximum number of argument heads—**6** in the default configuration, covering **95.2%** of real APIs in its benchmarks—and may need overflow handling for larger signatures [2603.00030]. ToolGen underperforms on unseen tools and thereby exposes a limitation of generative retrieval when tools are out of vocabulary [2410.03439]. ToolRegistry notes serialization constraints in process mode and only partial coverage of provider-specific features [2508.02979]. W&D reports that open-source models show smaller gains from width scaling, which suggests that effective parallel tool use is not yet a universal emergent capability [2602.07359].

## 7. Research directions and synthesis

Recent work converges on the view that parallel tool calling generation has at least four interlocking abstractions. The first is the **graph abstraction**, where tool calls are nodes with typed dependencies and parallelism emerges from level sets or ready sets [2312.04511; 2501.12432]. The second is the **schema abstraction**, where much of a tool call is deterministic structure that can be drafted, skipped, or compressed [2604.13519; 2603.00030]. The third is the **vocabulary abstraction**, where tools themselves become atomic tokens or constrained outputs over a finite action space [2410.03439]. The fourth is the **registry abstraction**, where heterogeneous protocols are normalized into a common schema and execution interface [2508.02979].

From these abstractions, several design principles recur. Explicit schemas, whether JSON Schema, function signatures, or normalized parameter models, make both decoding and validation easier [2604.13519; 2508.02979]. DAG or levelized traces are a strong supervision format for training parallel planners [2501.12432; 2601.19914]. Multi-tool data must balance diversity with coherence; ToolFlow shows that graph-based sampling improves diversity metrics while planned generation improves coherence metrics, and both correlate positively with BFCL performance [2410.18447]. Width should be scheduled, not merely maximized; W&D’s descending policy indicates that early wide exploration followed by narrower exploitation can outperform constant-width and ascending strategies [2602.07359].

A plausible implication is that future systems will combine these strands rather than choosing among them. One could imagine a pipeline in which a generative retriever emits candidate tool tokens, a planner organizes them into a DAG, a schema-aware decoder or factorized representation emits the structured calls with high \#MAT or compressed heads, and a protocol-agnostic registry executes the resulting independent calls in a mode-aware concurrent executor. The existing literature already contains each component in isolation. Parallel tool calling generation, in this broader sense, is therefore becoming a unifying problem statement for LLM agents: not merely how to choose tools, and not merely how to run them concurrently, but how to represent, generate, verify, and schedule structured tool interactions so that multi-step systems are both faster and more reliable [2604.13519].

Source: https://www.emergentmind.com/topics/parallel-tool-calling-generation