ToolComp: Benchmark for Multi-Tool Reasoning

• ToolComp is a benchmark for multi-tool, multi-step reasoning that evaluates not only final answers but also detailed intermediate reasoning processes.
• It uses a ReAct-style framework with human-verified process supervision to track each Thought, Action, and Observation during multi-tool execution.
• Empirical results reveal a significant gap between final-answer and intermediate reasoning accuracy, underscoring the need for process supervision in tool-use evaluation.

ToolComp is a benchmark for multi-tool, multi-step reasoning in LLMs, designed to evaluate not only whether a model reaches the correct final answer after using tools, but also whether its intermediate reasoning steps and tool actions are correct. It is positioned as a human-verified, text-only benchmark with both final-answer labels and process-supervision labels, addressing a gap left by earlier tool-use evaluations that emphasized single-tool invocation, shallow workflows, or final-answer-only scoring (Nath et al., 2 Jan 2025).

1. Motivation and conceptual scope

ToolComp was introduced in response to three limitations in prior tool-use evaluation. First, earlier benchmarks often concentrated on single-tool or shallow tool-use evaluation rather than chains of dependent tool calls. Second, benchmarks such as GAIA were described as including complex multi-hop tool-use questions but typically scored only the final answer, without step-wise labels that could identify where reasoning failed. Third, prior work on process supervision had been concentrated largely in mathematics and general reasoning rather than agentic tool use. ToolComp transfers process supervision into the tool-use setting, where correctness depends on a sequence of actions, tool outputs, and decisions (Nath et al., 2 Jan 2025).

This framing makes ToolComp more than a leaderboard dataset for function-calling accuracy. Its central object is the trajectory: the sequence of Thoughts, Actions, Action Inputs, and Observations that connects a prompt to a verifiable answer. The benchmark therefore treats tool use as a structured reasoning process rather than a single prediction.

A common misconception is that ToolComp is simply another function-calling benchmark. The benchmark definition suggests otherwise: prompts were designed so that solving them requires a chain of dependent tool calls, and the benchmark records whether individual steps in that chain are correct. Another misconception is that end-to-end accuracy alone is a sufficient proxy for tool competence. ToolComp explicitly rejects that assumption by pairing outcome supervision with process supervision and by evaluating models as judges of intermediate step quality as well as solvers of the underlying task (Nath et al., 2 Jan 2025).

2. Dataset composition and task design

ToolComp contains 485 human-verified prompts, 1731 step-level supervision labels, and 11 tools. The benchmark is divided into two subsets. ToolComp-Enterprise contains 287 examples and allows all 11 tools, with the intent of emulating richer enterprise-style agent settings. ToolComp-Chat contains 198 examples and allows only Google Search and Python, with the intent of reflecting general chatbot settings (Nath et al., 2 Jan 2025).

Split	Examples	Tool access
ToolComp-Enterprise	287	All 11 tools
ToolComp-Chat	198	Google Search and Python

The 11 supported tools are Date, Current Weather, Historical Weather, Calculator, Wiki Search, Google Search, Wolfram Alpha, Intraday Stock Info, Daily Stock Info, Stock Symbol Search, and Python. The tool set was chosen to cover retrieval, numerical computation, finance, general information lookup, and interactions between tools, while avoiding overly specialized capabilities (Nath et al., 2 Jan 2025).

Prompt creation followed two explicit constraints. Each prompt had to require a chain of dependent tool calls, and its final answer had to be programmatically verifiable. The construction pipeline began with manually created high-quality in-context examples with “processes” explaining how prompts are constructed. These examples were then used to generate candidate prompts with few-shot prompting. Candidate prompts were iteratively filtered, including the use of rejected examples as negative in-context examples, and the remaining prompts were refined by human annotators. This produced prompts intended to be challenging, unambiguous, and answerable with tools (Nath et al., 2 Jan 2025).

ToolComp adopts a ReAct-style format for trajectories. This format separates Thought, Action, Action Input, and Observation, making it amenable to step-wise supervision. The design choice is significant because it converts tool-use execution into a representation where intermediate states can be labeled, compared, and corrected.

3. Annotation protocol and process supervision

The benchmark’s annotation workflow is hybrid human-AI annotation. A policy model, FireFunction-V1, first produces an Action Plan, meaning a high-level plan for what tools to use and in what order. Human annotators validate or modify this plan. The policy model then generates ReAct steps, and annotators label each step as Correct or Incorrect for Thought, Action, and Action Input. Whenever a step is incorrect, annotators rewrite it into a correct version, and the corrected step is fed back into the trajectory so that annotation can continue from a repaired state (Nath et al., 2 Jan 2025).

This procedure yields three artifacts simultaneously: a valid full tool-use trajectory, a verified final answer, and per-step process labels together with corrected rewrites. ToolComp therefore provides both outcome supervision and process supervision. Outcome supervision concerns final answer correctness. Process supervision concerns step-wise labels over the trajectory, including corrected versions of incorrect steps (Nath et al., 2 Jan 2025).

The benchmark’s quality-control procedure is unusually extensive. All examples are manually inspected, and ambiguous prompts, wrong labels, or wrong answers are redone. The authors and three trusted annotators re-review all samples. The full benchmark is then run through GPT-4o, GPT-4 Turbo, Claude 3.5 Sonnet, and Llama 3.1 405B. Samples where all models disagree with the ground truth are re-refined, on the rationale that such cases may be mislabeled or exceptionally difficult (Nath et al., 2 Jan 2025).

This annotation protocol gives ToolComp a distinctive status among tool-use benchmarks. The process labels are not inferred automatically from environment state or derived solely from final correctness; they are human-verified at the level of individual Thoughts, Actions, and Action Inputs. A plausible implication is that ToolComp is especially suitable for research on trajectory auditing, reward modeling, and failure localization, because its supervision is aligned with the internal structure of an agent trace rather than with the endpoint alone.

4. Evaluation methodology

ToolComp evaluates models in two ways: final-answer evaluation and intermediate reasoning evaluation. Final-answer evaluation uses LLM grading rather than exact match, because many answers are structured and formatting should not dominate the score. A GPT-4 Turbo judge compares the question, the ground-truth answer, and the model answer, and labels the result as Incorrect, Correct, or Correct with Bad Formatting. The benchmark counts both Correct and Correct with Bad Formatting as wins. Exact match is also reported in the appendix, with normalization rules for sets, sorted lists, numbers within tolerance, and lowercased normalized strings (Nath et al., 2 Jan 2025).

Intermediate reasoning evaluation treats the model as a pairwise judge over process-supervision labels. For each sample, the model sees the human-corrected step and the original model-generated step and must choose which is better. To reduce position bias, the order is swapped, the model predicts twice, and ties are allowed. Scoring follows RewardBench-style aggregation, with loss scored as 0, tie as 0.5, and win as 1 (Nath et al., 2 Jan 2025).

This dual evaluation scheme matters because it separates two abilities that are often conflated: solving tool-use tasks end to end and recognizing better versus worse intermediate reasoning. ToolComp’s measurements indicate that these abilities overlap but are not identical. The paper reports a Pearson correlation between intermediate reasoning accuracy and final-answer accuracy of $r = 0.63$ with $p = 0.0084$, which indicates a statistically significant positive correlation but only a moderate one (Nath et al., 2 Jan 2025).

The benchmark also includes an explicit failure taxonomy in its appendix. Final-answer failures include Final Answer Missing Information, Called Incorrect Tool, Incorrect Tool Call Formatting, Terminated Early Unexpectedly, Hallucinated Information, Misunderstood Tool Info, Repeatedly Calling Same Tool, Action Plan Flawed, and Miscellaneous. For intermediate reasoning, mismatches are decomposed according to whether the difference lies in Thought, Action, or Action Input. A recurring issue is that models struggle especially when the main difference is in the Action Input, particularly in later trajectory stages (Nath et al., 2 Jan 2025).

5. Empirical results and the case for process supervision

ToolComp is difficult for contemporary models. Across evaluated model families, most models score below 50% overall on final-answer accuracy, and the average across evaluated models is 46.64%. The best reported final-answer result is o1-preview at 61.83%. Other strong results include GPT-4o (Aug 2024) at 58.68%, Claude 3.5 Sonnet at 58.03%, GPT-4 Turbo Preview at 57.61%, and Gemini 1.5 Pro (Aug 2024) at 56.61%. Weaker models include Llama 3.1 8B at 12.81% and Command R+ at 26.13% (Nath et al., 2 Jan 2025).

On intermediate reasoning, performance is substantially higher. The best pairwise judgment accuracy is o1-preview at 80.19%. Other notable results include Mistral Large 2 at 72.67%, GPT-4o (Aug 2024) at 72.61%, Llama 3.1 405B at 71.62%, and Gemini 1.5 Pro (Aug 2024) at 69.11%. The average is 67.54% (Nath et al., 2 Jan 2025).

These results support an important interpretation: models are better at judging step-wise correctness than at executing full tool-use tasks. This does not mean that process evaluation is easier in a trivial sense; rather, it suggests that recognition of better trajectories and generation of successful trajectories are related but distinct capabilities.

A central contribution of the ToolComp paper is its comparison between outcome-supervised reward models and process-supervised reward models. Synthetic data for this study were generated using Llama-3.1-8b-Instruct as the policy generator and GPT-4o as the critic. Dataset sizes were 11,654 for ORM, 14,932 for PRM, and 11,654 for SFT, after an initial generation of 75K prompts and retention of 17,369 final-answer-consistent prompts. Both ORM and PRM used a linear reward head on Llama-3.1-8b-Instruct with binary cross-entropy loss (Nath et al., 2 Jan 2025).

On base-model generations, PRM achieved 42.65% rank@1 accuracy, whereas ORM achieved 23.89%, a gap of about 19 percentage points. On fine-tuned model generations, PRM reached 60.25% rank@1 accuracy and exceeded ORM by about 11 percentage points. The paper’s headline conclusion is that PRMs generalize significantly better than ORMs for ranking tool-use trajectories (Nath et al., 2 Jan 2025).

Among PRM design choices, the best supervision granularity is Full Step with Observation, which attains 60.25% rank@1 accuracy. Other variants are Full Step without Observation at 55.43%, Sub Step with Observation at 49.48%, and Sub Step without Observation at 45.70%. For trajectory-level score aggregation, the best method is max at 60.25%, outperforming min at 23.68%, average at 25.74%, and product at 23.06% (Nath et al., 2 Jan 2025).

The paper interprets these findings as evidence that process supervision is especially suitable for tool-use reasoning because trajectories depend on many intermediate decisions, can recover from local mistakes, and incorporate dynamic feedback from tool outputs. Outcome-only supervision misses these internal distinctions. In ToolComp, process supervision is therefore not a secondary annotation layer but the main methodological claim.

6. Position within tool-use evaluation research and stated limitations

ToolComp occupies a specific niche within the broader literature on tool-using LLMs. Earlier work such as MultiTool-CoT demonstrated that chain-of-thought prompting can orchestrate multiple external tools in a single reasoning trajectory, reaching 85.85 accuracy on the 325-question test split of NumGLUE Task 2 and outperforming single-tool variants, but it did not provide ToolComp’s human-verified process-supervision framework (Inaba et al., 2023). This suggests a distinction between methods for multi-tool orchestration and benchmarks for diagnosing the quality of the resulting trajectories.

Subsequent benchmark work broadened the space of evaluation along different axes. CCTU evaluates tool use under 12 constraint categories spanning resource, behavior, toolset, and response dimensions, with 200 carefully curated cases and no model exceeding 20% Perfect Solve Rate under strict adherence to all constraints (Ye et al., 16 Mar 2026). ToolSense audits parametric tool retrieval and reports a knowledge-retrieval dissociation, including drops of about 50–64 percentage points on realistic retrieval queries relative to fully specified ToolBench-style benchmarks (Hathidara et al., 4 Jun 2026). Tool-DC addresses long-context tool-calling with a Try–Check–Retry framework and reports up to +25.10% average gains in extended noisy settings (Chen et al., 12 Mar 2026). TSCG instead treats tool schemas as a compilation problem and reports 52–57% token savings in scaling experiments while improving or preserving tool-use accuracy (Sakizli, 4 May 2026). Collectively, these works indicate that ToolComp’s distinctive contribution is not constraint modeling, retrieval diagnostics, long-context decomposition, or schema compilation, but human-verified process supervision for multi-tool reasoning.

The ToolComp paper states several limitations. It does not yet study reinforcement-learning policy optimization; the work is confined to reward models and ranking. The PRM and ORM experiments rely heavily on synthetic trajectories. The tool set is restricted mostly to retrieval and computation tools, and the benchmark is text-only, intentionally avoiding specialized modalities such as vision-oriented or translation-oriented agents (Nath et al., 2 Jan 2025).

These limitations define the benchmark’s scope rather than diminishing its role. ToolComp is best understood as a controlled, process-supervised evaluation environment for text-based multi-tool reasoning, with direct implications for reward modeling, trajectory ranking, and failure analysis. Its main conclusion is that multi-tool reasoning remains difficult for state-of-the-art models and that step-wise process supervision is a more informative signal than outcome-only supervision for evaluating and improving complex tool-use behavior (Nath et al., 2 Jan 2025).