WildToolBench: Real-World LLM Tool Benchmark

• WildToolBench is a benchmark for evaluating multi-turn, multi-step LLM tool-use based on real-world user behavior, emphasizing compositionality, vagueness, and variability.
• It employs a rigorous four-stage curation process that includes real-log extraction, tool set construction, multi-agent simulation, and quality control to create authentic dialogue scenarios.
• Empirical results across 57 LLMs reveal low session accuracy, underscoring challenges in orchestrating tool calls, long-context inference, and policy adaptation in dynamic interactions.

Searching arXiv for the specified paper and any directly relevant context. WildToolBench is a benchmark for LLM tool-use that is explicitly grounded in real-world user behavior patterns rather than simulated “perfect users” or hand-crafted single-step tasks. It was introduced in “Benchmarking LLM Tool-Use in the Wild” (Yu et al., 13 Feb 2026) to evaluate multi-turn, multi-step tool-use under three user-behavior challenges: compositional tasks, implicit intent, and instruction transition. The benchmark is designed around the claim that the central difficulty of practical tool-use is not synthetic task complexity in isolation, but the compositionality, vagueness, and variability of everyday interactions. Its empirical evaluations over 57 LLMs report that no model exceeds 15% session accuracy, framing WildToolBench as a robustness-oriented testbed for agentic behavior under realistic dialogue conditions (Yu et al., 13 Feb 2026).

1. Conceptual basis and motivating challenges

WildToolBench is motivated by the observation that fulfilling user needs through LLM multi-turn, multi-step tool-use is rarely a straightforward process. The benchmark identifies three key challenges from user behavior. The first is compositional tasks, in which real users issue compound requests that bundle multiple simple subtasks, such as finding popular movies, fetching ratings, and preparing a slide deck. These requests require non-linear orchestration of API calls, including trees of calls, rather than simple linear chaining (Yu et al., 13 Feb 2026).

The second challenge is implicit intent. In multi-turn dialogues, users may omit critical information, refer back through pronouns such as “one of them,” or rely on long-range context such as “what I asked two turns ago.” Under this condition, the model must infer missing parameters and latent goals by tracking context, coreference, and long-range dependencies. The third challenge is instruction transition, where task requests, clarifications, and casual conversation are interleaved within the same session. The agent must therefore switch policies dynamically: it may need to invoke a tool, ask a follow-up question, or answer without tools depending on the current turn type (Yu et al., 13 Feb 2026).

These design premises position WildToolBench against two tendencies in earlier tool-use evaluation: simulating idealized users and constructing artificially complex but narrowly specified tasks. The benchmark’s stated philosophy is that “The true challenge for LLM tool-use is not synthetic complexity, but the compositionality, vagueness, and variability inherent in everyday user interactions.” This suggests a shift from evaluating isolated tool-call competence toward evaluating tool-use as situated interaction.

2. Benchmark construction and curation pipeline

WildToolBench curates 256 realistic multi-turn scenarios comprising 1,024 total tasks through a four-stage pipeline (Yu et al., 13 Feb 2026). The first stage is Seed Scenario Extraction. Real user logs are uniformly sampled as few-shot examples, and their patterns are summarized into the three challenges. These samples, without leaking private data, are then used to prompt LLMs to generate draft scenarios.

The second stage is Tool Set Construction. The process starts from 1,600+ publicly available APIs (ToolAlpaca). Expert engineers verify, clean, and select approximately 400 tool lists, covering approximately 1,600 APIs total. The third stage is Multi-Agent Simulation & Annotation. A User-Agent generates four turns per scenario, covering single-tool, multi-tool, clarify, and chat behaviors. An Assistant-Agent executes tool calls; each call is automatically checked for syntax and types and then manually validated. Human experts annotate tool dependencies to build DAGs for orchestration metrics and inspect both contextual inference and policy switches (Yu et al., 13 Feb 2026).

The fourth stage is Quality Control. The benchmark undergoes four rounds of peer-review on random 20% samples each, with review focusing on natural language variability, policy transitions, and tool correctness. The final dataset is reported as 256 dialogues, 1,024 tasks, 100% human-verified (Yu et al., 13 Feb 2026).

This construction protocol is methodologically notable because it combines real-log grounding, synthetic scenario drafting, multi-agent generation, automatic checking, and expert review. A plausible implication is that WildToolBench is intended to occupy an intermediate position between fully naturalistic logged data and fully synthetic benchmark generation.

3. Dataset structure, task taxonomy, and tool coverage

Each user turn in WildToolBench is labeled as one of four task types: g_single, g_multi, g_clarify, and g_chat (Yu et al., 13 Feb 2026). The first denotes single-tool invocation tasks. The second denotes multi-step, multi-tool tasks and is further subdivided into sequential, parallel, and mixed topologies. The third captures requests that require the agent to ask for missing parameters. The fourth consists of casual, tool-free conversational turns.

The benchmark’s tool inventory spans 1,600 APIs across 8 major domains and 24 subcategories. Examples listed in the benchmark include Search & Listing (movies, news, products), Weather & Location, Document Operations (metadata, summarization), Media (images, audio), Financial & Data Analysis (calculators, charts), and Utilities (translation, unit conversion), along with other domains such as health, social, and government. Parameter types cover String, Integer, Float, Boolean, Enum, Array, Object, Nested (Yu et al., 13 Feb 2026).

The benchmark also reports several aggregate statistics. It contains 256 dialogue sessions and 1,024 tasks, with 4 tasks per session. The average turns per dialogue is 5.27, and the average tool-call steps per task is 1.92, compared with 1.68 in BFCL-V2. The approximate task-type distribution is reported as $P_{g_{single}} = 28.0\%$, $P_{g_{seq\_multi}} = 25.0\%$, $P_{g_{par\_multi}} = 15.7\%$, $P_{g_{clarify}} = 16.0\%$, and $P_{g_{chat}} = 15.3\%$. For hidden-intent challenges, the distribution is Partial Information: 16.0%, Coreferential Reference: 31.4%, and Long-Range Dependency: 2.4% (Yu et al., 13 Feb 2026).

Category	Definition
g_single	Single-tool invocation tasks
g_multi	Multi-step, multi-tool tasks
g_clarify	Requests requiring the agent to ask for missing parameters
g_chat	Casual, tool-free conversational turns

This taxonomy is central to the benchmark’s interpretability. Rather than evaluating a single undifferentiated notion of “tool-use,” WildToolBench decomposes performance into tool execution, clarification behavior, and non-tool conversational control.

4. Evaluation metrics and protocol

WildToolBench measures both task-level and session-level accuracy. For tool-use turns, the primary metric is accuracy defined as the number of correct tool-based responses over the total number of tool-use turns. Session accuracy is defined analogously over dialogue-level predictions, requiring all four tasks in a dialogue to be correct (Yu et al., 13 Feb 2026).

For compositional tasks, the benchmark introduces two secondary metrics. The first is Accomplishment Progress Rate (AP), defined as the fraction of correctly executed tool nodes over the total number of gold tool nodes. The second is Optimal Path Rate (OP). To compute OP, all valid execution paths in the task’s DAG are enumerated; if $L^*$ denotes the minimum-depth path length, OP is the fraction of executed paths of depth $L^*$ among all valid paths (Yu et al., 13 Feb 2026). AP therefore evaluates partial completion at the node level, while OP evaluates whether execution follows an optimal orchestration path under DAG constraints.

The evaluation protocol combines automatic and manual procedures. Under Automatic Matching, an incremental tree-matching algorithm locates each assistant API call in the enumerated decision trees, and syntax, parameter types, and nominal correctness are checked automatically. Under Manual Annotation, all gold tool call trajectories, DAG dependencies, and hidden-intent annotations are human-verified. In ambiguous cases, including proactive clarifications and ambiguous pronouns, expert raters adjudicate correctness (Yu et al., 13 Feb 2026).

This protocol is significant because it evaluates not only endpoint correctness but also structural alignment with the gold decision process. In that sense, WildToolBench treats tool-use as a trajectory-level problem rather than solely a final-answer problem.

5. Empirical results across models and task regimes

The benchmark evaluates 57 LLMs and reports that no model exceeds 15% session accuracy. Task-level accuracy is reported as mostly below 60%. Proprietary general-purpose models outperform open-source models, and reasoning-enhanced variants such as “Thinking” outperform their base counterparts (Yu et al., 13 Feb 2026).

Among the reported models, the highest session accuracies are Gemini-2.0-Thinking: 14.45%, Gemini-2.5-Pro: 14.06%, and Claude-4-Sonnet: 12.50%. At the lower end, the benchmark lists Doubao-1.5: 0.78%, Llama-3.3-70B-Instruct: 0.39%, and Hammer2.1-1.5B: 0.00% (Yu et al., 13 Feb 2026).

Model	Session Accuracy (%)
Gemini-2.0-Thinking	14.45
Gemini-2.5-Pro	14.06
Claude-4-Sonnet	12.50
Doubao-1.5	0.78
Llama-3.3-70B-Instruct	0.39
Hammer2.1-1.5B	0.00

Performance varies substantially across task subcategories. For sequential multi-tool tasks, task accuracy reaches up to 54.78% (A Claude 4). Parallel multi-tool performance is described as in the similar high-20s % range. Mixed topologies that combine serial and parallel structure drop to 16.67–25.00%. The OP Rate peaks at 42.74%, and the AP Rate peaks at 60.38%, leading to the conclusion that orchestrating mixed topologies remains a major deficiency (Yu et al., 13 Feb 2026).

For hidden-intent strategies, accuracy on Partial Information is 46–60%, on Coreferential Reference is 50–60%, and on Long-Range Dependency is only 30–45%, which is described as the largest performance gap across models. For instruction transitions, if $T$ denotes the number of task-type transitions in a four-task session, with $T \in \{0,1,2,3\}$, accuracy falls monotonically with $T$, with up to 30% drop when $P_{g_{seq\_multi}} = 25.0\%$0. The benchmark attributes this to self-conditioning, in which over-use of the previous policy biases subsequent decisions (Yu et al., 13 Feb 2026).

Taken together, these findings indicate that the principal failure modes are not exhausted by simple API invocation errors. The difficult cases are those requiring topology-sensitive orchestration, long-context inference, and rapid policy adaptation across heterogeneous turns.

6. Error characterization, model profiles, and research implications

The benchmark’s error analysis reports that action-level errors dominate. The named error classes include “Wrong Name” (8–30%), “Missing Info” (3–19%), and “Redundant Call” (13–23%), with the exact ranges varying by model. By contrast, parameter errors, including type mismatches and hallucinations, are reported as consistently low at approximately 2–7% (Yu et al., 13 Feb 2026).

Two model profiles are identified. A “Cautious” profile exhibits high refusal and low wrong-call rate. An “Eager” profile exhibits low refusal and high wrong-call rate (Yu et al., 13 Feb 2026). This characterization indicates that aggregate accuracy alone does not capture the qualitative trade-off between under-acting and over-acting in tool-use systems.

The benchmark also lists several recommendations for improving robustness. These include integrating explicit planning modules to build tool-call DAGs and select optimal paths, strengthening long-context encoding and coreference resolution to handle hidden intent, implementing dynamic policy-switching mechanisms to decide when to tool-call versus clarify versus chat, and incorporating uncertainty estimation to decide when to ask clarifying questions (Yu et al., 13 Feb 2026). These recommendations align directly with the three motivating challenges and indicate that improvement may require architectural and control-policy modifications rather than simple scaling of base models.

Future directions proposed in the benchmark include scaling via semi-automatic synthetic data combined with human-in-the-loop validation, developing benchmarks for longer-horizon tasks (8+ subtasks, hierarchical goals), evaluating multi-modal tools (vision, speech) and their orchestration, and exploring fine-tuning RL agents on WildToolBench to close the gap in policy adaptation (Yu et al., 13 Feb 2026). A plausible implication is that WildToolBench is intended not merely as a score-reporting benchmark but as a structured rubric for subsequent agent design, especially in settings where user behavior is heterogeneous, under-specified, and interactionally unstable.