ToolMaze: Benchmarking Robust Tool-Integrated Reasoning

Updated 6 July 2026

ToolMaze is a benchmark that assesses tool-integrated reasoning by simulating both explicit and implicit tool failures in controlled DAG environments.
It leverages a 2x2 perturbation taxonomy to differentiate between transient and permanent failures, enabling systematic analysis of recovery behaviors.
Empirical results reveal a significant gap between basic task execution and anomaly recovery, underscoring the need for advanced dynamic replanning in AI agents.

Searching arXiv for ToolMaze and closely related tool-use benchmarks to ground the article in current papers. ToolMaze is a benchmark for evaluating Tool-Integrated Reasoning (TIR) agents under realistic tool failures and non-linear tool dependencies, rather than only under “happy path” execution. It was introduced to measure whether an agent can detect anomalies, backtrack, and dynamically replan when tools fail or return corrupted outputs, in contrast to prior evaluations that largely assume stable and truthful tools (Zhu et al., 4 Jun 2026). The benchmark is organized along two controlled dimensions—DAG-based topological complexity and a $2 \times 2$ perturbation taxonomy—so that recovery behavior can be analyzed separately from ordinary task execution. In this formulation, robust tool use is treated as a distinct capability: not merely invoking tools correctly, but recognizing failure modes, selecting alternative paths, and recovering efficiently under uncertainty (Zhu et al., 4 Jun 2026).

1. Concept and motivation

ToolMaze was designed to address two gaps in prior tool-use evaluation. First, earlier benchmarks often leave the recovery space underspecified, making it difficult to determine whether a successful trajectory reflects genuine replanning or accidental substitution. Second, failures are often injected randomly, which obscures systematic comparison of recovery efficiency and path selection. ToolMaze instead makes both task topology and failure placement explicit, enabling controlled evaluation of dynamic path discovery and anomaly recovery (Zhu et al., 4 Jun 2026).

The benchmark is positioned against evaluations such as API-Bank, ToolLLM, ToolQA, and StableToolBench, which the ToolMaze paper characterizes as primarily testing straightforward tool invocation, static or shallow chains, environments without injected failures, or random fault settings that do not isolate recovery behavior (Zhu et al., 4 Jun 2026). This framing places ToolMaze within a broader shift in the literature from final-answer-centric tool evaluation toward process-sensitive and failure-aware analysis. ToolComp, for example, emphasizes step-wise supervision for multi-step multi-tool reasoning and shows that intermediate reasoning quality provides signal beyond final-answer accuracy (Nath et al., 2 Jan 2025). ToolMaze extends that concern from process correctness to recovery under perturbation.

A central claim of ToolMaze is that dynamic replanning is not reducible to ordinary tool execution. The reported scaling analysis shows that, for open-weight models, baseline non-perturbed Task Success Rate improves by 17.85 percentage points per order of magnitude in model size, whereas Perturbation Recovery Rate improves by only 4.88 percentage points per order of magnitude. The paper summarizes this as agentic fault-tolerance improving $3.66\times$ slower than basic task execution, suggesting a separate bottleneck in anomaly detection and recovery (Zhu et al., 4 Jun 2026).

2. Benchmark design: topology and perturbation

ToolMaze defines each benchmark instance by a coordinate $(\mathcal{C}, \mathcal{P})$ , where $\mathcal{C}$ denotes topological task complexity and $\mathcal{P}$ denotes perturbation mode (Zhu et al., 4 Jun 2026). This two-dimensional structure is the benchmark’s organizing principle.

DAG-based topological complexity

Tasks are constructed as directed acyclic graphs of tool calls. The DAG is built before natural-language task wording, so the valid solution space is known in advance (Zhu et al., 4 Jun 2026). ToolMaze uses a curated corpus of 270 deterministic simulated tools, manually designed from real-world APIs. These are divided into three functional categories—Source, Processor, and Action—and span six application domains: Financial, Travel, Office, Shopping, IoT, and General. The corpus contains 123 Action tools, 78 Source tools, and 69 Processor tools (Zhu et al., 4 Jun 2026).

A valid DAG must contain at least one Source node and at least one Action node, with Processor nodes used as intermediates; chaining must remain domain-consistent and acyclic (Zhu et al., 4 Jun 2026). The benchmark defines four complexity levels:

Level	Structure	Intended challenge
$\mathcal{C}1$	linear, single path, no alternatives	basic execution and failure handling
$\mathcal{C}2$	1-to-N alternatives	single-step rerouting
$\mathcal{C}3$	many-to-many multi-path	combinatorial recovery space
$\mathcal{C}4$	integrated multi-branch	multiple branching points

The construction pipeline has three stages. First, an LLM-based “DAG Architect” samples tools and builds a DAG matching the target complexity level; structural validation checks acyclicity and topology, while semantic validation uses a secondary LLM to verify that output fields from one node can meaningfully bind to inputs of the next (Zhu et al., 4 Jun 2026). Second, ToolMaze explicitly enumerates all valid recovery paths by clustering functionally equivalent tools or subgraphs into substitute sets. The ground-truth solution space is defined as $\mathcal{S} = \{s_1, \dots, s_k\}$ , and the shortest sequence $3.66\times$ 0 is chosen as the default unperturbed path (Zhu et al., 4 Jun 2026). Third, the DAG is converted into a realistic user query, followed by reverse-validation to ensure that the inferred dependencies still match the source DAG (Zhu et al., 4 Jun 2026).

ToolMaze reports 100 tasks per complexity level $3.66\times$ 1– $3.66\times$ 2, for 400 base tasks. Average path lengths increase with complexity: 5.30 for $3.66\times$ 3, 5.75 for $3.66\times$ 4, 6.66 for $3.66\times$ 5, and 11.00 for $3.66\times$ 6. The numbers of valid paths are 100, 200, 200, and 600, respectively. Mean Jaccard similarity of tool sets is reported as very low, up to only 0.06, indicating high diversity (Zhu et al., 4 Jun 2026).

The perturbation taxonomy

The second axis is a $3.66\times$ 7 failure taxonomy built from explicit vs. implicit and transient vs. permanent failures (Zhu et al., 4 Jun 2026). Together with the non-perturbed baseline, this yields four perturbation modes:

Mode	Meaning
$3.66\times$ 8	Explicit-Transient
$3.66\times$ 9	Explicit-Permanent
$(\mathcal{C}, \mathcal{P})$ 0	Implicit-Transient
$(\mathcal{C}, \mathcal{P})$ 1	Implicit-Permanent

Explicit failures are machine-readable errors such as HTTP 404, 429, 503, timeout, or forbidden responses. Implicit failures are structurally valid outputs whose content is semantically corrupted (Zhu et al., 4 Jun 2026). Transient failures are recoverable via retry, whereas permanent failures require rerouting or graceful termination (Zhu et al., 4 Jun 2026).

The appendix gives concrete classes. For $(\mathcal{C}, \mathcal{P})$ 2, examples include 503 Service Temporarily Unavailable, 504 Gateway Timeout, 429 Too Many Requests, 500 Internal Server Error, 502 Bad Gateway, 409 Maximum Concurrent Connections Reached, and 409 Resource Temporarily Locked. For $(\mathcal{C}, \mathcal{P})$ 3, examples include 404 Tool Not Found, 410 Tool Permanently Removed, 501 Not Implemented, 403 Access Forbidden, 451 Unavailable For Legal Reasons, 402 Account Quota Exhausted, 410 API Version Deprecated, 410 Feature Discontinued, 451 Geo-blocked, 403 Account Suspended, 402 License Invalid, and 410 Data Source Retired. For $(\mathcal{C}, \mathcal{P})$ 4 and $(\mathcal{C}, \mathcal{P})$ 5, corruption types include physics_violation, tool_leakage, irrelevant_response, statistical_inconsistency, internal_contradiction, missing_required_fields, logic_violation, temporal_contradiction, and fact_contradiction (Zhu et al., 4 Jun 2026).

A runtime perturbation engine injects failures deterministically at pre-specified nodes on the preferred path $(\mathcal{C}, \mathcal{P})$ 6. In multi-path tasks, the perturbation is tied to an alternative group, activated once any tool in that group is called, and then disabled for further tools in the same group (Zhu et al., 4 Jun 2026). This design ensures comparability across models because each $(\mathcal{C}, \mathcal{P})$ 7 pair induces the same failure condition.

3. Dataset, protocol, and metrics

ToolMaze expands its 400 base tasks into 2,000 instances by applying the non-perturbed baseline plus the four perturbation modes (Zhu et al., 4 Jun 2026). The tasks are generated with GPT-5.5 and Gemini-3.1-Pro-Preview, but grounded in the benchmark’s curated tool corpus (Zhu et al., 4 Jun 2026). Evaluation takes place in a sandboxed environment with a maximum of 25 agent steps, temperature = 1, and max tokens = 16,000 (Zhu et al., 4 Jun 2026).

The benchmark compares two prompting conditions: a standard tool-use prompt (“w/o hint”) and a failure-aware prompt (“w/ hint”). The failure-aware prompt explicitly warns that tools may fail or return unexpected data and instructs the model to check output coherence, stop if the workflow cannot be completed, and reroute or retry appropriately (Zhu et al., 4 Jun 2026).

ToolMaze uses three evaluation metrics.

Task Success Rate

Task Success Rate measures overall task completion:

$(\mathcal{C}, \mathcal{P})$ 8

It captures end-to-end success but does not isolate recovery ability (Zhu et al., 4 Jun 2026).

Perturbation Recovery Rate

Perturbation Recovery Rate measures successful recovery conditional on encountering a perturbation:

$(\mathcal{C}, \mathcal{P})$ 9

A trajectory is counted as recovered if the agent retries for transient faults, reroutes to an alternative path, or gracefully aborts when the task is unsolvable (Zhu et al., 4 Jun 2026). PRR is the benchmark’s main measure of fault tolerance.

Recovery Cost

Recovery Cost evaluates inefficiency after the first perturbed response:

$\mathcal{C}$ 0

where $\mathcal{C}$ 1 is the actual number of tool-call steps from perturbation to completion and $\mathcal{C}$ 2 is the theoretical minimum. The mode-level average is

$\mathcal{C}$ 3

RC near 0 indicates near-optimal recovery; RC near 1 indicates wasted steps or failed recovery (Zhu et al., 4 Jun 2026). This explicit efficiency measure distinguishes ToolMaze from benchmarks that record success without accounting for futile exploration.

4. Empirical findings and failure modes

ToolMaze evaluates both open-weight and proprietary models. The open-weight set includes GLM-5.1, Deepseek-V4-Pro, MiniMax-M2.7, Qwen3.5-35B-A3B, Qwen3.5-397B-A17B, and Qwen3.6-27B; the proprietary set includes GPT-5.5, Gemini-3.1-Pro-Preview, and Claude-Sonnet-4-6 (Zhu et al., 4 Jun 2026). Each model is tested under the non-perturbed baseline and perturbation modes $\mathcal{C}$ 4– $\mathcal{C}$ 5, with and without the failure-aware hint (Zhu et al., 4 Jun 2026).

Across nearly all models, perturbations substantially degrade performance relative to the non-perturbed baseline (Zhu et al., 4 Jun 2026). The most important empirical pattern is that implicit failures are much harder than explicit failures. The benchmark reports an overall implicit-explicit PRR gap of 37.15%, decomposed into 53.75% for transient conditions and 20.54% for permanent conditions (Zhu et al., 4 Jun 2026). The paper interprets this as a trust gap: models over-trust semantically corrupted outputs.

Failure-aware prompting improves results across all models, with reported gains ranging from roughly +1.5% to +20.8% depending on model and setting, but does not remove the underlying difficulty (Zhu et al., 4 Jun 2026). This suggests that prompting can modify surface behavior, but the deeper capability of anomaly detection and structured replanning remains limited.

Topology exerts a strong effect. Performance is reported as strongest around $\mathcal{C}$ 6, where alternatives exist and rerouting is feasible, but degrades in $\mathcal{C}$ 7 and $\mathcal{C}$ 8, where the search space grows and agents increasingly fall into branch-selection errors and trial-and-error loops (Zhu et al., 4 Jun 2026). PRR and RC move inversely: as PRR declines, RC rises. Under $\mathcal{C}$ 9, PRR can drop below 20% while RC exceeds 70%, indicating both low recovery and inefficient exploration (Zhu et al., 4 Jun 2026).

The appendix includes 16 case studies, covering each combination of $\mathcal{P}$ 0– $\mathcal{P}$ 1 with $\mathcal{P}$ 2– $\mathcal{P}$ 3 (Zhu et al., 4 Jun 2026). The reported successful patterns are structurally interpretable: retrying after explicit transient errors, halting after explicit permanent errors, re-querying after detecting implicit transient corruption, and rerouting or halting after persistent implicit corruption (Zhu et al., 4 Jun 2026). Failed trajectories, by contrast, are characterized by premature abandonment, blind persistence, silent propagation of poisoned values, and endless retry loops. The figure captions explicitly describe some of these behaviors as “sanity ignorance” (Zhu et al., 4 Jun 2026).

5. Relation to adjacent benchmarks and training paradigms

ToolMaze belongs to a wider research program on tool-use evaluation, but its emphasis differs from earlier work along three main axes: topology, perturbation control, and recovery-aware metrics.

ToolComp is the nearest precedent on process sensitivity. It provides 485 human-verified prompts, 1,731 step-wise supervision labels, and a ReAct-style evaluation setup in which each Thought, Action, and Action Input is judged correct or incorrect, then corrected by annotators if necessary (Nath et al., 2 Jan 2025). ToolComp shows that process-supervised reward models generalize better than outcome-supervised ones, with a 19% improvement in rank@1 accuracy on base-model trajectories and an 11% improvement on fine-tuned model trajectories (Nath et al., 2 Jan 2025). ToolMaze differs in that it does not merely ask whether intermediate steps are correct, but whether the agent can discover and execute alternate valid paths when the environment changes.

The benchmark also differs from m&m’s, which targets multi-step multi-modal planning across 33 tools, 4,427 raw query-plan examples, and 1,565 human-verified examples, with explicit comparison of step-by-step versus multi-step planning, JSON versus code, and parsing/verification/execution feedback (Ma et al., 2024). m&m’s demonstrates that executability and tool selection quality can diverge, and that feedback improves pass rate and argument naming while sometimes reducing tool-F1 (Ma et al., 2024). ToolMaze inherits the concern with real execution dynamics, but focuses specifically on replanning under injected faults.

A plausible implication is that ToolMaze operationalizes a stricter notion of robustness than either process-labeled reasoning benchmarks or executable-plan benchmarks. ToolComp asks whether the trajectory is reasoning-correct; m&m’s asks whether the plan is executable; ToolMaze asks whether the agent can recover when the environment invalidates the preferred plan. This suggests a layered evaluation hierarchy in which process correctness, executability, and recovery robustness are distinct but related competencies.

ToolMaze is also relevant to training frameworks that attempt to improve tool use through structured traces or interaction. MTR proposes a simulation-first framework in which tool ecosystems are replaced by schema-validated simulated observations, and reports that on four multi-hop QA benchmarks it reaches 29.38 average EM on Qwen2.5-7B-base and 29.53 on Qwen2.5-7B-Instruct, nearly matching live-API baselines while remaining fully API-free (Wang et al., 8 Oct 2025). Its reward includes answer correctness, internal consistency, and an efficiency penalty against repetitive loops (Wang et al., 8 Oct 2025). ToolMaster, by contrast, trains models to trial tools in the environment, then self-correct during execution, and reports more than 7% average improvement over baselines on StableToolBench and a 6.8% average margin over the strongest competitor on TMDB and ToolHop (Gao et al., 19 Jan 2026). These works indicate two contrasting approaches to robust tool use: simulated trace discipline versus interactive trial-and-execution. ToolMaze provides a benchmark on which those approaches can be differentiated in terms of replanning and anomaly recovery.

6. Interpretive significance and open issues

The benchmark’s main conceptual contribution is to redefine robust tool use as a problem of controlled search under uncertainty. In ToolMaze, dynamic replanning includes recognizing whether failure is explicit or implicit, determining whether retry is appropriate, backtracking when necessary, selecting another valid path, and stopping when the task is unsolvable (Zhu et al., 4 Jun 2026). This is broader than ordinary retry logic and narrower than unconstrained exploration: because the benchmark enumerates valid recovery paths in advance, recovery can be assessed against a known space rather than by ad hoc success criteria.

ToolMaze also clarifies the distinction between systematic replanning and blind trial-and-error. The explicit enumeration of solution spaces and the RC metric make it possible to penalize agents that eventually succeed but do so inefficiently or by flailing among branches (Zhu et al., 4 Jun 2026). This controlled design addresses a methodological ambiguity that affects many earlier tool-use evaluations.

Several broader concerns in the literature sharpen the significance of ToolMaze. ToolSense shows that strong benchmark retrieval can coexist with weak semantic understanding of tools, reporting a “knowledge-retrieval dissociation” in parametric retrieval systems and large collapses from verbose ToolBench queries to more realistic retrieval prompts (Hathidara et al., 4 Jun 2026). A plausible implication is that agents entering ToolMaze may fail not only because of weak replanning but also because they over-trust corrupted outputs or rely on shallow retrieval shortcuts rather than internalized tool semantics. Likewise, ToolComp’s error taxonomy—wrong tool choice, bad tool-call formatting, repeated calling of the same tool, misunderstanding tool outputs, early termination, and hallucinated information—overlaps with several failure patterns that ToolMaze exposes in perturbed settings (Nath et al., 2 Jan 2025).

The benchmark also raises a possible “reality gap” question. ToolMaze uses deterministic simulated tools and a controlled perturbation engine (Zhu et al., 4 Jun 2026). This controlled setting is advantageous for causal diagnosis and reproducibility, but it does not by itself establish performance under the full heterogeneity of production APIs. The MTR paper explicitly acknowledges a possible “reality gap” for simulated tool training (Wang et al., 8 Oct 2025). By analogy, ToolMaze’s deterministic design may best be understood as a diagnostic environment rather than a complete proxy for deployment. This suggests that strong ToolMaze performance is evidence of structured recovery capability, but not necessarily sufficient evidence of real-world operational robustness.

At the same time, the benchmark provides a concrete agenda for future agent design. The ToolMaze paper argues that tool-use evaluation must include failures, that semantic anomaly detection is a major missing capability, that recovery planning should be trained explicitly, that efficiency matters alongside success, and that benchmarks should expose full recovery spaces (Zhu et al., 4 Jun 2026). These conclusions align with ToolMaster’s emphasis on trialing and self-correction under environment interaction (Gao et al., 19 Jan 2026) and with ToolComp’s conclusion that process supervision is crucial for robust multi-step reasoning (Nath et al., 2 Jan 2025).

In this sense, ToolMaze functions both as an evaluation suite and as a problem statement. It isolates the proposition that current agents are often competent at executing known chains but substantially weaker at recognizing when the environment has invalidated those chains. The empirical asymmetry between TSR and PRR, the difficulty of implicit corruption, and the slower scaling of fault tolerance collectively support the view that dynamic replanning is a distinct research target rather than a byproduct of better prompting or larger models (Zhu et al., 4 Jun 2026).