When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents

Abstract: Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a $2 \times 2$ taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37\% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale $3.66\times$ slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze.

• The paper introduces ToolMaze, a novel benchmark that systematically evaluates dynamic replanning and anomaly recovery in LLM tool agents under explicit and implicit failures.
• It employs a 2x2 failure taxonomy combined with DAG-based task topologies to rigorously measure task success, perturbation recovery, and recovery cost.
• Experimental results reveal a significant performance decline in recovery rates for implicit failures, emphasizing the need for dedicated anomaly detection strategies.

ToolMaze: Systematic Benchmarking for Replanning and Anomaly Recovery in LLM Tool Agents

Motivation and Background

LLMs augmented with external tool invocation (TIR agents) have demonstrated utility in real-world information integration and automation. However, most benchmarks constrain evaluation to "happy paths"—idealized, failure-free tool invocation, ignoring realistic system-level instability such as network errors and silent data corruption. This results in a significant blind spot: LLMs can succeed when the environment is perfectly reliable, but may catastrophically fail in the presence of explicit or silent tool anomalies. ToolMaze addresses this by introducing a comprehensive benchmark for dynamic replanning and anomaly recovery, explicitly modeling different error types and complex task dependencies.

ToolMaze Framework

ToolMaze organizes robustness benchmarking as a structured product space along two axes: task topology (complexity) and perturbation mode (failure class). Each evaluation instance is anchored at a unique $(\mathcal{C}, \mathcal{P})$ coordinate, yielding a complete, systematic coverage of scenario diversity.

• Task Topology ($\mathcal{C}$): Four complexity levels are defined via DAGs—
  • $\mathcal{C}1$: Linear pipelines without alternatives
  • $\mathcal{C}2$: Single alternative nodes, enabling 1-to-N substitution
  • $\mathcal{C}3$: Multi-branch, many-to-many alternatives, capturing combinatorial replanning
  • $\mathcal{C}4$: Multiple nested branches, simulating deep, multi-level fallback graphs
• Perturbation Mode ($\mathcal{P}$): The $2\times2$ failure taxonomy considers
  1. Explicit (machine-detectable by error codes) vs. Implicit (structurally valid, semantically corrupt)
  2. Transient (recoverable by retry) vs. Permanent (necessitating reroute or halt)

This grid structures the evaluation space, ensuring isolation of core agentic recovery skills and avoiding confounding factors such as random failure sites or ambiguous solution criteria.

Figure 1: Overview of the ToolMaze framework, illustrating its topological complexity and the failure taxonomy underpinning perturbation design.

Dataset Construction and Evaluation Protocol

ToolMaze synthesizes tasks via a tool-first paradigm: DAG graphs are sampled from a corpus of 270 hand-curated APIs covering six domains. This approach guarantees semantic coherence and facilitates exhaustive enumeration of valid recovery (solution) paths. Each task instance is defined by a unique failure injection profile—perturbations are deterministic and localized at pre-specified tool nodes, ensuring full reproducibility and fair cross-model comparison.

The evaluation protocol decomposes robustness into three orthogonal metrics:

• Task Success Rate (TSR): Unconditional completion fraction, providing a baseline for classical utility

• Perturbation Recovery Rate (PRR): Conditional success given exposure to a perturbation, measuring true anomaly recovery
• Recovery Cost (RC): Relative tool-call inefficiency post-failure, penalizing trial-and-error loops or redundant execution

Experimental Findings

Impact of Failure Type and Task Complexity

All models exhibit a clear performance cliff when moving from the non-perturbed baseline to any perturbed evaluation. For instance, state-of-the-art proprietary LLMs such as Claude-Sonnet-4-6 and Gemini-3.1-Pro-Preview, which attain TSRs above 70% on clean pipelines, drop dramatically under perturbation modes, with even the most robust configurations averaging below 50% in the most challenging cases. This deficit is exacerbated as topological complexity increases, and is most acute for implicit, permanent failures.

Figure 2: Mean TSR, PRR, and RC across $\mathcal{C}1$–$\mathcal{C}4$ using both standard and failure-aware prompts. Higher complexity induces lower robustness and greater inefficiency.

Notably, recovery is strongly affected by perturbation semantics. Explicit, transient failures (e.g., HTTP 503) are generally handled reasonably (PRR exceeding 80% in top models), as agents reliably retry or reroute. However, implicit perturbations decimate recovery: across all models, average PRR collapses by as much as 37 percentage points under implicit variants compared to explicit ones—an effect primarily attributed to over-trust in corrupted—but format-compliant—tool outputs.

Figure 3: PRR drops sharply and RC increases moving from explicit to implicit failures, especially in permanent modes.

Model Scaling and Prompting

While model size correlates well with baseline TSR, improvement in PRR due to scaling is much slower—for each order of magnitude increase in parameters, fault tolerance improves only $\mathcal{C}$0 slower than baseline task completion. No model or prompting variant evidences strong dynamic replanning as an emergent behavior of scale alone.

Providing agents with explicit recovery hints boosts PRR and reduces RC by only a marginal amount—a maximum improvement of +20.8%—indicating that awareness of potential failures is a necessary but not sufficient condition for robust anomaly-handling.

Figure 4: Scaling of PRR (recovery) lags far behind TSR (base execution) as a function of model parameter count.

Qualitative Error Patterns

ToolMaze includes a suite of case studies which highlight characteristic behaviors. In explicit failure scenarios, unsuccessful agents often attempt downstream steps regardless of clear error signals, while successful models halt or reroute appropriately.

Figure 5: In explicit-transient failure, the robust agent efficiently retries; the unstable agent abandons the pipeline at the first error.

For implicit, persistent failures, a recurrent pathology is "sanity ignorance": agents blindly propagate corrupted values, never attempting retries or alternative execution even when the result is logically implausible.

Figure 6: Implicit-transient failure: Only the robust agent detects data anomaly and retries, the failing agent propagates the bad value.

Theoretical and Practical Implications

ToolMaze exposes a robust division between instruction-following/task completion (which scales with model size/prompting) and genuine agentic replanning/recovery, which does not. Current LLMs exhibit minimal grounding in semantic anomaly detection—reliance on explicit error signaling is necessary for satisfactory performance. This systematically leaves them vulnerable to silent data corruption, especially in realistic multi-branch dependency graphs where anomalies propagate and compound.

The practical implication is that TIR agents cannot be reliably deployed in real-world automation tasks without explicit architectural or algorithmic provisions for anomaly detection and structured replanning. The strong numerical gap between explicit and implicit PRR (mean gap of 37.15%) quantifies this limitation. This bottleneck persists regardless of advanced prompting or increased parameterization.

Limitations and Directions for Future Work

ToolMaze operates in controlled, DAG-structured topologies with mathematically complete solution sets and deterministic perturbation profiles. Although this yields rigorous coverage and analysis, it separates the benchmark from open-world agentic environments with inherently ambiguous or multi-modal solution trajectories. Moreover, the foundational $\mathcal{C}$1 failure taxonomy, while comprehensive, omits more adversarial or cascading failure modes (e.g., chained semantic drift, intent poisoning).

ToolMaze can serve as a foundation for future studies on open-domain, highly compositional agentic workflows, as well as for the investigation of architectural and training strategies for learned anomaly detection and revision (e.g., mechanism-based System 2 reasoning).

Conclusion

ToolMaze provides the first exhaustive, controlled evaluation suite for dynamic replanning and anomaly recovery in tool-augmented LLMs. Experimental evidence underscores a systemic lack of genuine anomaly awareness in contemporary models, most notably when tool failures are implicit and semantically subtle. Improvements in robustness do not simply emerge via model scaling or detailed prompting: agentic dynamic recovery is fundamentally distinct from instruction-following and must be directly targeted in both training and benchmarking. ToolMaze thus defines core desiderata for the next generation of agentic TIR benchmarks and for system architectures aspiring toward genuinely resilient, autonomous behavior.

Figure 7: Quantification of the implicit–explicit PRR gap demonstrates persistent over-trust in semantically perturbed responses (average gap: 37.15\%).