AgentFail: Root Cause Diagnosis in Agent Systems

• AgentFail is a dataset, taxonomy, and benchmark designed to diagnose root causes by shifting debugging from mere failure localization to causal explanation in multi-agent workflows.
• It employs counterfactual reasoning and a structured three-level taxonomy (agent, workflow, platform) to annotate and evaluate failure logs from diverse low-code systems.
• Empirical results show taxonomy guidance improves diagnostic accuracy by 15–20 percentage points, yet even the best models achieve only up to 33.6% accuracy.

AgentFail is a dataset, taxonomy, and benchmark for diagnosing failure root causes in platform-orchestrated agentic systems, defined as multi-agent workflows built on low-code or template-based platforms such as Dify and Coze. Its central objective is to shift debugging from identifying where a failure happened to explaining why it happened. The motivating claim is that prior work mostly emphasized failure attribution at the level of “which agent/step failed,” whereas meaningful repair requires deeper root-cause diagnosis across prompts, workflow orchestration, tools, and platform runtime behavior (Ma et al., 28 Sep 2025).

1. Problem definition and formalization

AgentFail targets platform-orchestrated agentic systems in which failures can arise from the underlying agent model, prompt design, workflow orchestration, or platform/runtime issues. In this setting, a failure log is not treated merely as evidence of an erroneous step, but as a trajectory requiring diagnosis of the decisive error and its underlying cause. The paper explicitly frames root-cause diagnosis as a prerequisite for debugging, repair, and reliability improvement, because labels such as “bad prompt,” “incorrect task decomposition,” “deadlock in workflow control,” “tool mismatch,” or “platform outage” are more actionable than simple step localization (Ma et al., 28 Sep 2025).

The paper includes a compact formalism for failure localization. For a trajectory $\tau$, failure is represented by a binary variable: $\phi(\tau)=1$ for failure and $\phi(\tau)=0$ for success. For an error $E(i,t)$, the decisive-error indicator is

$$\Delta_{E(i,t)}(\tau) = \begin{cases} 1, & \text{if } \phi(\tau) = 1 \text{ and } \phi(\tau^{E(i,t)}) = 0, \ 0, & \text{otherwise}. \end{cases}$$

If multiple decisive errors exist, the earliest one is selected as

$$E(i^*,t^*)=\arg\min_{E(i,t)\in C(\tau)} t,\quad \text{where } C(\tau)=\{E(i,t)\mid\Delta_{E(i,t)}(\tau)=1\}.$$

This “first decisive error” principle aligns the benchmark with counterfactual reasoning: a step is causally important only if correcting it changes the outcome from failure to success. The paper also distinguishes two outcome types for impact analysis: Execution Failure, where the run is interrupted, and Suboptimal Quality, where the run completes but output quality is inadequate (Ma et al., 28 Sep 2025).

2. Dataset construction and annotation procedure

AgentFail contains 307 failure logs collected from 10 platform-orchestrated agentic systems built on two representative platforms, Dify and Coze, with 5 systems from Dify and 5 systems from Coze. The paper reports that these failures were collected from 1,387 test runs in total. Each instance contains the Query, the Failure log, the System workflow and configuration, and the Annotations, including a root-cause label and explanation (Ma et al., 28 Sep 2025).

The corpus mixes two sources. One source is community-shared issues, consisting of publicly reported failures from users. The other is controlled runs by the authors, including executions of public benchmarks such as HumanEval and TravelPlanner, plus hand-crafted datasets tailored to each system. The ten systems span multiple task categories—software development, program repair, product QA, travel assistance, deep research, market research, industry analysis, and question answering—and multiple workflow structures: serial, parallel, branching, looping, and hybrid. This diversity is intended to capture failures caused by both task semantics and orchestration structure (Ma et al., 28 Sep 2025).

Annotation follows a Grounded Theory approach. Three expert annotators independently inspected the logs, identified the decisive error and corresponding root cause, and marked cases as fully confident or uncertain. Uncertain cases were resolved through discussion, followed by cross-review for consistency. Final inter-annotator agreement improved from Cohen’s $\kappa = 0.85$ initially to 1.0 after reconciliation. The result is a curated expert-consensus dataset rather than an automatically labeled corpus (Ma et al., 28 Sep 2025).

3. Three-level taxonomy of failure root causes

The core conceptual contribution of AgentFail is a three-level taxonomy that separates failures by where they originate: inside an individual agent, in workflow orchestration, or in the platform/runtime layer. The taxonomy is intended both as an annotation scheme and as structured guidance for automated diagnosis.

Level	Description	Subcategories
F1. Agent-level failures	Within a single agent or its interaction with local resources	F1.1 Tool or action planning error; F1.2 Response formatting error; F1.3 Response content deviation; F1.4 Knowledge or reasoning limitation; F1.5 Prompt design defect; F1.6 Language or encoding defect; F1.7 Tool invocation or KB retrieval error
F2. Workflow-level failures	Orchestration, control flow, or inter-agent coordination	F2.1 Missing input validation; F2.2 Unreasonable node dependency; F2.3 Loops and deadlock; F2.4 Faulty conditional judgment; F2.5 Improper task decomposition; F2.6 Context conflict; F2.7 Cross-agent tool or interface mismatch
F3. Platform-level failures	Platform/runtime infrastructure	F3.1 Network and resource fluctuation; F3.2 Service unavailability

Several distributional patterns are reported. Agent-level failures dominate overall. The most frequent categories are F1.4 knowledge/reasoning limitations and F1.5 prompt design defects, both with more than 50 instances each. Other common categories include F1.2 response formatting errors and F1.3 response content deviations. Workflow failures are less frequent overall, but F2.1 missing input validation and F2.2 unreasonable node dependency are especially common. Platform failures are the smallest share overall, consisting of F3.1 network/resource fluctuation and F3.2 service unavailability (Ma et al., 28 Sep 2025).

The paper also reports platform- and task-specific regularities. Coze shows more F2.2 unreasonable dependencies, attributed to very long serial workflows with many agents, which increase dependency-chain fragility. By task domain, software development is associated with reasoning limitations and prompt design issues; question answering with response formatting and tool or retrieval problems; task planning with language or encoding problems and dependency issues; and information insight tasks with response content deviation and reasoning limitations (Ma et al., 28 Sep 2025).

Impact analysis injects one failure type at a time and measures degradation in Execution termination, Suboptimal quality, and Sum. Reported high-impact cases include F3.2 service unavailability: 90.3%, F1.5 prompt design defect: 90.1%, F1.2 response formatting error: 89.5%, and F3.1 network/resource fluctuation: 89.6%. The paper further notes that F1.4 and F1.5 tend to produce suboptimal outputs, while F1.2, F1.6, and F1.7 often cause execution termination. At the workflow level, F2.3 loops/deadlock and F2.7 cross-agent interface mismatch often terminate execution, whereas F2.5 task decomposition and F2.6 context conflict more often degrade quality without stopping execution (Ma et al., 28 Sep 2025).

4. Counterfactual validation of annotation reliability

A notable methodological feature is the use of counterfactual reasoning-based repair strategy to validate annotations. For a labeled root cause $\mathcal{D}$, experts design a repair specifically intended to fix that cause. If applying that repair changes the case from failure to success, the original label is considered validated. This operationalizes the question: “If this root cause were fixed, would the system succeed?” (Ma et al., 28 Sep 2025)

The paper samples the top half of failure types by frequency for this verification. Example repairs include enforcing strict JSON output with validation and reprompting for response formatting error; semantic similarity checks with a correction prompt for response content deviation; switching to a stronger or more domain-adapted model for knowledge/reasoning limitation; a standardized prompt template for prompt design defect; encoding normalization for language/encoding defect; explicit checks for missing input validation; workflow redesign for unreasonable node dependency; and fallback branches or backup models for network/resource fluctuations (Ma et al., 28 Sep 2025).

The confusion matrix of repair effectiveness shows strong diagonal dominance, meaning that the repair aligned with the annotated cause usually works best. Reported examples include D1 response formatting error: 90.1%, D2 response content deviation: 95.6%, and D5 language/encoding defect: 96.3%. The paper notes some off-diagonal effects, indicating that some fixes can incidentally help multiple failure types. Even so, the repair study is presented as evidence that the labels are reliable rather than purely descriptive (Ma et al., 28 Sep 2025).

5. Benchmark design and empirical results

The AgentFail benchmark evaluates whether LLMs can identify the correct root-cause label from failure logs. The evaluated models are gpt-4o, GEMINI-2.5-PRO, CLAUDE-SONNET-4, LLaMA-3.1-70B, QWEN-32B, and DeepSeek-R1. Models are tested in two settings: without taxonomy, where the model relies on its own interpretation, and with taxonomy, where the taxonomy and definitions are supplied as guidance. To handle long logs, the benchmark uses three inference protocols: All-at-once, Step-by-step, and Binary search. The target is to predict the correct root-cause label, and the reported metric is accuracy (Ma et al., 28 Sep 2025).

The best reported result is 33.6%, achieved by DeepSeek-R1 under the with-taxonomy + step-by-step setting. Without taxonomy, models achieve about 8.3% to 13.0% accuracy. With taxonomy, accuracy rises to roughly 24.1% to 33.6%. The paper therefore reports an absolute gain of about 15–20 percentage points from taxonomy guidance. At the same time, the ceiling remains low: even the best system gets only about one-third of cases correct. The authors interpret this as evidence that failure logs are long, noisy, and causally tangled, so later symptoms may be visible while the true trigger remains difficult to trace (Ma et al., 28 Sep 2025).

A common misconception is that a high-quality taxonomy would make automated diagnosis nearly solved. The reported results do not support that interpretation. Rather, the taxonomy can largely improve the performance, thereby confirming its utility, while also showing that root cause identification reaches at most 33.6%, which indicates that the task remains challenging. In that sense, the benchmark functions both as an evaluation resource and as a measurement of current limitations in root-cause diagnosis (Ma et al., 28 Sep 2025).

6. Practical guidelines and design implications

Beyond annotation and benchmarking, AgentFail extracts system-design guidance from the observed failure modes. The paper recommends clear role specification and modular prompt design, explicit input/output validation, fallback mechanisms and secondary checks, and progressive workflow design that starts with simpler serial or parallel structures before adding complex branching, looping, or hybrid structures. These recommendations are tied directly to the taxonomy rather than offered as generic best practices (Ma et al., 28 Sep 2025).

The rationale is visible in the empirical patterns. Prompt defects, reasoning limitations, and response-formatting errors are frequent and high-impact; missing input validation and unreasonable dependencies are the most common workflow-level faults; and platform failures, although rarer, are highly destructive and usually terminate execution. A plausible implication is that reliability engineering for platform-orchestrated agentic systems should not be concentrated in a single layer. Prompt design, schema enforcement, workflow structure, and fallback runtime behavior all appear in the root-cause distribution (Ma et al., 28 Sep 2025).

The paper also notes qualitative findings that long serial workflows are especially vulnerable to dependency problems and error propagation, and that some failures are symptom-shared: fixing a reasoning issue may also resolve what was labeled as a formatting issue. This suggests that root causes can interact causally even when the taxonomy assigns a single decisive label for benchmarking purposes (Ma et al., 28 Sep 2025).

7. Position within agent-failure research

AgentFail belongs to a broader research shift from outcome-only evaluation toward trajectory-grounded diagnosis, but it occupies a distinct niche. Its emphasis is root-cause diagnosis for platform-orchestrated agentic systems, whereas several adjacent works focus on failure attribution or localization in other settings. A2P reframes automated failure attribution as a causal inference problem over long conversation logs and asks which single corrective intervention would have prevented failure (West et al., 12 Sep 2025). FAMAS treats failure attribution as a spectrum-based localization problem over repeated multi-agent executions, ranking the most suspicious agent-action-state tuple (Ge et al., 17 Sep 2025). AgentTrace reconstructs causal graphs from execution logs and ranks candidate root causes without requiring LLM inference at debugging time (Wang, 16 Mar 2026). POIROT turns the agents in a multi-agent system into the diagnostic layer itself through peer interrogation and weighted consensus on fault attribution (Varela et al., 1 Jun 2026).

Other work extends the diagnosis problem into software engineering and general agent execution traces. AgentRx diagnoses failed trajectories by synthesizing global and dynamic constraints, producing an auditable validation log, and using an LLM-based judge to localize the first unrecoverable failure step and its category (Barke et al., 2 Feb 2026). TrajAudit addresses repository-level coding trajectories by combining preliminary diagnosis from test-failure reports with semantic saliency folding and tool-based retrieval over long, noisy traces (Wang et al., 26 May 2026). These systems focus on step localization and critical failure identification; AgentFail instead foregrounds the taxonomy of underlying causes and the behavior of platform-built workflows (Ma et al., 28 Sep 2025).

This suggests a useful conceptual distinction. Failure attribution methods ask which step, node, or agent introduced the decisive error. AgentFail asks what structural, prompting, workflow, or platform-level cause made the system fail. The two questions are not interchangeable. The first supports localization; the second supports repair planning. AgentFail’s contribution is to make that second question explicit, annotated, and benchmarkable (Ma et al., 28 Sep 2025).