ErrorAtlas Taxonomy for LLM Failures

• ErrorAtlas Taxonomy is a comprehensive framework for characterizing and annotating LLM errors through distinct categories and impact assessments.
• It classifies failures in both agent trajectories and general LLM outputs using detailed metrics for diagnostic evaluation.
• The framework supports taxonomy-aware and taxonomy-blind annotation, enabling rigorous benchmarking and targeted error reduction.

The ErrorAtlas Taxonomy defines a comprehensive framework for characterizing, annotating, and analyzing the failure modes of LLMs and agentic systems. Emerging from critical insights that scalar task accuracy alone is insufficient to understand and improve complex model behavior, ErrorAtlas organizes observable errors into distinct, interpretable categories and provides protocols for systematic error collection and evaluation. Its two major instantiations—one agent trajectory-focused (AgentAtlas) and one general model-output-focused (via ErrorMap)—enable high-resolution diagnostics for both agentic and non-agentic benchmarks, supporting a transition from outcome-centric to explanation-aware evaluation in LLM research and deployment (Mazaheri et al., 19 May 2026, Ashury-Tahan et al., 22 Jan 2026).

1. ErrorAtlas for Agent Trajectories: Nine-Category Taxonomy

The AgentAtlas variant of ErrorAtlas introduces a nine-category taxonomy to classify failures at the level of agent trajectories, designed to capture the primary locus and nature of unrecoverable mistakes:

1. Goal misinterpretation: The agent misconstrues the user’s intent, operating on the wrong task or violating high-level constraints.
2. Wrong tool selection: Inappropriate tool, API, or action modality is chosen, resulting in ineffective or futile subsequent actions.
3. Wrong argument / wrong target: Given a correct action, execution fails due to argument, identifier, path, or parameter error.
4. Observation failure: The agent misreads or misinterprets environmental feedback or tool output, acting on stale or incorrect information.
5. Constraint violation: The agent disregards explicit user, policy, or permission constraints.
6. Recovery failure: The agent continues after a recoverable misstep without rolling back, replanning, or regrounding.
7. Looping or over-action: The agent executes excessive or repeated actions beyond the task's logical endpoint.
8. Unsafe trust of external content: The agent executes or trusts unvetted, potentially malicious outputs or data.
9. State or memory contamination: Persistent context or memory is contaminated by stale or adversarial information, corrupting future decisions.

Every failed trajectory is annotated at the precise step of the first unrecoverable error, assigning one and only one of these labels. Viable trajectories receive a “Valid” tag instead (Mazaheri et al., 19 May 2026).

2. Hierarchical Annotation Schema: Orthogonal Error Source and Impact

ErrorAtlas employs a two-field schema to capture both the nature and consequences of failures, enhancing interpretability and auditability:

• primary_error_source: Indicates which error class (the nine categories listed, plus “valid” for no-failure) is assigned.
• impact: Independently specifies the effect of the error, drawn from five classes:
  • unsafe_side_effect: Unintended, possibly destructive, changes to data or environment.
  • privacy_leak: Exposure of confidential or sensitive information.
  • wrong_final_state: Incorrect task result from the user’s perspective.
  • unnecessary_cost: Inefficient use of compute, time, or other resources.
  • no_impact: Error had no persistent effect.

These fields are orthogonal, supporting granular analysis of relationships such as which error sources have a propensity to cause high-severity impacts (Mazaheri et al., 19 May 2026).

Error Source (primary_error_source)	Example Consequences (impact)
wrong_tool	wrong_final_state, unnecessary_cost
recovery_failure	unsafe_side_effect
unsafe_trust	privacy_leak, unsafe_side_effect

3. Formal Notation and Metrics for Taxonomy Evaluation

The taxonomy supports rigorous evaluation using standard classification metrics, supporting both single-field and joint analysis. Let $N$ denote the number of trajectories; for trajectory $i$, $y_i$ is the gold primary label, $\hat{y}_i$ the model’s prediction, $z_i$ the gold impact label, and $\hat{z}_i$ the predicted impact. Key metrics:

• primary_src_accuracy: $\frac{1}{N}\sum_{i} [\hat{y}_i = y_i]$
• primary_src_macroF1: Macro-averaged $F_1$ over all nine error classes.
• impact_accuracy: $\frac{1}{N}\sum_{i} [\hat{z}_i = z_i]$
• impact_macroF1: Macro-averaged $F_1$ over all five impact classes.
• joint_accuracy: $i$0
• exact_step_acc (optional): Fraction of exact step localization matches.
• ±1_step_acc (optional): Fraction within one step of the gold critical step.

These enable comparability across models, labeling pipelines, and benchmarks (Mazaheri et al., 19 May 2026).

4. Annotation Methodology and Agreement

Annotation requires determination of (a) the first irrecoverable failure and (b) assignment of both primary error source and impact. AgentAtlas distinguishes “taxonomy-aware” and “taxonomy-blind” modes:

• Taxonomy-aware: Annotators (human or model) receive a closed-set menu of the nine error and five impact categories.
• Taxonomy-blind: Only a free-form failure description is produced, which must be mapped to the taxonomy post hoc via substring rules or learned mappers.

Empirically, taxonomy-aware prompting yields apparent accuracies in the $i$1 range, while taxonomy-blind mappings yield a much tighter and lower band of $i$2–$i$3, exposing the effect of prompt scaffolding (Mazaheri et al., 19 May 2026).

For human annotation, inter-annotator agreement (Cohen’s κ) reaches at least $i$4 in pilot studies (AgentRx, AgentProcessBench), provided labelers work with rigorous definitions, edge-case discussions, and consistent assignment of the “first unrecoverable” (Mazaheri et al., 19 May 2026).

5. Taxonomy Structure and Visualization

The error taxonomy is inherently hierarchical and can be compactly structured as follows:

$i$8

Each failed trajectory is located at the intersection of one error source and one impact in a $i$5 grid (Mazaheri et al., 19 May 2026).

6. Integration with Broader Error Taxonomies and LLM Evaluation

The ErrorAtlas taxonomy instantiated for agent trajectories is situated within a broader ecosystem of LLM failure taxonomies, including the 17-category hierarchy developed via ErrorMap (Ashury-Tahan et al., 22 Jan 2026). The latter addresses generic LLM outputs, supporting deeper subcategorization (such as Logical Reasoning Error, Computation Error, Missing Required Element, Tool/API Usage Error, etc.), and is derived through an LLM-assisted two-stage pipeline:

1. Label Extraction: For each model error, a “judge” LLM produces a structured short error label and supporting rationale.
2. Taxonomy Induction and Assignment: Candidate error clusters are induced and refined by prompting the LLM, followed by final category assignment per error instance.

Coverage over diverse datasets exceeds $i$6; agreement with meta-judges is $i$7 at the assignment level (Ashury-Tahan et al., 22 Jan 2026).

7. Practical Application in Model Evaluation, Benchmarking, and Analysis

Application proceeds in a structured workflow:

1. Prediction and Error Detection: For each instance, run the model and assign a pass/fail flag by comparing to the task metric.
2. Error Labeling and Taxonomy Assignment: For each error, extract a brief label (manual or LLM), then map to the fixed set of error categories.
3. Metrics and Visualization: Compute prevalence per error category. Visualize via bar charts or error fingerprints to enable model-to-model comparison and guide targeted debiasing or capability improvements.

Analysis at the category level empowers practitioners to monitor specific failure types longitudinally (e.g., tracking reduction in “wrong tool selection” after agent retraining) and to audit the behavioral axes emphasized (or neglected) by existing benchmarks (as shown by AgentAtlas benchmark-coverage audits) (Mazaheri et al., 19 May 2026).

---

References:

• AgentAtlas nine-category agent trajectory taxonomy, orthogonal impact labels, annotation protocols, evaluation methodology, and practical integration: (Mazaheri et al., 19 May 2026).
• ErrorMap methodology, 17-category ErrorAtlas for generic LLM outputs, derivation workflow, and empirical evaluation: (Ashury-Tahan et al., 22 Jan 2026).