---
title: ErrorAtlas Taxonomy for LLM Failures
url: https://www.emergentmind.com/topics/erroratlas-taxonomy
type: topic
---

# ErrorAtlas Taxonomy for LLM Failures

The ErrorAtlas Taxonomy defines a comprehensive framework for characterizing, annotating, and analyzing the failure modes of large language models (LLMs) and agentic systems. Emerging from critical insights that scalar task accuracy alone is insufficient to understand and improve complex model behavior, ErrorAtlas organizes observable errors into distinct, interpretable categories and provides protocols for systematic error collection and evaluation. Its two major instantiations—one agent trajectory-focused (AgentAtlas) and one general model-output-focused (via ErrorMap)—enable high-resolution diagnostics for both agentic and non-agentic benchmarks, supporting a transition from outcome-centric to explanation-aware evaluation in LLM research and deployment [2605.20530][2601.15812].

## 1. ErrorAtlas for Agent Trajectories: Nine-Category Taxonomy

The AgentAtlas variant of ErrorAtlas introduces a nine-category taxonomy to classify failures at the level of agent trajectories, designed to capture the primary locus and nature of unrecoverable mistakes:

1. **Goal misinterpretation**: The agent misconstrues the user’s intent, operating on the wrong task or violating high-level constraints.  
2. **Wrong tool selection**: Inappropriate tool, API, or action modality is chosen, resulting in ineffective or futile subsequent actions.
3. **Wrong argument / wrong target**: Given a correct action, execution fails due to argument, identifier, path, or parameter error.
4. **Observation failure**: The agent misreads or misinterprets environmental feedback or tool output, acting on stale or incorrect information.
5. **Constraint violation**: The agent disregards explicit user, policy, or permission constraints.
6. **Recovery failure**: The agent continues after a recoverable misstep without rolling back, replanning, or regrounding.
7. **Looping or over-action**: The agent executes excessive or repeated actions beyond the task's logical endpoint.
8. **Unsafe trust of external content**: The agent executes or trusts unvetted, potentially malicious outputs or data.
9. **State or memory contamination**: Persistent context or memory is contaminated by stale or adversarial information, corrupting future decisions.

Every failed trajectory is annotated at the precise step of the first unrecoverable error, assigning one and only one of these labels. Viable trajectories receive a “Valid” tag instead [2605.20530].

## 2. Hierarchical Annotation Schema: Orthogonal Error Source and Impact

ErrorAtlas employs a two-field schema to capture both the nature and consequences of failures, enhancing interpretability and auditability:

- **primary_error_source**: Indicates which error class (the nine categories listed, plus “valid” for no-failure) is assigned.
- **impact**: Independently specifies the effect of the error, drawn from five classes:
  - *unsafe_side_effect*: Unintended, possibly destructive, changes to data or environment.
  - *privacy_leak*: Exposure of confidential or sensitive information.
  - *wrong_final_state*: Incorrect task result from the user’s perspective.
  - *unnecessary_cost*: Inefficient use of compute, time, or other resources.
  - *no_impact*: Error had no persistent effect.

These fields are orthogonal, supporting granular analysis of relationships such as which error sources have a propensity to cause high-severity impacts [2605.20530].

| Error Source (primary_error_source) | Example Consequences (impact)        |
|-------------------------------------|--------------------------------------|
| wrong_tool                          | wrong_final_state, unnecessary_cost  |
| recovery_failure                    | unsafe_side_effect                   |
| unsafe_trust                        | privacy_leak, unsafe_side_effect     |

## 3. Formal Notation and Metrics for Taxonomy Evaluation

The taxonomy supports rigorous evaluation using standard classification metrics, supporting both single-field and joint analysis. Let $N$ denote the number of trajectories; for trajectory $i$, $y_i$ is the gold primary label, $\hat{y}_i$ the model’s prediction, $z_i$ the gold impact label, and $\hat{z}_i$ the predicted impact. Key metrics:

- **primary_src_accuracy**: $\frac{1}{N}\sum_{i} [\hat{y}_i = y_i]$
- **primary_src_macroF1**: Macro-averaged $F_1$ over all nine error classes.
- **impact_accuracy**: $\frac{1}{N}\sum_{i} [\hat{z}_i = z_i]$
- **impact_macroF1**: Macro-averaged $F_1$ over all five impact classes.
- **joint_accuracy**: $\frac{1}{N}\sum_{i} [\hat{y}_i = y_i \wedge \hat{z}_i = z_i]$
- **exact_step_acc** (optional): Fraction of exact step localization matches.
- **±1_step_acc** (optional): Fraction within one step of the gold critical step.

These enable comparability across models, labeling pipelines, and benchmarks [2605.20530].

## 4. Annotation Methodology and Agreement

Annotation requires determination of (a) the first irrecoverable failure and (b) assignment of both primary error source and impact. AgentAtlas distinguishes “taxonomy-aware” and “taxonomy-blind” modes:

- **Taxonomy-aware**: Annotators (human or model) receive a closed-set menu of the nine error and five impact categories.
- **Taxonomy-blind**: Only a free-form failure description is produced, which must be mapped to the taxonomy post hoc via substring rules or learned mappers.

Empirically, taxonomy-aware prompting yields apparent accuracies in the $[0.69, 0.95]$ range, while taxonomy-blind mappings yield a much tighter and lower band of $0.54$–$0.62$, exposing the effect of prompt scaffolding [2605.20530].

For human annotation, inter-annotator agreement (Cohen’s κ) reaches at least $0.88$ in pilot studies (AgentRx, AgentProcessBench), provided labelers work with rigorous definitions, edge-case discussions, and consistent assignment of the “first unrecoverable” [2605.20530].

## 5. Taxonomy Structure and Visualization

The error taxonomy is inherently hierarchical and can be compactly structured as follows:

```
Trajectory failure taxonomy:
├─ primary_error_source
│   ├ wrong_tool
│   ├ wrong_argument
│   ├ missed_constraint
│   ├ observation_misread
│   ├ failure_to_recover
│   ├ looping_or_over_action
│   ├ unsafe_trust
│   ├ state_memory_contamination
│   └ valid
└─ impact
    ├ unsafe_side_effect
    ├ privacy_leak
    ├ wrong_final_state
    ├ unnecessary_cost
    └ no_impact
```

Each failed trajectory is located at the intersection of one error source and one impact in a $9\times5$ grid [2605.20530].

## 6. Integration with Broader Error Taxonomies and LLM Evaluation

The ErrorAtlas taxonomy instantiated for agent trajectories is situated within a broader ecosystem of LLM failure taxonomies, including the 17-category hierarchy developed via ErrorMap [2601.15812]. The latter addresses generic LLM outputs, supporting deeper subcategorization (such as Logical Reasoning Error, Computation Error, Missing Required Element, Tool/API Usage Error, etc.), and is derived through an LLM-assisted two-stage pipeline:

1. **Label Extraction**: For each model error, a “judge” LLM produces a structured short error label and supporting rationale.
2. **Taxonomy Induction and Assignment**: Candidate error clusters are induced and refined by prompting the LLM, followed by final category assignment per error instance.

Coverage over diverse datasets exceeds $95\%$; agreement with meta-judges is $92\%$ at the assignment level [2601.15812].

## 7. Practical Application in Model Evaluation, Benchmarking, and Analysis

Application proceeds in a structured workflow:

1. **Prediction and Error Detection**: For each instance, run the model and assign a pass/fail flag by comparing to the task metric.
2. **Error Labeling and Taxonomy Assignment**: For each error, extract a brief label (manual or LLM), then map to the fixed set of error categories.
3. **Metrics and Visualization**: Compute prevalence per error category. Visualize via bar charts or error fingerprints to enable model-to-model comparison and guide targeted debiasing or capability improvements.

Analysis at the category level empowers practitioners to monitor specific failure types longitudinally (e.g., tracking reduction in “wrong tool selection” after agent retraining) and to audit the behavioral axes emphasized (or neglected) by existing benchmarks (as shown by AgentAtlas benchmark-coverage audits) [2605.20530].


---
**References:**  
- AgentAtlas nine-category agent trajectory taxonomy, orthogonal impact labels, annotation protocols, evaluation methodology, and practical integration: [2605.20530].  
- ErrorMap methodology, 17-category ErrorAtlas for generic LLM outputs, derivation workflow, and empirical evaluation: [2601.15812].

Source: https://www.emergentmind.com/topics/erroratlas-taxonomy