---
title: 'ErrorMap Pipeline: LLM Error Diagnosis'
url: https://www.emergentmind.com/topics/errormap-pipeline
type: topic
---

# ErrorMap Pipeline: LLM Error Diagnosis

ErrorMap Pipeline

The ErrorMap pipeline is a structured framework for systematic diagnosis, categorization, and quantification of failure modes in large language model (LLM) outputs. By translating raw, incorrect predictions into multi-tiered taxonomies and interpretable failure signatures, ErrorMap enables precise model debugging and cross-model comparability. The system underpins the construction of ErrorAtlas, a static, comprehensive taxonomy that exposes prevailing and underexplored LLM error types. ErrorMap operates purely on model outputs and ground-truth references, with no reliance on task-specific engineering or model internals, making it broadly applicable across LLMs and benchmarks [2601.15812]. 

## 1. Two-Stage Error Diagnosis and Taxonomy Induction

ErrorMap employs a two-stage procedure to extract structured diagnostics from model outputs and construct a robust, hierarchical taxonomy:

- **Stage 1: Per-Instance Error Analysis**
  - **Inputs:** For each failed model–dataset instance $(i, m)$, the system takes the input text $x_i$, gold answer $y_i$ (if available), up to $K$ informative correct predictions (ICPs) from other models, and one erroneous prediction $\hat{y}_i^m$ from the target model.
  - **Process:** The input is formatted into a prompt for a designated grading LLM (e.g., gpt-oss-120b). The LLM produces a structured JSON analysis including:
    - `required_criteria`: a list of necessary reasoning steps or sub-tasks, with evidence-based checks for satisfaction.
    - `error_title`: a concise label identifying the first major error.
    - `error_summary`: a brief diagnosis of root cause.
- **Stage 2: Taxonomy Hierarchy Building**
  - **2.a Category Generation:** All free-form `error_title` strings from Stage 1 are clustered into broader categories using iterative LLM-in-the-loop clustering (generate, refine, review).
  - **2.b Error Assignment:** Each per-instance error label is mapped to a taxonomy category, creating a populated hierarchy spanning categories, sub-categories, and instance mappings.

This twin-pipeline yields a per-run multi-level error taxonomy that can be held static for consistent future application or evolved with new data and models.

## 2. Formal Metrics and Failure Signature Representation

The ErrorMap formalism is grounded in clear mathematical definitions and metrics.

- **Failure Instance Detection:** For dataset $D=\{x_i,y_i\}_{i=1}^N$ and model $m$, the task-specific score $\mathrm{score}(m,x_i)$ is compared to threshold $\tau$. The failure indicator is defined:
  $$
  E_{m,i} = \begin{cases}
    1 & \text{if } \mathrm{score}(m,x_i)<\tau\\
    0 & \text{otherwise}
  \end{cases}
  $$
- **Taxonomy Construction** proceeds via assignment $\gamma: \mathcal{L} \rightarrow \{1,\ldots, K\}$ mapping free-form error labels to one of $K$ categories $\{C_j\}$.

- **Failure signature vector (for model $m$):**  
  Let $n_{m,j} = \sum_{i=1}^N E_{m,i}\cdot 1[\gamma(L_{m,i})=j]$.
  The total number of failures is $N_m^{\mathrm{fail}} = \sum_{i=1}^N E_{m,i}$.
  The normalized failure signature $f^m\in\Delta^{K-1}$ is given by:
  $$
  f_j^m = \frac{n_{m,j}}{N_m^{\mathrm{fail}}}
  $$
  which is a model-specific vector quantifying the relative prevalence of failure categories.

- **Aggregate and Comparative Metrics:**  
  - Overall error rate $err(m,D) = N_m^{\mathrm{fail}} / N$.
  - Model-to-model failure signature distance $d(m,m')=\|f^m-f^{m'}\|_1$ or $\|f^m-f^{m'}\|_2$.

## 3. Core Algorithms in ErrorMap

The principle algorithms are implemented in LLM-centric pipelines with minimal task-specific intervention.

- **Per-Instance Error Analysis (Algorithm 1):** Prompts the LLM with instance data to output a diagnostic JSON—including required criteria (each with fields `criterion`, `present`, `quality`, `evidence`, `comment`), `error_title`, and `error_summary`.
- **Taxonomy Generation (Algorithm 2):** Runs a clustering LLM prompt for category generation, refines via batches, and finalizes taxonomy with review.
- **Error Assignment (Algorithm 3):** Batch classifies each error title against the taxonomy, populates category→instance mappings.
- **Failure Signature Computation (Algorithm 4):** Canonically computes the normalized failure signature vector for each model given the error-label/category mappings.

All algorithms leverage LLM-based reasoning for both the extraction of fine-grained, context-dependent error types and the abstraction needed for robust clustering.

## 4. ErrorAtlas: Static Failure Taxonomy and Prevalence

Application of ErrorMap across 83 models and 35 datasets, with $\sim$7,000 sampled failure instances, led to the creation of ErrorAtlas—a fixed taxonomy that covers $\geq95\%$ of observed LLM failures.

| Category                     | Description                                                          |
|------------------------------|---------------------------------------------------------------------|
| Logical Reasoning Error       | Fails in inference or applying correct reasoning steps.             |
| Missing Required Element      | Omits mandatory sections, fields, or specified content.             |
| Computation Error             | Incorrect numerical, algebraic, or geometric results.               |
| Incorrect Identification      | Mislabels or misidentifies objects, concepts, or entities.          |
| Specification Misinterpret.   | Misunderstands task requirements or misformats outputs.             |
| Output Formatting Error       | Violates required structure, punctuation, case, or markup rules.    |
| Irrelevant/Extraneous Content | Generates off-topic or unnecessary information.                     |
| Counting/Enumeration Error    | Over-/under-counts, omits cases in combinatorial steps.             |
| Answer Selection Error        | Maps a solution to the wrong multiple-choice option.                |
| Incomplete Reasoning          | Omits essential explanation, proof steps, or justification.         |
| Factual Error                 | Supplies inaccurate or fabricated domain knowledge.                 |
| Tool/API Usage Error          | Misuses or omits required tool or API calls.                        |
| Naming/Symbol Error           | Uses incorrect symbols, variable names, or identifiers.             |
| Inappropriate Refusal         | Unjustifiably refuses to answer.                                    |
| Unit Conversion Error         | Incorrect unit or percentage conversions.                           |
| False Positive Detection      | Flags errors or anomalies that do not exist.                        |
| Error Detection Failure       | Fails to recognize existing mistakes (missed error).                |

Prevalence statistics (e.g., "Missing Required Element" appears in 31/35 datasets and 82/83 models, prevalence 15.6%; "Logical Reasoning Error" is in 25 datasets, 56 models, prevalence 9.1%) demonstrate wide coverage and specificity [2601.15812].

## 5. Representative Examples and Model Comparisons

Concrete applications elucidate the pipeline's interpretability and practical value:

- **Instance Diagnosis:** For the OmniMath dataset and Gemini 2.0 Flash Lite, when asked "Solve $a^3 + b^3 + c^3 = 2001$ in positive integers," the model listed only (1,10,10) with no completeness check. ErrorMap labeled this as "Incomplete solution set" and pinpointed partially-met criteria ("examine all feasible a", "list all permutations").
- **Failure Signature Comparisons:** Figure 2 contrasts models (e.g., Claude 3.5 Haiku vs. Gemini 2.0 Flash Lite) by per-category failure fraction, such as high "Missing Required Element" for Gemini and amplified "Logical Reasoning Error" for Claude, illustrating model-specific weaknesses.

## 6. Generalizability and Deployment Practices

ErrorMap is agnostic to the underlying model architecture and dataset specifics. Its sole requirements are input–output pairs (optionally with ground truth and correct references), a judge LLM, and standard prompt templates. The pipeline does not demand token-level alignments or model white-box access.

- **Immediate Applicability:** Any LLM or dataset, including emergent benchmarks, can be analyzed by running Stage 1 and Stage 2.b against ErrorAtlas to produce failure signatures.
- **Practical Deployment:** For expediency, users often bypass Stage 2.a (new taxonomy induction), relying on ErrorAtlas for robust, static categorization. Running Stage 1 for error titles and Stage 2.b for assignment enables rapid, cost-efficient diagnosis without loss of diagnostic integrity.

*This approach yields both detailed, per-instance explanations and condensed, model-level diagnostics, advancing interpretability and comparability in LLM evaluation frameworks* [2601.15812].

Source: https://www.emergentmind.com/topics/errormap-pipeline