ErrorMap: Hierarchical Error Analysis for ML

• ErrorMap Framework is a model-agnostic methodology that decomposes machine learning errors into interpretable, actionable categories.
• It integrates per-instance error analysis with hierarchical taxonomy building to systematically cluster and diagnose failures in LLMs and digital maps.
• It employs advanced techniques for failure signature extraction and probabilistic mode assignment, aiding model debugging, selection, and improvement.

ErrorMap Framework is a family of model-agnostic methodologies for error diagnosis and error localization in machine learning systems that integrates structured per-instance error analysis with hierarchical error taxonomy construction. Two instantiations are prominent: (i) charting failure sources in LLMs through a hierarchical error taxonomy and per-instance signature extraction (“ErrorMap” and “ErrorAtlas”), and (ii) identifying map errors in digital road networks via a semi-interacting multiple-model (sIMM) filter for on/off-road vehicle trajectory inference. In both contexts, ErrorMap frameworks deliver systematic procedures for transforming raw prediction failures into actionable insight, granular error categories, and improved downstream debugging and validation (Ashury-Tahan et al., 22 Jan 2026, Murphy et al., 2018).

1. Formal Objectives and Definitions

The primary objective of an ErrorMap framework is to decompose model prediction failures into interpretable, actionable categories. For LLMs, ErrorMap processes every mispredicted example $(x_i, y_i, \hat{y}_i)$ and delivers both a free-form error label $\ell_i$ and a structured failure breakdown. For map learning, the ErrorMap (sIMM) filter produces probabilistic mode assignments indicating whether observed data supports the current model (e.g., road map) or exposes a likely model defect (e.g., unmapped road segment). The objectives are to:

• Disentangle heterogeneous error sources (formatting, logical, computational, specification, or map-induced)
• Provide interpretable per-instance diagnosis and aggregate taxonomies
• Enable comparison across models, datasets, and tasks via a normalized error signature
• Flag latent or hidden model and data defects for validation and remediation

Mathematically, the LLM ErrorMap defines a set of error categories $F$ and computes the “failure signature” vector $S(M, D) \in \mathbb{N}^{|F|}$ such that $$S(M, D)[f] = |\{i : \delta(\hat{y}_i, y_i) = 1 \land \mathrm{error\ category\ of\ } i = f\}|$$, where $\delta$ is the primary failure indicator.

In the map learning case, the framework computes a mixture posterior $$p(X_t \mid y_{1:t}) = \sum_{m\in\{r,g\}} \mu_t^m\, p(X_t \mid M_t = m, y_{1:t})$$, with mode weights indicating map compatibility (Murphy et al., 2018).

2. Methodology: Stages and Algorithms

2.1. LLM ErrorMap: Two-stage Pipeline

• Stage 1: Per-instance Error Analysis
  • Required criteria (reasoning steps, formatting, factual checks)
  • Per-criterion assessment (presence, quality, evidence)
  • Short error title $\ell_i$
  • Concise error summary

$F$9

• Stage 2: Hierarchical Taxonomy Building
  • Category generation: iterative LLM clustering (≤25 clusters initially, refined in batches, reviewed for clarity)
  • Assignment: Each $\ell_i$2 assigned to a category $\ell_i$3 via LLM prompting

$S(M, D) \in \mathbb{N}^{|F|}$0

• Metrics: Counts, proportions, and distances on $\ell_i$4 for comparison.

2.2. Map Learning ErrorMap: sIMM Algorithm

• Mixture Model: Modes $\ell_i$5 (on-road, HMM), $\ell_i$6 (off-road, Kalman filter). Mode switches controlled by Markov chain, with semi-interaction to keep off-road filtering closed-form.
• Forward Filtering:

1. Independently update HMM (on-road) and Kalman filter (off-road) posteriors
2. Compute mode transition likelihoods $\ell_i$7
3. Blend mode weights $\ell_i$8 using cost-weights and normalization

• Backward MAP Sampling:

Maximizes joint trajectory and mode sequence posterior using recursion and matching $\ell_i$9 terms for all mode-switch combinations.

• Spatial Clustering:

Off-road mode assignments with high weight ($F$0) are spatio-temporally clustered to flag systematic map errors.

3. Taxonomy Construction and Failure Signature

In the LLM ErrorMap, the outcome is a hierarchical taxonomy (“ErrorAtlas”) with 17 top-level categories (e.g., Logical Reasoning Error, Missing Required Element, Computation Error, Specification Misinterpretation), each with 2–3 subcategories (depth 2–3). Every error instance maps to exactly one leaf in this taxonomy. Stopping criteria: clusters $F$1 labels or max depth 3; error sampling at $F$210% per model/dataset constrains resource requirements.

Failure signature $F$3 summarizes per-category error distributions, normalized to a probability distribution $F$4 for comparative analysis. Distances between model signatures (e.g., cosine similarity) enable global comparison. On the map learning side, trajectory segments with persistent high off-road mode weights or mismatches guide the flagging of map error types: missing roads, illegal turns, incorrect one-way directions.

Error Category	Description
Logical Reasoning Error	Fails logical inference or stepwise reasoning
Missing Required Element	Omits mandatory content/sections
Computation Error	Incorrect numerical/algebraic/geometric results
Specification Misinterpretation	Misreads task requirements or output format
...	...

Error Category	#Datasets	#Models	Prevalence %
Missing Required Element	31	82	15.6
Specification Misinterpretation	27	70	11.5
Logical Reasoning Error	25	56	9.1
...	...	...	...

4. Experimental Evaluation and Performance

The LLM ErrorMap was validated across 35 benchmarks (reasoning, medical QA, coding, etc.) and 83 models (open and proprietary). Representative thresholds: $F$5 if the score $F$670% of maximum; for classification, simply “wrong.” Resource requirements were $F$77200 model inferences (3 hours on 8×A100) for ErrorAtlas construction.

Metrics:

• Coverage: 95.2% of sampled errors mapped to one of the 17 categories
• Label accuracy: Meta-judge LLM agreed with category choices 92% of the time
• Robustness: Label-similarity 53% (45% semantic matches); taxonomy perturbation F1=0.88 (upper), 0.69 (lower)
• Statistical significance: Per-category differences analyzed using binomial tests

The map learning variant was tested on large GPS trace datasets, with off-road “excursions” successfully correlating to unmapped features.

5. Primary Insights and Applications

Key findings for LLMs include the systematic underestimation of errors not directly tied to target benchmark skills: for example, in reasoning tasks (MMLU-Pro, GPQA, OmniMath), $F$844% of errors are mechanical (computation, formatting, omission). Underexplored categories such as “Missing Required Element” (15.6%) and “Specification Misinterpretation” (11.5%) predominate over classic reasoning or factuality mistakes.

ErrorMap fingerprints models by failure type: LLMs such as Gemini 2.0 Flash Lite show high rates of omission; Claude 3.5 Haiku produces more logical errors; Mixtral 8×22b is susceptible to computation mistakes.

Primary applications:

• Model debugging: Detailed error profiles reveal where to focus improvement.
• Benchmark analysis: Hierarchical taxonomy enables ground-truth alignment and advances beyond aggregate accuracy.
• Model selection: Filtering candidates with fewer “Factual Errors” for high-risk domains (e.g., medicine, law).
• Domain-specific audits: Direct assessment of errors for regulatory and safety assurance.

For map learning, clusters of persistent off-road assignments enable automatic detection and localization of (1) missing roads, (2) illegal turn restrictions, and (3) wrong directionalities. Spatial clustering and trajectory context enable concrete proposals for map edits.

6. Limitations and Open Challenges

Several open issues and limitations are recognized:

• Prediction-only analysis: No visibility into model internal activations (hidden states, gradients)
• Soft category boundaries: Some errors straddle multiple categories, precluding crisp partitioning
• Dependence on generative models: Pure classification settings are not deeply analyzable
• LLM judgment circularity: LLMs are both subjects and evaluators of model errors, though empirical results indicate verification is more reliable than generation for error labeling
• Domain generality: Taxonomy may under-represent niche error modes outside benchmark coverage

A plausible implication is that extending ErrorMap frameworks to new domains and modalities will require further refinement of both per-instance diagnostic mechanisms and taxonomy expansion.

7. Broader Impact and Future Directions

By operationalizing “failure-centric” evaluation, ErrorMap frameworks transition model assessment from task-level accuracy to informative, structured explanation of why specific failures occur. This enables targeted model improvement, more meaningful benchmarking, and support for risk-sensitive model deployment. The opt-in update pipeline for ErrorAtlas and the modularity of sIMM for map error detection suggest long-term extensibility. Open challenges include integration into real-time, continuous evaluation infrastructure, transfer to new domains (e.g., vision, speech), and refinement of error category assignment when datasets or model families shift.

ErrorMap methodologies thus represent a foundational advance in systematic, interpretable model diagnosis, setting new standards for both LLM evaluation and robust map learning (Ashury-Tahan et al., 22 Jan 2026, Murphy et al., 2018).