Prediction Error Taxonomy in ML/AI

• Prediction Error Taxonomy is a systematic framework that defines and categorizes errors in ML/AI models based on their origins and fixability.
• It employs KL divergence metrics and gradient penalties to diagnose error accumulation in sequential and hierarchical models.
• The taxonomy distinguishes fixable errors from inherent unpredictability, guiding model improvements and risk-aware evaluation.

Prediction error taxonomy in machine learning and artificial intelligence provides a systematic framework for characterizing, quantifying, and interpreting different types and sources of errors in learning and inference systems. Such taxonomies are central for diagnostic purposes, development of new metrics, design of regularization losses, and proper evaluation of model performance—especially in sequential models and structured-output domains where errors are heterogeneous in their origin and impact.

1. Formal Differentiation of Prediction Errors

A rigorous prediction error taxonomy begins by distinguishing between error types according to their origin and their amenability to model improvement. For sequential autoregressive forecasting models, such as those used in atmospheric simulation, prediction errors may be formally represented by the divergence between model-generated output trajectories and reference (or ground-truth) trajectories at different lead times.

Let $x_{1:n}$ denote a trajectory drawn from the data distribution. Two conditional distributions are defined:

• $p_{(\mathrm{gen})}(x_{t+c} \mid x_{1:c})$: the autoregressive model whose error accumulation is under study,
• $p_{(\mathrm{truth})}(x_{t+c} \mid x_{1:c})$: the true conditional distribution (typically intractable).

The model's error at lead time $t$ is quantified as

$$\delta(t) \equiv \mathbb{E}_{x_{1:c}}[\,\mathrm{KL}(p_{(\mathrm{gen})}(x_{t+c} \mid x_{1:c}) \;\|\; p_{(\mathrm{truth})}(x_{t+c} \mid x_{1:c}))\,],$$

where $\mathrm{KL}(\cdot\|\cdot)$ is the Kullback-Leibler divergence. The growth of $\delta(t)$ as $t$ increases constitutes the error accumulation profile. In practice, the true conditional $p_{(\mathrm{truth})}$ is replaced by a continuous-time surrogate $p_{(\mathrm{cts})}$ that predicts directly without autoregressive rollouts, yielding the practical metric

$$\hat{\delta}(t) \equiv \mathbb{E}_{x_{1:c}}[\,\mathrm{KL}(p_{(\mathrm{gen})}(x_{t+c} \mid x_{1:c}) \;\|\; p_{(\mathrm{cts})}(x_{t+c} \mid x_{1:c}))\,].$$

This diagnostic is specifically designed to distinguish fixable model deficiencies from errors imposed by system-level unpredictability (chaos or under-resolution) (Parthipan et al., 2024).

2. Error Source Taxonomy: Fixable vs. Unfixable

Error taxonomy as defined in the context of ML atmospheric simulators systematically separates errors into:

• Fixable errors (model deficiencies):
  • Occur when the autoregressive model, due to compounding one-step prediction errors, enters regions of state space that a reference (CTS) never visits.
  • Typical manifestations include numerical blowups (NaNs), under-dispersive ensembles, and systematic biases away from the true attractor.
• Unfixable errors (fundamental unpredictability):
  • Arise from intrinsic system limitations such as chaos that bounds the predictability horizon, or from missing sub-grid variables and inadequate input resolution.
  • These are shared by the autoregressive model and CTS, hence $\hat{\delta}$ cancels their contribution, highlighting only model-improvable (fixable) discrepancies.

This taxonomy operationalizes the principle that error accumulation diagnostics and model development efforts should focus on amendable deficiencies rather than inherent limitations of the task (Parthipan et al., 2024).

3. Metrics and Diagnostic Methodologies

The taxonomy leads directly to the design of evaluation metrics that attribute error to specific sources:

• KL-based error accumulation metric $\hat{\delta}(t)$ quantifies, at each forecast horizon, the divergence between the generative model and its non-autoregressive surrogate reference.
• Gradient penalties for training: By incorporating a KL penalty term into the training objective—specifically targeting divergence from the CTS—the approach both diagnoses and regularizes against error accumulation due to rollouts:

$$\max_\theta \;\mathbb{E}_{x_{1:n} \sim \text{data}} \left[ \log p_{\mathrm{gen},\theta}(x_{c+1: c+n}|x_{1:c}) \right] - \frac{\lambda}{n}\sum_{t=1}^{n} \mathbb{E}[\mathrm{KL}(p_{\mathrm{gen},\theta}(x_{t+c}|x_{t+c-1}) \;\|\; p_{\mathrm{cts}}(x_{t+c}|x_{1:c}))]$$

Here, $\lambda>0$ balances one-step fit and long-run stability. No unproven metric or loss function is introduced; everything is derived as stated in (Parthipan et al., 2024).

• Practical computation: Ensembles are generated by rollouts from $p_{(\mathrm{gen})}$, marginal and reference distributions are approximated as Gaussians, and KL divergence is computed analytically.

4. Hierarchical Error Taxonomy in Classification

For classifiers and object detectors, prediction error can be further decomposed according to a task-specific class taxonomy. Rather than treating all misclassifications equally (flat scoring), hierarchical scoring metrics introduce a partial-credit structure based on the proximity of predicted and true classes within a tree-structured label space.

• Scoring tree: A directed, weighted tree $S=(V,E,w)$ rooted at $R$, with each path to a leaf summing to one. The topology and edge weights are domain-tuned.
• Hierarchical metrics:
  • Path Length Score (PL): Distance-based, symmetric, normalizes by tree diameter: $PL(T,P)=(H-d(T,P))/H$.
  • Lowest Common Ancestor Reward (L): Path-overlap-based, $L(T,P) = \sum_{e \in \vec{T}\cap\vec{P}} w(e)$.
  • Lowest Common Ancestor with Path Penalty (LPP): Combines path overlap and penalizes non-shared paths, normalized to $[0,1]$.
  • Standardized variants: True-path or predicted-path standardization ensures perfect score for any exact label match.
  • Hierarchical F1: Micro-averaged using per-class hierarchical precision/recall via LPP variants.

These metrics enable evaluators to distinguish between errors that are "near-miss" (e.g., predicting a sibling class node) and errors that are topologically distant in the class taxonomy, providing calibrated performance signals for model selection and risk-sensitive applications (Lanus et al., 6 Aug 2025).

5. Weighting Strategies and Impact on Error Interpretation

The hierarchical scoring framework permits the selection of edge-weighting strategies to tune the practical severity of different error types:

Weighting Strategy	Edge Distribution	Error Behavior
Decreasing	90% mass at root; remaining spread thinly	Subtree-coherence rewarded; deep misclassifications receive almost full credit if within correct subtree
Non-increasing	Mixed distribution	No strong favor to any level; encourages top-level correctness
Increasing	Little mass at root; mass increases toward leaves	Only exact leaf matches get high credit, sibling misclassifications are severely penalized

These strategies, as demonstrated in model evaluations, control whether the system favors cautious (higher-level) or aggressive (deeper, more specific) predictions. Choosing the appropriate tree and weighting scheme is central to aligning error measurement with operational requirements (Lanus et al., 6 Aug 2025).

6. Cross-Domain Generalization and Limitations

The error source taxonomy and the associated metricization, while developed for atmospheric simulators and structured classifiers, have broader relevance. Extension to other domains—such as robotics, finance, and language modeling—is possible provided:

• A suitable non-autoregressive reference (analogous to the CTS) can be specified or trained,
• Distribution divergences can be feasibly estimated (e.g., via analytical fit or learned critics),
• The reference model is of acceptable quality to prevent mis-guided regularization.

Limitations include computational cost of ensemble sampling, sensitivity to the auxiliary reference (especially in high-dimensional or multi-modal outputs), and assumptions about Gaussianity in one-step conditionals. Further sophistication could be obtained via alternative divergence metrics or adaptive noise perturbations. Nevertheless, the separation of fixable rollout errors and unavoidable system-level errors provides a principled, domain-agnostic basis for the diagnosis and mitigation of prediction errors throughout the ML/AI landscape (Parthipan et al., 2024, Lanus et al., 6 Aug 2025).

7. Experimental Findings and Practical Recommendations

Evidence from chaotic ODE systems (e.g., Lorenz 63, Lorenz 96) and real-world weather datasets shows that KL+noise-regularized models demonstrate reduced error accumulation and improved spread/skill characteristics compared to baselines. In hierarchical classification, micro-averaged hierarchical F1 and tunable detection offsets permit calibrated evaluation of models according to operational demands, with scoring tree configuration directly impacting model ranking and interpretability.

For implementation:

• Select tree topology and weighting reflecting domain risk preferences.
• Apply LPP with standardization (TPS or PPS) for mixed-level predictions.
• Use micro-averaged hierarchical F1 for overall comparison.
• Offsets for missed/ghost detections provide tuning for impact of detection errors.

Open-source Python implementations are available to automate hierarchical metric computation (Lanus et al., 6 Aug 2025).

---

Key references:

(Parthipan et al., 2024) Defining error accumulation in ML atmospheric simulators (Lanus et al., 6 Aug 2025) Hierarchical Scoring for Machine Learning Classifier Error Impact Evaluation