--- title: Exception Taxonomy in Agentic Artifacts url: https://www.emergentmind.com/topics/exception-taxonomy-across-agentic-artifacts type: topic --- # Exception Taxonomy in Agentic Artifacts Agentic artifacts—systems composed of large language models (LLMs) and external tools executing complex workflows—routinely experience exceptions that span reasoning, planning, coordination, and execution phases. Exception taxonomies provide formal schemes to classify, analyze, and address such failures, enabling systematic debugging, tooling, and empirical evaluation. Two leading efforts, TRAIL and SHIELDA, have introduced detailed taxonomies for exception analysis in agentic workflows, structured around cognitive, architectural, and operational axes [2505.08638] [2508.07935]. ## 1. The Motivating Complexity of Agentic Exceptions Exception analysis in agentic artifacts is uniquely challenging due to their multi-phase, multi-tool, and often multi-agent nature. Unlike traditional software, where exceptions can often be localized to a line of code or explicit stack trace, agentic traces intertwine LLM cognition (reasoning steps, prompt engineering), distributed orchestration (subtasks, memory, context), and system-level operations (tool APIs, external environments). The interplay among these layers introduces new failure modes and necessitates robust, scalable taxonomies to support annotation, diagnosis, and recovery [2505.08638]. A robust taxonomy must capture not only classic execution errors but also reasoning mistakes, planning flaws, and cross-artifact interactions [2508.07935]. ## 2. The TRAIL Taxonomy: Hierarchy and Error Classes TRAIL partitions agentic exceptions into three root categories: Reasoning Errors (R), Planning & Coordination Errors (P), and System Execution Errors (S), collectively covering all observed failure modalities: $$ E = R \cup P \cup S,\quad R \cap P = R \cap S = P \cap S = \emptyset $$ **1. Reasoning Errors (R):** Internal cognitive failures of the LLM, encompassing: - *Hallucinations* (language-only and tool-related) - *Information Processing* (poor retrieval, misinterpretation) - *Decision Making* (incorrect problem identification, tool selection errors) - *Output Generation* (syntactic formatting errors, instruction non-compliance) **2. Planning & Coordination Errors (P):** Failures in managing context, workflow, and subtask orchestration, including: - *Context Handling Failures* and *Resource Abuse* - *Goal Deviation* and *Task Orchestration Errors* **3. System Execution Errors (S):** Failures arising from interaction with the external environment, such as: - *Configuration Issues* (tool definition, environment setup) - *API and System Issues* (HTTP error codes, service failures) - *Resource Management* (exhaustion, timeouts) Each root is subdivided into fine-grained types, for example, output generation errors distinguish invalid structured outputs from instruction non-compliance. See the table below for an overview: | Category | Subcategory | Example | |------------------------------|--------------------------------------|-----------------------------------------------------------| | Reasoning (R) | Tool-Related Hallucination | Fictitious tool output field | | Planning & Coordination (P) | Task Orchestration Error | Patch generated before file read | | System Execution (S) | API Rate Limiting (429) | Search API overloaded due to polling | These categories are operationalized in the TRAIL dataset, consisting of 148 traces (spanning both single- and multi-agent systems), yielding 841 annotated errors [2505.08638]. Approximately 70% of errors fall into Reasoning, 20% into Planning & Coordination, and 8% into System Execution, with Output Generation (a Reasoning subcategory) contributing 42% of all errors. Notably, less frequent errors such as API/service failures are often high impact despite low prevalence. ## 3. The SHIELDA Taxonomy: Artifact-Centric Exception Enumeration SHIELDA introduces an orthogonal, artifact-driven taxonomy over 12 agentic workflow components (“artifacts”), each paired with one or more exception types (36 total), and each exception tagged by workflow phase: Reasoning/Planning (RP), Execution (E), or both (RP/E). The formal structure is: $$ A = \{a_1, \ldots, a_{12}\}, \quad E(a) = \{e_{a,1}, \ldots, e_{a,n_a}\}, \quad \phi: e \mapsto \{\mathrm{RP},\mathrm{E},\mathrm{RP/E}\} $$ with artifacts including Goal, Context, Reasoning, Planning, Memory, Knowledge Base (KB), Model, Tool, Interface, Task Flow, Other Agent, and External System. Key exception types include: - **Goal:** Ambiguous Goal (underspecified intent), Conflicting Goal - **Memory:** Poisoning, Outdated Memory, Misaligned Recall - **Model:** Token Limit Exceeded, Output Validation Failure, Output Handling Exception - **Tool:** Tool Invocation Exception, Tool Output Exception, Unavailable Tool - **Other Agent:** Communication Exception, Agent Conflict, Role Violation A fragment of SHIELDA’s taxonomy table: | Artifact | Exception Type | Phase | |---------------|-----------------------------|----------| | Goal | Ambiguous Goal | RP | | Model | Output Validation Failure | E | | Tool | Tool Invocation Exception | E | | Other Agent | Agent Conflict | E | Each type is accompanied by a definition, illustrative example, and identified root cause. For instance, “Memory Poisoning” (RP/E) refers to malicious/misleading entries in the memory store, e.g., adversarial-crafted demonstrations causing unsafe plans, typically arising from unfiltered writes or adversarial input. Exception modalities in SHIELDA span intent ambiguity, context loss, model syntax errors, inter-agent protocol violations, and external attacks [2508.07935]. ## 4. Comparative Structure, Granularity, and Coverage TRAIL adopts a phase-oriented hierarchy (Reasoning, Planning, Execution), with subtypes reflecting error process and agent cognition, suited for span-level annotation and empirical prevalence analysis. SHIELDA, in contrast, employs an artifact-phase matrix, enabling attribution of exceptions to specific architectural components and phases. SHIELDA’s granularity (36 types across 12 artifacts) facilitates fine-grained root-cause classification, essential for linking exception handling to concrete workflow state and recovery strategies. A plausible implication is that TRAIL excels in evaluating model-centric and cognitive error rates, especially in open-domain reasoning, while SHIELDA is tailored for cross-phase diagnostics and artifact-scoped handling patterns—each serving distinct but complementary purposes in end-to-end agentic workflow evaluation. ## 5. Statistical Prevalence and Empirical Impact Within TRAIL’s 148 real-world traces (GAIA and SWE-Bench), the dataset annotates 841 errors (average 5.68 per trace, median 5), with high inter-annotator agreement at the span level (95%). Reasoning dominates error classes, but low-prevalence categories (API failures, resource exhaustion) exhibit disproportionate impact and thus are critical detection targets [2505.08638]. SHIELDA does not present population statistics but provides artifact-by-artifact exception mapping, supporting targeted handling and escalation strategies [2508.07935]. Impact analysis in TRAIL reveals that, although Output Generation errors are most common, approximately 44% are low impact; conversely, rare categories such as authentication or resource outages are often high impact—motivating both frequency- and severity-aware evaluation frameworks. ## 6. Applicability Across Domains and Systems Both taxonomies demonstrate extensibility across workflow settings: - **Single-Agent Systems** (e.g., SWE-Bench): System Execution and Output Generation errors dominate, particularly in software engineering and code patching tasks. - **Multi-Agent Systems** (e.g., GAIA): Planning & Coordination failures (task orchestration, context handling) are frequent—often as agents hand off or synchronize subtasks. - **Tool-Augmented Reasoning:** Hallucinations may arise as either language-only or tied to external tool invocation; API and protocol issues surface in tool interactions. Domains empirically covered include software engineering (code editing, patch creation) and open-world information retrieval (web search, document inspection). Every exception root and artifact appears in both single- and multi-agent settings; only relative frequencies differ [2505.08638], [2508.07935]. ## 7. Formal Representation, Taxonomy Tables, and Figures TRAIL formalizes its hierarchy as: - $R = \{R_{1a}, R_{1b}, R_{2a}, R_{2b}, R_{3a}, R_{3b}, R_{4a}, R_{4b}\}$ - $P = \{P_{1a}, P_{1b}, P_{2a}, P_{2b}\}$ - $S = \{S_{1a}, S_{1b}, S_{2a}, S_{2b}, S_{2c}, S_{2d}, S_{3a}, S_{3b}\}$ SHIELDA's structure is succinctly captured as a mapping $A \to E(a)$ with each $e \in E(a)$ tagged with phase $\phi(e)$. Table representations in both frameworks organize artifacts and exceptions along with phase labels, supporting transparent annotation and downstream handling logic [2505.08638], [2508.07935]. TRAIL's Figure 1 and SHIELDA's Table 1 provide canonical visualizations of these structures. --- Exception taxonomies in LLM-driven agentic artifacts enable multi-granular failure localization, facilitate reproducible trace annotation, and underpin structured exception handling and recovery strategies. The TRAIL and SHIELDA frameworks together define the current state-of-the-art for exception analysis in this domain [2505.08638], [2508.07935]. Source: https://www.emergentmind.com/topics/exception-taxonomy-across-agentic-artifacts