TRAIL Benchmark: Trace Debugging & Reasoning
- TRAIL is a benchmark that introduces a formal error taxonomy and 148 human-annotated traces to localize failures in agentic workflows.
- It employs span-level error localization over structured OpenTelemetry traces for both single- and multi-agent systems.
- Empirical findings reveal modern LLMs achieve only 11% joint accuracy, underscoring challenges in debugging long-context reasoning.
TRAIL, short for Trace Reasoning and Agentic Issue Localization, is a benchmark for evaluating agentic workflow traces at the level of structured execution spans rather than only through coarse, end-to-end task outcomes. It was introduced to address the increasing adoption of agentic workflows across diverse domains and the corresponding need to scalably and systematically evaluate the complex traces these systems generate. The benchmark combines a formal taxonomy of error types with 148 large human-annotated traces constructed from established agentic benchmarks, with an emphasis on ecological validity in single- and multi-agent settings such as software engineering and open-world information retrieval. Its central finding is that modern long context LLMs perform poorly at trace debugging; the best reported model, Gemini-2.5-Pro, achieves only 11% joint accuracy on the benchmark (Deshpande et al., 13 May 2025).
1. Origins, motivation, and scope
TRAIL was designed in response to a specific limitation in the evaluation of LLM-based agents: current evaluation methods depend on manual, domain-specific human analysis of lengthy workflow traces, an approach that does not scale with the growing complexity and volume of agentic outputs. The motivating observation is that debugging agentic failures is complicated by the interplay of external tool outputs and LLM reasoning, which makes analysis more challenging than traditional software debugging.
The benchmark’s stated goals are threefold. First, it introduces a fine-grained error taxonomy covering reasoning, planning/coordination, and system-execution failures that arise in single- and multi-agent pipelines. Second, it curates a set of 148 real-world traces, comprising 1,987 spans in total, of which 575 are erroneous, and draws them from two established benchmarks while preserving the ecological validity of genuine agent logs. Third, it offers this dataset to drive research on “LLM-as-judge” approaches capable of localizing not just whether a trace is faulty, but where and why it went off the rails.
This positioning is significant because the benchmark is not limited to binary success or failure at the task level. Instead, it targets fine-grained localization and classification within OpenTelemetry-style traces. A plausible implication is that TRAIL is intended as an evaluation substrate for automated debugging systems rather than as a benchmark for task completion alone.
2. Error taxonomy
TRAIL partitions failures into three root categories: Reasoning Errors, System Execution Errors, and Planning & Coordination Errors. Each category is further subdivided into error types intended to provide a formal vocabulary for what goes wrong in agentic systems.
Reasoning Errors include four groups. Hallucinations cover Language-only Hallucination, where the agent fabricates or misstates facts, and Tool-related Hallucination, where the agent invents or misattributes outputs from a tool. Information Processing includes Poor Retrieval and Tool-Output Misinterpretation. Decision Making includes Incorrect Problem ID and Tool Selection Error. Output Generation includes Formatting Error and Instruction Non-compliance.
System Execution Errors include Configuration Issues, API and System Issues, and Resource Management. Configuration Issues comprise Incorrect Tool Definition and Environment Setup Error. API and System Issues comprise Rate Limiting (HTTP 429), Authentication Errors (401/403), Service Errors (500), and Resource Not Found (404). Resource Management comprises Resource Exhaustion and Timeout Issues.
Planning & Coordination Errors include Context Management and Task Management. Context Management comprises Context Handling Failure and Resource Abuse. Task Management comprises Goal Deviation and Task Orchestration Error.
The taxonomy is illustrated in the benchmark with representative examples. Language-only Hallucination is exemplified by an unsupported claim in an LLM reasoning span; Tool-related Hallucination by a mismatch between a tool call output and the agent’s observation; Poor Retrieval by a zero-hit retrieval followed by an unsupported inference; Tool-Output Misinterpretation by incorrect data type handling; Incorrect Problem ID by a plan that targets the wrong user problem; Tool Selection Error by invoking an unsuitable tool; Formatting Error by malformed structured output; Instruction Non-compliance by violating explicit user constraints; Incorrect Tool Definition by a misleading tool description; Environment Setup Error by an empty API key leading to an authentication failure; Rate Limiting by an HTTP 429 response; Resource Exhaustion by a memory error; Context Handling Failure by losing track of state across many steps; and Goal Deviation by wandering off-task.
Taken together, these categories formalize a mixed failure surface that spans language-model reasoning, tool interfaces, runtime conditions, and multi-step orchestration. This suggests that TRAIL is structured to evaluate failures that are specific to agentic systems rather than to isolated language-model responses.
3. Dataset composition and annotation procedure
TRAIL comprises 148 agent execution traces drawn from two established source benchmarks.
| Source benchmark | Traces | System type |
|---|---|---|
| GAIA | 118 | multi-agent |
| SWE-Bench Lite | 30 | single-agent |
Across these traces, the benchmark contains 1,987 spans in total, with 575 erroneous spans. The design emphasizes real-world applications such as web search, exemplified by GAIA, and software engineering, exemplified by SWE-Bench Lite. The benchmark description states that traces were curated from both single- and multi-agent systems to ensure ecological validity.
Annotation was performed by four experts in software engineering and log debugging. For each OpenTelemetry “span,” annotators inspected the span in context and marked span IDs, error categories from the taxonomy, evidence snippets, free-text descriptions, and an impact level of low, medium, or high. This procedure grounds the dataset in span-level judgments rather than document-level summaries.
A separate verification pass was conducted over 63 held-out traces by four additional ML researchers. In that pass, annotators modified only 5.31% of GAIA annotations, corresponding to 37 of 697 spans, and 5.63% of SWE-Bench spans, corresponding to 25 of 444 spans, which the benchmark describes as demonstrating high inter-annotator agreement. The most common revisions were Language-only Hallucinations, Resource Abuse, and Poor Information Retrieval (Deshpande et al., 13 May 2025).
The annotation design is consequential because the benchmark preserves both local evidence and contextual interpretation. A plausible implication is that TRAIL can support evaluation not only of label prediction but also of evidence-grounded explanation.
4. Task formulation and evaluation protocol
TRAIL is cast as a span-level error localization task. Given the raw JSON trace, a judge model must identify which spans contain an error and assign one of the taxonomy labels. This formulation makes the task simultaneously a localization problem and a categorical classification problem.
The benchmark reports three families of metrics. Category Precision, Recall, and F1 evaluate category assignment. Location Accuracy is the fraction of spans for which the model correctly predicts “error/no error.” Joint Accuracy is the fraction of spans where both the location and category are correct.
The 148 traces are evaluated in two splits corresponding to their source benchmarks: GAIA with 118 traces and SWE-Bench with 30 traces. This is described as mirroring how practitioners would focus on distinct real-world workflows. No retraining is involved; all models are tested in a zero-shot or few-shot setting.
This protocol distinguishes TRAIL from evaluations that score only final task outputs. Because the unit of analysis is the span, the benchmark measures whether a model can follow the internal structure of an agentic trace and recover both where a failure occurred and what type of failure it was.
5. Baseline models, results, and empirical findings
The benchmark surveys eight modern LLMs: OpenAI o1, o3, and GPT-4.1; Anthropic Claude-3.7-Sonnet; Google Gemini-2.5 Pro and Flash; and Meta’s Llama-4 Scout and Maverick. The headline result is that Gemini-2.5-Pro achieves the best joint accuracy, yet still only 11% overall, with 18% on GAIA and 5% on SWE-Bench. Category F1 scores range from 0.04 for Llama-4-Scout to 0.39 for Gemini-2.5-Pro.
A further constraint is context length. Three out of eight models are blocked from ingesting full traces because of context-length-exceeded failures. The per-category analysis shows that models perform moderately on Hallucination detection, with approximate F1 values in the 0.4–0.6 range, but near zero on Context Handling Failures, Tool Selection, and Task Orchestration. The benchmark also reports that Output Generation errors dominate in frequency, accounting for 42% of errors, while rare but critical categories such as API failures and goal deviations are among those where models falter most.
Several analytic findings are reported. Long contexts remain a major bottleneck, and performance is anti-correlated with input length, with for location accuracy. Chain-of-thought reasoning helps: models with “high” reasoning settings outperform “medium” and “low” ablations by 5–15 points in F1 and joint accuracy. The benchmark therefore identifies trace debugging as a difficult long-context reasoning problem rather than a task solved by current frontier models (Deshpande et al., 13 May 2025).
The benchmark’s stated future directions are extending TRAIL to multimodal tool traces, data augmentation for low-frequency, high-impact categories, specialized modules or retrieval-augmented methods to help LLM judges track structured logs over long horizons, and stronger integration of explicit reasoning chains to improve planning and coordination error detection.
6. Interpretation, use cases, and disambiguation
TRAIL is intended to support scalable evaluation for agentic workflows by providing both a practical taxonomy and an ecologically grounded dataset. Its immediate use case is the evaluation of “LLM-as-judge” systems for trace debugging, especially systems that must localize and classify failures in long, structured traces. Because the benchmark is grounded in GAIA and SWE-Bench Lite, it spans open-world information retrieval and software engineering rather than restricting itself to a single domain.
A common source of confusion is the acronym itself. “TRAIL” is also the name of an algorithm for cross-shard validation in cryptocurrency Byzantine shard protection, presented with PBFT for internal shard transaction processing and a modified version of PBFT for external cross-shard validation (Jacovetty et al., 2024). It is also the name of “TRAceability lInk cLassifier,” a machine-learning approach for maintaining software traceability links using historical traceability data, 131 normalized features, and supervised classification over candidate artifact pairs (Mills et al., 2018). These are distinct from Trace Reasoning and Agentic Issue Localization.
Within the literature on agentic systems, the specific contribution of TRAIL lies in formalizing trace debugging as span-level error localization over human-annotated OpenTelemetry-style traces. This suggests a shift from evaluating whether an agent succeeded to evaluating how and where its workflow failed.