- The paper introduces TELBench and DRIFT to isolate and diagnose error spans in deep-research agent trajectories.
- It employs a claim-centric auditing framework to track error propagation and analyze failure origins across semantic spans.
- Empirical results show significant F1 improvements and robust detection even in successful trajectories containing process errors.
Span-Level Error Localization in Deep-Research Agents: An Expert Analysis of "Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories" (2606.02060)
Introduction
This paper rigorously addresses a critical limitation in the evaluation and diagnosis of deep-research agents: the inability to localize and attribute the origins of failures at the process level. Traditional benchmarks and evaluation strategies predominantly focus on terminal outcomes but disregard the diagnostic challenge of pinpointing harmful errors within long, complex agent trajectories that involve multi-stage reasoning, exploration, and commitment formation. The authors present TELBench, a large-scale benchmark for span-level error localization, and DRIFT, a comprehensive claim-centric auditing framework, both of which advance process-level reliability research in the agentic paradigm.
The central focus is the diagnosis of harmful commitments within deep-research agent trajectories. Agent outputs are broken into ordered semantic spans, each corresponding to a semantically coherent execution unit (e.g., planning, retrieval, verification, extraction, decision, recovery, finalization). The core challenge lies in reliably attributing final answer failures—not to visible terminal mistakes—but to earlier erroneous commitments that may be reused or amplified in subsequent reasoning, often escaping detection by outcome-level evaluation or naive trajectory inspection.
Prior work on process-level evaluation either targets shorter, more structured tasks (math, API chains, code traces) or lacks the nuanced span-level ground truth required for pinpointing nonterminal errors. This work demonstrates that process errors are only partially correlated with final outcome failure: 36.9% of successful trajectories still contain harmful process errors, and multiple error spans often precede failure, underlining the importance of process-level annotation and diagnostics.
TELBench: Benchmark Construction and Taxonomy
A substantial contribution is the construction of TELBench, incorporating 2,790 annotated trajectories collected from three benchmarks (GAIA, XBench, BrowseComp), three cutting-edge LLM backbones (GPT-5, Gemini-2.5-Pro, Claude-Sonnet-4.5), and two agent frameworks (MiroFlow, OAgent). After curation and filtering, 1,000 instances comprise a verified test set with stable error boundaries and sufficient benign distractor behavior.
Semantic spans serve as the primary annotation units. Error spans are assigned binary (error/non-error) labels after LLM-assisted expert annotation and adjudication; they are further classified by a taxonomy of operation stages (eight categories such as planning, extraction, etc.) and primary fault mechanisms (18 fine-grained types in six families: constraint handling, retrieval, evidence, entity mapping, information processing, process control). Importantly, error spans are defined as introducing, relying on, or amplifying unsupported, contradicted, or prematurely committed judgments that affect the answer trajectory.
The dataset design enforces nontriviality by mixing benign spans (exploration, failed searches, tentative hypotheses) as distractors, and splits data into easy and hard—where the latter highlights long-range, subtle error propagation and implicitly evidenced faults.
Error Mechanism Analysis
Detailed analysis of annotated trajectories underscores the stage-structured and setting-sensitive nature of agentic failures:
- Stage risk is non-uniform: Normalized error incidence is highest in decision-making and finalization stages (e.g., 60.5% and 51.8% error rate, respectively), even though retrieval is the most frequent activity.
- Fault diversity matters: While evidence gaps dominate in overall error counts, rare mechanisms (missed checks, candidate-scoping, anchoring) are more predictive of ultimate task failure.
- Process errors are not equivalent to outcome errors: Many process errors occur in “successful” trajectories; agents may recover or reach the correct answer via alternative paths, rendering naive answer-based evaluation inadequate.
- Diagnostic heterogeneity across frameworks and LLMs: For example, GAIA trajectories bias toward information processing errors post-retrieval, Gemini is most constraint-error heavy, while GPT is evidence-error dominant, highlighting both agent and model-induced failure signatures.
DRIFT: Claim-Centric Trajectory Auditing
The authors propose DRIFT, a multi-stage auditable diagnostic agent that explicitly models claims and their provenance throughout the agent trajectory. DRIFT operates as follows:
- Claim Keeper: Extracts and maintains a ledger of consequential claims (entities, constraints, evidence interpretations, computations), tracking their inception, when they become commitments, and dependency propagation.
- Support Seeker: Evaluates evidential grounding for each consequential claim, labeling their support as DIRECT, WEAK, MISSING, or CONFLICTING based on actual trajectory context.
- Specialist Auditors: Perform type-directed audit passes (entity alignment, constraint satisfaction, retrieval quality, etc.) to validate weak/risky claims using the claim ledger and support structure.
- Dependency Tracer: Backtraces error propagation through claim use/dependency, marking as error those spans responsible for committing to, amplifying, or finalizing harmful unsupported/conflicting claims.
This structured, claim-centric workflow explicitly separates non-harmful exploratory or abandoned actions from true commitments that propagate risk downstream, enabling more reliable error localization.
Experimental Results
DRIFT consistently outperforms baseline and generic agentic auditing systems across all tested LLMs, with gains up to 30 percentage points in F1 and first-error accuracy. Noteworthy findings include:
- F1 gains are robust: Across easy and hard splits, and for all backbone models, DRIFT yields substantial improvements (e.g., macro-F1 >50%). These gains are not achieved by over-prediction, but by precise separation of harmful from benign spans.
- First-error localization is still challenging: There remains a substantial gap between overall span-level detection and first-error localization, especially as trajectory length and distractor density rise. This highlights the distinction between detecting unreliable regions and tracing the earliest causal error.
- Model scale is not a panacea: Increasing LLM backbone size does not yield monotonic improvements in error localization, underscoring that architectural capacity must be coupled with appropriate diagnostic structure to address long-horizon agent trajectories.
- Ablative contributions: Stepwise ablation demonstrates additive contributions from claim ledger formation, structured support checking, and dependency-based diagnosis, with the claim-centric paradigm accounting for the largest fraction of improvement.
- Efficiency trade-offs: DRIFT’s performance is achieved without dramatic increases in token usage relative to less structured auditing baselines, and generally lies on the efficiency-performance Pareto front except in some cases (e.g. Gemini's higher inference cost).
Error-Type and Setting Generalization
DRIFT exhibits consistent gains in recall across diverse error types, particularly for evidence and constraint grounding errors, which are often missed by simple full-context LLM prompting. Its generalization to multiple failure modes and benchmarks suggests that claim-centric diagnostic structures are necessary for robustly localizing diverse agentic failures in realistic, open-ended research settings.
Implications and Future Directions
The introduction of TELBench and DRIFT (code and data available at provided links) enables fine-grained process-level evaluation in open research- and tool-use tasks. The findings carry several implications:
- Process-level diagnosis should become standard for agentic evaluation, as answer-level metrics obscure the nuanced propagation and recovery dynamics of agent-based failures.
- Claim-centric auditing frameworks set a methodological template for future system evaluation, especially as agent frameworks grow in complexity and autonomy.
- Improved agent architectures may require built-in support or signals for claim commitment and dependency modeling, as naive scaling of LLMs is insufficient for failure localization in long-horizon tasks.
- Error correction and recovery research would benefit from integrating fine-grained span-level diagnosis to design self-healing or intervention-capable agents.
Open research directions include bridging the residual gap in first-error detection, diagnosing error propagation in even longer or multi-agent settings, and extending these methods to multimodal or interactive environments.
Conclusion
This work advances the study of agentic robustness by reframing the evaluation paradigm around explicit process-level error localization. Through the introduction of TELBench and the development of DRIFT, the authors demonstrate that reliable diagnosis in deep-research agents requires claim-centric, dependency-aware audit structures. Their framework achieves substantial empirical gains across agents and LLM backbones, and elucidates the importance of precise intermediate error detection in complex agent workflows. The tools, data, and analyses presented define a new standard for future research on long-horizon agent reliability.