TELBench: Error Localization Benchmark

• TELBench is a benchmark that defines semantic spans in deep-research agent trajectories to distinguish harmful error spans from benign exploration.
• It employs log normalization, LLM-assisted segmentation, and expert review to create a robust corpus for process-level error detection.
• Paired with the DRIFT framework, TELBench improves error localization metrics, enabling precise identification of the earliest harmful error spans.

TELBench, short for Trajectory Error Localization Benchmark, is a benchmark for span-level error localization in deep-research (DR) agent trajectories. It is built from real agent runs on complex information-seeking tasks and represents each trajectory as an ordered sequence of semantic spans, each annotated as error or non-error. Its central purpose is to identify harmful error spans, including the earliest harmful span, among normal exploration, failed searches, tentative hypotheses, and harmless noise. The benchmark is introduced alongside DRIFT, a claim-centric auditing framework for tracking unsupported or conflicting claims through an agent’s decision process, and is presented as a process-level alternative to evaluation schemes that consider only final answers (Wang et al., 1 Jun 2026).

1. Problem setting and benchmark definition

TELBench is situated in the study of deep-research agents: systems that solve complex information-seeking tasks by executing long, multi-step trajectories rather than a single prompt–response interaction. Such trajectories typically include planning and sub-goal decomposition, web search or tool calls, evidence inspection and extraction, intermediate hypotheses and candidate selection, decision-making under constraints, and final answer or report synthesis (Wang et al., 1 Jun 2026).

The benchmark is motivated by the claim that final-answer evaluation is insufficient for diagnosing reliability in these systems. Outcome-level benchmarks indicate whether an answer is correct, but they do not show where a trajectory became unreliable, which earlier commitment caused the failure, or how an unsupported claim propagated through later reasoning. The paper reports that 97.3% of failed trajectories contain at least one annotated error span, while 36.9% of successful trajectories also contain process errors, indicating that process reliability and final correctness are not equivalent (Wang et al., 1 Jun 2026).

TELBench therefore evaluates span-level error localization. A trajectory $T$ is represented as an ordered sequence of semantic spans,

$T = (s_1, s_2, \ldots, s_n),$

where each $s_j$ is a semantically coherent chunk such as planning, retrieval, verification, computation, or finalization. For each instance, the input is the task question $q$ and the ordered span sequence $T$; the output is a predicted set of error spans $\hat{E} \subset T$. Gold labels specify both the set of harmful error spans and the earliest harmful span (Wang et al., 1 Jun 2026).

An error span is defined as a span that introduces, relies on, amplifies, or finalizes a mistaken, unsupported, contradicted, or prematurely committed judgment that affects the answer path. A non-error span includes normal exploration, failed searches, tentative hypotheses, recognized and recovered mistakes, and harmless tool noise. The benchmark’s central distinction is therefore not between correct and incorrect text in the abstract, but between harmful committed reasoning and non-consequential exploratory behavior (Wang et al., 1 Jun 2026).

A useful contextual distinction is that this TELBench concerns trajectory error localization in research agents, and not telecom benchmarks with similar abbreviations, such as TeleResilienceBench, whose actual name is different and whose focus is reasoning resilience in telecommunications (Gajjar et al., 11 May 2026).

2. Corpus construction and annotation pipeline

TELBench is derived from a larger corpus of 2,790 real trajectories collected from 465 tasks × 3 models × 2 frameworks. The source benchmarks are GAIA-val, XBench, and BrowseComp-test, with BrowseComp downsampled to 200 tasks so that it does not dominate the corpus. The backbone models are GPT-5, Gemini-2.5-Pro, and Claude-Sonnet-4.5, while the agent frameworks are MiroFlow and OAgent (Wang et al., 1 Jun 2026).

The raw corpus contains 363,695 raw steps. Because raw logs are framework-specific and include retries, wrappers, and nested multi-agent traces, the paper introduces a conversion pipeline from logs to semantic spans. First, log normalization folds each tool call and result into a single execution unit for event-ordered logs, and reconstructs semantic execution order for nested multi-agent traces. Second, span segmentation groups contiguous execution units around a locally coherent objective, with boundaries determined by changes in search target, candidate set, time scope, verification criterion, or reasoning objective. Abnormal cases and stratified samples across framework, model, benchmark, outcome, and length are LLM-assisted and human-audited to refine boundaries (Wang et al., 1 Jun 2026).

This process yields 36,417 spans in the full corpus, with an average of approximately 13 spans per trajectory; in the final TELBench subset, the average is 11.95 spans per trajectory (Wang et al., 1 Jun 2026).

Annotation proceeds through an LLM-assisted, expert-verified pipeline. Two independent frontier LLMs first generate high-recall candidate error spans, including span indices, rationales, and referenced evidence spans. A pool of 7 expert annotators then reviews full trajectories and proposed candidates. For each trajectory, 2 experts read the complete span sequence, accept, reject, or modify candidate labels, adjust boundaries if needed, and add missing error spans. Disagreement and low-confidence cases are adjudicated. The paper states that each expert spent 300+ hours on reading, evidence checking, label revision, and adjudication (Wang et al., 1 Jun 2026).

From the 2,790 annotated trajectories, 1,890 trajectories have at least one error span, or 67.7% of the corpus. TELBench itself is a more selective Verified-1K benchmark: 1,000 trajectories chosen to satisfy three criteria—errors verifiable from trajectory-internal evidence, stable span boundaries suitable for evaluation, and enough benign distractor behavior to make localization non-trivial. Additional filtering removes missing or fragmented logs, extremely short or degenerate runs, hard-to-verify or ambiguous errors, and overrepresented simple patterns (Wang et al., 1 Jun 2026).

The final benchmark is split by difficulty into 600 easy and 400 hard instances. The hard split contains longer trajectories, more subtle or implicit errors, more benign exploration, and challenging patterns such as evidence overclaim, constraint miss, and candidate confusion (Wang et al., 1 Jun 2026).

3. Label structure and error typology

TELBench uses two orthogonal span labelings for analysis. Every span receives an operation stage label, while error spans also receive a primary fault label (Wang et al., 1 Jun 2026).

The operation stage taxonomy assigns each span to exactly one of the following stages: Plan, Retrieve, Source Verify, Extract, Compute, Decide, Reflect Recover, and Finalize. These labels characterize what the agent is doing at each point in the trajectory rather than whether the span is harmful (Wang et al., 1 Jun 2026).

The stage statistics reveal a strong asymmetry between retrieval and downstream use of retrieved information. Retrieval spans are numerous but comparatively low-risk: only 2.9% of retrieval spans are errors. By contrast, Decision spans have an error rate of 60.5%, and Finalize spans have an error rate of 51.8%. The paper interprets this as evidence that many failures arise not from search itself, but from how collected information is used (Wang et al., 1 Jun 2026).

The primary fault taxonomy groups error spans into six families:

Fault family	Subtypes named in the paper
Constraint Handling	Constraint Semantics Error; Constraint Check Omission; Constraint Relaxation; Answer Format Error
Search and Retrieval	Goal Drift; Candidate Scope Error; Retrieval Query Error
Evidence Grounding	Source Verification Error; Source Misuse Error; Unsupported Commitment
Entity Mapping	Entity Disambiguation Error; Entity Attribute Mapping Error; Memory Context Error
Information Processing	Extraction Parsing Error; Calculation Error; Aggregation Metric Error
Process Control	Overanchoring Error; Process Control Error

These fault types are described as induced from data using LLM-generated rationales and hierarchical clustering, then normalized manually (Wang et al., 1 Jun 2026).

The benchmark’s concept of harmfulness depends on impact on the answer path. A span is labeled as an error when it introduces or cements a wrong candidate, misinterprets or overclaims evidence, misapplies constraints, or finalizes an answer or no-answer decision on missing or contradicted support. By contrast, spans that explore a wrong hypothesis but later abandon it, or conduct a failed search without commitment, remain non-error (Wang et al., 1 Jun 2026).

The paper’s appendix includes two illustrative cases. In one snooker-match example, the annotated trajectory includes an early wrong_candidate commitment, later candidate reinforcement, an entity_role mismatch, and final error propagation. In another case, the agent outputs the correct final string, but the evidence chain supporting it is missing; TELBench still labels the relevant spans as errors because the process is unreliable even though the outcome is correct (Wang et al., 1 Jun 2026).

4. DRIFT: claim-centric auditing for TELBench

TELBench is introduced together with DRIFT, a claim-centric auditing framework designed specifically for span-level error localization. Rather than classifying spans independently, DRIFT treats a trajectory as a sequence of claims about entities, constraints, evidence, retrieval coverage, computations, and process decisions, and then tracks where those claims are introduced, become consequential, are reused, and are finalized (Wang et al., 1 Jun 2026).

Formally, given a question $q$ and a trajectory $T = (s_1, \ldots, s_n)$, DRIFT outputs a predicted set of error spans:

$\hat{E} = f_e(q, T),$

with final prediction

$\hat{E} = \{ s_j \in T \mid h(s_j) = 1 \},$

where $T = (s_1, s_2, \ldots, s_n),$0 indicates that the span commits to, reuses, amplifies, or finalizes a harmful claim (Wang et al., 1 Jun 2026).

DRIFT has three principal components. Claim Keeper constructs a claim ledger across the full trajectory. Each claim records its textual content, the span where it is introduced, the first span where it becomes consequential, the later spans that use it, its type, and its status. A core distinction is made between exploratory, tentative, consequential, and finalized claims, which allows the framework to avoid penalizing benign exploratory behavior (Wang et al., 1 Jun 2026).

Support Seeker then audits the support available within the trajectory for consequential claims. Support can be labeled DIRECT, WEAK, MISSING, or CONFLICTING. DIRECT means the decisive link is explicitly established in the trajectory; WEAK means related evidence exists but the decisive link is partial, implicit, snippet-based, or not checked against the exact question; MISSING means no shown evidence establishes the claim; and CONFLICTING means the evidence contradicts it. At this stage, DRIFT does not yet mark spans as errors; it identifies claims whose support is deficient (Wang et al., 1 Jun 2026).

Finally, Dependency Tracer, augmented with specialist auditors for evidence, constraints, entities, retrieval, computation, and process, determines which risky claims are actually harmful. A weakly or missingly supported claim is not automatically an error. It becomes harmful when it is committed, propagated, or finalized in a way that affects the answer path. DRIFT therefore marks spans as error when they commit to unsupported or conflicting claims, reuse or amplify them, or finalize them in the answer (Wang et al., 1 Jun 2026).

This architecture operationalizes the paper’s central thesis that deep-research trajectories should be audited as a recorded decision process in which claims are formed, inherited, and reused, rather than as a flat list of tool events (Wang et al., 1 Jun 2026).

5. Evaluation protocol and empirical results

TELBench evaluates span-level error localization and temporal localization of the first harmful span. All methods receive the same input—question plus ordered semantic spans—and must output a list of error span IDs such as ["s003","s007"]. Methods are run 3 times per setting (Wang et al., 1 Jun 2026).

The benchmark reports two primary metrics. The first is macro-averaged Precision, Recall, and F1 across trajectories for span-level error localization. The second is First-Error Accuracy (FEA), which measures whether the earliest harmful span is correctly identified among predicted error spans. The paper emphasizes that FEA is substantially harder than generic error detection because it tests temporal pinpointing rather than merely identifying some erroneous region (Wang et al., 1 Jun 2026).

The evaluation covers several model families—Qwen-3 series, GPT-5.4, DeepSeek-V3.2, Claude-Sonnet-4.6, and Gemini-2.5-Pro—under multiple auditing frameworks: Bare LLM, Codex, Claude Code, and DRIFT (Wang et al., 1 Jun 2026).

Across model families, DRIFT achieves the best macro-F1 and FEA. For GPT-5.4, the paper reports:

Method	F1	FEA
Bare	33.93	14.90
Codex	34.48	15.60
Claude Code	37.66	17.50
DRIFT	52.48	20.80

For Claude-Sonnet-4.6, the paper reports Bare: F1 = 21.89, FEA = 11.30 and DRIFT: F1 = 54.91, FEA = 24.10. For Gemini-2.5-Pro, it reports Bare: F1 = 31.01, FEA = 15.70 and DRIFT: F1 = 48.41, FEA = 19.90. For DeepSeek-V3.2, it reports Bare: F1 = 22.46, FEA = 10.30 and DRIFT: F1 = 50.51, FEA = 23.70 (Wang et al., 1 Jun 2026).

The headline summary is that DRIFT improves span-level F1 by up to 30 percentage points over bare prompting, and improves FEA by substantial margins, often 5–10 points over the bare baseline (Wang et al., 1 Jun 2026).

Several further findings shape the interpretation of TELBench. First, first-error localization remains hard: on the hard split, FEA can remain below approximately 20% even when F1 exceeds 40%, indicating that finding the first point of harmful commitment is a distinct capability from identifying erroneous regions in general (Wang et al., 1 Jun 2026). Second, scaling alone is insufficient: larger Qwen models do not monotonically improve macro-F1 or FEA, particularly on the hard split. Third, performance degrades as the number of spans grows, especially for trajectories with 10 or more spans, although DRIFT consistently outperforms bare prompting at all complexity levels (Wang et al., 1 Jun 2026).

Module ablations show a monotonic benefit from DRIFT’s structured components. Starting from Bare, adding Claim Keeper, then Support Seeker, and finally the full Dependency Tracer successively improves precision, recall, and F1. The largest gain comes from Claim Keeper, which suggests that claim-level representation itself is a major source of improvement (Wang et al., 1 Jun 2026).

Mechanism analysis further shows that first-error types vary by benchmark, framework, and model family. The paper states that GAIA has more processing errors, while other benchmarks lean toward evidence and constraint errors; OAgent is more evidence-error heavy, whereas MiroFlow is more constraint/search heavy; GPT models exhibit more evidence-heavy fingerprints, Gemini more constraint-heavy ones, and Claude a more balanced distribution (Wang et al., 1 Jun 2026).

6. Significance, limitations, and use as a research resource

TELBench provides what the paper calls a process-level lens on reliability in deep-research agents. By locating harmful spans within trajectories, it distinguishes benign exploratory errors from harmful committed errors, separates local mistakes from systematic claim propagation, and makes it possible to study how specific failures emerge across stages and fault types. This suggests a shift from evaluating agents solely by end-state correctness toward evaluating them as multi-step, auditable reasoning systems (Wang et al., 1 Jun 2026).

The benchmark also has practical research uses. The paper releases the dataset on HuggingFace at https://huggingface.co/datasets/NJU-LINK/TELBench and the DRIFT implementation on GitHub at https://github.com/NJU-LINK/DRIFT. The dataset includes Verified-1K trajectories with questions, ordered semantic spans, span-level error labels, and metadata such as benchmark, framework, and model family. Evaluation scripts compute macro precision, recall, F1, and First-Error Accuracy. Recommended baselines include Bare single-pass LLM, Codex, Claude Code, and DRIFT as a structured baseline (Wang et al., 1 Jun 2026).

The paper also outlines a workflow for applying TELBench-style analysis to a new deep-research agent: run the agent on GAIA, XBench, BrowseComp, or custom tasks; log full trajectories; convert logs to semantic spans using the paper’s criteria; apply DRIFT or another auditing system; visualize claim chains and support status; and compare against TELBench baselines (Wang et al., 1 Jun 2026). A plausible implication is that TELBench is intended not only as a static benchmark but also as a transferable methodology for auditing new agent systems.

Several limitations are stated explicitly. Verified-1K is sizable but not web-scale; annotation is labor-intensive; the domain coverage is centered on web-style deep research rather than coding, multimodal control, or enterprise workflows; the trajectories are drawn from MiroFlow and OAgent with specific toolchains and frontier LLMs; and determining when a claim is truly unsupported or conflicting can be subtle. The paper also notes that span boundaries, although stabilized through review, are not unique, and that LLM-assisted candidate generation can bias annotators if not carefully overseen, a risk mitigated through dual expert review and editing (Wang et al., 1 Jun 2026).

Future directions proposed in the paper include extending TELBench-style evaluation to enterprise deep-research, multimodal agents, code agents, and multi-agent systems; refining annotation and taxonomies; integrating span-level error labels into process reward models, self-refinement, self-verification, and agent routing; and developing new auditing frameworks beyond DRIFT, including alternative claim graphs, probabilistic support models, and differentiable training of auditors (Wang et al., 1 Jun 2026).

In that sense, TELBench occupies a specific place in the emerging benchmark landscape. Unlike clean-prompt outcome benchmarks, and unlike resilience benchmarks that start from flawed intermediate reasoning in another domain such as telecommunications (Gajjar et al., 11 May 2026), TELBench formalizes where deep-research trajectories go wrong and supplies both a dataset and an auditing framework for studying that question directly (Wang et al., 1 Jun 2026).