---
title: 'TELBench: Error Localization Benchmark'
url: https://www.emergentmind.com/topics/telbench
type: topic
---

# TELBench: Error Localization Benchmark

TELBench, short for **Trajectory Error Localization Benchmark**, is a benchmark for **span-level error localization** in **deep-research (DR) agent** trajectories. It is built from real agent runs on complex information-seeking tasks and represents each trajectory as an ordered sequence of **semantic spans**, each annotated as **error** or **non-error**. Its central purpose is to identify **harmful error spans**, including the **earliest harmful span**, among normal exploration, failed searches, tentative hypotheses, and harmless noise. The benchmark is introduced alongside **DRIFT**, a **claim-centric auditing framework** for tracking unsupported or conflicting claims through an agent’s decision process, and is presented as a process-level alternative to evaluation schemes that consider only final answers [2606.02060].

## 1. Problem setting and benchmark definition

TELBench is situated in the study of **deep-research agents**: systems that solve complex information-seeking tasks by executing **long, multi-step trajectories** rather than a single prompt–response interaction. Such trajectories typically include planning and sub-goal decomposition, web search or tool calls, evidence inspection and extraction, intermediate hypotheses and candidate selection, decision-making under constraints, and final answer or report synthesis [2606.02060].

The benchmark is motivated by the claim that **final-answer evaluation is insufficient** for diagnosing reliability in these systems. Outcome-level benchmarks indicate whether an answer is correct, but they do not show **where** a trajectory became unreliable, which earlier commitment caused the failure, or how an unsupported claim propagated through later reasoning. The paper reports that **97.3%** of failed trajectories contain at least one annotated error span, while **36.9%** of successful trajectories also contain process errors, indicating that process reliability and final correctness are not equivalent [2606.02060].

TELBench therefore evaluates **span-level error localization**. A trajectory \(T\) is represented as an ordered sequence of semantic spans,

\[
T = (s_1, s_2, \ldots, s_n),
\]

where each \(s_j\) is a semantically coherent chunk such as planning, retrieval, verification, computation, or finalization. For each instance, the input is the task question \(q\) and the ordered span sequence \(T\); the output is a predicted set of error spans \(\hat{E} \subset T\). Gold labels specify both the set of harmful error spans and the **earliest harmful span** [2606.02060].

An **error span** is defined as a span that introduces, relies on, amplifies, or finalizes a mistaken, unsupported, contradicted, or prematurely committed judgment that **affects the answer path**. A **non-error span** includes normal exploration, failed searches, tentative hypotheses, recognized and recovered mistakes, and harmless tool noise. The benchmark’s central distinction is therefore not between correct and incorrect text in the abstract, but between **harmful committed reasoning** and non-consequential exploratory behavior [2606.02060].

A useful contextual distinction is that this TELBench concerns **trajectory error localization in research agents**, and not telecom benchmarks with similar abbreviations, such as **TeleResilienceBench**, whose actual name is different and whose focus is reasoning resilience in telecommunications [2605.09929].

## 2. Corpus construction and annotation pipeline

TELBench is derived from a larger corpus of **2,790 real trajectories** collected from **465 tasks × 3 models × 2 frameworks**. The source benchmarks are **GAIA-val**, **XBench**, and **BrowseComp-test**, with BrowseComp downsampled to 200 tasks so that it does not dominate the corpus. The backbone models are **GPT-5**, **Gemini-2.5-Pro**, and **Claude-Sonnet-4.5**, while the agent frameworks are **MiroFlow** and **OAgent** [2606.02060].

The raw corpus contains **363,695 raw steps**. Because raw logs are framework-specific and include retries, wrappers, and nested multi-agent traces, the paper introduces a conversion pipeline from logs to semantic spans. First, **log normalization** folds each tool call and result into a single execution unit for event-ordered logs, and reconstructs semantic execution order for nested multi-agent traces. Second, **span segmentation** groups contiguous execution units around a locally coherent objective, with boundaries determined by changes in search target, candidate set, time scope, verification criterion, or reasoning objective. Abnormal cases and stratified samples across framework, model, benchmark, outcome, and length are **LLM-assisted and human-audited** to refine boundaries [2606.02060].

This process yields **36,417 spans** in the full corpus, with an average of approximately **13 spans per trajectory**; in the final TELBench subset, the average is **11.95 spans per trajectory** [2606.02060].

Annotation proceeds through an **LLM-assisted, expert-verified** pipeline. Two independent frontier LLMs first generate high-recall candidate error spans, including span indices, rationales, and referenced evidence spans. A pool of **7 expert annotators** then reviews full trajectories and proposed candidates. For each trajectory, **2 experts** read the complete span sequence, accept, reject, or modify candidate labels, adjust boundaries if needed, and add missing error spans. Disagreement and low-confidence cases are adjudicated. The paper states that each expert spent **300+ hours** on reading, evidence checking, label revision, and adjudication [2606.02060].

From the 2,790 annotated trajectories, **1,890 trajectories** have at least one error span, or **67.7%** of the corpus. TELBench itself is a more selective **Verified-1K** benchmark: **1,000 trajectories** chosen to satisfy three criteria—errors verifiable from trajectory-internal evidence, stable span boundaries suitable for evaluation, and enough benign distractor behavior to make localization non-trivial. Additional filtering removes missing or fragmented logs, extremely short or degenerate runs, hard-to-verify or ambiguous errors, and overrepresented simple patterns [2606.02060].

The final benchmark is split by difficulty into **600 easy** and **400 hard** instances. The hard split contains longer trajectories, more subtle or implicit errors, more benign exploration, and challenging patterns such as **evidence overclaim**, **constraint miss**, and **candidate confusion** [2606.02060].

## 3. Label structure and error typology

TELBench uses two orthogonal span labelings for analysis. Every span receives an **operation stage** label, while error spans also receive a **primary fault** label [2606.02060].

The **operation stage taxonomy** assigns each span to exactly one of the following stages: **Plan**, **Retrieve**, **Source Verify**, **Extract**, **Compute**, **Decide**, **Reflect Recover**, and **Finalize**. These labels characterize what the agent is doing at each point in the trajectory rather than whether the span is harmful [2606.02060].

The stage statistics reveal a strong asymmetry between retrieval and downstream use of retrieved information. Retrieval spans are numerous but comparatively low-risk: only **2.9%** of retrieval spans are errors. By contrast, **Decision** spans have an error rate of **60.5%**, and **Finalize** spans have an error rate of **51.8%**. The paper interprets this as evidence that many failures arise not from search itself, but from **how collected information is used** [2606.02060].

The **primary fault taxonomy** groups error spans into six families:

| Fault family | Subtypes named in the paper |
|---|---|
| Constraint Handling | Constraint Semantics Error; Constraint Check Omission; Constraint Relaxation; Answer Format Error |
| Search and Retrieval | Goal Drift; Candidate Scope Error; Retrieval Query Error |
| Evidence Grounding | Source Verification Error; Source Misuse Error; Unsupported Commitment |
| Entity Mapping | Entity Disambiguation Error; Entity Attribute Mapping Error; Memory Context Error |
| Information Processing | Extraction Parsing Error; Calculation Error; Aggregation Metric Error |
| Process Control | Overanchoring Error; Process Control Error |

These fault types are described as **induced from data** using LLM-generated rationales and hierarchical clustering, then normalized manually [2606.02060].

The benchmark’s concept of harmfulness depends on impact on the answer path. A span is labeled as an error when it introduces or cements a wrong candidate, misinterprets or overclaims evidence, misapplies constraints, or finalizes an answer or no-answer decision on missing or contradicted support. By contrast, spans that explore a wrong hypothesis but later abandon it, or conduct a failed search without commitment, remain **non-error** [2606.02060].

The paper’s appendix includes two illustrative cases. In one snooker-match example, the annotated trajectory includes an early **wrong_candidate commitment**, later **candidate reinforcement**, an **entity_role mismatch**, and final **error propagation**. In another case, the agent outputs the correct final string, but the evidence chain supporting it is missing; TELBench still labels the relevant spans as errors because the process is unreliable even though the outcome is correct [2606.02060].

## 4. DRIFT: claim-centric auditing for TELBench

TELBench is introduced together with **DRIFT**, a **claim-centric auditing framework** designed specifically for span-level error localization. Rather than classifying spans independently, DRIFT treats a trajectory as a sequence of **claims** about entities, constraints, evidence, retrieval coverage, computations, and process decisions, and then tracks where those claims are introduced, become consequential, are reused, and are finalized [2606.02060].

Formally, given a question \(q\) and a trajectory \(T = (s_1, \ldots, s_n)\), DRIFT outputs a predicted set of error spans:

\[
\hat{E} = f_e(q, T),
\]

with final prediction

\[
\hat{E} = \{ s_j \in T \mid h(s_j) = 1 \},
\]

where \(h(s_j) = 1\) indicates that the span **commits to, reuses, amplifies, or finalizes** a harmful claim [2606.02060].

DRIFT has three principal components. **Claim Keeper** constructs a **claim ledger** across the full trajectory. Each claim records its textual content, the span where it is introduced, the first span where it becomes consequential, the later spans that use it, its type, and its status. A core distinction is made between **exploratory**, **tentative**, **consequential**, and **finalized** claims, which allows the framework to avoid penalizing benign exploratory behavior [2606.02060].

**Support Seeker** then audits the support available within the trajectory for consequential claims. Support can be labeled **DIRECT**, **WEAK**, **MISSING**, or **CONFLICTING**. DIRECT means the decisive link is explicitly established in the trajectory; WEAK means related evidence exists but the decisive link is partial, implicit, snippet-based, or not checked against the exact question; MISSING means no shown evidence establishes the claim; and CONFLICTING means the evidence contradicts it. At this stage, DRIFT does not yet mark spans as errors; it identifies claims whose support is deficient [2606.02060].

Finally, **Dependency Tracer**, augmented with specialist auditors for evidence, constraints, entities, retrieval, computation, and process, determines which risky claims are actually harmful. A weakly or missingly supported claim is not automatically an error. It becomes harmful when it is committed, propagated, or finalized in a way that affects the answer path. DRIFT therefore marks spans as error when they commit to unsupported or conflicting claims, reuse or amplify them, or finalize them in the answer [2606.02060].

This architecture operationalizes the paper’s central thesis that deep-research trajectories should be audited as a **recorded decision process** in which claims are formed, inherited, and reused, rather than as a flat list of tool events [2606.02060].

## 5. Evaluation protocol and empirical results

TELBench evaluates **span-level error localization** and **temporal localization of the first harmful span**. All methods receive the same input—question plus ordered semantic spans—and must output a list of error span IDs such as `["s003","s007"]`. Methods are run **3 times per setting** [2606.02060].

The benchmark reports two primary metrics. The first is macro-averaged **Precision**, **Recall**, and **F1** across trajectories for span-level error localization. The second is **First-Error Accuracy (FEA)**, which measures whether the **earliest harmful span** is correctly identified among predicted error spans. The paper emphasizes that FEA is substantially harder than generic error detection because it tests temporal pinpointing rather than merely identifying some erroneous region [2606.02060].

The evaluation covers several model families—**Qwen-3 series**, **GPT-5.4**, **DeepSeek-V3.2**, **Claude-Sonnet-4.6**, and **Gemini-2.5-Pro**—under multiple auditing frameworks: **Bare LLM**, **Codex**, **Claude Code**, and **DRIFT** [2606.02060].

Across model families, **DRIFT** achieves the best macro-F1 and FEA. For **GPT-5.4**, the paper reports:

| Method | F1 | FEA |
|---|---:|---:|
| Bare | 33.93 | 14.90 |
| Codex | 34.48 | 15.60 |
| Claude Code | 37.66 | 17.50 |
| DRIFT | **52.48** | **20.80** |

For **Claude-Sonnet-4.6**, the paper reports **Bare: F1 = 21.89, FEA = 11.30** and **DRIFT: F1 = 54.91, FEA = 24.10**. For **Gemini-2.5-Pro**, it reports **Bare: F1 = 31.01, FEA = 15.70** and **DRIFT: F1 = 48.41, FEA = 19.90**. For **DeepSeek-V3.2**, it reports **Bare: F1 = 22.46, FEA = 10.30** and **DRIFT: F1 = 50.51, FEA = 23.70** [2606.02060].

The headline summary is that DRIFT improves span-level F1 by up to **30 percentage points** over bare prompting, and improves FEA by substantial margins, often **5–10 points** over the bare baseline [2606.02060].

Several further findings shape the interpretation of TELBench. First, **first-error localization remains hard**: on the hard split, FEA can remain below approximately **20%** even when F1 exceeds **40%**, indicating that finding the first point of harmful commitment is a distinct capability from identifying erroneous regions in general [2606.02060]. Second, **scaling alone is insufficient**: larger Qwen models do not monotonically improve macro-F1 or FEA, particularly on the hard split. Third, performance degrades as the number of spans grows, especially for trajectories with **10 or more spans**, although DRIFT consistently outperforms bare prompting at all complexity levels [2606.02060].

Module ablations show a monotonic benefit from DRIFT’s structured components. Starting from **Bare**, adding **Claim Keeper**, then **Support Seeker**, and finally the full **Dependency Tracer** successively improves precision, recall, and F1. The largest gain comes from **Claim Keeper**, which suggests that claim-level representation itself is a major source of improvement [2606.02060].

Mechanism analysis further shows that first-error types vary by benchmark, framework, and model family. The paper states that **GAIA** has more **processing** errors, while other benchmarks lean toward **evidence** and **constraint** errors; **OAgent** is more **evidence-error heavy**, whereas **MiroFlow** is more **constraint/search** heavy; **GPT** models exhibit more **evidence-heavy fingerprints**, **Gemini** more **constraint-heavy** ones, and **Claude** a more balanced distribution [2606.02060].

## 6. Significance, limitations, and use as a research resource

TELBench provides what the paper calls a **process-level lens** on reliability in deep-research agents. By locating harmful spans within trajectories, it distinguishes benign exploratory errors from harmful committed errors, separates local mistakes from systematic claim propagation, and makes it possible to study how specific failures emerge across stages and fault types. This suggests a shift from evaluating agents solely by end-state correctness toward evaluating them as multi-step, auditable reasoning systems [2606.02060].

The benchmark also has practical research uses. The paper releases the dataset on **HuggingFace** at `https://huggingface.co/datasets/NJU-LINK/TELBench` and the DRIFT implementation on **GitHub** at `https://github.com/NJU-LINK/DRIFT`. The dataset includes Verified-1K trajectories with questions, ordered semantic spans, span-level error labels, and metadata such as benchmark, framework, and model family. Evaluation scripts compute macro precision, recall, F1, and First-Error Accuracy. Recommended baselines include **Bare single-pass LLM**, **Codex**, **Claude Code**, and **DRIFT** as a structured baseline [2606.02060].

The paper also outlines a workflow for applying TELBench-style analysis to a new deep-research agent: run the agent on GAIA, XBench, BrowseComp, or custom tasks; log full trajectories; convert logs to semantic spans using the paper’s criteria; apply DRIFT or another auditing system; visualize claim chains and support status; and compare against TELBench baselines [2606.02060]. A plausible implication is that TELBench is intended not only as a static benchmark but also as a transferable methodology for auditing new agent systems.

Several limitations are stated explicitly. **Verified-1K** is sizable but not web-scale; annotation is labor-intensive; the domain coverage is centered on **web-style deep research** rather than coding, multimodal control, or enterprise workflows; the trajectories are drawn from **MiroFlow** and **OAgent** with specific toolchains and frontier LLMs; and determining when a claim is truly unsupported or conflicting can be subtle. The paper also notes that span boundaries, although stabilized through review, are not unique, and that LLM-assisted candidate generation can bias annotators if not carefully overseen, a risk mitigated through dual expert review and editing [2606.02060].

Future directions proposed in the paper include extending TELBench-style evaluation to **enterprise deep-research**, **multimodal agents**, **code agents**, and **multi-agent systems**; refining annotation and taxonomies; integrating span-level error labels into **process reward models**, **self-refinement**, **self-verification**, and **agent routing**; and developing new auditing frameworks beyond DRIFT, including **alternative claim graphs**, **probabilistic support models**, and **differentiable training of auditors** [2606.02060].

In that sense, TELBench occupies a specific place in the emerging benchmark landscape. Unlike clean-prompt outcome benchmarks, and unlike resilience benchmarks that start from flawed intermediate reasoning in another domain such as telecommunications [2605.09929], TELBench formalizes **where deep-research trajectories go wrong** and supplies both a dataset and an auditing framework for studying that question directly [2606.02060].

Source: https://www.emergentmind.com/topics/telbench