---
title: 'T2L-ARVO: Trace-to-Line Benchmark'
url: https://www.emergentmind.com/topics/t2l-arvo
type: topic
---

# T2L-ARVO: Trace-to-Line Benchmark

Searching arXiv for recent papers on T2L-ARVO, ARVO, and T2L-Agent.
T2L-ARVO is a benchmark for agentic, line-level vulnerability localization in real-world open-source software. It was introduced together with T2L-Agent as a curated subset of ARVO and is designed to evaluate two related tasks: coarse-grained detection over project chunks and fine-grained localization to exact vulnerable lines. The benchmark contains 50 expert-verified, reproducible C/C++ cases spanning five crash families, and couples AST-based project segmentation with Dockerized runtime artifacts such as sanitizer traces and stack information, so that systems are assessed under project-scale, trace-aware conditions rather than snippet-level static classification alone [2510.02389].

## 1. Provenance within the ARVO ecosystem

T2L-ARVO is derived from ARVO, the “Atlas of Reproducible Vulnerabilities for Open-source software,” which was created to make real OSS-Fuzz vulnerabilities consistently rebuildable, triggerable, and analyzable across versions. In its 2024 release, ARVO reproduced **5,651 / 8,934** candidate vulnerabilities, identified precise patch commits for **5,001 / 5,651** reproduced cases, and covered **273** C/C++ projects [2408.02153]. In the later 2026 release, ARVO expanded to **6,138 real vulnerabilities** across **311 projects**, with **81%** reproduction success and **89.4%** patch-location accuracy [2606.17283].

The T2L-ARVO paper describes its parent corpus at the time of benchmark construction as containing **4,993 reproducible vulnerabilities in 288 OSS projects** [2510.02389]. T2L-ARVO should therefore be understood not as the full ARVO dataset, but as a deliberately curated benchmark extracted from it for evaluating vulnerability localization systems. This distinction is central: ARVO is a large-scale reproducible vulnerability dataset, whereas T2L-ARVO is a small, expert-verified benchmark specialized for trace-to-line reasoning.

ARVO’s relevance to T2L-ARVO is structural rather than merely nominal. ARVO provides reproducible builds, PoCs, patch locations, and versioned environments; T2L-ARVO adds benchmark-specific chunking, line-level ground truth, and evaluation protocols for LLM-based localization. This suggests that T2L-ARVO functions as a task-specific layer on top of ARVO’s broader reproducibility infrastructure.

## 2. Benchmark construction and case composition

T2L-ARVO contains **50 vulnerabilities**, with **10 cases per crash family**, selected to balance coverage across five failure categories while preserving real-world project complexity [2510.02389]. The benchmark’s crash families are as follows:

| Crash family | Representative subtypes |
|---|---|
| Buffer Overflow | heap-buffer-overflow, stack-buffer-overflow, global-buffer-overflow, index-out-of-bounds |
| Uninitialized Access | use-of-uninitialized-value, Segv on unknown address, Null-dereference |
| Memory Lifecycle | heap-use-after-free, heap-double-free, use-after-poison, invalid-free |
| Type Safety / Parameter Validation | bad-cast, negative-size-param, memcpy-param-overlap, object-size violations, incorrect-function-pointer-type, non-positive VLA bound |
| System Runtime | unknown signal, check failed, sanitizer configuration errors |

Case selection proceeds from the larger ARVO corpus through clustering by failure mechanism and scoring candidates using both structural and semantic criteria. The structural criteria include the number of files changed in the patch, architectural spread, and directory depth; the semantic criteria include cross-module coupling, interface changes, and concurrency touchpoints. A dual validation layer is then applied: manual expert assessment confirms reproducibility and security relevance, while LLM-assisted checks gauge agent-facing difficulty [2510.02389].

The resulting benchmark spans imaging, networking, compression, multimedia, scripting, toolchain, security, and document-processing projects. The intent is not to maximize scale, but to preserve “production patterns rather than a single project or bug family.” T2L-ARVO therefore occupies a specific point in the benchmark design space: smaller than ARVO, but more tightly controlled for localization research.

## 3. Task formulation and scoring

T2L-ARVO supports two primary tasks. The first is **chunk-level detection**, in which a system must identify the project chunk or module containing the vulnerability. The second is **line-level localization**, in which the system must predict the exact vulnerable line or lines corresponding to the patch [2510.02389].

To define chunk-level units, the benchmark introduces AST-based segmentation via a `chunk_case` tool built on Tree-sitter. Chunks are intended to be function-aligned, semantically coherent units that remain manageable within LLM context windows. Each chunk entry includes `file_path`, `chunk_kind`, `symbol`, `start_line`, `end_line`, `source`, `ast_type`, and `imports`. A separate `mark_diff` stage identifies which chunks intersect the patch, producing the ground-truth chunk set for detection scoring.

The benchmark’s scoring semantics are binary at the case level. If \(C_i^*\) denotes the ground-truth chunk set for case \(i\) and \(P_i^C\) the predicted chunk set, detection is counted as correct when \(P_i^C \cap C_i^* \neq \emptyset\). The benchmark-level detection rate is
$$
\mathrm{Detection\ Rate} = \frac{1}{N} \sum_{i=1}^{N} \mathrm{Det}_i.
$$
For localization, if \(L_i^*\) denotes the patched line set in the vulnerable revision and \(P_i^L\) the predicted line set, localization is counted as correct when \(P_i^L \cap L_i^* \neq \emptyset\). The localization rate is
$$
\mathrm{Localization\ Rate} = \frac{1}{N} \sum_{i=1}^N \mathrm{Loc}_i.
$$

A correct line-level prediction must exactly match at least one patched line in the old vulnerable code; the benchmark does not report a \(\pm k\)-line tolerance [2510.02389]. This makes localization substantially stricter than conventional file- or function-level vulnerability detection.

## 4. Runtime evidence, AST structure, and trace-aware evaluation

T2L-ARVO is explicitly designed for trace-aware localization rather than static inspection of isolated code. Each case is packaged in a Dockerized environment exposing sanitizer and debugger artifacts, including ASAN, MSAN, and UBSAN outputs, stack traces, and optional GDB traces. The benchmark tooling includes `run_san`, `run_gdb`, `static_analysis`, `llm_analyze`, and `compare_llm_metrics`, enabling a system to correlate runtime symptoms with repository structure [2510.02389].

This coupling of runtime evidence with AST chunking is the defining methodological property of T2L-ARVO. The benchmark assumes that effective localization in real OSS requires more than source-code pattern matching: the crash point may be far from the root cause, stack traces may be noisy, and the relevant code may span multiple files. One case study cited in the benchmark description involves a project segmented into **5,441 AST chunks**, illustrating the long-context navigation problem that T2L-ARVO is intended to stress.

The benchmark therefore encodes several difficulty dimensions simultaneously. Sanitizer traces can be long and partially irrelevant; memory bugs often crash far from the original validation lapse; system/runtime failures may produce sparse or unstable dynamic evidence; and large projects make exhaustive search infeasible. This suggests that T2L-ARVO is less a static dataset than a controlled experimental environment for testing multi-step diagnostic workflows.

## 5. Empirical results and what they measure

T2L-ARVO was introduced together with T2L-Agent, whose central component is the Agentic Trace Analyzer (ATA). On the benchmark, T2L-Agent achieves up to **58.0% detection** and **54.8% line-level localization**, substantially outperforming the reported baselines [2510.02389].

Without the advanced refinement and divergence mechanisms, baseline performance already shows that coarse detection is easier than exact localization. Reported overall averages include **48.0% detection** and **38.5% localization** for GPT-4.1, **44.3%** and **41.7%** for GPT-5, and markedly lower results for some open models. Familywise behavior is uneven: Buffer Overflow and Memory Lifecycle cases are relatively easier because they provide stronger runtime signals, whereas System Runtime cases are the hardest, with detection rarely exceeding about **21%** in the reported tables [2510.02389].

The ablation results are especially revealing. Removing the Agentic Trace Analyzer drives both GPT-5 and Claude 4 Sonnet to **0.0% detection** and **0.0% localization**, showing that static prompt-only inspection is ineffective under the benchmark’s conditions. Adding **detection refinement** improves localization substantially for several models, and adding **divergence tracing** yields the best results, including **58.0% detection** for GPT-5 and **54.8% localization** for Qwen3 Next 80B [2510.02389].

These numbers do not imply that T2L-ARVO is “solved.” On the contrary, they indicate that even strong agentic systems fail on roughly half the benchmark at line level. The benchmark is therefore calibrated to expose meaningful headroom rather than to certify mature deployment readiness.

## 6. Scope, misconceptions, and terminological context

Several points of clarification are necessary for accurate interpretation. First, T2L-ARVO is **not** the full ARVO dataset. It is a **50-case benchmark** derived from ARVO, intended primarily as a held-out evaluation suite rather than a large training corpus [2510.02389]. Second, it is **not** a generic static vulnerability benchmark: runtime evidence is integral to the task definition, and the published ablations show that removing trace-aware analysis collapses performance. Third, its current scope is limited to memory-safety and type/runtime issues in reproducible C/C++ OSS projects; it does not cover non-crashing logic vulnerabilities, web application flaws, or large-scale concurrency bugs [2510.02389].

The benchmark’s limitations follow directly from its design. Its scale is only **50 cases**, although those cases are expert-verified and balanced across crash families. Human verification and environment preparation remain bottlenecks. The paper also notes missing fine-grained metadata and the continued cost of running full trace-aware agent pipelines, even under controlled budgets [2510.02389].

The name also invites acronymic confusion. In T2L-ARVO, **T2L** denotes **Trace-to-Line**, as in T2L-Agent [2510.02389]. This is unrelated to other uses of “T2L” in recent literature, including **text-to-layout** in story visualization and change detection [2305.18247] [2412.15541], and **Text-to-LoRA** for hypernetwork-based adapter generation [2506.06105]. The benchmark name is therefore specific to line-level vulnerability localization over ARVO-derived cases, not to text-conditioned generation or model adaptation.

T2L-ARVO’s main significance lies in how it redefines evaluation granularity for LLM-based vulnerability analysis. Rather than asking whether a model can label a file or function as vulnerable, it asks whether an agent can navigate a real project, use dynamic evidence, and reach the patched line. In that sense, T2L-ARVO marks a shift from coarse vulnerability identification toward precision diagnostics grounded in reproducible software artifacts [2510.02389].

Source: https://www.emergentmind.com/topics/t2l-arvo