T2L-ARVO: Trace-to-Line Benchmark

• T2L-ARVO is a curated benchmark for agentic, line-level vulnerability localization, extracting 50 expert-verified cases from the ARVO dataset.
• It couples AST-based project chunking with Dockerized runtime artifacts, enabling both coarse detection over project modules and precise line-level localization.
• Empirical results show models achieving up to 58% detection and 54.8% localization, revealing significant headroom for refining vulnerability diagnostics.

Searching arXiv for papers on T2L-ARVO, ARVO, and T2L-Agent. T2L-ARVO is a benchmark for agentic, line-level vulnerability localization in real-world open-source software. It was introduced together with T2L-Agent as a curated subset of ARVO and is designed to evaluate two related tasks: coarse-grained detection over project chunks and fine-grained localization to exact vulnerable lines. The benchmark contains 50 expert-verified, reproducible C/C++ cases spanning five crash families, and couples AST-based project segmentation with Dockerized runtime artifacts such as sanitizer traces and stack information, so that systems are assessed under project-scale, trace-aware conditions rather than snippet-level static classification alone (Xi et al., 30 Sep 2025).

1. Provenance within the ARVO ecosystem

T2L-ARVO is derived from ARVO, the “Atlas of Reproducible Vulnerabilities for Open-source software,” which was created to make real OSS-Fuzz vulnerabilities consistently rebuildable, triggerable, and analyzable across versions. In its 2024 release, ARVO reproduced 5,651 / 8,934 candidate vulnerabilities, identified precise patch commits for 5,001 / 5,651 reproduced cases, and covered 273 C/C++ projects (Mei et al., 2024). In the later 2026 release, ARVO expanded to 6,138 real vulnerabilities across 311 projects, with 81% reproduction success and 89.4% patch-location accuracy (Mei et al., 15 Jun 2026).

The T2L-ARVO paper describes its parent corpus at the time of benchmark construction as containing 4,993 reproducible vulnerabilities in 288 OSS projects (Xi et al., 30 Sep 2025). T2L-ARVO should therefore be understood not as the full ARVO dataset, but as a deliberately curated benchmark extracted from it for evaluating vulnerability localization systems. This distinction is central: ARVO is a large-scale reproducible vulnerability dataset, whereas T2L-ARVO is a small, expert-verified benchmark specialized for trace-to-line reasoning.

ARVO’s relevance to T2L-ARVO is structural rather than merely nominal. ARVO provides reproducible builds, PoCs, patch locations, and versioned environments; T2L-ARVO adds benchmark-specific chunking, line-level ground truth, and evaluation protocols for LLM-based localization. This suggests that T2L-ARVO functions as a task-specific layer on top of ARVO’s broader reproducibility infrastructure.

2. Benchmark construction and case composition

T2L-ARVO contains 50 vulnerabilities, with 10 cases per crash family, selected to balance coverage across five failure categories while preserving real-world project complexity (Xi et al., 30 Sep 2025). The benchmark’s crash families are as follows:

Crash family	Representative subtypes
Buffer Overflow	heap-buffer-overflow, stack-buffer-overflow, global-buffer-overflow, index-out-of-bounds
Uninitialized Access	use-of-uninitialized-value, Segv on unknown address, Null-dereference
Memory Lifecycle	heap-use-after-free, heap-double-free, use-after-poison, invalid-free
Type Safety / Parameter Validation	bad-cast, negative-size-param, memcpy-param-overlap, object-size violations, incorrect-function-pointer-type, non-positive VLA bound
System Runtime	unknown signal, check failed, sanitizer configuration errors

Case selection proceeds from the larger ARVO corpus through clustering by failure mechanism and scoring candidates using both structural and semantic criteria. The structural criteria include the number of files changed in the patch, architectural spread, and directory depth; the semantic criteria include cross-module coupling, interface changes, and concurrency touchpoints. A dual validation layer is then applied: manual expert assessment confirms reproducibility and security relevance, while LLM-assisted checks gauge agent-facing difficulty (Xi et al., 30 Sep 2025).

The resulting benchmark spans imaging, networking, compression, multimedia, scripting, toolchain, security, and document-processing projects. The intent is not to maximize scale, but to preserve “production patterns rather than a single project or bug family.” T2L-ARVO therefore occupies a specific point in the benchmark design space: smaller than ARVO, but more tightly controlled for localization research.

3. Task formulation and scoring

T2L-ARVO supports two primary tasks. The first is chunk-level detection, in which a system must identify the project chunk or module containing the vulnerability. The second is line-level localization, in which the system must predict the exact vulnerable line or lines corresponding to the patch (Xi et al., 30 Sep 2025).

To define chunk-level units, the benchmark introduces AST-based segmentation via a chunk_case tool built on Tree-sitter. Chunks are intended to be function-aligned, semantically coherent units that remain manageable within LLM context windows. Each chunk entry includes file_path, chunk_kind, symbol, start_line, end_line, source, ast_type, and imports. A separate mark_diff stage identifies which chunks intersect the patch, producing the ground-truth chunk set for detection scoring.

The benchmark’s scoring semantics are binary at the case level. If $C_i^*$ denotes the ground-truth chunk set for case $i$ and $P_i^C$ the predicted chunk set, detection is counted as correct when $P_i^C \cap C_i^* \neq \emptyset$. The benchmark-level detection rate is

$$\mathrm{Detection\ Rate} = \frac{1}{N} \sum_{i=1}^{N} \mathrm{Det}_i.$$

For localization, if $L_i^*$ denotes the patched line set in the vulnerable revision and $P_i^L$ the predicted line set, localization is counted as correct when $P_i^L \cap L_i^* \neq \emptyset$. The localization rate is

$$\mathrm{Localization\ Rate} = \frac{1}{N} \sum_{i=1}^N \mathrm{Loc}_i.$$

A correct line-level prediction must exactly match at least one patched line in the old vulnerable code; the benchmark does not report a $\pm k$-line tolerance (Xi et al., 30 Sep 2025). This makes localization substantially stricter than conventional file- or function-level vulnerability detection.

4. Runtime evidence, AST structure, and trace-aware evaluation

T2L-ARVO is explicitly designed for trace-aware localization rather than static inspection of isolated code. Each case is packaged in a Dockerized environment exposing sanitizer and debugger artifacts, including ASAN, MSAN, and UBSAN outputs, stack traces, and optional GDB traces. The benchmark tooling includes run_san, run_gdb, static_analysis, llm_analyze, and compare_llm_metrics, enabling a system to correlate runtime symptoms with repository structure (Xi et al., 30 Sep 2025).

This coupling of runtime evidence with AST chunking is the defining methodological property of T2L-ARVO. The benchmark assumes that effective localization in real OSS requires more than source-code pattern matching: the crash point may be far from the root cause, stack traces may be noisy, and the relevant code may span multiple files. One case study cited in the benchmark description involves a project segmented into 5,441 AST chunks, illustrating the long-context navigation problem that T2L-ARVO is intended to stress.

The benchmark therefore encodes several difficulty dimensions simultaneously. Sanitizer traces can be long and partially irrelevant; memory bugs often crash far from the original validation lapse; system/runtime failures may produce sparse or unstable dynamic evidence; and large projects make exhaustive search infeasible. This suggests that T2L-ARVO is less a static dataset than a controlled experimental environment for testing multi-step diagnostic workflows.

5. Empirical results and what they measure

T2L-ARVO was introduced together with T2L-Agent, whose central component is the Agentic Trace Analyzer (ATA). On the benchmark, T2L-Agent achieves up to 58.0% detection and 54.8% line-level localization, substantially outperforming the reported baselines (Xi et al., 30 Sep 2025).

Without the advanced refinement and divergence mechanisms, baseline performance already shows that coarse detection is easier than exact localization. Reported overall averages include 48.0% detection and 38.5% localization for GPT-4.1, 44.3% and 41.7% for GPT-5, and markedly lower results for some open models. Familywise behavior is uneven: Buffer Overflow and Memory Lifecycle cases are relatively easier because they provide stronger runtime signals, whereas System Runtime cases are the hardest, with detection rarely exceeding about 21% in the reported tables (Xi et al., 30 Sep 2025).

The ablation results are especially revealing. Removing the Agentic Trace Analyzer drives both GPT-5 and Claude 4 Sonnet to 0.0% detection and 0.0% localization, showing that static prompt-only inspection is ineffective under the benchmark’s conditions. Adding detection refinement improves localization substantially for several models, and adding divergence tracing yields the best results, including 58.0% detection for GPT-5 and 54.8% localization for Qwen3 Next 80B (Xi et al., 30 Sep 2025).

These numbers do not imply that T2L-ARVO is “solved.” On the contrary, they indicate that even strong agentic systems fail on roughly half the benchmark at line level. The benchmark is therefore calibrated to expose meaningful headroom rather than to certify mature deployment readiness.

6. Scope, misconceptions, and terminological context

Several points of clarification are necessary for accurate interpretation. First, T2L-ARVO is not the full ARVO dataset. It is a 50-case benchmark derived from ARVO, intended primarily as a held-out evaluation suite rather than a large training corpus (Xi et al., 30 Sep 2025). Second, it is not a generic static vulnerability benchmark: runtime evidence is integral to the task definition, and the published ablations show that removing trace-aware analysis collapses performance. Third, its current scope is limited to memory-safety and type/runtime issues in reproducible C/C++ OSS projects; it does not cover non-crashing logic vulnerabilities, web application flaws, or large-scale concurrency bugs (Xi et al., 30 Sep 2025).

The benchmark’s limitations follow directly from its design. Its scale is only 50 cases, although those cases are expert-verified and balanced across crash families. Human verification and environment preparation remain bottlenecks. The paper also notes missing fine-grained metadata and the continued cost of running full trace-aware agent pipelines, even under controlled budgets (Xi et al., 30 Sep 2025).

The name also invites acronymic confusion. In T2L-ARVO, T2L denotes Trace-to-Line, as in T2L-Agent (Xi et al., 30 Sep 2025). This is unrelated to other uses of “T2L” in recent literature, including text-to-layout in story visualization and change detection (Gong et al., 2023, Zang et al., 2024), and Text-to-LoRA for hypernetwork-based adapter generation (Charakorn et al., 6 Jun 2025). The benchmark name is therefore specific to line-level vulnerability localization over ARVO-derived cases, not to text-conditioned generation or model adaptation.

T2L-ARVO’s main significance lies in how it redefines evaluation granularity for LLM-based vulnerability analysis. Rather than asking whether a model can label a file or function as vulnerable, it asks whether an agent can navigate a real project, use dynamic evidence, and reach the patched line. In that sense, T2L-ARVO marks a shift from coarse vulnerability identification toward precision diagnostics grounded in reproducible software artifacts (Xi et al., 30 Sep 2025).