TraceBench: Diverse Trace-Centric Benchmarks

• TraceBench is a collection of trace-centric evaluation benchmarks that cover domains like code reasoning, agentic analysis, and HPC I/O diagnosis.
• It employs diverse methods such as execution-verified call graphs in TraceEval, structured error analysis in TraceSIR, and expert-labeled issues in IOAgent.
• The benchmarks shift focus to traces as primary artifacts, enabling detailed semantic, performance, and diagnostic evaluations.

Searching arXiv for the cited papers and related uses of “TraceBench”. TraceBench is an overloaded term in recent arXiv literature. In one usage, it denotes the prompt-level referent answered by TraceEval, an execution-verified, multi-language benchmark for code semantic reasoning that evaluates recovery of runtime call structure from source code (Li et al., 10 May 2026). In other usages, it denotes a failure-centric benchmark for structured analysis and reporting of agentic execution traces (Yang et al., 28 Feb 2026), the first open diagnosis test suite for HPC I/O performance diagnosis from Darshan traces (Egersdoerfer et al., 25 Feb 2026), and, in a different sense, the benchmark driver/artifact accompanying a framework for budgeted dynamic trace structures (Alpay et al., 20 May 2026). The term therefore names not one fixed benchmark, but a family of trace-oriented evaluation artifacts in which traces themselves, rather than only final outputs, become the primary object of measurement.

1. Terminological scope

Recent usage of the term spans several technically distinct artifacts.

Usage	Domain	Core object of evaluation
TraceEval	Code semantic reasoning	Runtime call-graph edges
TraceBench in TraceSIR	Agentic systems	Diagnostic and reporting quality over failed traces
TraceBench in IOAgent	HPC I/O diagnosis	Expert-labeled I/O issues in Darshan traces
TraceBench artifact in BDTS	Trace data structures	Synthetic-trace and token-budget benchmarks

The most direct answer to the label “TraceBench” in the code-reasoning setting is TraceEval, which the relevant paper presents as the benchmark aligned with that prompt wording. However, the same label is also used by the TraceSIR and IOAgent papers for two unrelated benchmarks, and by the BDTS paper for an ancillary benchmark driver rather than a benchmark standard. A plausible implication is that “TraceBench” functions in current literature as a descriptive label for trace-centric benchmarking rather than as a uniquely identifying proper name (Li et al., 10 May 2026, Yang et al., 28 Feb 2026, Egersdoerfer et al., 25 Feb 2026, Alpay et al., 20 May 2026).

2. TraceEval as an execution-verified benchmark for code semantic reasoning

TraceEval is a multi-language, execution-verified benchmark for code semantic reasoning, focused on recovering a program’s runtime call structure from source code rather than generating code that merely passes tests. Its core task is call graph construction. A program $P$ has functions $F=\{f_1,\ldots,f_n\}$, and its call graph is $G=(F,E)$, where $(f_i,f_j)\in E$ iff some invocation site in $f_i$ resolves to $f_j$ at runtime. Given source code $S$, a system predicts an edge set $\hat{E}$, and evaluation is at the edge level using

$$\text{Precision}=\frac{|\hat{E}\cap E^\star|}{|\hat{E}|},\qquad \text{Recall}=\frac{|\hat{E}\cap E^\star|}{|E^\star|},\qquad \text{F1}=\frac{2PR}{P+R}.$$

A positive edge is correct only if it is observed during execution under the benchmark’s validation inputs. The ground truth is therefore the execution-covered call graph rather than static-tool output or hand annotation (Li et al., 10 May 2026).

The execution-verified property is central. Labels come from direct runtime observation: Python uses sys.settrace, JavaScript uses AST instrumentation, and Java uses source-level method injection. A traced edge is a witnessed call. Programs are accepted only if they run successfully end-to-end, produce at least two traced edges, are deterministic across repeated tracer runs, and conform to the unified JSON schema. The paper explicitly frames this as a way to avoid annotator disagreement, label noise, and the precision/recall tradeoff of static analyzers.

The corpus contains 10,583 real-world programs, with a test split of 2,129 and a train split of 8,454. By language, the full corpus contains 3,820 Python programs, 3,226 JavaScript programs, and 3,537 Java programs. The split is repository-level, with 769 Python, 649 JavaScript, and 711 Java programs in test, and 3,051 Python, 2,577 JavaScript, and 2,826 Java programs in train. Instances were extracted from 1,600+ permissively licensed GitHub repositories; the candidate pool was sampled from 1,655 repositories drawn from a 10,183-row GitHub GraphQL search over repositories with at least 50 stars and permissive licenses.

The construction pipeline has three stages. First, harness generation prompts GPT-5.4 once per source file to add a driver or entry point, lightweight stubs for external dependencies, and to keep the original call relationships intact so that the file becomes self-contained and runnable. Second, language-specific tracers execute the rewritten program and record caller$\rightarrow$callee edges actually exercised. Third, validation accepts only programs whose execution succeeds, whose trace has at least two edges, and which satisfy determinism and schema-validity checks. The same reproducible pipeline can be applied to new repositories, so the benchmark is also an instance-generation mechanism.

The zero-shot evaluation covers 10 LLMs on the held-out test split, all run at temperature 0.0: five frontier proprietary models and five open-weight models. The strongest model is Claude-Opus-4.6, with overall average F1 of 72.9%; its language-specific scores are 79.1 for Python, 64.8 for JavaScript, and 74.9 for Java. Other reported averages are 64.5 for Claude-Sonnet-4.6, 60.0 for GPT-5.4, 56.5 for GPT-5.4-mini, 62.1 for DeepSeek-v3.2, 44.3 for Llama-3.3-70B-Instruct, 46.6 for Qwen2.5-Coder-32B-Instruct, 27.1 for Qwen2.5-Coder-7B-Instruct, and 9.5 for Qwen2.5-Coder-1.5B-Instruct. The spread from 72.9 to 9.5 is reported as a 7.7× difference. The paper identifies three recurring false-positive sources for the strongest model: untaken-branch hallucination, declared-vs-runtime-type dispatch confusion, and class-name-as-callee output-schema mismatch. It also reports a clear precision–recall tradeoff, with Claude-Opus-4.6’s advantage over Claude-Sonnet-4.6 coming mainly from higher precision.

To demonstrate the utility of the training split as a supervision substrate, the authors LoRA-fine-tune Qwen2.5-Coder-1.5B, 7B, and 32B on the 8,454-program training split, using teacher-generated reasoning traces and the format source $F=\{f_1,\ldots,f_n\}$0 trace $F=\{f_1,\ldots,f_n\}$1 call graph, with LoRA rank $F=\{f_1,\ldots,f_n\}$2 and learning rate $F=\{f_1,\ldots,f_n\}$3. Reported F1 scores are 65.1 for 1.5B, 68.4 for 7B, and 71.2 for 32B, corresponding to gains of +55.6, +41.3, and +24.6 respectively. The tuned Qwen2.5-Coder-32B reaches 71.2%, within 1.7 F1 points of zero-shot Claude-Opus-4.6. The paper also notes that tuned Qwen2.5-Coder-1.5B surpasses several zero-shot frontier models other than Claude-Opus-4.6.

The stated limitations are that the current benchmark covers only Python, JavaScript, and Java. Planned future languages are Go, Rust, and TypeScript. The authors also propose replacing teacher-distilled chain-of-thought supervision with verifier-driven post-training, in which the model generates its own rationales and is rewarded according to whether the predicted call graph matches tracer ground truth. Released artifacts include the benchmark, the reproducible pipeline, baseline runners, tuned-model checkpoints, and a datasheet.

3. TraceBench in structured analysis of agentic execution traces

In the TraceSIR paper, TraceBench is a unified, failure-centric benchmark for automated analysis and reporting of agentic execution traces. Its motivation is the claim that existing agentic benchmarks are largely outcome-oriented and therefore discard the behavioral evidence needed for issue localization, root-cause analysis, and cross-case reporting. The benchmark is designed to compare systems on their ability to structure traces, diagnose errors, explain causes, and generate actionable reports rather than merely predict final success or failure (Yang et al., 28 Feb 2026).

TraceBench covers three representative real-world agentic scenarios, each tied to an established benchmark: BrowseComp for deep research, Tau2Bench for function calling, and SWE-bench for agentic coding. For each scenario, the authors run GLM-4.6 on the official test split, retain the complete execution traces in the standard OpenAI message format, and randomly sample 50 failure cases with gold_score = 0. The resulting benchmark contains 150 failed task instances in total, 50 per scenario. Each instance includes a required oid field and a messages field, with optional auxiliary fields task, gold_score, gold_judge, and other.

TraceSIR does not analyze these traces only as raw message lists. It first applies TraceFormat, a deterministic abstraction

$F=\{f_1,\ldots,f_n\}$4

where each round contains the agent’s Thought $F=\{f_1,\ldots,f_n\}$5, Action $F=\{f_1,\ldots,f_n\}$6, and Observation $F=\{f_1,\ldots,f_n\}$7. StructureAgent then further compresses long trace elements by applying a length-aware abstraction operator

$F=\{f_1,\ldots,f_n\}$8

with threshold $F=\{f_1,\ldots,f_n\}$9 set to 100 words or 1,000 characters. The paper emphasizes that this preserves temporal order and causal alignment while making long traces tractable for downstream analysis.

TraceBench is paired with ReportEval, the evaluation protocol for the reports generated over these traces. ReportEval scores each report on five equally weighted 0–10 dimensions: Overall Structure, Error Analysis, Root Cause Analysis, Optimization Analysis, and Overall Impact. The overall report score is the equally weighted sum of the five dimension scores, normalized to the range 0–100. Evaluation is hybrid: six expert LLM-agent engineers or researchers conduct blinded human evaluation, and GPT-5 is used as an LLM judge, with each LLM-judge evaluation run three times and averaged.

The baseline is ClaudeCode, evaluated with GLM-5 and Claude-4.6 backbones; TraceSIR is evaluated with the same backbones over the same 50 sampled failure cases per scenario. Under human evaluation, the reported overall scores are 55.0, 66.0, 65.0, and 81.0 on Deep Research; 40.0, 74.0, 53.0, and 77.0 on Function Calling; and 57.0, 77.0, 62.0, and 89.0 on Agentic Coding, for ClaudeCode + GLM-5, ClaudeCode + Claude-4.6, TraceSIR + GLM-5, and TraceSIR + Claude-4.6 respectively. The paper summarizes this as a 9.7% average relative improvement under human evaluation, with the largest relative gain reaching 26.0%. Under GPT-5-based judging, the corresponding scores are 82.7, 90.0, 88.0, and 91.3 for Deep Research; 80.7, 88.0, 89.3, and 91.3 for Function Calling; and 58.7, 90.0, 84.7, and 90.7 for Agentic Coding, with an average 7.5% relative improvement over ClaudeCode. The largest dimension-level gains are reported for error analysis and root cause analysis.

The benchmark is therefore both a dataset and an evaluation scaffold. It supports instance-level diagnosis, because each failed trace is analyzed individually, and scenario-level aggregation, because ReportAgent produces one report per scenario after all 50 cases have been processed.

4. TraceBench in HPC I/O performance diagnosis

In the IOAgent paper, TraceBench is a labeled benchmark suite for HPC I/O performance diagnosis from Darshan traces. The paper presents it as the first open diagnosis test suite for this problem and uses it to compare diagnosis systems on accuracy, utility, and interpretability. The benchmark exists because the authors identify a lack of publicly available trace datasets with labeled I/O performance issues, making fair comparison of diagnosis systems difficult (Egersdoerfer et al., 25 Feb 2026).

TraceBench is built entirely from Darshan traces, specifically the original Darshan I/O traces rather than Darshan DXT, because DXT is higher overhead and typically not enabled by default. The benchmark draws from three sources. Simple-Bench contributes 10 labeled Darshan logs generated by 10 small C scripts that intentionally introduce at least one specific I/O issue from the label set. IO500 contributes 21 traces collected from 21 unique configurations of the IO500 benchmark. Real Applications contributes 9 samples from real production HPC applications, with additional recollected traces for E2E and OpenPMD after fixing the primary issue. The paper reports 40+ traces and 182 issues total.

The label space consists of 11 diagnosis categories, some split into read and write variants: High Metadata Load; Misaligned Read Requests; Misaligned Write Requests; Random Access Patterns on Read; Random Access Patterns on Write; Shared File Access; Small Read I/O Requests; Small Write I/O Requests; Repetitive Data Access on Read; Server Load Imbalance; Rank Load Imbalance; Multi-Process Without MPI; No Collective I/O on Read; No Collective I/O on Write; Low-Level Library on Read; and Low-Level Library on Write. The labels are expert-driven: each trace is manually labeled, and each I/O issue is confirmed by at least two experts.

The paper provides a source-wise label distribution. For example, totals are 24 for Server Load Imbalance, 19 for Shared File Access, 18 each for Misaligned Write Requests, Small Write I/O Requests, and No Collective I/O on Read, 17 for Small Read I/O Requests, 16 for Misaligned Read Requests, 15 for No Collective I/O on Write, 13 for Multi-Process Without MPI, 7 each for Random Access Patterns on Read and on Write, 5 for High Metadata Load, 2 for Rank Load Imbalance, and 1 each for Repetitive Data Access on Read, Low-Level Library on Read, and Low-Level Library on Write. This construction makes the benchmark explicitly multi-label, because many traces contain overlapping issues.

Evaluation uses three metrics. Accuracy measures how well the diagnosis matches the ground-truth labels. Utility measures how useful the diagnosis is for understanding I/O behavior, identifying issues, and figuring out how to address them. Interpretability measures readability and understandability for users with different levels of HPC I/O expertise. For Utility and Interpretability, the paper uses GPT-4o as a judge to rank anonymized tool outputs from 1 to 4, with 1 best and 4 worst, while rotating tool names, rank labels, and diagnosis order to reduce position bias. The reported scoring formulas are

$G=(F,E)$0

$G=(F,E)$1

and a normalized score $G=(F,E)$2 obtained by dividing by the maximum possible score over the subset.

The compared systems are IOAgent-gpt-4o, IOAgent-llama-3.1-70B, ION, and Drishti. Reported overall averages are 0.632 for IOAgent-gpt-4o, 0.550 for IOAgent-llama-3.1-70B, 0.447 for Drishti, and 0.383 for ION. IOAgent-gpt-4o is reported as best overall on accuracy and interpretability, while IOAgent-llama-3.1-70B is notably strong on utility and on simpler cases. Within the paper’s argument, TraceBench is the benchmarking scaffold that connects Darshan traces, expert-labeled issue categories, competing diagnosis systems, and the three scoring dimensions.

5. Adjacent trace-centric benchmarks and artifacts

Several nearby papers clarify what TraceBench is not, and what neighboring trace-oriented benchmark design looks like.

The paper "TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation" introduces TRACE, not TraceBench, as a benchmark for execution efficiency in cross-language code translation. It contains 1,000 efficiency-critical tasks across C++, Java, and Python, covers six translation directions, adds 10 generated stress tests per task to the 10 default tests from TransCoder-Test, and evaluates efficiency with Execution Time and Peak Memory through the Beyond score from Mercury. The paper’s main claim is that correctness is not a reliable proxy for efficiency, with Claude-4-think leading correctness at 95.5% pass rate but only mid-level time efficiency, while smaller open-source models such as Qwen2.5-Coder-14B-Instruct do better on time efficiency (Gong et al., 17 Mar 2026). This makes TRACE conceptually adjacent to TraceEval: both reject correctness-only evaluation, but they probe different latent properties.

The BDTS paper uses “TraceBench” differently again. There, TraceBench is the benchmark driver/artifact for an ancillary Rust implementation of budgeted dynamic trace structures rather than a community benchmark standard. The paper evaluates synthetic traces with 10,000, 20,000, and 40,000 vertices; reports build times of 0.582 ms, 1.169 ms, and 2.722 ms; full descendant-enumeration times of 0.243 ms, 0.589 ms, and 1.415 ms; and compaction of histories from 350,000, 1,030,000, and 2,710,000 approximate tokens down to 1,048, 2,072, and 4,120 approximate tokens. It also reports model-tokenizer measurements in which raw strings of 3,359–3,360 tokens are reduced to 432–433 tokens for distilbert/distilgpt2, gpt2, and facebook/opt-125m (Alpay et al., 20 May 2026). In this usage, TraceBench is not the scientific contribution but the measurement harness.

Other trace-driven benchmark constructions reinforce the same pattern. MapReplayBench is generated by tracing HashMap API usage from Java applications and replaying it in a standalone JMH harness, with the main reported case study showing strong positive Pearson correlation $G=(F,E)$3 between replay and application benchmark speedup ratios and a reduction in experimentation time from about 72 hours to about 8 hours per default-initial-capacity configuration (Schiavio et al., 14 Mar 2026). TorchTraceAP is presented as the first benchmark dataset specifically designed to evaluate and improve ML models’ ability to detect anti-patterns in PyTorch traces; it contains 610 traces, split into 460 training and 150 testing traces, and defines window-level detection and window-level classification tasks over anti-pattern regions in hierarchical CPU/GPU traces (Chen et al., 16 Dec 2025).

These neighboring works show that trace-centric benchmarking is now distributed across code reasoning, code translation efficiency, agentic systems, HPC I/O, systems data structures, library performance engineering, and ML-systems profiling. The label “TraceBench” therefore sits inside a broader methodological movement rather than naming a single benchmark lineage.

6. Common methodological themes and recurrent misconceptions

A recurring misconception is that TraceBench denotes one benchmark. The literature summarized here does not support that view. The same label names at least three distinct benchmarks and one benchmark driver, with different domains, tasks, and oracles. TraceEval measures edge-level recovery of execution-covered call graphs from source code. TraceSIR’s TraceBench measures the quality and usability of reports over failed agentic traces. IOAgent’s TraceBench measures diagnosis quality on expert-labeled Darshan traces. BDTS’s TraceBench measures algorithmic performance and token-budget behavior of a trace data-structural framework.

A second misconception is that these benchmarks are merely variants of correctness or pass-rate evaluation. The papers consistently argue otherwise. TraceEval contrasts semantic reasoning about execution structure with test-passing code benchmarks. TRACE argues that correct translations may still be inefficient. TraceSIR argues that final task outcomes discard trace evidence needed for issue localization and root-cause analysis. IOAgent argues that raw traces require expert-grounded diagnosis rather than simple parsing or success/failure summaries. This suggests that trace-centric benchmarking is motivated by inadequacy of coarse end metrics when the research goal concerns semantics, efficiency, diagnosis, or explanation.

A third distinction concerns the oracle. TraceEval uses direct runtime observation as ground truth. IOAgent uses expert-confirmed issue labels. TraceSIR uses a balanced report-evaluation protocol combining expert judgment and GPT-5 judging. TRACE uses stress tests and normalized Beyond scores over reference implementations. BDTS uses formal invariants, asymptotic bounds, synthetic traces, and tokenizer or forward measurements. A plausible implication is that “trace benchmark” is best understood as a family resemblance term: the shared object is the trace, but the correctness criterion depends on whether the trace is being treated as behavioral evidence, diagnostic substrate, performance workload, or compacted state representation.

Across these usages, the technical role of traces also changes. In TraceEval, traces witness runtime call edges. In TraceSIR, traces preserve behavioral evidence for analysis and report generation. In IOAgent, traces are the observable substrate from which I/O pathologies must be inferred. In BDTS, traces are mutable, budgeted data structures. In adjacent works such as MapReplayBench and TorchTraceAP, traces become replay workloads or anti-pattern localization targets. The result is a heterogeneous but coherent research area in which trace collection, trace abstraction, and trace-grounded evaluation increasingly function as primary methodological choices rather than auxiliary tooling.