CodeTraceBench: A Trace-Centered Benchmark

Updated 4 July 2026

CodeTraceBench is a multi-purpose benchmark suite defined over code-agent trajectories, runtime call-graphs, and code translation tasks.
It utilizes detailed, step-level annotations and a two-pass protocol to precisely localize failure onsets in heterogeneous execution traces.
Empirical evaluations show that hierarchical trace indexing and diagnostic replay yield significant improvements in failure localization and translation efficiency.

CodeTraceBench is a benchmark name that appears in multiple 2026 arXiv preprints, but its most explicit and developed use is in "CodeTracer: Towards Traceable Agent States," where it denotes a large-scale, step-level annotated benchmark of code-agent execution traces for failure onset localization and diagnostic replay (Li et al., 13 Apr 2026). In the same year, the name is also used in adjacent summaries for two distinct benchmark lines: TraceEval, an execution-verified benchmark for call-graph recovery from source code, is "also referred to below as CodeTraceBench," and TRACE is presented as "CodeTraceBench (TRACE)" for efficiency-oriented code-translation evaluation (Li et al., 10 May 2026, Gong et al., 17 Mar 2026). This suggests a shared emphasis on trace-grounded evaluation rather than outcome-only scoring, but the datasets, targets, and metrics are materially different.

1. Nomenclature and benchmark scope

A common source of confusion is that "CodeTraceBench" does not denote a single, universally fixed benchmark in the 2026 literature. Instead, the label is attached to three distinct evaluation settings: code-agent trajectory diagnosis, dynamic call-graph recovery, and execution-efficiency assessment for code translation. The most direct use of the standalone name occurs in the CodeTracer work, where CodeTraceBench is defined as "the first large-scale, step-level annotated benchmark of code-agent execution traces" (Li et al., 13 Apr 2026).

Usage of the name	Primary target	Reported scale
CodeTraceBench in CodeTracer	Failure localization in code-agent trajectories	full split 3,320; verified split 1,060
TraceEval, also referred to as CodeTraceBench	Runtime call-graph recovery from source	10,583 programs
CodeTraceBench (TRACE)	Execution efficiency in code translation	1,000 efficiency-critical tasks

The distinction matters because the supervised object differs across the three uses. In CodeTracer, the object is a heterogeneous multi-stage trajectory with raw run artifacts, stage spans, and incorrect or unproductive steps. In TraceEval, the object is a source program paired with a dynamically witnessed set of caller-to-callee edges. In TRACE, the object is a translation task whose outputs are scored by execution time and peak memory under stress tests. Consequently, performance figures are not directly comparable across the three benchmarks even when the same model family appears in multiple evaluations.

2. Construction of the code-agent trajectory benchmark

In the CodeTracer formulation, CodeTraceBench is constructed from executed trajectories drawn from five public software-engineering benchmarks and four code-agent frameworks (Li et al., 13 Apr 2026). The source benchmarks are SWE-bench Verified, SWE-bench Pro, MultiSWE-bench, SWE-PolyBench, and TerminalBench. These benchmarks cover repository-level bug fixing, multi-file refactoring and transformation, and long-horizon CLI interaction. The trajectories are produced under SWE-Agent, MiniSWE-Agent, OpenHands, and Terminus 2, paired with five model backbones: Claude-sonnet-4, GPT-5, DeepSeek-V3.2, Qwen3-Coder-480 B, and Kimi-K2-Instruct.

The detailed construction pipeline reports a raw pool of 5,284 trajectories after filtering out timeouts, truncated logs, and environment misconfigurations. Short trivial runs, defined as correct trajectories with fewer than ten normalized steps, are then removed, leaving 4,354 trajectories for annotation. Tasks span bug fixing, multi-file refactoring, dependency installation and system configuration, scripting and build-system maintenance, and complex command-line interfaces.

The benchmark contains 236 unique tasks organized into 26 semantic categories, with examples including testing harnesses, dependency resolution, regex transformations, and configuration fixes. Each task is tagged with "Easy," "Medium," or "Hard" difficulty according to trajectory length and observed failure complexity. After curating for long-horizon runs with clear failure cascades and deduplicating near-identical executions, the release is organized as a "full split" of 3,320 instances and a higher-quality "verified split" of 1,060 instances. Each instance includes raw run artifacts, stage boundaries, gold failure-critical stage labels, and step-level annotations of every incorrect or unproductive action.

The benchmark is therefore not merely a collection of logs. It is a supervised corpus in which each trajectory is normalized into an analyzable object with explicit temporal structure and error provenance. This makes it suitable for quantitative evaluation of failure localization systems rather than only qualitative debugging case studies.

3. Annotation schema and failure-localization task

CodeTraceBench in the CodeTracer work uses a two-pass annotation protocol consisting of stage labeling and step-level failure tracing (Li et al., 13 Apr 2026). Every command in a trajectory is assigned to one of five ordered workflow phases: environment verification, dependency installation, inspection/debugging, patching, and verification. Contiguous commands of the same phase form a stage span, and agents may revisit earlier phases, producing multiple non-contiguous spans with the same stage label.

Step-level supervision is richer and separates successful from failed trajectories. For successful runs, annotators mark redundant or trial-and-error steps that do not contribute to the final solution. For failed runs, annotation follows a chain-based backward tracing protocol: starting from the failing test output, annotators recursively identify the immediately preceding command whose side-effect produced the observed error, continue tracing upstream, and terminate at the earliest "error-critical" step whose action triggered the downstream cascade. The downstream chain of commands that depend on that error is also flagged as supporting evidence.

The error-critical step is assigned one of six controlled error types: environment/setup issues, dependency resolution failures, mislocalized edits, incorrect hypotheses, verification misinterpretation, and unproductive looping. This design explicitly separates the onset of failure from later manifestations of the failure, which is important in long-horizon agent trajectories where many downstream actions may be locally plausible but globally dependent on an earlier wrong commitment.

Reliability is measured through double annotation on a random 15% subset of trajectories. Cohen’s $\kappa$ on the error-critical step label is reported as $0.73$, indicating substantial agreement under the annotation guidelines. A plausible implication is that the benchmark’s supervision is intended to support both span-level and causal analyses of agent failure, rather than only coarse success/failure classification.

4. Evaluation protocol, baselines, and empirical results

CodeTraceBench supports two principal localization tasks: stage-level failure onset localization and step-level evidence retrieval (Li et al., 13 Apr 2026). The stage-level task asks for the single stage span that contains the earliest error-critical decision. The step-level task asks for the set of incorrect steps whose actions contribute directly to failure. Evaluation uses standard precision, recall, and F1 at the step level, macro-averaged over trajectories, and total LLM tokens consumed during diagnosis are reported as a cost measure. Analogous metrics can be defined at the stage level by collapsing each stage into one atomic unit, but the main reported results focus on step-level evaluation.

Three localization methods are compared under matched budgets. The first is Bare LLM, or direct prompting over raw unstructured run logs. The second is Mini-CodeTracer, which standardizes heterogeneous logs into a flat JSON but does not use hierarchical indexing. The third is the full CodeTracer system, which adds evolving extraction and tree indexing to build a hierarchical trace and then performs structured evidence queries along the trace tree.

On the full evaluation set of 3,320 trajectories, CodeTracer substantially outperforms the lighter baselines. With Claude-sonnet-4, Bare LLM achieves 16.2 F1 using 105.1k tokens, Mini-CodeTracer achieves 19.2 F1 using 82.4k tokens, and CodeTracer reaches 46.6 F1 using 56.8k tokens. With GPT-5, the corresponding F1 values are 18.8, 19.3, and 48.0, with token usage of 58.5k, 44.8k, and 31.1k. With DeepSeek-V3.2, the F1 values are 16.3, 19.2, and 46.1, with token usage of 83.4k, 63.8k, and 44.6k. The reported aggregate comparison states that Mini-CodeTracer recovers only part of the gap, improving F1 by roughly 2–3 percentage points over direct prompting, whereas hierarchical trace indexing yields the largest gain, improving F1 by 27–29 percentage points while reducing token cost by 15–45%.

These results position CodeTraceBench as a benchmark for structured diagnostic reasoning rather than for code generation in the narrow sense. The dominant variable is not whether the model can summarize a log, but whether it can recover a failure chain from heterogeneous, partially redundant trajectory artifacts.

5. Diagnostic replay and recovery of failed runs

The CodeTracer work extends CodeTraceBench beyond passive localization by introducing diagnostic replay on originally failed trajectories (Li et al., 13 Apr 2026). The procedure first generates structured localization diagnostics, including the error-critical stage and supporting evidence steps. The same backbone is then reinvoked under the identical per-run iteration and token budget, but with a concise hint summarizing the failure onset prepended to the new run.

The diagnosis tokens are counted outside the replay budget. Average diagnosis-token costs are reported as 8.4k for Claude-sonnet-4, 5.2k for GPT-5, and 7.1k for DeepSeek-V3.2. Under this setup, replay yields consistent absolute gains of 8–12 percentage points in Pass@1 on the subset of originally failed runs. The average boost is described as approximately $0.10$, that is, about a 10 percentage-point increase.

This replay protocol is significant because it turns failure localization into an intervention signal. Instead of using annotations only to score a post hoc explanation, the framework tests whether accurate localization can change downstream task outcomes under matched budgets. The reported gains support the narrower claim that early, precisely localized failure cues can help agents revise wrong commitments rather than repeat unproductive exploration.

Two other 2026 benchmark lines use "CodeTraceBench" as an alternate or descriptive label, but they target different technical objects. In "An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoning," TraceEval is explicitly "also referred to below as CodeTraceBench" and is defined as an execution-verified benchmark for recovering a program’s runtime call structure from source code (Li et al., 10 May 2026). Its pipeline comprises LLM-assisted harness generation, dynamic tracing, and mechanical validation. A program is accepted only if it executes under its harness with no runtime errors, yields at least two cross-function edges, is deterministic across three runs, and produces valid output JSON. The released corpus contains 10,583 programs, split into 8,454 train and 2,129 test instances across Python, JavaScript, and Java. On the held-out test split, Claude-Opus-4.6 reaches an average F1 of 72.9%, and LoRA-tuned Qwen2.5-Coder-32B reaches 71.2%, within 1.7 F1 of zero-shot Claude-Opus-4.6.

In "TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation," the benchmark is presented as "CodeTraceBench (TRACE)" and is defined as the first benchmark explicitly designed to expose and evaluate execution efficiency in LLM-based cross-language code translation (Gong et al., 17 Mar 2026). TRACE builds on TransCoder-Test, generates stress tests through a progressive synthesizer loop, and retains 1,000 efficiency-critical tasks across C++, Java, and Python. Efficiency is measured by execution time and peak memory, with Beyond scores computed relative to a task-specific reference spectrum. The benchmark reports that 23.5% of correct translations are more than $2\times$ slower or more memory-hungry than the best, and it organizes inefficiencies into three top-level classes: algorithm implementation discrepancy, language construct mismatch, and resource management inefficiency.

Taken together, these three uses of the name reveal a broader pattern. CodeTraceBench in the narrow sense refers to an annotated code-agent trajectory benchmark for localizing hidden error chains. TraceEval and TRACE extend the same trace-oriented intuition to execution-grounded semantics and efficiency analysis, respectively. A common misconception is therefore that the term names a single benchmark with a single metric family; the 2026 literature instead uses it for several trace-centered evaluation programs that differ in supervision, scale, and operational target.