CodeTracer Architecture: Modular Tracing & Diagnostics

• CodeTracer architecture is a modular framework that standardizes heterogeneous execution artifacts into a hierarchical trace tree with persistent memory.
• It enables systematic failure onset localization and reflective replay to optimize agent debugging and recovery.
• Benchmark evaluations demonstrate improved precision, recall, and reduced token cost compared to bare LLMs across diverse agent workflows.

CodeTracer architecture is a modular tracing and diagnostic framework developed for analyzing, explaining, and debugging complex code agent executions at scale. The system parses heterogeneous execution artifacts, reconstructs agent state transitions as a hierarchical trace tree with memory, and performs systematic failure onset localization. It enables precise post-mortem diagnosis, supports reflective replay for recovery, and sets benchmarks for evaluation on diverse large-scale agent workflows (Li et al., 13 Apr 2026).

1. Architectural Overview and System Pipeline

CodeTracer is structured as a multi-stage pipeline, integrating robust extraction, hierarchical state tracking, and diagnostic inference. The overall data flow is as follows:

$m_i$4

The pipeline processes raw artifacts generated by diverse code agent frameworks, producing:

• A formal trace tree capturing all agent state transitions.
• Step- and stage-localized diagnosis of failure onsets and associated evidence.
• Optionally, reflective replay plans for improved agent reruns (Li et al., 13 Apr 2026).

2. Evolving Extractors: Artifact Normalization

The first stage implements an extractor registry for automating extraction across heterogeneous agent outputs:

• Registry Lookup: Each encountered directory/file schema is matched against existing parser templates using fingerprint similarity (e.g., filename and JSON key overlap).
• Synthesis & Registration: If an artifact format is not recognized, CodeTracer synthesizes and registers a new extractor (using prompt-driven or template-based routines).
• Output Schema: Each step is normalized into a record with action, observation, diff, and verification outcome fields.

No retraining is required for new formats: extraction capacity grows with the system as further formats are encountered, greatly improving scalability relative to manual curation (Li et al., 13 Apr 2026).

3. Hierarchical Trace Tree Construction

CodeTracer formalizes the agent run as a directed hierarchical trace tree $T = (V, E)$:

• Node ($v_i$): Represents a distinct agent and environment state, including a compact memory summary $m_i$ and an exploration step set $S_i$.
• Transition ($E$): Edge $(v_{i-1}, v_i)$ exists if the step results in a nontrivial state change (nonzero code diff per $\Delta(\mathrm{artifact}_i)$).
• State Evolution: Iteratively, $v_i = f(v_{i-1}, \mathrm{artifact}_i)$ for each state-changing step; exploration-only steps are aggregated within the current node.
• Memory Accumulation: Each node's memory $m_i$ is updated via $m_i = U(m_{i-1}, \tau_i)$, carrying forward accumulated regressed tests, changed files, and observed errors.

This hierarchical representation encodes not only linear execution but also complex branching, backtracking, and exploratory subpaths common in agent-based coding workflows (Li et al., 13 Apr 2026).

4. Persistent Memory Module

The persistent memory module incrementally tracks and aggregates diagnostic signals:

• For each new node $v_i$0, the persistent memory $v_i$1 merges prior context $v_i$2 with new facts (such as failed tests or file/regression statistics).
• This memory is extended across multiple runs (for the same or related tasks) to facilitate recognition of recurring failure motifs and to optimize extractor invocation and evidence propagation.
• The persistent memory design enables both intra-run and inter-run contextualization, supporting advanced diagnostic inference and efficient template reuse (Li et al., 13 Apr 2026).

5. Failure Onset Localization Algorithm

Failure localization proceeds in two stages: stage-level prioritization and intra-stage step selection.

• Stage Scoring: For each stage $v_i$3, a weighted scoring function is applied:

$v_i$4

where $v_i$5 is the number of regressed tests, $v_i$6 is lines changed, $v_i$7 counts diagnostic backtracking, and $v_i$8 is the exploratory step fraction.

• Stage Selection: Identify $v_i$9 as the putative failure onset.
• Evidence Extraction: Rank steps within $m_i$0 by similar local signals, producing a minimal evidence set $m_i$1 of failure-relevant steps.

The algorithm operates with total $m_i$2 complexity, where $m_i$3 is the number of steps, supporting scalability to long agent traces (Li et al., 13 Apr 2026).

6. Benchmarking and Quantitative Evaluation

Evaluation is conducted via CodeTraceBench, comprising 3,326 filtered agent trajectories drawn from 7,936 raw runs on four code agent frameworks and multiple backbones (Claude, GPT-5, DeepSeek, etc.).

• Metrics: Step-level macro Precision/Recall/F1 against gold failure-relevant steps, prompt token cost, replay recovery rate.
• Results: On GPT-5 runs, CodeTracer achieves 48.0% F1 (vs. 18.8% for bare LLM), 45.0% macro-precision, and 51.5% macro-recall, while reducing average token cost to 31,100. Reflective replay using localized CodeTracer diagnostics recovers +10–15 percentage points in Pass@1 success over baseline agent reruns.

A summary table of results on the Full split for three backbones:

Method / Backbone	Precision (%)	Recall (%)	F1 (%)	Token Cost (k)
Bare LLM (GPT-5)	16.7	21.5	18.8	58.5
Mini-CodeTracer (GPT-5)	26.0	21.4	19.3	44.8
CodeTracer (GPT-5)	45.0	51.5	48.0	31.1

Similar gains are reported for other foundation models (Claude, DeepSeek) (Li et al., 13 Apr 2026).

7. Context, Significance, and Theoretical Implications

CodeTracer advances the state of the art in error attribution and diagnostic analysis for code agent frameworks by:

• Parsing and normalizing heterogeneous agent run artifacts via dynamically evolving extractors, eliminating format brittleness.
• Formally reconstructing fine-grained agent state transitions as a hierarchical trace tree with persistent context.
• Leveraging composite statistical scoring to accurately localize failure onset at both the stage and step levels.
• Enabling reflective replay workflows whereby localized diagnosis can seed improved agent reruns with substantially lower token cost and higher robustness.

A plausible implication is that adoption of hierarchical tracing with persistent memory and lightweight statistical localization could become a standard for benchmarking and scaling multi-agent, multi-stage coding systems, especially for complex tasks involving high rates of cascading errors or exploratory dead-ends.

Benchmarks and methodology introduced in (Li et al., 13 Apr 2026) have set a reference point for future research in trace-based debugging and agent workflow introspection in automated software engineering.