TraceFormat: Structured Execution Trace

• TraceFormat is a structured representation that converts raw execution logs into TAO steps, enabling precise behavioral diagnosis and issue localization.
• It employs deterministic parsing with LLM-based abstraction to reduce verbosity while maintaining essential causal and temporal details.
• The format supports cross-case aggregation and error analysis in agentic systems, enhancing the evaluation of deep research and coding workflows.

Searching arXiv for the cited TraceSIR paper and closely related work on trace formats, trace encodings, and verifiable formatting. TraceFormat is the core representation introduced in TraceSIR for structured analysis of agentic execution traces. In that framework, raw execution logs in standard OpenAI messages JSON are deterministically mapped into a sequence of Thought–Action–Observation steps, then selectively abstracted to remain within practical context limits while preserving temporal order and causal alignment. The representation is motivated by the fact that agentic systems for deep research, tool calling, and coding produce long, intricate traces containing thousands of tool calls and sub-agent interactions, verbose chain-of-thought reasoning, and large code blocks or tool outputs; in that setting, manual inspection does not scale, directly prompting LLMs over raw logs is unstable, and outcome-only evaluation discards the behavioral evidence needed for issue localization and root cause analysis (Yang et al., 28 Feb 2026).

1. Definition and representational scope

TraceFormat is formally defined by a deterministic parsing function from the original message sequence to a structured trace: $$\Phi: \mathcal{M} \rightarrow \mathcal{T} = \{ (t_i, a_i, o_i) \}_{i=1}^{N}$$ where $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$ is the original message sequence in OpenAI format, and each tuple $(t_i, a_i, o_i)$ is a Thought–Action–Observation step (Yang et al., 28 Feb 2026).

A trace case consists of a required oid, a messages field containing the agent’s execution trace in standard OpenAI chat format, and optional auxiliary fields task, gold_score, gold_judge, and other. Each raw message $m_k$ has a role and content. TraceFormat standardizes this heterogeneous log into a stepwise behavioral representation.

Component	Source in raw trace	Function
Thought $t_i$	assistant messages	intermediate reasoning, planning, or internal explanation
Action $a_i$	assistant messages	tool invocation, API call, code execution, or explicit no-op/final response
Observation $o_i$	role: tool or equivalent	tool return values, search results, or execution outputs

The representation preserves the full step-wise causal structure, the actual actions and their arguments, tool or environment outputs needed to understand behavior, and temporal order. It discards or abstracts redundant or excessively verbose content such as overly long chain-of-thought segments, huge code snippets, large tool outputs, and non-essential chat wrapper text. The paper notes that TraceFormat can be rendered as a table over step index, Thought, Action, and Observation (Yang et al., 28 Feb 2026).

This positioning is central: TraceFormat is neither a raw log dump nor an outcome summary. It is a standardized abstraction intended to retain enough behavioral detail for diagnosis while remaining compact enough for reliable downstream analysis.

2. Formal structure and abstraction mechanism

The parser $\Phi$ is deterministic at the parsing stage. The resulting structured trace $\mathcal{T}$ is then passed through a length-aware abstraction operator: $\mathcal{T}' = \mathcal{A}_{\theta}(\mathcal{T})$ where $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$0 is a length threshold, set in the experiments to 100 words or 1,000 characters (Yang et al., 28 Feb 2026).

The abstraction operates at the granularity of individual steps and individual fields. A single overlong $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$1 may be compressed while leaving $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$2 and $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$3 intact. The same mechanism may also be applied to the task field when necessary. Its stated purpose is to substantially reduce redundancy while remaining faithful to the original execution semantics.

The prioritization policy is explicit. The system keeps crucial decisions, including which tool was invoked and with which parameters; key error messages, exceptions, and wrong assumptions; and tool outputs that directly inform subsequent decisions. It compresses repeated text, low-information parts of outputs such as long logs, and overlong intermediate reasoning that does not change the behavior profile (Yang et al., 28 Feb 2026).

Two guarantees define the semantics of the format. First, temporal order is preserved: $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$4 follow the original message order. Second, causal alignment is preserved: each $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$5 is the reasoning that led to $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$6, and $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$7 is the observation resulting from that action. The paper explicitly treats these properties as the basis for downstream reasoning.

The compression stage is not fully deterministic because the summarization step is LLM-based, but the paper states that in practice the behavior is stable enough for diagnosis and reporting. This yields a representation that is deterministic in structural parsing and adaptive in length control.

3. Function inside TraceSIR

TraceFormat is the common representational layer for the three agents in TraceSIR: StructureAgent, InsightAgent, and ReportAgent (Yang et al., 28 Feb 2026).

StructureAgent receives JSON input with oid, messages, and optional metadata, applies $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$8 to produce $\mathcal{M} = \{ m_1, m_2, \dots, m_K \}$9, applies $(t_i, a_i, o_i)$0 to produce $(t_i, a_i, o_i)$1, and emits the compressed structured trace together with a standardized id using a TraceBench prefix. The abstraction operator is run as an LLM-based tool, and its purpose is to ensure that resulting traces fit into downstream context windows without losing critical information.

InsightAgent consumes the compressed structured trace $(t_i, a_i, o_i)$2, the associated task field $(t_i, a_i, o_i)$3, and optionally gold_score and gold_judge. It outputs a structured diagnosis

$(t_i, a_i, o_i)$4

where $(t_i, a_i, o_i)$5 is an overall completion score from 0 to 100, $(t_i, a_i, o_i)$6 contains detected errors, $(t_i, a_i, o_i)$7 contains weaknesses and optionally strengths, $(t_i, a_i, o_i)$8 is root cause analysis, and $(t_i, a_i, o_i)$9 contains optimization suggestions including fine-tuning samples (Yang et al., 28 Feb 2026).

The TAO organization is presented as the reason TraceFormat is sufficient for diagnosis. It provides a clean, step-wise causal narrative in which one can compare intent $m_k$0, action $m_k$1, and resulting observation $m_k$2. The paper explicitly uses this to distinguish earlier reasoning failure, action-selection failure, and tool or environment failure. In that sense, TraceFormat supports issue localization not merely by compression but by causal factorization.

ReportAgent operates at the aggregate level over cases $m_k$3. It consumes the per-case diagnostics $m_k$4, the standardized id, and the structured trace $m_k$5. It computes error-type frequencies

$m_k$6

and score distributions over intervals

$m_k$7

where $m_k$8 is gold_score if available or $m_k$9 otherwise. It then generates a Markdown report combining quantitative statistics and qualitative patterns, scans the report for referenced standardized ids, and appends the corresponding structured trace data in an appendix (Yang et al., 28 Feb 2026).

This trace-level traceability is one of the format’s defining properties. Reported claims are backed by specific TAO segments rather than by detached summaries.

4. Evaluation, comparative position, and common misconceptions

TraceFormat is not evaluated in isolation; its empirical effect is assessed through the full TraceSIR pipeline on TraceBench, which contains 150 failed traces from BrowseComp, Tau2Bench, and SWE-bench, spanning Deep Research, Function Calling, and Agentic Coding (Yang et al., 28 Feb 2026).

The evaluation protocol, ReportEval, scores reports along five dimensions from 0 to 10: OS (Overall Structure), EA (Error Analysis), RCA (Root Cause Analysis), OA (Optimization Analysis), and OI (Overall Impact). The overall report score is the sum of the five dimensions, normalized to 0–100. Under human evaluation, the average overall score improvement is 9.7%. Under LLM-as-a-judge evaluation, the average improvement is 7.5%, with the largest gain, 26.0%, occurring in Agentic Coding with a weaker backbone (Yang et al., 28 Feb 2026).

The paper’s comparative claims are explicit. Direct LLM reading of raw traces is described as unreliable because long traces are too long and noisy, exceed context limits, and reduce reliability. Outcome-only evaluation is described as insufficient because success or failure scores cannot support issue localization or root cause analysis. Simple summarization or clustering-based aggregation is described as losing fine-grained, step-level evidence and producing coarse, weakly grounded reports. Related representations based on selected states or summarized steps are described as often discarding substantial context, not being explicitly aligned with Thought–Action–Observation, and not being designed for cross-case reporting (Yang et al., 28 Feb 2026).

Three recurring misconceptions are therefore addressed by the framework’s design. First, TraceFormat is not merely a shortened trace; it is a causally aligned TAO abstraction. Second, it is not equivalent to evaluating final answers or scores; it is designed precisely because behavioral evidence is needed for diagnosis. Third, it is not a generic summary format detached from execution; it is explicitly trace-centric, with standardized schema and step-level evidence linking conclusions to specific execution steps.

In the paper’s capability comparison, the TraceFormat-based system is credited with full execution trace modeling with explicit Thought–Action–Observation, structured trace abstraction preserving causal and temporal dependencies, and trace-level evidence linking conclusions to specific execution steps, whereas ClaudeCode is marked as lacking these capabilities (Yang et al., 28 Feb 2026).

5. TraceFormat in the broader research landscape

In the broader literature, “trace format” denotes several distinct but related representational regimes. One line treats a trace format as an arbitrary ASCII or binary file format whose logical structure should be declared explicitly. In that setting, DFDL is based on XML Schema and describes how bytes on disk are mapped to an XML Infoset; Defuddle is a generic parser that consumes DFDL-annotated schemas and raw files and outputs XML, optionally extended through GRDDL and XSLT to RDF/OWL. The architecture explicitly separates bits, format description, and logical content, and is motivated by long-term semantic preservation of log or trace data (0910.3152).

A second line treats trace format as a generic, implementation-independent explanatory trace. In the CHR and CHR$t_i$0 setting, a generic trace is defined as a trace that can be produced by different implementations of a software component and used independently from the traced component. There, Simple Fluent Calculus is used to specify observational semantics, virtual and actual traces, and a concrete XML schema for trace events such as initialState, introduce, solve, apply, and fail (Deransart et al., 2010).

A third line makes trace format sample-centric and verifiable. FG-Trac defines five log types—User Mapping Log, Training Role Log, Modality Attention Log, Sample Contribution Score Log, and Training Action Log—realized as NDJSON records keyed by a pseudonymous identifier $t_i$1. These records are anchored by Merkle-tree commitments on a blockchain, so that per-sample traces become tamper-evident and auditable (Chen et al., 21 Jan 2026).

A fourth line treats trace format as a budgeted dynamic structure. BDTS maintains rooted, status-labeled trace graphs and append-only histories under explicit byte or token budgets, using status-filtered reachability, cursor pagination, soft-capped recency logs, reference-counted observation keys, delta overlays, bounded cost caches, and summary-plus-suffix compaction. In that framework, a long history is replaced by a summary item plus the longest suffix that fits within budget (Alpay et al., 20 May 2026).

A fifth line uses the term at the bitstream level. DPTC defines a lossless compressed trace representation for flash-ADC data, storing the first sample explicitly, then encoding differences in groups of four with short or long headers and packing them into 32-bit words; the paper reports a typical storage cost around 4 to 5 bits per sample (Bruni et al., 2019).

A sixth line treats traces as objects to be embedded into numerical feature spaces. In process mining, trace encoding is the preprocessing step that maps event logs into numerical representations for predictive monitoring, anomalous case detection, and trace clustering; the survey benchmarks 27 methods in terms of expressivity, scalability, correlation, and domain agnosticism (Jr. et al., 2023).

Taken together, these works suggest that “trace format” spans declarative schemas, explanatory operational traces, cryptographically anchored lifecycle logs, budgeted compact representations, binary compression formats, and numerical encodings. Within that landscape, TraceFormat in TraceSIR is distinctive for making causal behavioral structure—Thought, Action, and Observation—the explicit unit of abstraction (Yang et al., 28 Feb 2026).

6. Design principles, implementation, and limitations

The design principles stated or implied for TraceFormat are faithfulness, minimality or compression, interpretability, modularity, and scalability. Faithfulness means preserving temporality and causality through aligned TAO triplets. Minimality means removing redundancy and overlong content through $t_i$2 while keeping behavioral significance. Interpretability follows from the TAO structure being easy for both humans and LLMs to inspect. Modularity means that the format is decoupled from the underlying agent architecture, tool suite, and domain. Scalability refers both to long-horizon traces through per-step abstraction and to cross-case aggregation through standardized ids and structured fields (Yang et al., 28 Feb 2026).

The practical interface reflects this modular design. In the demonstration system, users upload either single-case JSON or multi-case ZIP files. The JSON schema includes oid, messages, and optional task, gold_score, gold_judge, and other. TraceSIR creates a job, assigns standardized ids with a TraceBench prefix, runs StructureAgent to convert traces into TraceFormat, runs InsightAgent to compute $t_i$3 per case, and runs ReportAgent to generate reports and append referenced TraceFormat cases. Users can query job status, download analysis results, and rerun tasks by resuming or regenerating reports without reprocessing traces (Yang et al., 28 Feb 2026).

The format also has explicit limitations. The abstraction $t_i$4 and even TAO parsing depend on LLM-based tools, so weak backbones may misrepresent details, especially for complex code or specialized technical content. The current design is stated to expect a moderate number of cases; for very large trace collections, further hierarchical aggregation or multi-stage reporting may be needed. The structuring and step-wise abstraction are token-expensive and slow on huge traces. Because abstraction uses LLMs, some run-to-run variability remains (Yang et al., 28 Feb 2026).

A further practical implication is that optional fields are auxiliary. The paper emphasizes that the core pattern is: $t_i$5 This suggests adaptation to other agentic systems whenever Thought-like content, Action-like content, and Observation-like content can be identified. In that sense, TraceFormat is not simply a serialization convention; it is a representational discipline for turning verbose execution logs into compact, causally faithful evidence suitable for diagnosis, aggregation, and report generation (Yang et al., 28 Feb 2026).