---
title: 'TraceFormat: Structured Execution Trace'
url: https://www.emergentmind.com/topics/traceformat
type: topic
---

# TraceFormat: Structured Execution Trace

Searching arXiv for the cited TraceSIR paper and closely related work on trace formats, trace encodings, and verifiable formatting.
TraceFormat is the core representation introduced in TraceSIR for structured analysis of agentic execution traces. In that framework, raw execution logs in standard OpenAI `messages` JSON are deterministically mapped into a sequence of Thought–Action–Observation steps, then selectively abstracted to remain within practical context limits while preserving temporal order and causal alignment. The representation is motivated by the fact that agentic systems for deep research, tool calling, and coding produce long, intricate traces containing thousands of tool calls and sub-agent interactions, verbose chain-of-thought reasoning, and large code blocks or tool outputs; in that setting, manual inspection does not scale, directly prompting LLMs over raw logs is unstable, and outcome-only evaluation discards the behavioral evidence needed for issue localization and root cause analysis [2603.00623].

## 1. Definition and representational scope

TraceFormat is formally defined by a deterministic parsing function from the original message sequence to a structured trace:
\[
\Phi: \mathcal{M} \rightarrow \mathcal{T} = \{ (t_i, a_i, o_i) \}_{i=1}^{N}
\]
where \(\mathcal{M} = \{ m_1, m_2, \dots, m_K \}\) is the original message sequence in OpenAI format, and each tuple \((t_i, a_i, o_i)\) is a Thought–Action–Observation step [2603.00623].

A trace case consists of a required `oid`, a `messages` field containing the agent’s execution trace in standard OpenAI chat format, and optional auxiliary fields `task`, `gold_score`, `gold_judge`, and `other`. Each raw message \(m_k\) has a `role` and `content`. TraceFormat standardizes this heterogeneous log into a stepwise behavioral representation.

| Component | Source in raw trace | Function |
|---|---|---|
| Thought \(t_i\) | assistant messages | intermediate reasoning, planning, or internal explanation |
| Action \(a_i\) | assistant messages | tool invocation, API call, code execution, or explicit no-op/final response |
| Observation \(o_i\) | `role: tool` or equivalent | tool return values, search results, or execution outputs |

The representation preserves the full step-wise causal structure, the actual actions and their arguments, tool or environment outputs needed to understand behavior, and temporal order. It discards or abstracts redundant or excessively verbose content such as overly long chain-of-thought segments, huge code snippets, large tool outputs, and non-essential chat wrapper text. The paper notes that TraceFormat can be rendered as a table over step index, Thought, Action, and Observation [2603.00623].

This positioning is central: TraceFormat is neither a raw log dump nor an outcome summary. It is a standardized abstraction intended to retain enough behavioral detail for diagnosis while remaining compact enough for reliable downstream analysis.

## 2. Formal structure and abstraction mechanism

The parser \(\Phi\) is deterministic at the parsing stage. The resulting structured trace \(\mathcal{T}\) is then passed through a length-aware abstraction operator:
\[
\mathcal{T}' = \mathcal{A}_{\theta}(\mathcal{T})
\]
where \(\theta\) is a length threshold, set in the experiments to 100 words or 1,000 characters [2603.00623].

The abstraction operates at the granularity of individual steps and individual fields. A single overlong \(o_i\) may be compressed while leaving \(t_i\) and \(a_i\) intact. The same mechanism may also be applied to the `task` field when necessary. Its stated purpose is to substantially reduce redundancy while remaining faithful to the original execution semantics.

The prioritization policy is explicit. The system keeps crucial decisions, including which tool was invoked and with which parameters; key error messages, exceptions, and wrong assumptions; and tool outputs that directly inform subsequent decisions. It compresses repeated text, low-information parts of outputs such as long logs, and overlong intermediate reasoning that does not change the behavior profile [2603.00623].

Two guarantees define the semantics of the format. First, temporal order is preserved: \((t_1,a_1,o_1), (t_2,a_2,o_2), \dots\) follow the original message order. Second, causal alignment is preserved: each \(t_i\) is the reasoning that led to \(a_i\), and \(o_i\) is the observation resulting from that action. The paper explicitly treats these properties as the basis for downstream reasoning.

The compression stage is not fully deterministic because the summarization step is LLM-based, but the paper states that in practice the behavior is stable enough for diagnosis and reporting. This yields a representation that is deterministic in structural parsing and adaptive in length control.

## 3. Function inside TraceSIR

TraceFormat is the common representational layer for the three agents in TraceSIR: StructureAgent, InsightAgent, and ReportAgent [2603.00623].

StructureAgent receives JSON input with `oid`, `messages`, and optional metadata, applies \(\Phi\) to produce \(\mathcal{T}\), applies \(\mathcal{A}_{\theta}\) to produce \(\mathcal{T}'\), and emits the compressed structured trace together with a standardized `id` using a TraceBench prefix. The abstraction operator is run as an LLM-based tool, and its purpose is to ensure that resulting traces fit into downstream context windows without losing critical information.

InsightAgent consumes the compressed structured trace \(\mathcal{T}'\), the associated task field \(q\), and optionally `gold_score` and `gold_judge`. It outputs a structured diagnosis
\[
\mathcal{D} = \{ s, E, W, R, O \}
\]
where \(s\) is an overall completion score from 0 to 100, \(E\) contains detected errors, \(W\) contains weaknesses and optionally strengths, \(R\) is root cause analysis, and \(O\) contains optimization suggestions including fine-tuning samples [2603.00623].

The TAO organization is presented as the reason TraceFormat is sufficient for diagnosis. It provides a clean, step-wise causal narrative in which one can compare intent \(t_i\), action \(a_i\), and resulting observation \(o_i\). The paper explicitly uses this to distinguish earlier reasoning failure, action-selection failure, and tool or environment failure. In that sense, TraceFormat supports issue localization not merely by compression but by causal factorization.

ReportAgent operates at the aggregate level over cases \(i \in \{1,\dots,M\}\). It consumes the per-case diagnostics \(\mathcal{D}_i\), the standardized `id`, and the structured trace \(\mathcal{T}'_i\). It computes error-type frequencies
\[
P(\ell) = \frac{1}{M} \sum_{i=1}^{M} \mathbb{I}(\ell_i = \ell)
\]
and score distributions over intervals
\[
P(b) = \frac{1}{M} \sum_{i=1}^{M} \mathbb{I}(\hat{s}_i \in b),
\]
where \(\hat{s}_i\) is `gold_score` if available or \(s_i\) otherwise. It then generates a Markdown report combining quantitative statistics and qualitative patterns, scans the report for referenced standardized `id`s, and appends the corresponding structured trace data in an appendix [2603.00623].

This trace-level traceability is one of the format’s defining properties. Reported claims are backed by specific TAO segments rather than by detached summaries.

## 4. Evaluation, comparative position, and common misconceptions

TraceFormat is not evaluated in isolation; its empirical effect is assessed through the full TraceSIR pipeline on TraceBench, which contains 150 failed traces from BrowseComp, Tau2Bench, and SWE-bench, spanning Deep Research, Function Calling, and Agentic Coding [2603.00623].

The evaluation protocol, ReportEval, scores reports along five dimensions from 0 to 10: OS (Overall Structure), EA (Error Analysis), RCA (Root Cause Analysis), OA (Optimization Analysis), and OI (Overall Impact). The overall report score is the sum of the five dimensions, normalized to 0–100. Under human evaluation, the average overall score improvement is 9.7%. Under LLM-as-a-judge evaluation, the average improvement is 7.5%, with the largest gain, 26.0%, occurring in Agentic Coding with a weaker backbone [2603.00623].

The paper’s comparative claims are explicit. Direct LLM reading of raw traces is described as unreliable because long traces are too long and noisy, exceed context limits, and reduce reliability. Outcome-only evaluation is described as insufficient because success or failure scores cannot support issue localization or root cause analysis. Simple summarization or clustering-based aggregation is described as losing fine-grained, step-level evidence and producing coarse, weakly grounded reports. Related representations based on selected states or summarized steps are described as often discarding substantial context, not being explicitly aligned with Thought–Action–Observation, and not being designed for cross-case reporting [2603.00623].

Three recurring misconceptions are therefore addressed by the framework’s design. First, TraceFormat is not merely a shortened trace; it is a causally aligned TAO abstraction. Second, it is not equivalent to evaluating final answers or scores; it is designed precisely because behavioral evidence is needed for diagnosis. Third, it is not a generic summary format detached from execution; it is explicitly trace-centric, with standardized schema and step-level evidence linking conclusions to specific execution steps.

In the paper’s capability comparison, the TraceFormat-based system is credited with full execution trace modeling with explicit Thought–Action–Observation, structured trace abstraction preserving causal and temporal dependencies, and trace-level evidence linking conclusions to specific execution steps, whereas ClaudeCode is marked as lacking these capabilities [2603.00623].

## 5. TraceFormat in the broader research landscape

In the broader literature, “trace format” denotes several distinct but related representational regimes. One line treats a trace format as an arbitrary ASCII or binary file format whose logical structure should be declared explicitly. In that setting, DFDL is based on XML Schema and describes how bytes on disk are mapped to an XML Infoset; Defuddle is a generic parser that consumes DFDL-annotated schemas and raw files and outputs XML, optionally extended through GRDDL and XSLT to RDF/OWL. The architecture explicitly separates bits, format description, and logical content, and is motivated by long-term semantic preservation of log or trace data [0910.3152].

A second line treats trace format as a generic, implementation-independent explanatory trace. In the CHR and CHR\(^\vee\) setting, a generic trace is defined as a trace that can be produced by different implementations of a software component and used independently from the traced component. There, Simple Fluent Calculus is used to specify observational semantics, virtual and actual traces, and a concrete XML schema for trace events such as `initialState`, `introduce`, `solve`, `apply`, and `fail` [1001.2188].

A third line makes trace format sample-centric and verifiable. FG-Trac defines five log types—User Mapping Log, Training Role Log, Modality Attention Log, Sample Contribution Score Log, and Training Action Log—realized as NDJSON records keyed by a pseudonymous identifier \(h_i = \mathrm{SHA256}(u_i)\). These records are anchored by Merkle-tree commitments on a blockchain, so that per-sample traces become tamper-evident and auditable [2601.14971].

A fourth line treats trace format as a budgeted dynamic structure. BDTS maintains rooted, status-labeled trace graphs and append-only histories under explicit byte or token budgets, using status-filtered reachability, cursor pagination, soft-capped recency logs, reference-counted observation keys, delta overlays, bounded cost caches, and summary-plus-suffix compaction. In that framework, a long history is replaced by a summary item plus the longest suffix that fits within budget [2605.22879].

A fifth line uses the term at the bitstream level. DPTC defines a lossless compressed trace representation for flash-ADC data, storing the first sample explicitly, then encoding differences in groups of four with short or long headers and packing them into 32-bit words; the paper reports a typical storage cost around 4 to 5 bits per sample [1903.10984].

A sixth line treats traces as objects to be embedded into numerical feature spaces. In process mining, trace encoding is the preprocessing step that maps event logs into numerical representations for predictive monitoring, anomalous case detection, and trace clustering; the survey benchmarks 27 methods in terms of expressivity, scalability, correlation, and domain agnosticism [2301.02167].

Taken together, these works suggest that “trace format” spans declarative schemas, explanatory operational traces, cryptographically anchored lifecycle logs, budgeted compact representations, binary compression formats, and numerical encodings. Within that landscape, TraceFormat in TraceSIR is distinctive for making causal behavioral structure—Thought, Action, and Observation—the explicit unit of abstraction [2603.00623].

## 6. Design principles, implementation, and limitations

The design principles stated or implied for TraceFormat are faithfulness, minimality or compression, interpretability, modularity, and scalability. Faithfulness means preserving temporality and causality through aligned TAO triplets. Minimality means removing redundancy and overlong content through \(\mathcal{A}_{\theta}\) while keeping behavioral significance. Interpretability follows from the TAO structure being easy for both humans and LLMs to inspect. Modularity means that the format is decoupled from the underlying agent architecture, tool suite, and domain. Scalability refers both to long-horizon traces through per-step abstraction and to cross-case aggregation through standardized `id`s and structured fields [2603.00623].

The practical interface reflects this modular design. In the demonstration system, users upload either single-case JSON or multi-case ZIP files. The JSON schema includes `oid`, `messages`, and optional `task`, `gold_score`, `gold_judge`, and `other`. TraceSIR creates a job, assigns standardized `id`s with a TraceBench prefix, runs StructureAgent to convert traces into TraceFormat, runs InsightAgent to compute \(\mathcal{D}\) per case, and runs ReportAgent to generate reports and append referenced TraceFormat cases. Users can query job status, download analysis results, and rerun tasks by resuming or regenerating reports without reprocessing traces [2603.00623].

The format also has explicit limitations. The abstraction \(\mathcal{A}_{\theta}\) and even TAO parsing depend on LLM-based tools, so weak backbones may misrepresent details, especially for complex code or specialized technical content. The current design is stated to expect a moderate number of cases; for very large trace collections, further hierarchical aggregation or multi-stage reporting may be needed. The structuring and step-wise abstraction are token-expensive and slow on huge traces. Because abstraction uses LLMs, some run-to-run variability remains [2603.00623].

A further practical implication is that optional fields are auxiliary. The paper emphasizes that the core pattern is:
\[
\texttt{messages} \rightarrow \text{TAO via } \Phi \rightarrow \text{compressed via } \mathcal{A}_{\theta}.
\]
This suggests adaptation to other agentic systems whenever Thought-like content, Action-like content, and Observation-like content can be identified. In that sense, TraceFormat is not simply a serialization convention; it is a representational discipline for turning verbose execution logs into compact, causally faithful evidence suitable for diagnosis, aggregation, and report generation [2603.00623].

Source: https://www.emergentmind.com/topics/traceformat