TraceSIR: A Multi-Agent Trace Analysis Framework

• TraceSIR is a multi-agent framework that structures, diagnoses, and reports LLM agent execution traces by converting raw logs into actionable insights.
• It employs a sequential pipeline with StructureAgent, InsightAgent, and ReportAgent to parse trace data, diagnose errors, and generate comprehensive reports.
• The framework introduces TraceFormat to maintain the Thought–Action–Observation structure, enabling precise error localization and root cause analysis.

Searching arXiv for the TraceSIR paper to ground the article in the latest record. TraceSIR is a multi-agent framework for structured analysis and reporting of agentic execution traces. It is designed for LLM-based agents that use external tools and iterative decision making in settings such as deep research, function calling, and coding, where long and intricate traces make failure diagnosis and root cause analysis difficult. Rather than reducing evaluation to final pass/fail outcomes, TraceSIR analyzes full execution traces, introduces a structured abstraction called TraceFormat, and organizes analysis through three specialized agents—StructureAgent, InsightAgent, and ReportAgent—together with a benchmark of real traces, TraceBench, and a report-centric evaluation protocol, ReportEval (Yang et al., 28 Feb 2026).

1. Problem formulation and motivation

TraceSIR addresses a failure mode of contemporary agent evaluation: outcome-only assessment discards the behavioral signal required for issue localization. In the problem setting considered by the framework, an agentic system is an LLM-based system that operates over multiple steps, calls tools or APIs, browses the web, manipulates code, maintains internal reasoning and planning, and adjusts decisions based on observations. A single query can therefore produce a long execution trace containing user messages, assistant intermediate reasoning, tool calls, tool outputs, and possibly nested sub-agents. These traces can be thousands of tokens long, with hundreds or thousands of tool calls (Yang et al., 28 Feb 2026).

The diagnostic difficulty follows directly from this trace structure. Manual inspection does not scale across large trace collections, and cross-case comparison further multiplies the effort. Direct LLM summarization of raw logs is unreliable because traces often exceed context windows and naive summarization can hallucinate or ignore important steps. A recurrent misconception in this setting is that agent analysis can be reduced to final success or failure plus a short judge message. TraceSIR is built on the opposite premise: partial progress, mistaken assumptions, planning errors, repeated tool misuse, and related intermediate behaviors are the primary evidence needed for root cause analysis and systematic improvement.

The framework formulates the central technical problem as one of abstraction under constraint. A usable representation must fit within LLM context budgets, preserve the causal structure of behavior, and remain analyzable across many cases so that coherent and actionable reports can be generated. TraceSIR’s contribution is to treat these requirements as a unified systems problem rather than as isolated summarization or judging steps.

2. Multi-agent pipeline and system architecture

TraceSIR operates on one or more raw traces in OpenAI-style messages JSON format, with optional metadata such as task, gold_score, and gold_judge. The pipeline then proceeds through three agents in a fixed order: StructureAgent, InsightAgent, and ReportAgent (Yang et al., 28 Feb 2026).

Component	Input	Function and output
StructureAgent	Raw messages trace and optional metadata	Parses traces into TraceFormat and outputs a compact structured trace $\mathcal{T}'$
InsightAgent	Structured trace $\mathcal{T}'$, task, optional gold labels	Produces a structured diagnostic object $\mathcal{D}$
ReportAgent	Diagnostics $\{\mathcal{D}_1,\dots,\mathcal{D}_M\}$	Generates a Markdown report with statistics, recurring patterns, and recommendations

The separation of roles is deliberate. StructureAgent makes traces analyzable, InsightAgent performs per-case diagnosis, and ReportAgent summarizes the system-level picture. This modularization suggests a design principle of staged abstraction: raw behavior is first normalized into a trace representation, then interpreted case by case, and only then aggregated into a report. A plausible implication is that TraceSIR is intended to reduce error propagation that would otherwise occur if a single model attempted to summarize raw logs, diagnose failures, and infer recurring patterns in one pass.

The framework is report-centric rather than metric-centric. It does compute statistics, including error frequencies and score distributions, but the terminal artifact is a report intended to be coherent, informative, and actionable. That emphasis distinguishes TraceSIR from pipelines that stop at trajectory scoring, clustering, or pass/fail judgments.

3. TraceFormat and length-aware trace abstraction

TraceFormat is the framework’s core abstraction for execution traces. A raw trace case contains an original identifier oid, a message sequence

$\mathcal{M} = \{ m_1, m_2, \dots, m_K \},$

and optional fields such as task, gold_score, gold_judge, and other. TraceSIR defines a deterministic parsing function

$$\Phi: \mathcal{M} \rightarrow \mathcal{T} = \{ (t_i, a_i, o_i) \}_{i=1}^{N},$$

where each triple $(t_i, a_i, o_i)$ corresponds to one interaction round: Thought, Action, and Observation (Yang et al., 28 Feb 2026).

The representation preserves strict chronological order and explicit alignment between what the agent thought, what it did, and what happened. In raw OpenAI messages, assistant reasoning and tool calls are often mixed in a single assistant message, tool outputs may be long and noisy, and there is no explicit decomposition into the classic agent loop “Thought → Action → Observation.” TraceFormat makes that loop first-class. The case also receives a standardized id with a TraceBench prefix, which supports consistent cross-referencing in tools and reports.

StructureAgent then applies a length-aware abstraction operator

$\mathcal{T}' = \mathcal{A}_{\theta}(\mathcal{T}),$

where $\theta$ is a length threshold and $\mathcal{A}_\theta$ selectively abstracts overlong elements while preserving essential behavioral information through automatic tool invocation. The operator acts independently on Thought, Action, and Observation fields. In experiments, $\mathcal{T}'$0 is set to 100 words or 1,000 characters. Short fields are preserved as is; overlong thoughts, actions, observations, or even the task field can be summarized through tool-mediated interaction with the backbone LLM.

The abstraction is selective rather than truncational. TraceSIR preserves temporal order, causal links between thoughts, actions, and observations, high-level planning content, intermediate decisions, and key metadata. It compresses or removes verbose repeated chain-of-thought text, long code or logs that are not directly needed for understanding behavior, and redundant tool outputs. The significance of this design is methodological: if the structuring stage erases the wrong details, subsequent diagnosis and reporting cannot recover them. TraceFormat is therefore not merely a convenience format but the condition of possibility for the later stages.

4. Per-case diagnosis and cross-case report generation

InsightAgent performs fine-grained diagnosis on each structured trace $\mathcal{T}'$1. Given $\mathcal{T}'$2, the task $\mathcal{T}'$3, and optional gold_score and gold_judge, it outputs a structured diagnostic object

$\mathcal{T}'$4

where $\mathcal{T}'$5 is an overall task completion score from 0 to 100, $\mathcal{T}'$6 denotes detected errors, $\mathcal{T}'$7 denotes weaknesses, $\mathcal{T}'$8 denotes root cause analysis, and $\mathcal{T}'$9 denotes optimization suggestions (Yang et al., 28 Feb 2026).

The error field identifies a primary task-critical error together with secondary errors. The weakness field captures recurring patterns such as poor planning or misreading tool outputs, and may also mention strengths. The root-cause field is the framework’s explanatory center: it accounts for why errors occurred rather than merely listing where they appeared. The optimization field contains concrete recommendations for prompt changes, tool design, or agent policy, including fine-tuning samples derived from the trace. The paper further indicates that InsightAgent supports issue localization and canonical error labels such as reasoning error, tool misuse or misinterpretation, planning failure, and insufficient exploration, although it does not define a fixed taxonomy.

Optional gold metadata functions as an auxiliary supervisory signal rather than as a prerequisite. gold_score indicates ground-truth success or failure; in TraceBench, only failures with gold_score = 0 are used. gold_judge provides textual feedback from the benchmark’s automatic evaluator. InsightAgent can incorporate these signals to align scoring and interpretation, but it can also operate on $\mathcal{D}$0 alone.

ReportAgent aggregates diagnostics across cases. For $\mathcal{D}$1 cases it receives

$\mathcal{D}$2

with $\mathcal{D}$3. It computes canonical error labels $\mathcal{D}$4 and estimates error frequencies by

$\mathcal{D}$5

For performance analysis it partitions scores into intervals $\mathcal{D}$6, defines $\mathcal{D}$7 if available and $\mathcal{D}$8 otherwise, and estimates

$\mathcal{D}$9

The resulting Markdown report includes an overall structure and introduction, quantitative summaries, qualitative patterns in weaknesses, root causes, and optimization suggestions, concrete examples with trace snippets referencing specific case IDs, and actionable recommendations for practitioners. After generating the report, ReportAgent automatically identifies which TraceBench case IDs are referenced and appends their structured traces to an appendix. This evidential appendix is important because it grounds report-level conclusions in specific trace steps rather than generic claims. In the experiments, report generation is triggered every 10 cases, although evaluation uses only the final report after all 50 cases per scenario; in practical deployments, a final report is generated even for a single case.

5. TraceBench, ReportEval, and empirical results

TraceSIR is evaluated on TraceBench, a benchmark built from three real-world agentic scenarios: BrowseComp for deep research, Tau2Bench for function calling, and SWE-bench for agentic coding. The target agent model is GLM-4.6. For each benchmark, GLM-4.6 is run on the official test split, complete execution traces are recorded in OpenAI message format, and the benchmark’s official evaluator provides a binary gold_score. From each benchmark, 50 failure cases with gold_score = 0 are randomly sampled, producing 150 failed traces in total, split evenly across Deep Research, Function Calling, and Agentic Coding (Yang et al., 28 Feb 2026).

Report quality is assessed with ReportEval, which scores each report on five dimensions from 0 to 10: Overall Structure (OS), Error Analysis (EA), Root Cause Analysis (RCA), Optimization Analysis (OA), and Overall Impact (OI). The overall report score is

$\{\mathcal{D}_1,\dots,\mathcal{D}_M\}$0

Human evaluation uses 6 expert engineers or researchers in LLM agents, blinded to method and backbone, with at least 2 experts evaluating each scenario’s reports. LLM-as-a-judge evaluation uses GPT-5, run 3 times per report and averaged, with the same five ReportEval dimensions.

The principal baseline is ClaudeCode, instantiated with the same two backbones used for TraceSIR: GLM-5 and Claude-4.6. All methods use the same 50 failure traces per scenario and generate exactly one report per scenario per backbone. Under human evaluation, the reported overall scores are as follows. In Deep Research, ClaudeCode + GLM-5 scores 55.0, ClaudeCode + Claude-4.6 scores 66.0, TraceSIR + GLM-5 scores 65.0, and TraceSIR + Claude-4.6 scores 81.0. In Function Calling, the corresponding scores are 40.0, 74.0, 53.0, and 77.0. In Agentic Coding, they are 57.0, 77.0, 62.0, and 89.0. Across scenarios and backbones, TraceSIR improves overall report quality by ~9.7% under human evaluation, with the largest gains in Error Analysis and Root Cause Analysis.

GPT-5 yields the same ranking. For each scenario, TraceSIR with Claude-4.6 is best: 91.3 in Deep Research, 91.3 in Function Calling, and 90.7 in Agentic Coding. With the GLM-5 backbone, TraceSIR also outperforms ClaudeCode: 82.7 to 88.0 in Deep Research, 80.7 to 89.3 in Function Calling, and 58.7 to 84.7 in Agentic Coding, including a +26.0 point improvement in the last case. On average, TraceSIR yields a 7.5% relative improvement under LLM-based evaluation. Human and LLM-based evaluators differ in absolute scores, but they agree on ranking: TraceSIR’s reports are consistently more structured, comprehensive, and actionable than ClaudeCode’s.

6. Limitations, broader context, and practical significance

TraceSIR’s limitations follow from its architecture. All three agents are LLM-based, so quality depends on the capabilities of the underlying backbone models; weak models can miss nuance during abstraction or diagnosis, especially in complex coding or domain-specific technical settings. The current design assumes a moderate number of cases, exemplified by 50-case evaluations, and more sophisticated hierarchical aggregation or incremental reporting may be needed for thousands of traces. The multi-agent pipeline also incurs token cost, latency, and output variability, so reports are not perfectly reproducible (Yang et al., 28 Feb 2026).

Within the broader ecosystem, TraceSIR is positioned at the intersection of agent debugging, monitoring and evaluation, and interpretability and analysis. Compared with outcome-oriented evaluations, it uses full traces, emphasizes behavioral diagnosis and root cause analysis, and produces cross-case reports rather than only metrics. Compared with methods such as DREAM, TRACE, TRAIL, MAS-FIRE, and reward-based approaches, the paper characterizes prior work as often focusing on specific metrics, partial signals, selected states, or trajectory scoring, which can lose the context needed for deep root cause analysis. Compared with naive summarization or clustering, TraceSIR retains explicit linkage between report claims and trace-level evidence.

Its distinctive features are therefore threefold. First, TraceFormat makes the Thought–Action–Observation structure explicit while preserving causal dependencies under compression. Second, the multi-agent architecture separates structuring, per-case diagnosis, and cross-case reporting into distinct computational roles. Third, ReportEval evaluates reports directly in terms aligned with practitioner needs rather than treating reports as incidental by-products. The appendix comparison described in the paper further attributes to TraceSIR native, systematic support for TAO-based trace modeling, structured trace abstraction preserving causal dependencies, cross-case aggregation with statistics, fine-grained per-case diagnostics, explicit linkage from report claims to trace evidence, generation of SFT samples, and robust performance with weaker backbones.

In practical terms, TraceSIR requires traces in or convertible to OpenAI messages format, an LLM API such as GLM-5 or Claude-4.6, and optionally task, gold_score, and gold_judge fields for richer analysis. Its immediate significance is not that it replaces benchmark evaluation, but that it converts messy execution logs into structured, evidence-grounded analyses at both the single-run and multi-run levels. The future directions suggested by the paper—more structured or symbolic trace representations beyond TAO, hierarchical aggregation, automated remediation, and tighter integration into training and monitoring pipelines through SFT data or RL signals—indicate that TraceSIR is best understood as a framework for operational diagnosis of agentic behavior rather than as a single summarization technique.