Agentic Trace Analyzer (ATA)

• Agentic Trace Analyzer (ATA) is a trace-centric layer that records detailed execution trajectories for debugging, accountability, and evaluation in agentic AI.
• ATA employs multi-level analysis methods including LLM-judge evaluation, post-hoc causal localization, and specification-driven compliance to uncover process violations.
• These techniques enhance system governance by reconstructing causal chains and assuring compliance through structured logging, telemetry, and real-time trace analysis.

In the literature summarized here, an Agentic Trace Analyzer (ATA) denotes a trace-centric layer for agentic AI that instruments or consumes execution trajectories and uses them for debugging, evaluation, accountability, root-cause analysis, assurance, or governance, rather than relying only on final outcomes or raw logs. Recent systems describe this role in several ways: as multi-level evaluation above the observability layer, as a structured logging and telemetry substrate, as a causal root-cause localizer, as a specification-driven compliance checker, and as a contract-enforced assurance mechanism with replay and governance (Yehudai et al., 21 May 2026, AlSayyad et al., 7 Feb 2026, Sharma et al., 25 Mar 2026, Paduraru et al., 18 Mar 2026).

1. Conceptual basis and problem setting

Agentic systems are described as increasingly capable systems in which agents define strategies, take actions, and interact with different environments. The resulting autonomy creates serious challenges for overseeing and assessing behavior, because the relevant evidence is distributed across long, stochastic, tool-using traces rather than concentrated in a single final answer (Yehudai et al., 21 May 2026). Related work on runtime observability makes the same point more operationally: autonomous agents do not behave like conventional software, because their behavior is stochastic, multi-step, tool-using, and often internally reasoned through hidden or semi-structured intermediate states; static auditing, proxy filtering, and conventional logging therefore miss the cognitive trajectory and the causal chain linking thought to action (AlSayyad et al., 7 Feb 2026).

The motivating failure model is consistently trace-centric. In deployed multi-agent workflows, visible errors are often far downstream from the actual mistake because of long execution traces, hidden dependencies, and cascading failures. In this setting, “nearest to the error” is a weak heuristic, and backward tracing through execution logs becomes more informative than inspecting only the final error manifestation (Wang, 16 Mar 2026). A related line of work argues that outcome-only benchmarks miss procedural failures such as incorrect workflow routing, unsafe tool usage, and violations of prompt-specified rules; a trace may end in the right database state or correct final answer while still containing serious process violations (Sharma et al., 25 Mar 2026).

Formalizations in this literature reflect the same shift. Agentic CLEAR models a dataset of tasks as $\mathcal{D}=\{x_n\}_{n=1}^N$, an agentic system as $A$, and the trace for task $x_n$ as $t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$, where each step is an LLM call with input $i_k$, output $o_k$, and producing node $a_k$ (Yehudai et al., 21 May 2026). AgenTracer formalizes a multi-agent trajectory as $\tau=(s_0,a_0,s_1,a_1,\ldots,s_T)$ and defines the target of failure attribution as the decisive error: the earliest step whose correction would turn failure into success (Zhang et al., 3 Sep 2025). The trace-based assurance framework models a run as $\tau=(s_0,a_0,o_1,s_1,\dots,s_T)$ and treats failures as trace properties checked over the run, not just final-answer properties (Paduraru et al., 18 Mar 2026).

2. Trace substrates and representations

ATA systems differ first in what they record. Some frameworks are primarily observability substrates, some are abstraction layers, and some are assurance records with contracts and provenance. What they share is an attempt to preserve temporal order, causal linkage, and enough intermediate structure to support diagnosis or verification.

Framework	Trace unit	Preserved structure
AgentTrace (AlSayyad et al., 7 Feb 2026)	operational, cognitive, and contextual spans	public method execution, LLM prompts/completions and extracted reasoning, external I/O linked by trace IDs and span IDs
TraceSIR (Yang et al., 28 Feb 2026)	$\mathcal{T}=\{(t_i,a_i,o_i)\}_{i=1}^{N}$	Thought–Action–Observation rounds, temporal order, causal alignment, optional metadata
MAT assurance framework (Paduraru et al., 18 Mar 2026)	$A$0	acting agent, state snapshot, action, observation, provenance, contracts, verdicts
AgentSim / ATC (Zerhoudi et al., 29 Apr 2026)	thought $A$1 action $A$2 observation chains	retrieved document IDs, evidence spans, grounding and validation metadata
Agentic CLEAR (Yehudai et al., 21 May 2026)	$A$3	stepwise LLM-call inputs, outputs, node identity, trace-level and rubric-level judgments

AgentTrace is explicit about a three-surface schema. The operational surface records public method calls, arguments, return values, start/complete/error status, and timing information; the cognitive surface records raw prompts, model completions, extracted reasoning chains such as Chain-of-Thought, confidence estimates, and sometimes structured fields like plan or reflection; the contextual surface records HTTP requests, database queries, cache accesses, vector-store operations, file-system I/O, and other tool or data-access events. Trace IDs and span IDs link these surfaces causally and temporally, so that a sequence such as method call $A$4 prompt $A$5 reasoning excerpt $A$6 tool call $A$7 returned data $A$8 completion can be reconstructed (AlSayyad et al., 7 Feb 2026).

TraceSIR introduces a different representation objective: compression without loss of behavioral semantics. A deterministic parsing function $A$9 converts OpenAI-format messages into structured interaction rounds, after which an abstraction operator $x_n$0 compresses overly long Thoughts, Actions, and Observations independently; the appendix reports $x_n$1 as 100 words or 1,000 characters (Yang et al., 28 Feb 2026). The trace-based assurance framework instead makes provenance and machine-checkable verdicts first-class, attaching supports, returns, or derived_from relations to MAT records (Paduraru et al., 18 Mar 2026).

AgentSim’s representations are oriented toward grounded RAG traces. The corpus is distributed as full traces, compact trajectories, and supervised pairs, with the full-trace format exposing thought $x_n$2 action $x_n$3 observation chains together with full LLM inputs and outputs. This preserves the link between retrieval behavior, evidence use, synthesis, and abstention (Zerhoudi et al., 29 Apr 2026).

3. Core analytical methods

A prominent ATA pattern is multi-level LLM-judge evaluation. Agentic CLEAR implements a two-stage pipeline: trace evaluation followed by system-level aggregation. For each task, the agent is executed to produce $x_n$4; each step is critiqued by a step judge $x_n$5; the whole trajectory is judged holistically by $x_n$6; task-specific rubrics are generated by $x_n$7 and then validated by $x_n$8; and the accumulated critiques are summarized by $x_n$9 and $t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$0. The prompts distinguish step-level dimensions such as correctness, completeness, and clarity from trace-level dimensions such as execution quality and the final deliverable, and each prompt elicits a brief textual justification prior to the score (Yehudai et al., 21 May 2026).

A second pattern is post-hoc causal localization without LLM inference at debugging time. The causal-graph version of AgentTrace models a failed multi-agent execution as a directed acyclic graph $t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$1 whose edges are reconstructed from logs via sequential edges, communication edges, and data dependency edges. It begins at the observed error node, performs breadth-first backward traversal through parent links to obtain a candidate ancestor set, and then ranks nodes by a weighted linear scoring function over five feature groups: Position, Structure, Content, Flow, and Confidence. The learned weights are $t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$2, $t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$3, $t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$4, $t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$5, and $t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$6, with Position dominating but the remaining groups adding gains in the ablation study (Wang, 16 Mar 2026).

A third pattern is specification-driven compliance checking. AgentPex normalizes heterogeneous traces, extracts explicit-only behavioral rules from the system prompt, tool schema, and task description, and evaluates the trace against output specifications, transition specifications, forbidden edges, argument groundedness, argument schema compliance, predicted plan, and predicted final state. Scores are produced by specialized evaluators using a standardized judge prompt, then aggregated with a gated-minimum rule,

$t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$7

so that strong performance on secondary dimensions cannot hide a serious failure on a critical rule. Predicted Plan and Output Spec are always critical; Transition and Argument Spec are important; Forbidden edges are checked separately and score 0 if any forbidden transition occurs, 100 otherwise (Sharma et al., 25 Mar 2026).

Counterfactual replay and intervention define another ATA family. AgenTracer treats failure attribution as identifying who caused the failure and when the decisive error occurred. For failed trajectories it uses an analyzer agent to propose minimally invasive corrected actions and replays the remainder of the trajectory; for successful trajectories it injects programmed faults into selected steps and labels the injected step as decisive when the perturbed run fails. The resulting TracerTraj examples have the form $t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$8, combining an agent-level label with a step-level label (Zhang et al., 3 Sep 2025).

Trace abstraction and report synthesis form yet another variant. TraceSIR coordinates StructureAgent, InsightAgent, and ReportAgent. StructureAgent converts raw traces into TraceFormat; InsightAgent produces an overall completion score $t_n=\{(i_k,o_k,a_k)\}_{k=1}^{K_n}$9, detected errors $i_k$0, weaknesses $i_k$1, root cause analysis $i_k$2, and optimization suggestions $i_k$3; ReportAgent aggregates $i_k$4 across task instances, estimates error frequencies and score distributions, and writes a consolidated Markdown report with an appendix of referenced cases (Yang et al., 28 Feb 2026).

Probabilistic verification adds a distinct analytical regime. TriCEGAR learns a predicate-tree abstraction $i_k$5 from typed runtime snapshots, maps concrete states to abstract states via $i_k$6, constructs an abstract MDP $i_k$7, and model checks quantitative reachability properties such as $i_k$8 and $i_k$9. It also scores abstract runs by likelihood, $o_k$0, and uses low log-likelihood as an anomaly signal (Koohestani et al., 30 Jan 2026).

4. Assurance, governance, and human supervision

A substantial branch of ATA research treats trace analysis as part of a broader assurance stack. The metrologically grounded TRACE framework for operationally critical domains organizes trustworthy agentic AI into four layers: L1 — Deterministic / physics-based core, L2 — Learned-component inventory, split into L2a — Classical / specialized ML and L2b — Generative / LLM validators, L3 — Stateful orchestration-and-escalation policy, and L4 — Bounded human supervision. TRACE defines six principles—Evidence Traceability, Bounded Human Supervision, Staged Autonomy, Bounded Context, Metrological Accountability, and Model Parsimony / Right-Sized Components—and a trust-metric suite of 17 metrics: 12 layer-wise, 4 cross-cutting, and 1 parsimony metric. Its Computational Parsimony Ratio (CPR) compares the resource cost of the most economical model that adequately solves a sub-task to the resource cost of the deployed model, with $o_k$1 interpreted as optimal model choice and $o_k$2 as architectural overhead or over-engineering (Zabolotnii, 5 May 2026).

The trace-based assurance framework for orchestration makes the assurance layer more operational. It instruments executions as Message-Action Traces, evaluates step contracts $o_k$3 and trace contracts $o_k$4, localizes the first violating step, and stores seed, perturbation schedule, tool stubs or cached outputs, retrieved artifacts, and violated contract IDs for deterministic replay. It extends trace checking with budgeted counterexample search over bounded perturbations, structured fault injection at service, retrieval, and memory boundaries, and a governance mediator

$o_k$5

which enforces per-agent capability limits and language-to-action mediation (Paduraru et al., 18 Mar 2026).

Human supervision is typically bounded and selective rather than pervasive. In TRACE’s L4, humans are final reviewers with workload limits and veto rights, and entry to L4 is determined by the L3 policy rather than requested directly by learned components (Zabolotnii, 5 May 2026). AgentSim uses a disagreement-driven Analyst–Critic–Judge loop, in which two Critic models independently evaluate or revise each step, a Judge computes a Divergence Score,

$o_k$6

and steps with $o_k$7 are sent to human review; the paper reports $o_k$8, reviewer decisions of Promote, Revise, or Discard, and 10% double annotation with adjudication (Zerhoudi et al., 29 Apr 2026). A related domain-specific variant appears in the EHR TRACE framework, where the Auditor is triggered only when uncertainty exceeds a threshold or a safety-critical action is proposed, and then checks contraindications, protocol violations, and hallucinated patient facts (Qu et al., 13 Feb 2026).

5. Domain-specific instantiations and applications

Some ATA systems are general-purpose, but several papers embed trace analysis into highly specific workflows. In software vulnerability localization, T2L-Agent uses an Agentic Trace Analyzer to bridge static code analysis and runtime behavior. ATA ingests crash points, stack traces or backtraces, sanitizer reports, allocation traces, register states, variable-state snapshots, and control-flow cues; aligns them with Tree-sitter / AST-based chunks, source slices, and symbol and line-number metadata; ranks candidate regions and lines by syntax, semantics, and execution-trace alignment; and then supports iterative refinement until exact vulnerable lines are found. The reported ablation is unusually stark: w/o Agentic Trace Analyzer, GPT-5 and Claude 4 Sonnet both fall to 0.0% detection, 0.0% localization, whereas the full T2L-Agent reaches up to 58.0% detection and 54.8% localization on T2L-ARVO (Xi et al., 30 Sep 2025).

In streaming clinical reasoning, TRACE (“Temporal Reasoning via Agentic Context Evolution”) replaces long-context accumulation and retrieval-based augmentation with a dual-memory architecture and a four-agent control loop. The inference state is $o_k$9, where $a_k$0 is a static Global Protocol of institutional rules, $a_k$1 is a dynamic Individual Protocol tracking patient-specific state, and $a_k$2 is a short rolling buffer of recent event bundles. Router selects relevant rules deterministically, Reasoner predicts the next bundle, Auditor conditionally verifies uncertain or high-risk outputs, and Steward performs structured state compression (“Mitosis”). On MIMIC-IV v2.2, protocol adherence is reported as about 92.1%–94.7%, and Auditor activation as about 5.15%–9.42% of timesteps (Qu et al., 13 Feb 2026).

In grounded information-seeking, AgentSim is both a trace generator and a trace-quality control framework. Its Agent-Trace Corpus contains 103,567 reasoning trace steps, 20,548 supervised training pairs, 199,968 unique retrieved documents, and 26,176 total queries generated across MS MARCO, Quasar-T, and CausalQA. The paper reports verification over 5,320 trace entries, including 1,385 substantive answers, with 100% grounding rate on substantive answers, mean token coverage 0.872, and combined quality rate 99.5% for grounded answers plus correct refusals (Zerhoudi et al., 29 Apr 2026).

Operationally critical domains supply a different application profile. The trustworthy-AI TRACE framework instantiates the same four-layer architecture in clinical decision support, industrial multi-domain operations, and a judicial AI assistant. The industrial case uses a sub-domain × layer matrix, with technology and operations described as L2a-dominant and administrative workflows as L1 + L2b-dominant; the judicial case introduces hallucination verification and mandatory review for high-stakes confident recommendations; and the paper cites a 17% hallucination rate in commercial legal RAG systems as motivation for multilayer oversight (Zabolotnii, 5 May 2026).

6. Empirical evidence, misconceptions, and limitations

The strongest quantitative evidence for ATA-style evaluation comes from systems that compare trace-based diagnostics with human annotations or downstream success. Agentic CLEAR evaluates four benchmarks, seven settings, and tens of thousands of LLM calls. Its generated issues map cleanly to human error taxonomies: all 15 GPT-5 issues and 12 OSS-120B issues map to at least one TRAIL category, collectively covering 12 and 10 of the 12 relevant categories. At instance level, GPT-5 full+partial reaches macro F1 = 0.459 and micro F1 = 0.497, while OSS-120B full+partial reaches macro F1 = 0.374 and micro F1 = 0.427. For success prediction, trace-level evaluation is reported as the strongest predictor overall; on AppWorld / GPT-5 / trace score, the reported value is 0.890 AUC, versus 0.823 for step-wise and 0.828 for rubric (Yehudai et al., 21 May 2026).

Causal root-cause localization yields another strong result profile. On 550 synthetic failure scenarios across 10 domains, the causal-graph AgentTrace achieves 94.9% Hit@1, 98.4% Hit@3, and 0.97 MRR, compared with 68.5% Hit@1 and 0.74 MRR for the GPT-4 LLM Analysis baseline. Mean runtime is 0.12 s for AgentTrace versus 8.3 s for LLM-based analysis, reported as about a 69× speedup (Wang, 16 Mar 2026). TraceSIR’s report-centric evaluation is closer to engineering practice than to localization. On TraceBench and ReportEval, it reports an average human-evaluated overall improvement of 9.7%, a largest relative gain up to 26.0%, and an average automated-judge improvement of 7.5% over ClaudeCode, with stronger gains especially in error analysis and root cause analysis (Yang et al., 28 Feb 2026).

A recurring misconception in this literature is that successful end states imply correct behavior. AgentPex directly contests that assumption: among 58 Claude traces with perfect $a_k$3 reward, 48 traces (83%) still contain at least one procedural violation, including simultaneous text plus tool call, omitted calculation, and skipped post-cancellation verification. Its output-spec evaluator yields ROC-AUC = 0.680 for binary classification of $a_k$4 failure, and at a threshold below 65 flags 48% of $a_k$5-failed traces (Sharma et al., 25 Mar 2026). A second misconception is that ATA must always be an LLM-judge pipeline. The causal-graph AgentTrace and TriCEGAR show alternative designs in which graph reconstruction, backward traversal, weighted ranking, online MDP induction, model checking, and likelihood-based anomaly scoring can operate without LLM inference at debugging time or without hand-crafted state abstractions (Wang, 16 Mar 2026, Koohestani et al., 30 Jan 2026).

Limitations are equally consistent. Agentic CLEAR notes that rubrics generated from task descriptions alone can miss hidden task requirements, that many $a_k$6-Bench tasks are adversarial and the correct behavior is to refuse, that step-wise and rubric assumptions do not hold uniformly, that judge choice materially changes the failure descriptions, and that the current pipeline focuses mainly on LLM interactions rather than full system execution (Yehudai et al., 21 May 2026). The observability version of AgentTrace does not provide a full formal ontology of agent actions, beliefs, goals, or risk states, and does not provide detailed empirical overhead benchmarks; the causal-graph version assumes single-root-cause scenarios, accurate execution logging, and shows weaker performance for unusual late-stage failures (AlSayyad et al., 7 Feb 2026, Wang, 16 Mar 2026). TraceSIR reports dependence on underlying LLM quality, a moderate-size assumption for report generation, latency and token cost, and output variability (Yang et al., 28 Feb 2026). AgentPex identifies limited benchmark coverage, computational cost, extraction limitations for complex final-state prediction, rule overlap and double counting, and potential alert fatigue under continuous deployment (Sharma et al., 25 Mar 2026). TriCEGAR depends on the coverage and representativeness of traces, may overfit if refinement is unconstrained, and still treats probabilistic CEGAR refinement as future work (Koohestani et al., 30 Jan 2026).

Taken together, these systems define ATA not as a single algorithm but as a family of trace-native methods. The common invariant is that behavior is analyzed as a structured trajectory—sometimes as spans, sometimes as Thought–Action–Observation rounds, sometimes as causal graphs, predicate-tree abstractions, or Message-Action Traces—and that diagnosis, assurance, or governance is grounded in those trajectories rather than collapsed into a final scalar outcome.