Reproducible Protocols for Agent Traces and Leakage-Robust Evaluation

Establish reproducible protocols for collecting complete agent interaction traces (including prompts, tool calls, arguments, outputs, and outcomes), filtering them, and performing leakage-robust evaluation to enable comparable training and assessment across tool-using AI agents.

Background

Trace-first development is central to agent improvement, yet current practices vary widely in what data is logged, how it is sanitized, and how evaluation avoids information leakage. Without standardized, reproducible protocols, results are hard to compare and improvements may be confounded by data artifacts.

Leakage-robust evaluation is especially important for retrieval- and tool-using agents where training and test distributions can overlap. Clear protocols would support continual refinement, fair benchmarking, and stronger scientific rigor in agent system studies.

References

Establishing reproducible protocols for trace collection, filtering, and leakage-robust evaluation remains an open research problem.

— AI Agent Systems: Architectures, Applications, and Evaluation  (2601.01743 - Xu, 5 Jan 2026) in Section 7.2 (Long-Term Memory, Context Management, and Continual Improvement)

Studying the effect of such redaction on the induction quality is left to future work.

— Inducing Task Models from Computer-Use Traces  (2608.20319 - Jiang et al., 20 Aug 2026) in Limitations section

AXNav's emphasis on replay provides a precedent for treating execution as material that people can inspect. Our trace example demonstrates the availability of a sequence, while leaving open whether the current log format is sufficient for a professional to reproduce and assess the finding efficiently.

— Agentic Web Accessibility Auditing: Authoring and Evaluating Per-Criterion Worker Agents for WCAG  (2609.09379 - Mishra et al., 8 Sep 2026) in Section 7.4, subsection “Inspectable evidence for human auditors”

We reproduced 45 cross-session IPI cases from the stored prompt-injection study by Xie et al. . AgentTracer achieved more than 85\% injection-point accuracy in this exploratory experiment. However, the sample size and covered scenarios are insufficient to support a general conclusion. We therefore exclude this result from the formal evaluation and leave a broader evaluation of persistent entities and cross-session tracing to future work.

— AgentTracer: Tracing Indirect Prompt Injection Attack through Fine-Grained Intention-Execution Alignment  (2610.09935 - Yao et al., 7 Oct 2026) in Section 6, Discussion, subsection “Applicability and Extensions,” paragraph “Extension to Persistent IPI”

And branch points flatten: the search that TreeOfThoughtsStrategy performs and the retry that ReflexionStrategy performs are both linearized into the accepted path, so the rejected branches, which are exactly the negative examples a preference method wants, are not recoverable from the log. Extending the schema to record them is the most substantial open item in its design.

— AgentTune: A Unified Training Lifecycle for Tool-Using LLM Agents  (2610.13335 - Lyngkhoi et al., 9 Oct 2026) in Section 3.3.4, “Schema properties and their limits”