Reproducible Protocols for Agent Traces and Leakage-Robust Evaluation

Establish reproducible protocols for collecting complete agent interaction traces (including prompts, tool calls, arguments, outputs, and outcomes), filtering them, and performing leakage-robust evaluation to enable comparable training and assessment across tool-using AI agents.

Background

Trace-first development is central to agent improvement, yet current practices vary widely in what data is logged, how it is sanitized, and how evaluation avoids information leakage. Without standardized, reproducible protocols, results are hard to compare and improvements may be confounded by data artifacts.

Leakage-robust evaluation is especially important for retrieval- and tool-using agents where training and test distributions can overlap. Clear protocols would support continual refinement, fair benchmarking, and stronger scientific rigor in agent system studies.

References

Establishing reproducible protocols for trace collection, filtering, and leakage-robust evaluation remains an open research problem.

— AI Agent Systems: Architectures, Applications, and Evaluation  (2601.01743 - Xu, 5 Jan 2026) in Section 7.2 (Long-Term Memory, Context Management, and Continual Improvement)

Studying the effect of such redaction on the induction quality is left to future work.

— Inducing Task Models from Computer-Use Traces  (2608.20319 - Jiang et al., 20 Aug 2026) in Limitations section

AXNav's emphasis on replay provides a precedent for treating execution as material that people can inspect. Our trace example demonstrates the availability of a sequence, while leaving open whether the current log format is sufficient for a professional to reproduce and assess the finding efficiently.

— Agentic Web Accessibility Auditing: Authoring and Evaluating Per-Criterion Worker Agents for WCAG  (2609.09379 - Mishra et al., 8 Sep 2026) in Section 7.4, subsection “Inspectable evidence for human auditors”