Reproducible Protocols for Agent Traces and Leakage-Robust Evaluation

Establish reproducible protocols for collecting complete agent interaction traces (including prompts, tool calls, arguments, outputs, and outcomes), filtering them, and performing leakage-robust evaluation to enable comparable training and assessment across tool-using AI agents.

Background

Trace-first development is central to agent improvement, yet current practices vary widely in what data is logged, how it is sanitized, and how evaluation avoids information leakage. Without standardized, reproducible protocols, results are hard to compare and improvements may be confounded by data artifacts.

Leakage-robust evaluation is especially important for retrieval- and tool-using agents where training and test distributions can overlap. Clear protocols would support continual refinement, fair benchmarking, and stronger scientific rigor in agent system studies.