Direct comparison of self‑evolving agent systems on a shared benchmark

Establish a standardized, shared benchmark and evaluation protocol that enable direct comparison of self‑evolving agent design and automated multi‑agent architecture search systems on scientific computing tasks that require heterogeneous tool use, structured task decomposition, and domain‑grounded validation.

Background

Work on automated agent design has mostly been validated on coding and general reasoning tasks, whereas scientific computing introduces different constraints such as heterogeneous tools, structured workflows, and domain-grounded evaluation. Because prior systems optimize different design spaces (e.g., unconstrained agent code versus DAG-structured workflows), there is no unified basis for comparison. A shared scientific benchmark would enable rigorous, apples-to-apples evaluation across methods.

References

Direct comparison on a shared benchmark remains an open challenge.

— Mimosa Framework: Toward Evolving Multi-Agent Systems for Scientific Research  (2603.28986 - Legrand et al., 30 Mar 2026) in Subsubsection 'Automated Agent Design and Self-Evolving Systems', Section 2.2

However, a benchmark jointly evaluating the Physical RSI methods remains an open need.

— Who Improves the Robot? Toward Physical Recursive Self-Improvement  (2610.18842 - Zhou et al., 13 Oct 2026) in Section 6, subsection “Benchmarks and Evaluation Resources”