AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility

Abstract: Agent systems are advancing quickly across domains, but their evaluation remains fragmented. Most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, create test-production mismatch, and limit fair comparison across diverse agent designs. The root problem is the lack of an open, agent-agnostic assessment interface. We advocate Agentified Agent Assessment (AAA), where evaluation is performed by judge agents and all participants interact through standardized protocols: A2A for task management and MCP for tool access. Conventional benchmarking defines two separate interfaces, one for the benchmark and one for the agent, while AAA only needs one; this yields a generic, unified framework that separates assessment logic from agent implementation and enables reproducible, interoperable, and multi-agent evaluation. We further introduce AgentBeats as a concrete realization of AAA: we identify five practical operation modes that make standardized assessment compatible with real-world constraints on openness, privacy, and reproducibility. To evaluate our design at scale, we conduct two studies: a five-month open competition that drew 298 judge agents across 12 categories together with 467 subject agents from independent participants, showing that AAA applies across a heterogeneous range of benchmarks; and a case study on coding agents that confirms agentified evaluation preserves fidelity with the public record while surfacing previously missing head-to-head results, yielding research insights about agent design. Combining a community-scale field study and a controlled coding case study, we verify that AAA delivers coverage, practicality, and fidelity across heterogeneous scenarios at scale. Together, AAA and AgentBeats offer a clear path toward open, standardized, and reproducible agent assessment.

• The paper introduces the AAA paradigm, reducing integration complexity from O(N×M) to O(N+M) via standardized agent-to-agent protocols.
• The paper details the AgentBeats platform which supports multiple operational modes to enable plug-and-play, reproducible evaluations in real-world settings.
• The paper validates the framework through a large-scale community competition, offering actionable insights on cost-performance trade-offs and tool usage.

AgentBeats: A Standardized Paradigm for Agent Benchmarking and Assessment

Motivation and Problem Formalization

The proliferation of LLM–based agent systems has resulted in increasingly diverse architectures and deployment patterns. However, agent evaluation frameworks have not kept pace. Most benchmarks rely on tightly coupled, LLM-centric harnesses that require extensive, custom integration efforts, sharply limiting the breadth and reproducibility of agent assessments. Each new agent-benchmark combination typically necessitates bespoke engineering, often resulting in a combinatorial explosion of integrations when evaluating $M$ agents across $N$ benchmarks.

Figure 1: AAA reduces integration complexity from $N \times M$ to $N + M$ via protocol standardization and complete separation between benchmark and agent components.

Traditional benchmarking workflows are further hampered by test–production mismatch and limited support for heterogeneity in agent logic, communication, and tool access. These limitations impede fair comparison, limit open participation in evaluation campaigns, and undermine reproducibility.

The Agentified Agent Assessment (AAA) Paradigm

To address these systematic deficits, the paper introduces the Agentified Agent Assessment (AAA) paradigm. In AAA, benchmarks are re-conceptualized as "judge agents" that communicate with subject (evaluatee) agents over standardized production protocols—specifically, the A2A protocol for agent-to-agent task flow and the MCP protocol for tool access.

Figure 2: Under AAA, benchmarks (now judge agents) and subject agents are separated by a standard protocol, allowing the entire evaluated agent to be swapped, not merely the LLM.

AAA collapses evaluation workflows from dual interfaces—one for the benchmark and one for the agent—into a single, generic protocol, achieving clear separation between assessment logic and agent implementation. This approach allows arbitrary A2A/MCP-compatible agents to be directly evaluated on any AAA-benchmarked task, yielding plug-and-play interoperability and strict decoupling between the assessment process and agent codebases.

The AAA workflow specifies clear roles: the delegator (which submits evaluation requests), the judge agent (encapsulating the benchmark and assessment policy), and the subject agent(s) (the SUT). Task distribution, environment preparation, execution, scoring, and result reporting are all formalized as agentic interactions.

Figure 3: AAA interaction schema—delegator, judge agent, and subject agent(s) exchange standardized messages, supporting both single- and multi-agent workflows.

Instantiating AAA: The AgentBeats Platform

The AgentBeats system provides a concrete realization of AAA while accommodating the operational requirements of openness, privacy, and reproducibility. To match the diversity of agent development and deployment scenarios, AgentBeats supports five operation modes: local, remote, hosted, proxy, and CI (continuous integration). Each mode represents a distinct mapping of agent deployment, evaluation orchestration, and result reporting.

Figure 4: AgentBeats assessment lifecycle—agent construction, registration, and execution are modularized to support varied real-world assessment workflows.

By leveraging standardized A2A/MCP protocol support, platform-managed agent instantiation, and flexible orchestration strategies, AgentBeats enables reproducible, production-compatible evaluations for both open-source and closed-source agents. The system further provides utility extensions such as agent registries, leaderboards, tool authentication, control plane integration, and seamless interoperability with third-party infrastructure.

Empirical Evaluation: Community Adoption at Scale

A five-month open community competition serves as a large-scale validation of AAA and AgentBeats. The platform fielded 298 judge agent submissions across 12 benchmarking categories and 467 distinct subject agents, with contributions spanning finance, research, coding, games, and healthcare.

Figure 5: Left—categorical distribution of judge and subject agents; right—temporal submission activity throughout the competition.

Agentified versions of existing benchmarks such as Tau2-Bench, MedAgentBench, SWE-Bench Pro, OfficeQA Pro, PersonaGym, and OSWorld were contributed, demonstrating that AAA retains broad expressivity and is compatible with both tool-use and multi-agent evaluation settings. The prevalence of Python (98–99% of submissions) as the implementation language underscores the protocol-driven, language-agnostic nature of AAA, which also welcomed agents built in TypeScript and Rust.

A notable fraction of judge agents (78.3%) implemented at least part of their assessment logic via natural language prompts, highlighting the value of semantic internalization and confirming that the paradigm supports rapid, low-friction prototyping of novel benchmarks.

Fidelity and Informative Insights: Coding Agent Case Study

To assess whether agentified benchmarks preserve canonical benchmark semantics and yield new research insights, the authors present a detailed case study benchmarking four representative coding agents—Claude Opus 4.7/Claude Code, GPT-5.4/Codex CLI, Gemini 3.1 Pro/OpenCode, and Qwen3.5-397B-A17B/MSA—across DevEval, SWE-Bench Pro, and Terminal-Bench 2.0. The experiments utilize unified, containerized environments controlled by judge agents, decoupling agents from historically idiosyncratic harnesses.

Contrary to expectations, no single agent system achieves dominance across benchmarks: GPT-5.4/Codex CLI attains the best solve rate on DevEval (94.8%), whereas Claude Opus 4.7/Claude Code leads on SWE-Bench Pro (69.1%) and Terminal-Bench 2.0 (68.5%). Per-instance costs cluster within a moderate range, confirming the practicability of large-scale, multi-agent evaluations and clarifying cost-performance trade-offs.

Harness Swapping Experiment

A harness-swapping ablation—pairing GPT-5.4 with Claude Code and Opus 4.7 with Codex—reveals consistent performance drops relative to native pairings on five out of six cells, confirming the existence of co-adaptation between model and harness. Notably, discrepancies in token consumption and solve speed emerge, revealing nontrivial interactions between agent orchestration logic and model-level inference optimization.

Fine-Grained Tool Use and Cost Analyses

Additional behavioral analysis quantifies tool-call distributions and resource impact, decomposed by task, agent, and outcome. For all benchmarks, succeeded instances consistently require fewer tool invocations and incur lower mean spend than failed instances.

Figure 6: Tool call distributions for DevEval, SWE-Bench Pro, and Terminal-Bench: normalized across command/shell, edit/write, read, and other tool categories.

Figure 7: Mean API spend per instance, stratified by outcome, for each agent-benchmark cell in the main experiment.

Figure 8: Average tool-call counts per instance for successful versus failed runs, capturing behavioral differentials across agent systems.

Error frequency analysis shows that tool-call failures remain low in leading proprietary agents but are notably higher for Gemini 3.1 Pro on certain tasks, elucidating failure mechanisms and highlighting the systemic value of standardized behavioral logging.

Implications, Limitations, and Future Directions

AgentBeats and the AAA paradigm articulate and resolve key obstacles to scalable, open, and fair agent benchmarking. By redefining benchmark–agent interactions in terms of production-facing, generic protocols, AAA enables:

• Elimination of bespoke, per-benchmark integration code and dramatic reduction in development overhead.
• Strict separation between agent code and benchmark logic, facilitating reproducibility and supporting both single- and multi-agent evaluation modalities.
• Community-driven extensibility, with rapid adaptation of existing benchmarks and effortless prototyping of novel assessment logic via prompts or modular subagents.
• Cross-benchmark, production-aligned evaluation that exposes meaningful, head-to-head performance comparisons across heterogeneous agent systems.

Theoretical implications include the possibility of benchmarking general-purpose, protocol-compatible agents in a domain-agnostic fashion, directly supporting research into agent alignment, governance, safety, and robustness. Practically, the system enables plug-and-play evaluation infrastructure suitable for both industrial deployment pipelines and public community benchmarks.

Unresolved challenges remain in the areas of standardizing prompt specification, mitigating co-adaptation risks, and extending protocol support to increasingly multimodal or real-time evaluation settings. Ongoing protocol evolution and cross-platform agent registries (e.g., CUBE integration) are natural directions for further standardization.

Conclusion

The AgentBeats platform and the Agentified Agent Assessment paradigm jointly represent a significant architectural advance in agent evaluation, providing a protocol-centric, interoperable, and community-scalable solution. The platform achieves broad adoption among practitioners and supports fidelity-preserving, insightful assessment workflows. The demonstrated reduction in integration cost, the flexibility afforded to both agent developers and benchmark designers, and the empirical support for robust, reproducible, and extensible evaluation collectively establish AgentBeats and AAA as foundational infrastructure for next-generation agent research and deployment (2606.13608).