ADK Arena: Automated Evaluation for LLM Agents
- ADK Arena is a fully automated framework that benchmarks Agent Development Kits by replacing human developers with an LLM coding agent.
- It uses a controlled LLM-as-developer methodology to measure API usability via generation effort and execution performance on standardized benchmarks.
- The system employs Docker-isolated environments and a three-step validation pipeline to ensure reproducible, fair comparisons across 51 Python ADK frameworks.
ADK Arena is a fully automated evaluation framework for Agent Development Kits (ADKs), defined as SDK-level frameworks for building LLM-powered autonomous agents. Its central objective is to make framework comparison controlled, scalable, and reproducible by replacing human developers with an LLM coding agent that learns each framework’s API from documentation, writes agent code, and iteratively repairs it through a validate-and-feedback loop until tests pass. By holding the developer constant and varying only the framework, ADK Arena treats generation effort as a quantitative proxy for API usability and execution results as a controlled measure of framework effectiveness (Huang et al., 4 Jun 2026).
1. Scope and naming context
The label “Arena” is used across several unrelated research systems, including a toolkit for multi-agent reinforcement learning (Wang et al., 2019), a spectral swarm-robotics method for arena-shape identification (Cazenille et al., 2024), a tool for measuring and analysing the energy efficiency of Android apps (Anwar, 2 Oct 2025), an Arena-style benchmarking platform for visual generative models (Li et al., 2024), and a ROS 2 development and benchmarking platform for human-centric navigation (Shcherbyna1 et al., 2024). Within that broader naming landscape, ADK Arena is specifically concerned with the empirical evaluation of ADKs rather than with reinforcement learning environments, robotics arenas, energy measurement, or visual-model leaderboards.
The problem ADK Arena addresses is the lack of an ecosystem-wide framework benchmark for ADKs. The paper argues that developer surveys capture opinions rather than runtime behavior, agent benchmarks usually compare models rather than frameworks, and prior framework comparisons are small-scale and hand-written. Manual comparison scales as an problem, where is the number of frameworks and is the number of benchmarks, making broad comparison slow, expensive, and biased by evaluator familiarity (Huang et al., 4 Jun 2026).
2. LLM-as-a-Developer as evaluation methodology
The methodological core of ADK Arena is LLM-as-a-Developer. Rather than using an LLM as an evaluator, the system uses it as a developer proxy that reads the framework’s documentation, inspects source code, writes a task-solving agent, validates it, repairs it based on diagnostics, and repeats until the agent passes validation or time runs out. The paper presents this as a controlled way to ask two distinct questions: how hard it is to produce a valid agent implementation for a framework, and how well that implementation performs once deployed (Huang et al., 4 Jun 2026).
This methodology yields two principal signals. Generation effort comprises tokens, number of LLM calls, and validation failures needed to produce a working agent; the paper uses this as a proxy for API usability / complexity. Execution performance measures how well the generated agent resolves benchmark tasks after deployment. A central assumption is that the developer LLM is held constant across frameworks, so differences in generation cost, validation success, and benchmark outcomes are attributable to the framework rather than to developer variation.
The paper also draws an important distinction between validation and benchmark success. Passing validation means that the generated artifact is executable and benchmark-ready, but it does not guarantee strong downstream task performance. This prevents conflating framework learnability with framework-enabled task effectiveness.
3. System architecture and validation pipeline
Operationally, ADK Arena consists of three stages: environment setup, agent generation, and benchmarking. The framework collection comprises 51 Python ADK frameworks drawn from academic venues, GitHub topic/keyword searches with at least 1,000 stars, and curated awesome lists, while filtering out agent applications, no-code platforms, and non-Python projects. Each framework is built into its own Docker image based on a shared base image, with benchmark-specific tooling layered on top as needed. This per-framework Docker isolation is used to ensure reproducibility and prevent cross-framework contamination (Huang et al., 4 Jun 2026).
Inside each container, the LLM developer receives access to two information channels. Curated documentation is collected from the framework’s docs/ directory and exposed through explore_docs. Source code is mounted read-only and accessed through explore_source and search_api. All LLM traffic is routed through a local proxy so that every framework is evaluated against the same backend and the same telemetry infrastructure.
A key normalization decision is that every generated agent is reduced to a single Python entry point,
This decouples generation from evaluation, because every benchmark adapter can invoke the agent through the same interface regardless of the underlying ADK.
The generation loop runs under a 5-minute budget per agent. The LLM developer explores docs and source, writes code, validates the current artifact, repairs it based on diagnostics, and repeats. Validation is implemented as a three-level validation pipeline. Step 1: Static analysis performs py_compile, import verification, framework usage checks, and 40+ AST/regex patterns for common anti-patterns; it also rejects raw OpenAI/httpx/requests fallbacks, dummy imports that never actually use the framework, and incorrect async signatures. Step 2: Real LLM smoke test executes the agent with a real LLM through a token-recording proxy to ensure that model communication and framework runtime integration are valid. Step 3: Real benchmark task runs one real task from the official task set under realistic conditions, checking survival and expected core behavior rather than full task completion. When validation fails, the system matches failures against a library of known error signatures and returns a structured diagnostic hint, such as advising use of ADK_BASE_URL after a ConnectionRefusedError.
4. Benchmarks, adapters, and measurement
ADK Arena standardizes evaluation through benchmark adapters. Each adapter takes the generated agent’s solve() function, a task prompt, and a working directory, and then handles benchmark-specific setup, execution, and scoring. The four integrated benchmarks are summarized below (Huang et al., 4 Jun 2026).
| Benchmark | Role |
|---|---|
| SWE-bench Verified | Software engineering benchmark based on fixing real GitHub issues |
| -bench | Benchmark for conversational tool use and real-world task interaction |
| Terminal-Bench | Benchmark for hard command-line and terminal tasks |
| MCP-Atlas | Benchmark for tool-use competence with real MCP servers |
The evaluation spans 51 Python ADK frameworks, 4 benchmarks, and 2 generated agents per framework-benchmark pair, giving
and
The two developer models are GPT-5.4 (Codex) and Opus-4.6 (Claude Code), while both generated agents are executed on the same backbone model, GPT-5.4 Nano. This separation is designed to isolate the effect of the framework and the coding model from the execution model.
Measurement is divided into generation performance and execution performance. Generation performance includes validation pass/fail, input/output/cache tokens, and generation cost. Execution performance is the benchmark success rate after deployment. The paper defines Pass as the number of benchmarks, out of 4, for which the generated agent passes all three validation levels. It also distinguishes generation cost from resolution rate, with the former reflecting developer effort and the latter reflecting task-solving success.
A further infrastructural component is the transparent token proxy, which performs protocol translation between Anthropic-style and OpenAI-style message formats and records input tokens, output tokens, cached tokens, latency, request/response contents, tool-call sequences, and per-call breakdowns. This makes it possible to compare frameworks on the same backbone model while still capturing detailed telemetry.
5. Empirical findings
The paper reports that generation succeeds for 57% of runs: out of 408 generated agents, 232 pass all three validation levels, corresponding to
Generation cost varies 5.6 across frameworks, from about \$0.6 to \$3.4 per agent, and is interpreted as a proxy for API complexity / usability. The paper stresses, however, that cost alone does not predict success: some cheap frameworks are cheap because the model gives up quickly, while some expensive frameworks are expensive because they expose large or unclear APIs rather than because they necessarily fail (Huang et al., 4 Jun 2026).
Framework performance is highly uneven. The best single-benchmark ADK agents resolve up to 80% of tasks, while the median framework resolves only 32%. The result is explicitly summarized as “no single framework dominates.” This suggests that framework choice materially affects both buildability and execution quality, but that gains are benchmark-dependent rather than universal.
The identity of the developer model also matters. Although both generated agents run on the same backbone model, Opus-4.6-authored agents resolve roughly twice as many tasks as GPT-5.4-authored agents, with reported means of 41% vs. 22% and medians of 46% vs. 8%. The paper interprets this as evidence that code quality at generation time strongly affects downstream task success.
Benchmark-specific behavior is also differentiated. Among agents that actually run, 0-bench is the easiest, with 64% median resolution and 80% for the best system. SWE-bench and Terminal-Bench are harder, with medians of 18% and 14%, respectively, because they require extended, iterative work in which early mistakes compound over long traces. MCP-Atlas lies between these cases, with median performance around 39% among frameworks with execution data.
The paper also compares ADK-generated agents against five frontier coding agents: Claude Code, GitHub Copilot, Codex CLI, OpenHands, and OpenCode, all run with the same backbone model. The strongest average performer is GitHub Copilot at 56% average resolution across the four benchmarks; other reported averages are 48.5% for OpenHands, 45% for OpenCode, 40.5% for Claude Code, and 39.5% for Codex. The most accurate systems are also the most token-hungry: Copilot uses about 3.9M input tokens and OpenHands about 4.7M, whereas Claude Code achieves 40.5% resolution with about 0.48M tokens and Codex is the cheapest at roughly \$N$10.02 per task. ADK-generated agents are not uniformly worse: the paper gives the example that Agno reaches 68% on MCP-Atlas with far fewer tokens than Copilot, which gets 58%.
6. Information-source ablations, limitations, and implications
A central ablation study examines what information the LLM developer actually needs in order to build a valid native agent. The three conditions are Docs only, Source only, and None, where “None” means no reference material at all beyond the model’s parametric knowledge. The primary metric is genuine framework usage, meaning that the generated agent truly uses the framework’s native API rather than falling back to a thin wrapper or raw API calls. The reported rates remain within a narrow band: 40% for Source only, 33% for None, and 28% for Docs only. The paper therefore concludes that documentation, source code, and parametric knowledge are largely substitutable rather than any one being a hard bottleneck, and that Docs only is the weakest condition (Huang et al., 4 Jun 2026).
Several assumptions shape interpretation. All generated agents run on GPT-5.4 Nano, which isolates framework effects but does not evaluate framework–model interactions across many backbones. The methodology assumes that the developer LLM is a sufficiently capable proxy for a human first-time developer. The authors also note a training-data bias: popular frameworks are more likely to appear in pretraining data, which may advantage them, although the paper argues that this is not purely a confound because it reflects real-world adoption and ecosystem maturity. Finally, passing validation is described as necessary but not sufficient: many validated agents still fail tasks.
The broader implication is that ADK Arena converts framework evaluation from a manual, artisanal exercise into a benchmarkable pipeline. It provides a measurable proxy for API usability, a controlled measure of framework effectiveness, and a mechanism for studying how documentation, source code, and prior model knowledge influence agent creation. A plausible implication is that ADK selection should be based not only on ecosystem visibility or adoption, but also on how robustly a framework supports benchmark-ready, task-solving agents under controlled developer conditions.
