---
title: 'ADK Arena: Automated Evaluation for LLM Agents'
url: https://www.emergentmind.com/topics/adk-arena
type: topic
---

# ADK Arena: Automated Evaluation for LLM Agents

ADK Arena is a fully automated evaluation framework for Agent Development Kits (ADKs), defined as SDK-level frameworks for building LLM-powered autonomous agents. Its central objective is to make framework comparison controlled, scalable, and reproducible by replacing human developers with an LLM coding agent that learns each framework’s API from documentation, writes agent code, and iteratively repairs it through a validate-and-feedback loop until tests pass. By holding the developer constant and varying only the framework, ADK Arena treats generation effort as a quantitative proxy for API usability and execution results as a controlled measure of framework effectiveness [2606.05548].

## 1. Scope and naming context

The label “Arena” is used across several unrelated research systems, including a toolkit for multi-agent reinforcement learning [1907.09467], a spectral swarm-robotics method for arena-shape identification [2403.17147], a tool for measuring and analysing the energy efficiency of Android apps [2510.01754], an Arena-style benchmarking platform for visual generative models [2408.14468], and a ROS 2 development and benchmarking platform for human-centric navigation [2409.12471]. Within that broader naming landscape, ADK Arena is specifically concerned with the empirical evaluation of ADKs rather than with reinforcement learning environments, robotics arenas, energy measurement, or visual-model leaderboards.

The problem ADK Arena addresses is the lack of an ecosystem-wide framework benchmark for ADKs. The paper argues that developer surveys capture opinions rather than runtime behavior, agent benchmarks usually compare models rather than frameworks, and prior framework comparisons are small-scale and hand-written. Manual comparison scales as an \(O(N \times M)\) problem, where \(N\) is the number of frameworks and \(M\) is the number of benchmarks, making broad comparison slow, expensive, and biased by evaluator familiarity [2606.05548].

## 2. LLM-as-a-Developer as evaluation methodology

The methodological core of ADK Arena is **LLM-as-a-Developer**. Rather than using an LLM as an evaluator, the system uses it as a developer proxy that reads the framework’s documentation, inspects source code, writes a task-solving agent, validates it, repairs it based on diagnostics, and repeats until the agent passes validation or time runs out. The paper presents this as a controlled way to ask two distinct questions: how hard it is to produce a valid agent implementation for a framework, and how well that implementation performs once deployed [2606.05548].

This methodology yields two principal signals. **Generation effort** comprises tokens, number of LLM calls, and validation failures needed to produce a working agent; the paper uses this as a proxy for API usability / complexity. **Execution performance** measures how well the generated agent resolves benchmark tasks after deployment. A central assumption is that the developer LLM is held constant across frameworks, so differences in generation cost, validation success, and benchmark outcomes are attributable to the framework rather than to developer variation.

The paper also draws an important distinction between validation and benchmark success. Passing validation means that the generated artifact is executable and benchmark-ready, but it does not guarantee strong downstream task performance. This prevents conflating framework learnability with framework-enabled task effectiveness.

## 3. System architecture and validation pipeline

Operationally, ADK Arena consists of three stages: **environment setup**, **agent generation**, and **benchmarking**. The framework collection comprises **51 Python ADK frameworks** drawn from academic venues, GitHub topic/keyword searches with at least 1,000 stars, and curated awesome lists, while filtering out agent applications, no-code platforms, and non-Python projects. Each framework is built into its own Docker image based on a shared base image, with benchmark-specific tooling layered on top as needed. This per-framework Docker isolation is used to ensure reproducibility and prevent cross-framework contamination [2606.05548].

Inside each container, the LLM developer receives access to two information channels. **Curated documentation** is collected from the framework’s `docs/` directory and exposed through `explore_docs`. **Source code** is mounted read-only and accessed through `explore_source` and `search_api`. All LLM traffic is routed through a local proxy so that every framework is evaluated against the same backend and the same telemetry infrastructure.

A key normalization decision is that every generated agent is reduced to a single Python entry point,
\[
\texttt{solve(prompt: str, workdir: str) -> str}.
\]
This decouples generation from evaluation, because every benchmark adapter can invoke the agent through the same interface regardless of the underlying ADK.

The generation loop runs under a **5-minute budget** per agent. The LLM developer explores docs and source, writes code, validates the current artifact, repairs it based on diagnostics, and repeats. Validation is implemented as a **three-level validation pipeline**. **Step 1: Static analysis** performs `py_compile`, import verification, framework usage checks, and 40+ AST/regex patterns for common anti-patterns; it also rejects raw OpenAI/httpx/requests fallbacks, dummy imports that never actually use the framework, and incorrect async signatures. **Step 2: Real LLM smoke test** executes the agent with a real LLM through a token-recording proxy to ensure that model communication and framework runtime integration are valid. **Step 3: Real benchmark task** runs one real task from the official task set under realistic conditions, checking survival and expected core behavior rather than full task completion. When validation fails, the system matches failures against a library of known error signatures and returns a structured diagnostic hint, such as advising use of `ADK_BASE_URL` after a `ConnectionRefusedError`.

## 4. Benchmarks, adapters, and measurement

ADK Arena standardizes evaluation through **benchmark adapters**. Each adapter takes the generated agent’s `solve()` function, a task prompt, and a working directory, and then handles benchmark-specific setup, execution, and scoring. The four integrated benchmarks are summarized below [2606.05548].

| Benchmark | Role |
|---|---|
| **SWE-bench Verified** | Software engineering benchmark based on fixing real GitHub issues |
| **\(\tau^2\)-bench** | Benchmark for conversational tool use and real-world task interaction |
| **Terminal-Bench** | Benchmark for hard command-line and terminal tasks |
| **MCP-Atlas** | Benchmark for tool-use competence with real MCP servers |

The evaluation spans **51 Python ADK frameworks**, **4 benchmarks**, and **2 generated agents per framework-benchmark pair**, giving
\[
51 \times 4 = 204 \text{ framework-benchmark pairs}
\]
and
\[
204 \times 2 = 408 \text{ generated agents}.
\]
The two developer models are **GPT-5.4 (Codex)** and **Opus-4.6 (Claude Code)**, while both generated agents are executed on the same backbone model, **GPT-5.4 Nano**. This separation is designed to isolate the effect of the framework and the coding model from the execution model.

Measurement is divided into **generation performance** and **execution performance**. Generation performance includes validation pass/fail, input/output/cache tokens, and generation cost. Execution performance is the benchmark success rate after deployment. The paper defines **Pass** as the number of benchmarks, out of 4, for which the generated agent passes all three validation levels. It also distinguishes **generation cost** from **resolution rate**, with the former reflecting developer effort and the latter reflecting task-solving success.

A further infrastructural component is the **transparent token proxy**, which performs protocol translation between Anthropic-style and OpenAI-style message formats and records input tokens, output tokens, cached tokens, latency, request/response contents, tool-call sequences, and per-call breakdowns. This makes it possible to compare frameworks on the same backbone model while still capturing detailed telemetry.

## 5. Empirical findings

The paper reports that generation succeeds for **57% of runs**: out of **408 generated agents**, **232** pass all three validation levels, corresponding to
\[
\frac{232}{408} \approx 57\%.
\]
Generation cost varies **5.6\(\times\)** across frameworks, from about **\$0.6 to \$3.4 per agent**, and is interpreted as a proxy for API complexity / usability. The paper stresses, however, that cost alone does not predict success: some cheap frameworks are cheap because the model gives up quickly, while some expensive frameworks are expensive because they expose large or unclear APIs rather than because they necessarily fail [2606.05548].

Framework performance is highly uneven. The best single-benchmark ADK agents resolve **up to 80%** of tasks, while the **median framework resolves only 32%**. The result is explicitly summarized as “no single framework dominates.” This suggests that framework choice materially affects both buildability and execution quality, but that gains are benchmark-dependent rather than universal.

The identity of the developer model also matters. Although both generated agents run on the same backbone model, **Opus-4.6-authored agents resolve roughly twice as many tasks as GPT-5.4-authored agents**, with reported means of **41% vs. 22%** and medians of **46% vs. 8%**. The paper interprets this as evidence that code quality at generation time strongly affects downstream task success.

Benchmark-specific behavior is also differentiated. Among agents that actually run, **\(\tau^2\)-bench** is the easiest, with **64% median resolution** and **80%** for the best system. **SWE-bench** and **Terminal-Bench** are harder, with medians of **18%** and **14%**, respectively, because they require extended, iterative work in which early mistakes compound over long traces. **MCP-Atlas** lies between these cases, with median performance around **39%** among frameworks with execution data.

The paper also compares ADK-generated agents against five frontier coding agents: **Claude Code**, **GitHub Copilot**, **Codex CLI**, **OpenHands**, and **OpenCode**, all run with the same backbone model. The strongest average performer is **GitHub Copilot** at **56% average resolution** across the four benchmarks; other reported averages are **48.5%** for OpenHands, **45%** for OpenCode, **40.5%** for Claude Code, and **39.5%** for Codex. The most accurate systems are also the most token-hungry: Copilot uses about **3.9M input tokens** and OpenHands about **4.7M**, whereas Claude Code achieves **40.5%** resolution with about **0.48M** tokens and Codex is the cheapest at roughly **\$0.01–\$0.02 per task**. ADK-generated agents are not uniformly worse: the paper gives the example that **Agno** reaches **68% on MCP-Atlas** with far fewer tokens than Copilot, which gets **58%**.

## 6. Information-source ablations, limitations, and implications

A central ablation study examines what information the LLM developer actually needs in order to build a valid native agent. The three conditions are **Docs only**, **Source only**, and **None**, where “None” means no reference material at all beyond the model’s parametric knowledge. The primary metric is **genuine framework usage**, meaning that the generated agent truly uses the framework’s native API rather than falling back to a thin wrapper or raw API calls. The reported rates remain within a narrow band: **40%** for Source only, **33%** for None, and **28%** for Docs only. The paper therefore concludes that documentation, source code, and parametric knowledge are largely substitutable rather than any one being a hard bottleneck, and that **Docs only is the weakest condition** [2606.05548].

Several assumptions shape interpretation. All generated agents run on **GPT-5.4 Nano**, which isolates framework effects but does not evaluate framework–model interactions across many backbones. The methodology assumes that the developer LLM is a sufficiently capable proxy for a human first-time developer. The authors also note a training-data bias: popular frameworks are more likely to appear in pretraining data, which may advantage them, although the paper argues that this is not purely a confound because it reflects real-world adoption and ecosystem maturity. Finally, passing validation is described as **necessary but not sufficient**: many validated agents still fail tasks.

The broader implication is that ADK Arena converts framework evaluation from a manual, artisanal exercise into a benchmarkable pipeline. It provides a measurable proxy for API usability, a controlled measure of framework effectiveness, and a mechanism for studying how documentation, source code, and prior model knowledge influence agent creation. A plausible implication is that ADK selection should be based not only on ecosystem visibility or adoption, but also on how robustly a framework supports benchmark-ready, task-solving agents under controlled developer conditions.

Source: https://www.emergentmind.com/topics/adk-arena