ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developer

Abstract: The rapid proliferation of Agent Development Kits (ADKs), SDK-level frameworks for building LLM-powered autonomous agents, has outpaced any empirical understanding of how framework choice affects agent performance. We propose \textbf{LLM-as-a-Developer}, a methodology that replaces human developers with an LLM coding agent that learns each framework's API from documentation, writes agent code, and iteratively repairs it through a validate-and-feedback loop until tests pass. By holding the developer constant and varying only the framework, generation effort becomes a quantitative proxy for API usability and the resulting agents provide a controlled measure of framework effectiveness. We implement this in \textbf{ADK Arena}, a fully automated pipeline with per-framework Docker isolation, a three-level validation pipeline, and benchmark adapters for SWE-bench, $τ^2$-bench, Terminal-Bench, and MCP-Atlas. Evaluating all 51 popular Python ADK frameworks (204 agent--benchmark pairs), we find that: (1)~generation succeeds for 57\% of runs, and its cost varies 5.6$\times$ across frameworks (\$0.6 to \$3.4 per agent), a quantitative proxy for API complexity, though cost alone does not predict success; (2)~no single framework dominates: the best single-benchmark ADK agents resolve up to 80\% of tasks and can even \emph{beat} general-purpose frontier coding agents at a fraction of the cost, yet the median framework resolves only 32\%; (3)~across information-source ablations, genuine framework usage stays within a narrow 28--40\% band (highest with raw source access and still 33\% with no reference material at all), indicating that documentation, source code, and parametric knowledge are largely substitutable rather than any one being a hard bottleneck.

• The paper introduces the LLM-as-a-Developer paradigm to standardize evaluation of ADKs, reducing experimenter bias in measuring framework usability and task performance.
• The paper demonstrates that generation cost, reflected in a 5.6× token spread, serves as a proxy for API complexity and affects agent code efficacy.
• The paper highlights ecosystem vulnerabilities through inter-ADK dependency analysis, revealing risks from centralized hubs and undeclared code dependencies.

LLM-as-a-Developer for Empirical Evaluation of Agent Development Kits

Introduction and Motivation

The proliferation of Agent Development Kits (ADKs)—SDK-level Python frameworks for constructing LLM-driven autonomous agents—has rapidly expanded the design space available to developers. However, there has been an absence of rigorous, empirical methodology for comparing such frameworks in a way that isolates the framework’s impact on agent usability and task performance. The conventional approach, which requires domain experts to manually implement benchmarks for each framework, introduces substantial experimenter bias and lacks scalability. This paper addresses this methodological gap by formalizing the "LLM-as-a-Developer" paradigm, which leverages LLMs as standardized proxy developers. By automating the development, debugging, and validation cycle purely through LLM action, the paper provides a means to hold developer skill constant and measure framework effects at ecosystem scale.

Methodology: LLM-as-a-Developer

The core methodological innovation is the replacement of the human developer with a controlled LLM agent, instantiated here as OpenAI Codex (GPT-5.4) or Claude Code (Opus-4.6), that explores framework documentation and/or source code, writes the necessary agent code, and iteratively repairs it until passing rigorous validation. This controlled experiment keeps the developer’s capabilities fixed, making per-framework variance in API usability and effectiveness quantifiable.

Figure 1: LLM-as-a-Developer framework, replacing human developers with a coding LLM for repeatable, scalable ADK evaluation.

The LLM is provided with:

• Curated documentation (if available)
• Read-only source code
• Command-line utilities for exploration, code generation, validation, and diagnostic feedback This set of affordances enables the agent to replicate a realistic developer onboarding and debugging loop, minimizing the risk of framework-specific experimenter familiarity.

ADK Arena Pipeline Design

The paper introduces ADK Arena, a fully automated pipeline that operationalizes this evaluation methodology.

Figure 2: The ADK Arena pipeline outlining per-framework Docker isolation, agent generation via LLM, and multi-benchmark validation and scoring.

Key features:

• Per-Framework Isolation: Each ADK is installed into its own Docker container, running the LLM-as-a-Developer with tightly controlled access to its documentation and source.
• Multi-Level Validation: The generated agent must pass static analysis, a live LLM smoke test, and solution of a real task from four diverse, high-fidelity benchmarks (SWE-bench, $\tau^2$-bench, MCP-Atlas, Terminal-Bench).
• Reproducibility and Scale: The pipeline supports high-throughput evaluation (51 frameworks × 4 benchmarks), requiring only Dockerized ADK repositories as input.
• Unified LLM Proxy: All agent code, regardless of native SDK or API endpoint format, is executed against a fixed LLM backend (GPT-5.4 Nano) with all protocol translation and metric capture via a central proxy.

This design allows both the LLM that authors the agent and the LLM that executes it to be decoupled and controlled, with precise token- and telemetry-based instrumentation.

Empirical Results and Analysis

Generation Efficiency and API Usability

ADK Arena’s LLM-as-a-Developer succeeds in generating and validating agents for 57% of runs (232/408). There is a pronounced 5.6× spread in token/cost to generate agents, with per-agent development cost ranging from $0.6 to$3.4. Generation effort, as measured by token consumption and LLM calls, serves as a strong proxy for API complexity. However, higher generation spend does not guarantee agent efficacy, and some lightweight APIs are among the least successful due to insufficient affordance for the target benchmarks.

Task Resolution and Framework Effectiveness

Execution results reveal considerable variance in benchmark success. The strongest ADKs enable agents generated by Opus-4.6 to resolve up to 80% of tasks on individual benchmarks—on par with, or exceeding, some hand-engineered frontier agents, but at dramatically lower cost. The median framework, however, only achieves a 32% task resolution rate. These results establish that:

• No single framework dominates across all benchmarks—there is no universally superior ADK architecture.
• Agent code quality—determined almost entirely at LLM codegen time—is a critical factor. Opus-authored agents consistently outperform GPT-authored ones when evaluated on the same GPT-5.4 Nano backbone (41% vs. 22%), indicating that developer LLM selection can outweigh framework LLM selection.

Information Source Ablation

Ablation studies provide a strong, contradictory signal against naive assumptions regarding documentation:

Genuine framework usage—i.e., where agents invoke native abstractions rather than relying on thin wrappers—remains in a narrow 28–40% band across information conditions. Uniquely, access to curated documentation alone leads to the lowest rate of genuine usage (28%), while access to source only (40%), or even no reference material (33%), is more effective. This suggests that documentation worked examples tend to anchor LLMs towards synthesizing wrapper code as opposed to deeper integration, while source and parametric knowledge are largely substitutable across modern LLMs. The result is that practical onboarding bottlenecks are not strictly addressed by investing in documentation quality.

Ecosystem Dependency Structure

The inter-ADK dependency graph reveals extreme supply-chain fragility and centralization:

Figure 3: Inter-ADK dependency network showing hub-and-spoke topology and latent supply-chain risks.

LangChain and LlamaIndex form the central hubs, with 14 and 10 inbound dependencies respectively, and pydantic is a transitive dependency for three-quarters of the ADK ecosystem. A large percentage (74%) of inter-ADK dependencies are code-only and undeclared, creating runtime breaking risk that is invisible during installation. This highlights the potential for catastrophic failures and upgrade brittleness, as upstream changes can percolate through much of the ecosystem.

Practical and Theoretical Implications

This work positions LLM-as-a-Developer as a rigorously defensible strategy for ecosystem-level, controlled-variable evaluation of software frameworks in domains where manual baseline construction is intractable or inherently biased. Key implications include:

• Benchmark Development: Benchmarking SDKs and agent orchestration platforms should account not only for run-time task completion, but also for the usability cost as exposed by LLM developer effort signals.
• Framework Selection: No single ADK is optimal; empirical evaluation must match framework affordances to task distribution, and "community lock-in" on popular options (e.g., LangChain) is not inherently justified by technical superiority.
• Documentation Strategy: There is a need to revisit investments in documentation, with evidence that code-exploration affordances (e.g., indexed source, discoverable APIs) may be more effective than polished examples, at least for LLM-driven agent onboarding.
• Ecosystem Engineering: The fragility exposed by dependency analysis mandates robust ecosystem engineering practices, strict dependency declarations, and infrastructure for controlled upgrades/testing.

Further, the paradigm is transferable—future work may generalize LLM-as-a-Developer to other rapidly evolving infrastructure (prompt engineering platforms, agent compilers, system orchestration SDKs) and can sweep additional LLM developer populations for robust cross-model evaluation.

Conclusion

This paper codifies LLM-as-a-Developer as a scalable, reproducible methodology for empirical comparison of agent framework usability and effectiveness. The ADK Arena pipeline operationalizes the methodology at ecosystem-scale, underpinning a large-scale analysis of 51 ADKs over four competitive agent benchmarks. Numerical results reveal a high variance in both agent generation effort and effectiveness, with no dominant framework and notable information source substitution effects. A comprehensive analysis of ADK dependencies exposes critical ecosystem risks. The methodology provides a new standard for multi-framework AI evaluation, with direct implications for agent SDK engineering, LLM developer training, and the design of future agent ecosystems.

(2606.05548)