AutoCodeArena: LLM Code Benchmark

• AutoCodeArena is an automated benchmarking framework for LLM code generation that leverages execution feedback to reflect real-world performance.
• It employs an LLM-as-a-Judge methodology combined with Elo-style ranking to provide objective, reproducible evaluations.
• The framework reduces human annotation costs and scales efficiently by automating pairwise comparisons across diverse coding environments.

AutoCodeArena is an automated benchmarking and evaluation framework for LLM code generation, designed to align closely with execution-centric human preferences while delivering scalable, reproducible, and quantitative ratings of model performance. Developed as an extension and complement to execution-enabled human evaluation platforms, AutoCodeArena deploys an “LLM-as-a-Judge” methodology and Elo-style ranking to provide a transparent, minimally human-in-the-loop standard for comparing the coding capabilities of current and emerging models (Zhuo et al., 9 Oct 2025).

1. Foundational Motivation and Problem Statement

AutoCodeArena addresses several critical requirements in LLM code evaluation. Traditional human-centric pairwise evaluation frameworks, while robust in capturing nuanced preferences, are time-intensive, costly, and challenging to scale, especially as the pace of new model releases accelerates. At the same time, benchmarks restricted to static datasets, unit-test pass rates, or synthetic tasks fail to reflect the complex interactive demands of real-world code generation and execution.

AutoCodeArena is motivated by empirical findings from BigCodeArena (Zhuo et al., 9 Oct 2025), which demonstrate that human preferences in code evaluation become significantly more reliable and reproducible when supported by code execution feedback. However, fully automating this execution-informed evaluation requires the replacement of the human rater with an automated judge that can emulate pairwise human preferences, aggregate outcomes statistically, and rank models via a standard Elo or Bradley–Terry framework.

2. Methodology and Evaluation Pipeline

AutoCodeArena’s evaluation methodology consists of five principal steps, each inspired by manual human-in-the-loop arena designs but operating entirely automatically:

1. Prompt Selection and Task Definition: The benchmark uses a curated pool of 600 representative coding prompts selected in proportion with the distributions found in a 4.7K-sample conversation corpus. The focus is on real-world task distributions as provided by the BigCodeArena dataset, capturing both diversity and practical relevance.
2. Code Generation and Execution: For each prompt, candidate models generate code, which is then executed within diverse, sandboxed environments representative of production usage. The system covers 10 programming languages and 8 execution environments.
3. Automated Pairwise Comparison via LLM-as-a-Judge: Outputs from different models, along with their execution traces and artifacts (logs, UI screenshots, etc.), are submitted to an automated judge—typically a state-of-the-art LLM (e.g., Claude-3.7-Sonnet). The judge delivers a pairwise preference for each comparison, based on criteria that closely mirror the rationale used by human annotators in BigCodeArena.
4. Aggregation and Statistical Modeling: Pairwise win rates are collected and aggregated using a Bradley–Terry model to estimate relative win probabilities:

$$p_{ij} = \frac{e^{\beta_i}}{e^{\beta_i} + e^{\beta_j}}$$

Elo ratings and confidence intervals are generated by bootstrapping over the aggregate comparisons, providing robust, population-level model strength estimates.

1. Leaderboard Production and Analysis: A standard reference model or system baseline (e.g., GPT-4.1) anchors the comparison, and all participant models receive updated Elo rankings based on their relative win rates.

This automated, closed-loop pipeline ensures high-throughput, adaptive, and reproducible LLM code evaluation that is tightly coupled to real-world execution outcomes.

3. Comparison with Human-Centric Evaluation Platforms

AutoCodeArena was directly inspired by, and empirically validated against, the BigCodeArena platform (Zhuo et al., 9 Oct 2025), which collected >14,000 model-centric conversation sessions and >4,700 high-quality human preference pairs. Key observations include:

• Correlation with Human Preferences: Elo-based rankings produced by the LLM-judged AutoCodeArena benchmark closely mirror those derived from human preferences in BigCodeArena, validating the approach’s fidelity.
• Dependency on Execution Feedback: Human judgments in isolated code review are shown to be unreliable without execution artifacts; the same principle holds for LLM judges.
• Reproducibility and Turnaround: AutoCodeArena significantly reduces annotation cost and evaluation latency. Human-in-the-loop assessments can take days to weeks for large-scale benchmarks; automated pipelines can complete equivalent evaluations within hours.

The reliance on execution-based feedback—especially including logs, screenshots, and interactive outcomes—ensures that model comparisons account not only for syntactic correctness but also for semantic nuances and UI/UX-level factors relevant to real-world deployments.

4. Technical Formulations and System Integration

AutoCodeArena’s technical architecture incorporates several mathematical and procedural innovations:

• Prompt Sampling: Tasks are selected with sampling weights $w_i$, and the pairwise sampling probability for any model pair $(i, j)$ is:

$p(i, j) = \frac{w_i w_j}{\sum_{k<\ell} w_k w_\ell}$

• Rating Aggregation: Final rankings use the Bradley–Terry model and bootstrapped intervals, ensuring robustness against sampling variance and prompt selection bias.
• Execution Environment Coverage: By spanning 10 programming languages and 8 execution environments, AutoCodeArena enables heterogeneous evaluation and tracks performance variations attributable to language or environment-specific challenges.
• Baseline Standardization: Model win rates and Elo increases are interpreted relative to a chosen, stable baseline (e.g., GPT-4.1), permitting cross-benchmark and cross-release comparison.

Integration with the broader BigCodeArena ecosystem ensures that prompt selection, sandboxing rules, and even input-output formats remain consistent, facilitating like-for-like cross-modality analysis.

5. Benchmarked Results and Observed Trends

AutoCodeArena’s most recent results reveal several salient properties of the LLM code generation landscape (Zhuo et al., 9 Oct 2025):

• Model Leadership: Proprietary LLMs, exemplified by GPT-5 and the Claude-Opus-4/Sonnet-4 series, lead in head-to-head code generation, with state-of-the-art models achieving sizable win-rate advantages.
• Execution-Dependent Ranking: Even small differences in execution quality or UI rendering can cause significant leaderboard repositioning, highlighting the necessity of execution-level evaluation.
• Alignment with Human Preferences: Rankings generated through AutoCodeArena are reported to match the preference structure captured in BigCodeArena’s human-in-the-loop analysis.
• Language and Framework Effects: Differential performance across languages or frameworks is observed as a secondary analysis output; these trends can be further explored to diagnose model limitations.

Pairwise win rates, Elo scores, and bootstrapped confidence intervals are used as the primary metrics for platform reporting.

6. Role in the Automated Code Evaluation Ecosystem

AutoCodeArena, through its automated, execution-grounded methodology, contributes a transparent, scalable, and reproducible framework for model assessment. In synergy with other developments such as CodeArena (Du et al., 3 Mar 2025) and Copilot Arena (Chi et al., 13 Feb 2025), it advances the following ecosystem functions:

• Continuous Model Tracking: AutoCodeArena enables weekly or even daily comparison among new model entrants, adapting quickly as LLMs evolve.
• Human Annotation Cost Reduction: By automating preference judgments, annotation bottlenecks are removed, democratizing large-scale comparative benchmarking.
• Benchmark Extension: The framework can be adapted for new language environments and execution modes; further, BigCodeArena data allows expansion into multi-turn interaction assessments.
• Statistical Foundations: Landmark use of Bradley–Terry modeling permits deeper statistical inferences about win rates, performance variance, and model strengths.

Its findings inform both the design of next-generation LLMs and the methodology of future code evaluation platforms.

7. Limitations and Prospects for Future Research

While AutoCodeArena’s automated pipeline closely approximates human preferences, certain limitations are acknowledged:

• LLM-as-a-Judge Limits: Some aspects of human judgment, particularly nuanced reasoning, creativity, or domain-dependent subtlety, may not be captured by model-based judges, especially as tasks stray from standard execution feedback paradigms.
• Execution Environment Challenges: Instabilities or inaccuracies in sandboxed execution—particularly for multimodal or highly interactive tasks—can introduce assessment noise.
• Scalability of Coverage: The current system, while spanning 10 languages and 8 environments, remains limited; expansion to broader frameworks or live prompt refreshing is proposed.
• Trajectory Depth: Incorporation of multi-turn, interactive testing (beyond single or pairwise runs) is an ongoing research target, and statistical bootstrapping methods remain an area of refinement.
• Model-Driven Bias: Reliance on LLM judgment may induce bias if judge models themselves differ in capability or latent alignment with the candidates.

Future work envisions live, dynamically updated benchmarks; richer interaction modeling; improved integration of program analysis for automatic test case synthesis; and the development of even more robust statistical and agentic evaluation pipelines.

---

In summary, AutoCodeArena represents the current archetype of fully automatic, execution-reflective evaluation for LLM code generation. By combining rigorous statistical modeling, execution-based validation, and seamless integration with open human-centric platforms, it establishes a transparent, scalable, and reproducible standard for progress in the field of automated program synthesis and evaluation (Zhuo et al., 9 Oct 2025).