RealDevBench: Benchmark for Repository Code Quality

• RealDevBench is a benchmark for repository-level evaluation that tests full application generation through interactive runtime and GUI-based assessments.
• It comprises 194 open-ended tasks across domains like display, data, analysis, and game, combining natural language requirements with multimodal materials.
• The benchmark emphasizes dynamic, acceptance-level evaluation to capture integrated software quality beyond static code or unit test metrics.

RealDevBench is a benchmark for evaluating whether LLMs and coding agents can generate production-ready software repositories whose correctness depends on graphical interfaces, interactive logic, and runtime behavior rather than isolated unit-test success. It is the benchmark component of the RealDevWorld framework and consists of 194 open-ended software engineering tasks spanning multiple domains, with task definitions organized around natural-language requirements, structured feature lists, and supplementary multimodal materials. Its central premise is that repository-level software quality cannot be established reliably through static checks alone: applications must be built, deployed, and exercised through realistic GUI interactions to determine whether they satisfy user-facing functionality (Bian et al., 17 Aug 2025).

1. Definition and conceptual scope

RealDevBench is designed for repository-from-scratch evaluation. Systems are expected to produce full repositories rather than fill templates or patch existing code, and the target artifacts are applications with GUIs, runtime workflows, and, in a subset of cases, multimodal inputs such as images, audio, and tabular data. Within RealDevWorld, RealDevBench specifies the development tasks, while AppEvalPilot supplies the interactive evaluation mechanism that launches the resulting application, generates feature-grounded test cases, and judges execution outcomes (Bian et al., 17 Aug 2025).

The benchmark is positioned against two limitations in prior evaluation practice. First, function-level benchmarks such as BigCodeBench, LiveCodeBench, and NaturalCodeBench focus on isolated functions with unit tests and therefore do not determine whether a complete application works from a user’s perspective. Second, many repository-level benchmarks emphasize maintenance, retrieval, or template completion and rarely test GUI-level interactions, visual layout, or runtime workflows. RealDevBench instead targets development tasks in Python, JavaScript, and TypeScript environments, with explicit emphasis on open-ended application construction and dynamic acceptance-level behavior (Bian et al., 17 Aug 2025).

A recurrent misconception is that RealDevBench is merely a dataset of repository specifications. In practice, it is inseparable from an execution-centered evaluation philosophy: the benchmark is intended to be paired with interactive GUI testing, because many failure modes only emerge when an application is actively used. A later diagnostic study formalized this point by treating interactive software correctness as a graph-level reachability property over latent UI state-transition graphs, while observing that an evaluator typically sees only a single trajectory and can therefore confuse evaluator-side execution error with genuine software defect (Hong et al., 17 May 2026).

2. Domain coverage and task composition

RealDevBench comprises 194 tasks across four domains chosen to reflect web-centric and data-intensive applications (Bian et al., 17 Aug 2025).

Domain	Share of tasks	Typical applications
Display	50.0%	Portfolio sites, link-tree pages, galleries, documentation sites
Data	14.4%	Finance dashboards, stock data viewers
Analysis	18.6%	Traffic analytics, product review analysis dashboards
Game	17.0%	Card games, memory games, racing-style games

The Display domain centers on presentation-heavy web applications. Representative tasks include a professional portfolio website with fixed header navigation, smooth scroll, interactive project cards, dynamic tag-cloud skills visualization, social links with animations, and responsive layout, as well as a social link tree with category filtering, light/dark theme toggle, and QR-code generation. The Analysis domain focuses on transforming raw data into dashboards and insights, with tasks such as blog traffic analysis from CSV and product review analysis with rating distributions, keyword extraction, monthly trends, and sentiment breakdowns. The Data domain emphasizes information dashboards and visual analytics, including personal finance tracking and stock dashboards with candlestick charts, technical indicators, sentiment visualization, and export functions. The Game domain covers interactive entertainment, from simple card battle games to more complex racing applications such as TurboRally, which requires vehicle selection, physics, dynamic weather, checkpoint tracking, and real-time race dashboards (Bian et al., 17 Aug 2025).

Task sourcing was described as drawing requirements from SRDD and real freelance platforms such as Upwork and Freelancer, while features were extracted from high-quality GitHub repositories with more than 1000 stars via LLM-assisted summarization. This combination gives the benchmark a mixed provenance: the task statements are open-ended and realistic, but the feature lists are structured enough to support fine-grained evaluation hooks (Bian et al., 17 Aug 2025).

3. Task representation and multimodal specification

Each RealDevBench task is structured as a triplet

$Q = (D, F, M),$

where $D$ is a natural-language requirement description, $F$ is a structured feature list, and $M$ is a collection of supplementary materials. The requirement description specifies the application’s purpose and narrative; the feature list enumerates functional goals in a user-story-like format; and the supplementary materials provide the external assets the resulting repository is expected to ingest or expose at runtime (Bian et al., 17 Aug 2025).

The benchmark’s open-endedness derives from what is not supplied. No partially completed repositories or templates are provided, and multiple implementations can be acceptable so long as they satisfy the feature list and integrate the materials meaningfully. This means models must make architectural and interaction-design decisions, including routing, state management, layout, data ingestion, and repository organization. The resulting evaluation target is therefore broader than exact code synthesis or patch generation (Bian et al., 17 Aug 2025).

The materials $M$ are multimodal. They include images, CSV datasets, PDFs, markdown files, and, at the benchmark-definition level, audio. The intent is not merely to decorate prompts with extra modalities, but to require their use in application behavior. A portfolio task may need to serve a resume PDF and render a profile photo; an analysis task may need to read a CSV and generate charts; a link-tree task may need to parse a markdown file of URLs; some qualitative examples refer to audio-based functionality such as a language spelling application. This feature-grounded, multimodal construction differentiates RealDevBench from benchmarks that reduce application development to static text-to-code mapping (Bian et al., 17 Aug 2025).

4. Interactive evaluation with AppEvalPilot

RealDevBench is evaluated through an end-to-end pipeline in which a generator $\mathcal{A}$ receives $Q$ and produces a repository $R$, after which the framework deploys $R$, executes interactive tests, and aggregates results: $$(Q, R) \xrightarrow{\text{Identify}} C \xrightarrow{\text{Execute}} T \xrightarrow{\text{Judge}} S.$$ Here $D$0 denotes test cases derived from the requirements and feature list, $D$1 denotes the resulting execution traces, and $D$2 is the final software quality score (Bian et al., 17 Aug 2025).

AppEvalPilot is the agent-as-a-judge component responsible for this process. It generates domain-appropriate test cases using few-shot prompting and domain-specific knowledge, with the prompt framing it as a professional test engineer. The generation stage is constrained to cover all user requirements and typically produces 15–20 main test cases per task. Test execution uses a hierarchical action space with four primitives—Open(app), Run(code), Tell(answer), and Stop—and consumes both screenshots and accessibility-tree XML. The execution policy follows a Plan–Act paradigm, and the framework requires at least five steps per test case to reduce shallow or hallucinated judgments (Bian et al., 17 Aug 2025).

Judgment is performed at both test-case and feature levels. At the item level, the binary scoring rule is

$D$3

For human feature-level quality, the benchmark uses

$D$4

where $D$5 is the human label for feature $D$6. Alignment between agent and human scores is measured with Pearson correlation (Bian et al., 17 Aug 2025).

The reported human-ground-truth protocol used 49 tasks, with 12 experts in total: 9 QA specialists and 3 senior experts. At the feature level, human quality averaged 0.74 and AppEvalPilot’s quality estimate averaged 0.73, with alignment 0.85. At the test-case level, AppEvalPilot reached alignment 0.81 and accuracy 0.92 against human judgments. The reported time per application was 9.0 minutes, with relative cost 0.26; the framework was also reported to reduce cost by 77% relative to Browser-Use in comparable settings (Bian et al., 17 Aug 2025).

5. Empirical results and observed difficulty

RealDevBench is difficult for contemporary frontier systems. In an experiment on 54 test tasks, raw LLMs evaluated by Agent Quality were markedly weaker than higher-level agents and platforms. Among the listed LLMs, Claude-3.7 reached 0.31, Gemini-2.5 Pro 0.29, Kimi-K2 0.39, DeepSeek-V3 0.29, Qwen3-Coder-480B 0.53, and Qwen3-235B 0.33. Among agentic systems, OpenHands reached 0.50, Lovable 0.74, Bolt 0.54, MGX 0.60, and MGX BoN-3 0.78. The reported interpretation was that RealDevBench remains hard even for frontier models, with raw LLMs rarely exceeding 0.4–0.5 Agent Quality and agent frameworks substantially outperforming them on average (Bian et al., 17 Aug 2025).

A second important result is the mismatch between static proxies and runtime functional quality. The benchmark reports Static Code Quality, Static Visual Quality, and Agent Quality, and notes that code or screenshot quality can look relatively high even when interactive functionality is deficient. This is a central methodological claim of RealDevBench: static inspection misrepresents application correctness, especially for systems with stateful interactions, form workflows, or game mechanics (Bian et al., 17 Aug 2025).

The benchmark’s empirical profile therefore favors agentic systems that can plan, deploy, and interact, rather than models that primarily optimize for code appearance or local syntactic plausibility. A plausible implication is that RealDevBench rewards integration competence—routing, state transitions, data flow, and GUI affordances—more strongly than benchmarks dominated by function-level semantics. That implication is consistent with the benchmark’s design, but the benchmark itself reports the phenomenon through the gap between Agent Quality and static metrics rather than through a separate ablation on architectural choices (Bian et al., 17 Aug 2025).

6. Reliability, diagnostic evaluation, and benchmark context

Subsequent work has shown that evaluating RealDevBench itself is nontrivial. DiagEval models interactive software as a latent state-transition graph

$D$7

with success defined as reachability of a target state, while the evaluator observes only a single trajectory

$D$8

On this view, a failed rollout rules out only one realized path, so negative judgments are intrinsically ambiguous between evaluator-side failure and genuine software defect. Applied to RealDevBench, DiagEval reported that, on its full evaluation sets, accuracy improved from 65.0% to 81.6%, and that on false-negative cases it recovered 47.0% of failures after one diagnostic round and 62.1% after two. In that study’s RealDevBench usage, 662 cases had valid paired human scores; after removing deployment failures, 41 formed a pilot set and 429 formed the main test set. The human annotation protocol used two independent test engineers with adjudication by an algorithm reviewer, yielding observed agreement $D$9 and Cohen’s $F$0 (Hong et al., 17 May 2026).

These findings sharpen a second misconception: a low score on RealDevBench does not necessarily mean poor software generation alone; it may partly reflect evaluator limitations in exploring the UI. DiagEval’s results suggest that reliable RealDevBench measurement requires not only stronger generators but also stronger evaluators and post-failure diagnosis (Hong et al., 17 May 2026).

Within the broader benchmark ecosystem, RealDevBench occupies the branch concerned with open-ended repository construction and runtime interaction. DevEval evaluates code generation in practical software projects and contains 2,690 samples from 119 projects across 10 domains (Li et al., 2024). ProjDevBench evaluates end-to-end project development from requirements on 20 problems across 8 categories, with output repositories compiled and tested on an Online Judge (Lu et al., 2 Feb 2026). DesignBench targets front-end engineering with 900 webpage samples across generation, edit, and repair, covering React, Vue, Angular, and vanilla HTML/CSS (Xiao et al., 6 Jun 2025). RealClawBench converts 76,155 real OpenClaw sessions into 281 executable tasks using reconstructed execution environments and deterministic verifiable scorers (Lv et al., 2 Jun 2026). Relative to these, RealDevBench is distinctive in combining open-ended repository development, multimodal task materials, and GUI-agent evaluation aimed at production-style software behavior (Bian et al., 17 Aug 2025).