RealDevWorld: Interactive Software Benchmark

• RealDevWorld is an evaluation framework that assesses production-ready repositories using dynamic, interactive GUI testing rather than static code reviews.
• It integrates RealDevBench’s 194 open-ended tasks with AppEvalPilot’s automated, agent-driven testing to capture functional, visual, and runtime behavior.
• The framework simulates real user interactions like clicking and typing to provide detailed diagnostic feedback that enhances software quality assessment.

Searching arXiv for the specified paper and closely related benchmarks to ground the article in current literature. {"queries":[{"term":"(Bian et al., 17 Aug 2025) RealDevWorld Automated GUI Testing for Production-Ready Software Evaluation","limit":5},{"term":"AppWorld benchmarking interactive coding agents arXiv (Trivedi et al., 2024)","limit":5},{"term":"SWE-Dev Evaluating and Training Autonomous Feature-Driven Software Development arXiv (Du et al., 22 May 2025)","limit":5},{"term":"DevEval Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories arXiv (Li et al., 2024)","limit":5},{"term":"DevBench Realistic Developer-Informed Benchmark for Code Generation Models arXiv (Golnari et al., 17 Jan 2026)","limit":5}]} ArXiv search results obtained. I’ll use the returned metadata to cite the primary RealDevWorld paper and situate it relative to AppWorld, SWE-Dev, DevEval, and DevBench. RealDevWorld is an evaluation framework for assessing whether LLMs and code agents can generate production-ready repositories from scratch rather than merely isolated code artifacts. Its central claim is that repository quality cannot be established by static inspection alone: “you don’t know until you click.” In this framework, functional correctness, visual fidelity, and runtime behavior are treated as emergent properties of interactive use, so evaluation is performed by launching applications, interacting with their GUIs, and judging observed outcomes. RealDevWorld consists of two coupled components—RealDevBench, a 194-task benchmark of open-ended software engineering tasks with multimodal materials, and AppEvalPilot, an agent-as-a-judge system that executes GUI-based acceptance testing and produces structured diagnostic feedback (Bian et al., 17 Aug 2025).

1. Conceptual basis and problem setting

RealDevWorld is motivated by a specific failure mode in prior software-evaluation practice: function-level tests, unit tests, similarity metrics, pass/fail scripts, and static LLM-as-a-judge analyses do not reliably capture the properties that determine whether an application behaves like production software. According to the framework’s problem statement, these methods miss end-to-end workflows, front-end interactions, dynamic state transitions, and runtime errors, especially in repositories with graphical interfaces and heterogeneous navigation flows (Bian et al., 17 Aug 2025).

The framework therefore adopts a GUI-driven notion of software evaluation. Instead of assuming a fixed execution path, it evaluates what a user actually experiences by clicking, typing, scrolling, navigating, and observing the application under real execution. This shifts the target of assessment from isolated code correctness to holistic application behavior, including workflow completion, layout integrity, and failure handling. A plausible implication is that RealDevWorld treats evaluation as a form of adaptive acceptance testing rather than as a static proxy for implementation quality.

RealDevWorld’s scope is specifically end-to-end assessment of LLMs and agents that generate software repositories from scratch. This differs from benchmarks centered on code completion, small bug-fixing patches, or API-call planning. The framework’s emphasis is not merely whether required code exists, but whether a generated repository can be exercised as a usable application under realistic interaction.

2. System structure and formal workflow

RealDevWorld is organized around a task specification, a generated repository, and an interactive evaluation pipeline. The task specification is defined as a triplet

$Q = (D, F, M),$

where $D$ is the requirement description, $F$ is the feature list, and $M$ is the set of materials such as images, audio, datasets, and documents. A generator—human or LLM—produces a repository $R$ from $Q$. AppEvalPilot then derives test cases from $(Q, R)$, executes them through GUI interaction, collects traces, and produces scores and structured feedback (Bian et al., 17 Aug 2025).

The paper formalizes the evaluation pipeline as

$$(Q, R) \rightarrow \text{Identify } C \rightarrow \text{Execute } T \rightarrow \text{Judge } S.$$

Here, $C$ denotes the identified test cases, $T$ the collected traces, and $D$0 the judged outcomes. The traces include screens, DOM or accessibility text, and logs. This pipeline makes the repository itself part of the test-generation process, which is important because generated repositories differ widely in layout, navigation flow, and execution paths.

The two principal components are summarized below.

Component	Role	Core content
RealDevBench	Task suite	194 open-ended software engineering tasks
AppEvalPilot	Agent-as-a-judge	GUI interaction, evidence gathering, scoring

This architecture allows RealDevWorld to move beyond binary success/failure judgments. Because test cases are inferred from the requirement structure and the generated repository, the framework can report failures at the level of navigation, form handling, responsiveness, module presence, or runtime error categories rather than only declaring a repository incorrect.

3. RealDevBench: benchmark composition and task realism

RealDevBench contains 194 tasks spanning four domains: Display (50.0%), Data (14.4%), Analysis (18.6%), and Game (17.0%) (Bian et al., 17 Aug 2025). The benchmark is explicitly open-ended: repositories are constructed from scratch and are not based on templates. Its materials are multimodal, including text, images, tabular datasets, audio, and documents such as PDFs and markdown files.

The task suite is intended to mirror realistic development scenarios. Examples listed in the benchmark include a portfolio site with navigation, hero section, project cards, skill tags, social links, secure PDF download, and responsive design; a link tree with avatar, link list, category filters, theme toggle, and QR code generation; a blog traffic analysis dashboard with PV/UV trends, dwell time, sources, bounce rate, and keyword cloud; a stock dashboard with candlesticks, volume bars, technical indicators, news sentiment, correlation heatmap, and export; a personal finance tracker with monthly export; and games such as a card battle system and Turbo Rally with vehicle selection, physics engine, dynamic weather, race tracking, three tracks, and a live performance dashboard (Bian et al., 17 Aug 2025).

Its curation process combines several sources and constraints. The domain taxonomy is drawn from WebDev Arena, requirements are sampled from SRDD and real freelancer platforms including Upwork and Freelancer, and repositories were mined from GitHub with 1000+ stars, active development, documentation, and clear feature specifications. Structured feature extraction was performed with Claude-3.5-Sonnet to produce testable feature lists and evaluation criteria. Task definitions are made reproducible through the triplet structure $D$1, with materials drawn from public datasets such as Kaggle, public image libraries such as Unsplash, and hand-crafted documents (Bian et al., 17 Aug 2025).

The benchmark expects repositories in Python, JavaScript, and TypeScript, with a primary focus on web-first workflows while still accommodating desktop and OS-level interactions when needed. This suggests that RealDevBench is designed to stress interface-level variability and multimodal integration rather than only backend logic.

4. AppEvalPilot: agent-as-a-judge methodology

AppEvalPilot is the framework’s execution and judgment engine. Its architecture combines few-shot test-case generation, a Plan-Act execution loop, adaptive refinement based on interaction history and UI conditions, multimodal perception, and structured reporting. Perception is based on screenshots and icons on the visual side, and on accessibility-tree text, page content, and logs on the textual side. The action space includes Open(app), Run(code), Tell(answer), and Stop. GUI control and OS-level interaction are implemented through PyAutoGUI, while cross-application UI understanding relies on screen observation plus accessibility text (Bian et al., 17 Aug 2025).

The evaluation process has three stages. First, test-case generation uses few-shot learning to map requirements to cases and to encode domain-specific knowledge such as gameplay mechanics or data security rules; the case count is capped at approximately 15–20. Second, test-case execution turns each case into a multi-step plan, executes it through click/type/scroll/navigation operations, adapts to UI variation, and enforces at least 5 steps per case for robustness. Third, judging records actions and observations and assigns Pass, Fail, or Uncertain with supporting evidence, then aggregates outcomes at the test-case and feature levels (Bian et al., 17 Aug 2025).

The scoring scheme is conservative. At the item level,

$D$2

so Uncertain is treated as failure. Human feature quality is defined as

$D$3

where $D$4 indicates feature pass/fail. The aggregate agent quality is the pass-rate over evaluated items. The paper also presents an illustrative multi-dimensional decomposition,

$D$5

but states that AppEvalPilot does not adopt fixed weights in practice; instead, it reports stratified evidence across functional, visual, and runtime aspects (Bian et al., 17 Aug 2025).

The resulting diagnostic layer is unusually granular for a benchmark. Per-case reports include concrete evidence such as whether a PDF download triggers a file save and whether the resulting file opens without errors. Diagnostic categories include navigation/linking, form/validation, media rendering, responsiveness, runtime error, data integrity, and module presence. Step-by-step action logs and screenshots are retained to support downstream debugging.

5. Empirical performance and human alignment

On 49 RealDevBench tasks, AppEvalPilot with a Claude backbone achieved Feature-level Quality 0.73, Feature-level Alignment with human 0.85, Test-case-level Quality 0.74, Test-case-level Alignment 0.81, and Accuracy 0.92. Its average efficiency was 9.0 minutes per app at cost 0.26 under the study’s accounting. The framework also reports a correlation of 0.85 with expert human assessments and describes this as human-aligned evaluation with substantially reduced reliance on manual review (Bian et al., 17 Aug 2025).

The comparison baselines show substantial degradation under alternative evaluation strategies. Browser-Use (Claude) achieved Feature Align 0.58, Test-case Align 0.61, Accuracy 0.76, Time 13.50, and Cost 1.13; WebVoyager (Claude) achieved Feature Align 0.43, Test-case Align 0.55, Accuracy 0.74, Time 1.60, and Cost 0.10; and a Claude-3.5-Sonnet model-only baseline reached Feature Quality 0.27, Align 0.23, and Test-case Accuracy 0.68 (Bian et al., 17 Aug 2025).

The paper’s distribution analysis reports Agent Quality overlap with human: 0.96, compared with static code (0.75) and visual aesthetic (0.55). This is presented as evidence that interactive GUI evaluation aligns more closely with expert judgment than static code scoring or purely visual assessment. On 54 tasks evaluating generation systems, agent frameworks also outperform direct LLM generation on Agent Quality: MGX (BoN-3) scored 0.78, Lovable 0.74, Bolt 0.54, and OpenHands 0.50, whereas Qwen3-Coder-480B scored 0.53, Kimi-K2 0.39, and Claude-3.7-Sonnet 0.31. The reported gap is approximately 0.27 in favor of agent frameworks for functional, interactive completeness (Bian et al., 17 Aug 2025).

Ablation results identify several failure modes: missing materials such as absent audio causing Uncertain judgments, hallucinated reasoning such as incorrect numeric bounds, misaligned test cases, tasks requiring advanced memory or planning, real-time lag, and disagreements over whether a requirement implies display-only versus full functionality. The paper also reports that using AppEvalPilot’s test cases improves baseline GUI-agent performance by approximately 0.17 on average, which suggests that case-generation quality is itself a major contributor to reliable evaluation (Bian et al., 17 Aug 2025).

6. Relation to adjacent evaluation paradigms

RealDevWorld is situated against several neighboring benchmark traditions. Relative to unit tests and scripted checks, it targets dynamic, user-driven workflows that unit tests cannot reach and that brittle scripted flows fail to generalize across diverse generated UIs. Relative to repository-level benchmarks such as SWE-Bench, RepoBench, and FrontendBench, it adds adaptive, GUI-based acceptance testing aligned with ISO/IEC/IEEE 29119 acceptance level. Relative to static LLM-as-a-judge and aesthetic scoring, it argues that static code and visual assessments miss runtime issues and align poorly with human judgments (Bian et al., 17 Aug 2025).

This positioning becomes clearer when compared with other recent benchmarks. "AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents" provides a controllable multi-app environment with state-based unit tests and detects collateral damage, but it is API-driven and explicitly notes that it does not yet include UI interaction (Trivedi et al., 2024). "SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development" addresses feature addition in large existing repositories with Dockerized environments and developer-authored executable unit tests, emphasizing repository integration rather than GUI-level interaction (Du et al., 22 May 2025). "DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories" measures repository-level code generation with execution-based testing and dependency-oriented metrics, but its evaluation remains centered on repository context and unit-test outcomes rather than interactive application behavior (Li et al., 2024). "DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models" emphasizes telemetry-derived ecological validity for code completion and uses a triad of functional correctness, similarity, and LLM-judge assessments, again without GUI execution as the primary acceptance mechanism (Golnari et al., 17 Jan 2026).

Taken together, these comparisons indicate that RealDevWorld occupies a distinct niche: it evaluates production-ready software as experienced through interaction, whereas related frameworks primarily evaluate code synthesis, repository integration, or API-mediated task completion. This suggests that RealDevWorld is complementary rather than substitutive within the current benchmark ecosystem.

7. Limitations, validity threats, and extensions

The paper identifies several threats to validity. The execution environment is brittle with respect to OS shortcuts and window focus, and there is cross-platform variance. Non-determinism and timing issues are significant in race conditions and real-time games. Agent-side failures include hallucinations, insufficient planning, and ambiguous standards for visual fidelity. In addition, the visual metrics are intentionally basic: they verify presence and rendering of required modules, but may under-specify nuanced aesthetic quality or accessibility quality (Bian et al., 17 Aug 2025).

Known failure cases include audio-verification tasks without accessible audio, memory-intensive games, lag-sensitive interactions, and cases where the agent misinterprets module presence as fully functional behavior. These limitations are structurally important because they show that GUI evaluation, while more realistic than static checks, remains sensitive to instrumentation, perception quality, and temporal control.

The proposed future work focuses on richer multimodal perception, including OCR, semantic layout understanding, and event trace instrumentation; better planning and memory for complex interaction sequences; standardized scoring layers for functional, visual, and runtime dimensions with validated weights; and accessibility compliance checks (Bian et al., 17 Aug 2025). A plausible implication is that the framework is moving toward a more formalized acceptance-testing stack in which GUI interaction, perceptual grounding, and software-quality criteria become jointly measurable.

RealDevWorld’s code is available through the AppEvalPilot repository, and benchmark tasks are defined reproducibly through the $D$6 triplet structure. This reproducibility matters because the framework’s contribution is not only a dataset, but a concrete methodology for automated GUI-centric evaluation of generated software under realistic execution conditions.