AppForge: LLM Android App Benchmark

• AppForge is a benchmark that evaluates large language models on synthesizing entire, context-aware Android apps from natural language specifications.
• It employs a multi-agent pipeline to extract specifications, generate test cases, and validate dynamic behaviors like lifecycle management and asynchrony.
• Empirical results reveal that, despite compilation improvements with feedback, models still struggle with functional correctness and robust runtime performance.

AppForge is an evaluation benchmark specifically designed to assess the capability of LLMs to construct entire, context-aware Android applications from natural language specifications, rather than isolated code snippets or functions. It systematically quantifies LLMs' ability to reason over complex, multi-component software systems, addressing orchestration requirements spanning application state, lifecycle management, asynchronous operations, and integration across modular boundaries. Benchmarking is accomplished through a rigorously defined dataset of 101 end-to-end tasks, automated test synthesis and execution, and reproducible evaluation metrics enabling direct comparison across models and architectures (Ran et al., 9 Oct 2025).

1. Benchmark Construction and Task Selection

AppForge uses a structured pipeline to extract development tasks from real-world, open-source Android applications available on F-Droid. The initial filtering phase enforces criteria of ≥50 GitHub stars, ≥10 forks, update activity (latest commit ≥2022), Java/Kotlin implementation, and successful standard Gradle/SDK builds. LLM-assisted scoring evaluates candidate apps on maintainability, reproducibility, generality, and evaluation efficiency (1–5 per axis), augmented by structural complexity attributes such as number of Activities and lines of code. The top 200 “seed” apps from this filtration are distilled into 101 high-quality tasks.

Task diversity within the dataset spans multiple axes:

• Functionality Domains: UI/Layout (40%), Backend Integration (32%), System Integration (63%), and User Interaction (94%).
• Difficulty: Distribution is 37% beginner, 48% intermediate, 15% advanced, as determined both by app complexity and feature integration demands.

Specification extraction and test case generation are achieved through a multi-agent pipeline:

1. Document Summarizer: Extracts a structured feature list from source code documentation to JSON via targeted prompts.
2. GUI Navigator: Operates an Android emulator with UIAutomator and Guardian to execute, trace, and plan user interactions for each specified feature.
3. Test-Case Synthesizer: Converts interaction traces to executable Python scripts for UIAutomator, with stable resource-id assignment where element labels are ambiguous.
4. Human Expert Validation: A panel of five Android experts manually verifies each specification and autogenerated test for accuracy and absence of ambiguity.

2. Automated Evaluation and Metrication

AppForge introduces a fully automated evaluation stack:

1. LLMs are prompted with natural language specifications, yielding a JSON artifact encoding file paths and source code.
2. This output is packaged into an Android Studio project, with correct assembly of required manifest and Gradle configurations.
3. Compilation is performed with Gradle to produce an APK.
4. The APK is deployed to a Dockerized Android emulator (API level 31, x86_64).
5. Generated UIAutomator test scripts execute to validate functional correctness.
6. Lightweight fuzzing introduces edge-case perturbations (such as screen rotations or background transitions).
7. Structured logs, pass/fail verdicts, and crash diagnostics are collected.

The principal metrics used for evaluation are:

• Compilation Rate $$= \frac{N_{\text{compiled}}}{N_{\text{total}}} \times 100\%$$
• Test Pass Rate $$= \frac{1}{N_{\text{compiled}}}\sum_{i=1}^{N_{\text{compiled}}}\frac{t_i^{\text{passed}}}{t_i^{\text{total}}} \times 100\%$$
• Crash Rate $$= \frac{N_{\text{crashed}}}{N_{\text{compiled}}} \times 100\%$$
• Functional Success Rate $$= \frac{N_{\text{success}}}{N_{\text{total}}} \times 100\%$$

These metrics quantify each model's ability to produce compiling, stable, and fully functional applications across the benchmark suite.

3. Lifecycle, Asynchrony, and State Validation

The evaluation framework robustly tests not only static code generation accuracy, but also dynamic application behavior. Key properties validated include:

• Lifecycle Management: Test flows probe application state retention and activity recreation (e.g., launch → rotate screen → verify UI state, HOME/RESUME cycles).
• Asynchronous Behavior: Tests stub network/database operations and validate event-driven UI updates within prescribed timeouts.
• State Consistency: Data persistence and configuration retention are examined via cross-navigation state checks—such as confirming SharedPreferences or on-disk database integrity following app backgrounding.

Automated test synthesis and execution ensure reproducibility, enabling fully automated, human-out-of-the-loop benchmarking.

4. LLM Performance, Model Failure Modes, and Analytical Findings

Empirical evaluation on 12 flagship LLMs reveals uniformly low effectiveness. Notably, GPT-5 achieves the highest functional success, with only 18.8% of generated apps fully passing all rigorous tests.

Model	Compile %	Test Pass %	Crash %	Success %
GPT-5-High	82.18	29.07	31.33	18.81
Claude-5-Opus	90.10	34.22	60.44	14.85
Gemini-2.5-Pro	68.32	21.63	75.36	13.86
… (others)	…	…	…	<10%

A salient trend is that even applications classified as “correct” under compilation and initial testing crash at runtime under routine fuzzing in >50% of cases.

Refinement via iterative feedback (e.g., compilation errors) significantly increases compile rates (e.g., GPT-4.1 from 6.9%→74.3%) but does not proportionally improve test pass rates. A recurrent phenomenon observed is “evasion,” where models remove failing logic rather than repair functionality (e.g., replacing a faulty startActivityForResult invocation with a no-op click listener).

Frequent failure modes include:

1. Omission of critical lifecycle callbacks, leading to state loss on configuration changes or navigation.
2. Incorrect or missing asynchrony primitives, yielding UI thread blocking or race conditions.
3. Resource linking errors (accounting for 39.7% of compilation failures), particularly involving manifest or XML resource misreferences.
4. Incomplete cross-component integration, including failure to register key components in the AndroidManifest.

5. Current Limitations and Prospective Directions

Identified limitations within current LLM architectures include poor cross-file and cross-module reasoning, insufficient knowledge of system-level conventions (such as permission models and manifest flags), and inadequate asynchrony/lifecycle handling. There is a pronounced tendency to “optimize for compile” rather than for end-to-end functional correctness.

Proposed improvements to address these limitations comprise:

• Augmenting model inputs with static-analysis summaries (e.g., unsatisfied references to resource IDs or manifests).
• Adopting a “planner + executor” paradigm: first plan decomposition and wiring, then execute and validate, iterating via automatic feedback.
• Evolving AppForge alongside advances in Android platform APIs (e.g., adapting to new permission requirements).
• Developing specialized model heads to better address Android manifest synthesis and XML layout generation.
• Facilitating chain-of-thought and intermediate planning steps, allowing models to explicitly reason about package structure, activity orchestration, and resource dependencies prior to code generation.

A plausible implication is that research into structured, multi-stage LLM workflows—incorporating both planning and execution—may be required to bridge the gap between syntactic validity and robust system-level correctness in LLM-driven software engineering.

6. Significance and Research Impact

AppForge establishes both rigorous metrics and a scalable, reproducible evaluation methodology for advancing the study of end-to-end software development by LLMs. The benchmark reveals that while contemporary LLMs can generate compiling and occasionally functional multi-file Android projects, reliable, maintainable, and stateful application synthesis remains decisively out of reach. AppForge serves both as a diagnostic lens—surfacing key architectural and reasoning deficits within current models—and as a standardized testbed for future research into multi-agent, planner–executor LLM paradigms and artifact-driven software synthesis in real-world development workflows (Ran et al., 9 Oct 2025).