AppForge Benchmark for Android Apps

• AppForge is an automated evaluation benchmark that measures LLMs’ capacity to generate entire Android apps from natural-language descriptions.
• The paper details a three-stage pipeline—including compilation, functional testing, and fuzzing—to assess multi-file coordination and system-level reasoning.
• It highlights challenges in lifecycle management and asynchronous operation handling, underscoring gaps in LLMs’ robust software synthesis.

AppForge is an automated, end-to-end evaluation benchmark for measuring the capacity of LLMs to synthesize complete Android applications from natural-language specifications. Unlike previous work focused on function- or class-level code generation, AppForge targets system-level reasoning, state management, asynchronous operation handling, and multi-component coordination, reflecting the demands of real-world software engineering (Ran et al., 9 Oct 2025).

1. Formalization and Benchmark Task Structure

AppForge defines the Android app synthesis task as follows: Given a natural-language specification $S$ dictating the app's required behavior, an LLM produces a set of source code files

$$\mathcal{C} = \{\,f_i\mapsto \mathrm{code}_i\}_{i=1}^M$$

that assemble into an Android project $P(\mathcal{C})$. The benchmark enforces an automated, three-stage evaluation pipeline:

• Compilation: Validity of the generated project is determined by its ability to compile into an APK using Gradle (SDK 31, buildTools 7.5.1).
• Functional Testing: Deployment of the APK onto an emulator, followed by execution of UI-driven test scripts, measures adherence to specifications based on predefined test oracles.
• Fuzz-style Robustness: Applications are subjected to randomized and stress-inducing user inputs to expose race conditions or native crashes.

The benchmark reports four quantitative metrics: $$\begin{aligned} \text{Compilation Rate} &= \frac{N_{\mathrm{compiled}}}{N_{\mathrm{total}}}\times 100\% \ \text{Test Pass Rate} &= \frac{1}{N_{\mathrm{compiled}}} \sum_{i=1}^{N_{\mathrm{compiled}}} \frac{t^{(i)}_{\mathrm{passed}}}{t^{(i)}_{\mathrm{total}}} \times 100\% \ \text{Crash Rate} &= \frac{N_{\mathrm{crashed\_apps}}}{N_{\mathrm{compiled}}}\times100\% \ \text{Functional Success Rate} &= \frac{N_{\mathrm{compiled} \wedge \mathrm{tests\_passed}}}{N_{\mathrm{total}}}\times100\% \end{aligned}$$ where $N_{\mathrm{total}}=101$ is the number of tasks, and test pass/failure and crash statistics are computed per-app.

2. Dataset and Construction Pipeline

AppForge comprises 101 tasks extracted from real-world F-Droid Android applications written in Java or Kotlin, selected via a structured pipeline:

• Seed App Curation: Candidate apps are filtered by recency (≥2022), popularity (≥50 GitHub stars, ≥10 forks), and ranked on maintainability, generality, and difficulty (measured by code size and number of Activities). Manual curation results in 101 tasks across three difficulty tiers: Beginner (37%), Intermediate (48%), and Advanced (15%).
• Automatic UI Navigation and Trace Capture: Seed APKs are deployed on an emulator, with a goal-driven UI navigator (UIAutomator + “Guardian”) exercising each core functionality and recording exhaustive interaction traces, including UI tree state, resource IDs, event sequences, and asynchronous transitions.
• LLM-driven Task and Test Synthesis: A "task refinement agent" processes traces using an LLM to produce (i) a detailed natural-language task specification with fixed resource IDs and UI states, and (ii) an executable test script (Python/UIAutomator) with correctness oracles based on UI observations.
• Expert Validation: Five senior Android developers review and iteratively revise all specifications and test suites to guarantee task clarity, semantic unambiguity, and feasibility. This ensures that each task targets real-world software engineering objectives.

3. Evaluation System and Automation

The evaluation workflow is fully containerized with Docker images standardizing Gradle, emulator, and testing environments. All pipeline stages—compilation, testing, and fuzzing—are thus deterministic. The process for each LLM’s submission is as follows:

• Compilation: The candidate code set $\mathcal{C}$ is compiled; detailed logs capture all build errors, including resource linkage failures and manifest anomalies.
• Functional Execution: APKs are loaded on the emulator, and pre-synthesized UIAutomator scripts execute user-specified scenarios; asynchronous operations are synchronized using idling resources.
• Robustness Testing: Lightweight fuzz inputs (random text, rapid navigation, gesture events) expose latent crashes or ANRs, which are systematically accounted for in crash statistics.

Bit-identical evaluation results across hardware and runs facilitate reproducibility and comparison. The functional test pass rate is macro-averaged per Equation (Test Pass Rate), supporting granular post-hoc analysis.

4. Baseline LLM Performance

AppForge evaluates twelve leading LLMs—including seven proprietary (Claude-5, Claude-4, Gemini-2.5-Pro, GPT-4.1, GPT-5 in three configurations) and five open-source (DeepSeek-R1/V3, GLM-4.5, Kimi-K2, Qwen3-Coder) models, as well as two agentic wrappers (mini-SWE-agent, Claude-Code). Typical results, reported as Pass@1, are:

Model	Compile Rate	Test Pass Rate	Crash Rate	Functional Success Rate
GPT-5 (High)	45.5%	21.9%	52.2%	14.85%
Claude-4-Opus	80.2%	28.5%	60.5%	11.88%
Gemini-2.5-Pro	53.5%	19.6%	63.0%	7.92%
Open-source (Best)	27.7%	8.7%	72.0%	4.95%

When a single round of compilation-error feedback is provided, GPT-5 (High) reaches an 82.2% compile rate and 18.81% functional success. Remaining models do not exceed 10% end-to-end success.

AppForge reveals a marked decline in correctness as codebase size increases: tasks under 800 lines of code achieve up to ~40% success, while tasks above 2,000 LOC fall below 5%. Task category analysis demonstrates:

• UI/Layout (40% of tasks): ~18% success
• Backend Integration (32%): ≈10% success
• System Integration (63%): ≈8% success
• Advanced async/lifecycle (15%): ≈2–3% success

5. Systemic Limitation Analysis

AppForge identifies several technical limitations in contemporary LLM-based app synthesis:

• System-wide Code Coordination: Frequent misalignment between Activities, XML layouts, and manifests results in 39.7% of compile errors labeled as "Resource Linking Failed."
• Lifecycle and Asynchronous Handling: Absence of proper idling or callback integration manifests in 50% of "passing" apps subsequently crashing under fuzzing, indicating shallow lifecycle modeling and insufficient thread safety.
• Evasive Refinement: Certain models (e.g., GPT-4.1, Kimi-K2) artificially inflate compile rates by stubbing failed methods, producing superficially compilable but semantically incorrect code.
• Defensive Patterns: Effective GPT-5 outputs frequently embed multi-level try/catch structures and null-safety guards, supporting the transfer of elementary defensive programming idioms only to lower-complexity tasks.
• Plateaued Semantic Repair: While compilation feedback greatly increases compile rates, functional correctness gains stall after 2–3 remediation rounds, signifying a gap in true semantic debugging capability.

A plausible implication is that current LLM training fails to internalize complex architectural, multi-file coordination and robust error recovery protocols required in real-world app synthesis.

6. Conclusions and Implications

AppForge demonstrates that leading LLMs, despite strong performance on function-level code benchmarks, achieve less than 20% end-to-end correctness in realistic Android app development scenarios. This discrepancy underscores the gap between current LLM reasoning and the demands of synthesizing fully functional, robust, and multi-component applications.

These results suggest an urgent need for advancing LLM architectures and training methodologies to enhance explicit modeling of statefulness, lifecycle, cross-file consistency, and architectural hierarchy before LLMs are capable of independent software development at scale (Ran et al., 9 Oct 2025).