SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?

Abstract: Can LLM agents develop industry-level mobile applications? We introduce \textbf{SWE-Bench Mobile}, a benchmark for evaluating coding agents on realistic software engineering tasks derived from a production iOS codebase. Unlike existing benchmarks that focus on isolated problems or bug fixes, SWE-Bench Mobile captures the full complexity of industrial development: multi-modal inputs (PRDs and Figma designs), a large-scale mixed Swift/Objective-C codebase, and comprehensive test suites. We evaluate 22 agent-model configurations across four coding agents -- three commercial (Cursor, Codex, Claude Code) and one open-source (OpenCode) -- and find that even the best configurations achieve only 12\% task success rate. Our analysis reveals that (1) agent design matters as much as model capability -- the same model shows up to 6$\times$ performance gap across agents, (2) commercial agents consistently outperform open-source alternatives, and (3) simple ``Defensive Programming'' prompts outperform complex ones by 7.4\%. These findings highlight a significant gap between current agent capabilities and industrial requirements, while providing actionable insights for practitioners and researchers. We release SWE-Bench Mobile as a \textit{hosted benchmark challenge} to prevent data contamination and ensure fair evaluation. The public leaderboard and development toolkit are available at https://swebenchmobile.com.

• The paper introduces a 50-task benchmark using product requirements, Figma designs, and a 500,000-line Swift/Objective-C codebase to evaluate feature development rather than isolated bug fixes.
• The best agent configuration completed only 12% of tasks, while success fell from 18% on small changes to 2% on tasks requiring seven or more files, exposing severe cross-file reasoning limits.
• The study finds that scaffolding can matter as much as model choice and that missing feature flags caused 54% of failures, highlighting practical targets for improving production-ready coding agents.

Motivation and positioning

Existing coding benchmarks diverge substantially from professional software engineering practice. HumanEval and MBPP evaluate isolated algorithmic functions; SWE-Bench and its extensions (Multimodal, Multilingual, Pro) derive tasks from GitHub issues, which emphasize bug fixing over feature development, use text-only inputs, involve localized changes to one or two files, and concentrate on Python—a language heavily represented in LLM training corpora. SWE-Bench Mobile addresses this gap by evaluating agents on feature implementation from product specifications within a production mobile codebase. The authors argue that mobile development constitutes a distinct evaluation paradigm for three reasons: multi-modal dependency (implementation is grounded in Figma designs rather than text alone), event-driven complexity (stateful handling of asynchronous events, network changes, and OS lifecycle callbacks), and client-side constraints (framework-heavy iOS SDK environments with domain-specific APIs).

Benchmark construction

Each task is a triplet $$\mathcal{T} = (\mathcal{I}, \mathcal{O}, \mathcal{E})$$ consisting of an input context, an expected output (a unified diff patch), and an evaluation configuration. The benchmark comprises 50 tasks sourced from real product requirements at XiaoHongShu Inc., a social media platform with over 300 million monthly active users. Key statistics:

Metric	Value
Total tasks	50
Total test cases	449 (avg. 9.1 per task)
Tasks with Figma designs	35 (70%)
Tasks with reference images	46 (92%)
Avg. PRD length	450 words
Codebase	~500K LoC Swift/Objective-C (~5GB)
Avg. files modified per task	4.2

Tasks are labeled Easy/Medium/Hard by files modified (1–2 / 3–5 / 6+), lines changed (<50 / 50–150 / >150), and architectural complexity. UI Components constitute the largest category (36%), followed by data management, gesture/interaction, media, and architecture refactoring tasks.

A notable methodological choice is the diff-based evaluation strategy: pytest suites inspect the patch text directly without compiling or running the iOS application. Tests verify goal-oriented modification patterns, feature entry points, removal of blocking behavior, cross-file cohesion, and semantics-aware matching tolerant to identifier naming and refactoring style. This avoids build overhead and simulator nondeterminism at scale, but—as the authors concede—it validates structural correctness and architectural compliance only; runtime behaviors such as UI rendering, memory management, and concurrency issues cannot be detected. Two metrics are reported: Task Success Rate (all tests pass) and Test Pass Rate (individual cases passed).

To prevent data contamination given the proprietary codebase, the benchmark is hosted: the test set is private, with a sanitized development kit and public leaderboard.

Main results

Across 22 agent-model configurations spanning four agents—Cursor, Codex, Claude Code (commercial), and OpenCode (open-source)—and nine backbone models (Claude Opus/Sonnet/Haiku 4.5, GLM 4.6/4.7, GPT-5/5.1/5.2, Gemini 3 Pro), performance is uniformly low:

Configuration	Task Success (%)	Test Pass (%)	Cost ($$/task)</th> <th>Time (min)</th> </tr> </thead><tbody><tr> <td>Cursor + Opus 4.5</td> <td><strong>12.0</strong></td> <td><strong>28.1</strong></td> <td>3.50</td> <td>15.0</td> </tr> <tr> <td>Cursor + Sonnet 4.5</td> <td>12.0</td> <td>26.7</td> <td>2.00</td> <td>14.2</td> </tr> <tr> <td>Codex + GLM 4.6</td> <td>12.0</td> <td>19.6</td> <td>1.30</td> <td>13.3</td> </tr> <tr> <td>Claude Code + GLM 4.6</td> <td>10.0</td> <td>26.7</td> <td>1.30</td> <td>11.7</td> </tr> <tr> <td>OpenCode + GLM 4.6</td> <td>8.0</td> <td>17.8</td> <td>0.13</td> <td>32.5</td> </tr> <tr> <td>OpenCode + Opus 4.5</td> <td>2.0</td> <td>12.0</td> <td>9.33</td> <td>8.2</td> </tr> </tbody></table></div> <p>Three findings stand out. First, even the best configuration solves only 12$$1.30/task, less than half the cost of Cursor + Opus 4.5, suggesting that mid-tier models paired with strong scaffolding can match frontier-model performance on these tasks. Robustness checks show moderate variance (Claude Code + Opus 4.5: 6%, 8%, 6% across three runs, σ = 1.15%; Codex + Opus 4.5 perfectly stable at 4%). Complexity scaling Performance degrades sharply with task complexity. Success falls from 18% for tasks requiring 1–2 file modifications to 2% for tasks requiring 7+ files, and from 20% for patches under 50 lines to 3% for patches exceeding 200 lines. By difficulty tier, Easy tasks achieve 18.5% pass rate versus 5.8% for Hard. This indicates that cross-file reasoning and coordinated multi-module changes—not code generation per se—are the binding constraint. Qualitative case studies reinforce this: in one failure example, an agent correctly implemented server-driven configuration for an emoji limit change but modified only one file when the PRD required coverage "across all emoji-saving scenarios," failing scope analysis despite correct core logic. Prompt engineering ablation A systematic ablation of 12 prompt strategies using Claude Code + GLM 4.6 produced two counterintuitive findings. First, a simple "Defensive Programming" prompt emphasizing edge-case handling achieved the highest Test Pass Rate (26.7% vs. 19.3% baseline, a 7.4-point improvement) at unchanged Task Success Rate (10%)—it improves partial-completion quality rather than full-task completion. Second, complexity hurts: elaborate multi-step prompts ("Comprehensive," "Structured Checklist," "Context Rich") reduced Task Success from 10% to 4%. Quality-focused prompts outperformed workflow-focused ones, and test-driven prompting underperformed (6%). One caveat: the ablation covers a single agent-model configuration, so transferability across models is unverified. Error analysis Failure categorization reveals that 54% of failures stem from missing feature flags—the production practice of gating features behind toggles for gradual rollout and A/B testing, which agents do not spontaneously adopt. Missing data models account for 22%, incomplete file coverage and missing UI components for 11–15% each, missing required methods for 9%, and incomplete multi-file implementation for 4–7% overall (rising to 15–20% among Hard-task failures). The feature-flag dominance is arguably the paper's most actionable finding: it identifies a specific, teachable gap between generic code generation ability and knowledge of deployment practices, rather than a diffuse capability deficit. Limitations The authors identify several constraints. The benchmark draws from a single iOS codebase, limiting generalization to Android/Kotlin or cross-platform frameworks. Diff-based evaluation cannot detect runtime-only defects. The prompt study covers one configuration. With 50 tasks, the benchmark is a snapshot that omits feature types such as real-time communication, payments, and accessibility. Additionally, the patch-level tests verify structural intent via pattern matching, which—despite tolerance mechanisms—may admit implementations that would fail at runtime or reject valid alternative architectures. Conclusion SWE-Bench Mobile demonstrates that current coding agents fall well short of industry-level mobile development: a 12% ceiling on task success, steep degradation on multi-file work, and systematic ignorance of production conventions like feature flags. Its contributions are a contamination-resistant hosted benchmark combining PRDs, Figma designs, and a large proprietary Swift/Objective-C codebase, plus evidence that scaffolding quality rivals model capability as a performance determinant. The open questions it leaves include whether simulator-based runtime evaluation would materially change rankings, how agents perform on Android ecosystems, and whether targeted training on production practices can close the feature-flag gap. Paper to Video (Beta) No one has generated a video about this paper yet. Whiteboard No one has generated a whiteboard explanation for this paper yet. Paper Prompts Sign up for free to create and run prompts on this paper. Top Community Prompts Explain it Like I'm 14  off on Knowledge Gaps  off on Practical Applications  off on Glossary  off on Conceptual Simplification  off on Open Problems We haven't generated a list of open problems mentioned in this paper yet. Continue Learning How does SWE-Bench Mobile’s diff-based evaluation compare with simulator- or device-based testing for measuring real iOS application quality?  Why do feature flags account for most agent failures, and how could training or tool support improve this weakness?  What agent scaffolding techniques best improve cross-file reasoning and implementation completeness in large mobile codebases?  Would coding agents achieve similar results on Android, Kotlin, or cross-platform frameworks such as Flutter and React Native?  Find recent papers about AI agents for mobile software engineering benchmarks.  Content Paper Summary Paper to Video Whiteboard Paper Prompts Open Problems Continue Learning Stay informed about trending AI papers: About Whiteboards Videos Labs Email Digest Chrome Extension RSS Terms Privacy Contact Twitter Discord YouTube