AppEval: A Unified Benchmark for LLM-Based Mobile Application Repair in ArkTS, Swift, and Kotlin

Abstract: Repository-level LLM agents are typically evaluated on projects whose tests run on the build host. It remains unclear whether their repairs survive the mobile build-install-launch-test boundary, where a missing SDK, offline device, or pre-assertion crash can be mistaken for a program failure. We present AppEval, a benchmark and native-toolchain evaluation framework for mobile application repair across HarmonyOS/ArkTS, iOS/Swift, and Android/Kotlin. Each task separates a hidden behavior test from the reference production fix and is accepted only when the same installed-app target reaches an assertion failure on the defective revision and passes after the fix; infrastructure failures remain a distinct outcome. A common schema maps this contract to each platform's build system, runtime, and test runner. The audited Android partition contains 200 accepted instrumentation tasks from 24 independently buildable repositories. On these tasks, five agents achieve Pass@1 between 22.00% and 90.50%, a 68.50-percentage-point spread under the same dynamic oracle. These results show that mobile repair performance depends strongly on the evaluated agent while demonstrating why runtime-aware acceptance is necessary for meaningful comparison. The quantitative findings in this paper are Android-specific; audited iOS and HarmonyOS results are required before drawing cross-platform generalization conclusions.

• The paper introduces AppEval, a platform-native benchmark that accepts a repair only when the same installed-app test fails on the defective state and passes after the fix.
• The Android evaluation covers 200 tasks and shows Pass@1 results from 22.00% to 90.50% across five agent/model configurations, with zero unresolved infrastructure errors.
• The paper finds near-zero agreement between systems on solved tasks, highlighting heterogeneous agent capabilities and the need for runtime-aware, stratified evaluation beyond compilation or host-based tests.

AppEval addresses a specific evaluation gap in repository-level LLM agent research: existing repair benchmarks such as SWE-bench validate patches on the build host, whereas mobile application defects become observable only after a patch is built, installed, launched, and exercised inside a platform runtime. The paper introduces AppEval as both a benchmark and an evaluation framework spanning HarmonyOS/ArkTS, iOS/Swift, and Android/Kotlin, with a strict two-state dynamic acceptance protocol that distinguishes genuine behavioral repair from infrastructure artifacts such as missing SDKs, offline devices, or pre-assertion crashes (2608.18588).

Motivation and problem statement

The central concern is that fail-to-pass evidence can be invalidated at the mobile build–install–launch–test boundary. A target test that never reaches its assertion may have failed for reasons unrelated to the defect — unavailable SDK, installation conflict, offline device, application crash, or timeout — while conversely, repairing build configuration can turn a command green without correcting user-visible behavior. The authors argue that the meaningful question is not whether an agent produces compilable code, but whether its patch changes intended behavior under a controlled runtime while unrelated infrastructure failures are diagnosed separately.

This positions AppEval relative to SWE-bench and SWE-agent, which established repository-level issue resolution with executable tests but do not model device-bound execution. It also differs from mobile interaction benchmarks such as AndroidWorld and AndroidInTheWild, which assess device-control policies rather than source repair whose effect must be observed in an installed application.

Task definition and dynamic acceptance

Each benchmark record is a tuple containing a repository, an immutable defective base commit, a natural-language issue description, known defect files, a hidden test-only patch $p_t$, a reference production fix $p_f$, one target fail-to-pass method, a regression-test set, and a fully specified execution plan fixing toolchain, build tasks, artifact paths, package identifiers, runtime destination, runner, timeouts, and expected outcomes. The evaluator constructs two states: a defective state $S_{bug} = c_b \oplus p_t$ and a fixed state $S_{fix} = c_b \oplus p_t \oplus p_f$.

A record is accepted only if the same target test builds, installs, launches, executes, and terminates in an assertion failure attributable to the tested behavior on $S_{bug}$, and passes on $S_{fix}$ under the identical frozen execution plan. Compilation errors, dependency failures, installation errors, crashes before assertion, ANRs, offline devices, and timeouts are retained as infrastructure evidence and never satisfy acceptance. A skipped or empty test selection on the fixed state is explicitly not a pass. Patch separation is enforced by hygiene rules: the test patch may change only test sources, must not rely on debug hooks or fix-only backdoors introduced by the reference fix, and both patches must apply cleanly to the base commit.

Agents observe only $(R, c_b, d)$ and return a ranked localization list followed by edits. Localization is scored with Hit@$k$ and Recall@$k$; repair is scored with Pass@1 over eligible tasks after excluding unresolved infrastructure errors, with the denominator policy fixed before evaluation. Retries rerun the exact same candidate patch and never grant the agent a new sample.

Cross-platform execution contract

A common schema maps semantic operations — build, install, launch, targeted run, evidence capture — onto each platform's native toolchain without replacing it: Gradle, APKs, ADB, and instrumentation (Espresso/UI Automator) for Android; Xcode schemes, XCTest bundles, and simulator destinations for iOS; Hvigor, HAP artifacts, HDC, and ArkXTest for HarmonyOS. The Android verifier binds every ADB command to one exact serial, validates API level and ABI, waits for boot completion and a settle interval, wakes and unlocks the device, removes stale packages, installs fresh APKs, clears data, and invokes only the recorded instrumentation class and method, preserving full logcat output. Repository locks serialize same-repository verification while distinct repositories run concurrently on separately locked devices, eliminating overlapping Git resets and ambiguous device selection as irreproducibility sources.

Dataset characteristics

The audited Android partition contains 200 accepted installed-app instrumentation tasks across 24 independently buildable repositories associated with 6 upstream organizations. Defects derive from two sources: issue- and pull-request-derived cases preserving real maintenance intent, and controlled behavior mutations covering text, visibility, navigation, persistence, layout, accessibility, and callback ordering — admitted only when a test proves the behavior on a running application. Each record maps to exactly one target method under a "one issue, one test" rule that rejects tests inspecting newly added tags or fix-only backdoors.

Patch-size statistics show a median reference fix of two changed lines (mean 6.33, max 49) against a median test patch of 28 lines (mean 32.35). The authors caution against reading this concentration of small fixes as task triviality: agents must still identify the correct production locus, respect lifecycle and resource semantics, and survive the full build-install-run path. They also flag the long tail of lifecycle, persistence, permissions, and multi-file cases as motivation for stratified reporting so aggregate scores cannot hide weaknesses.

Experimental results

Five agent/model configurations were evaluated on all 200 Android tasks with one sample per task:

Agent/model	Passed	Failed	Infra.	Pass@1
Qwen3.7-Max	181	19	0	90.50%
GPT-5.6 Sol	172	28	0	86.00%
DeepSeek V4 Pro	84	116	0	42.00%
GPT-5.6 Terra	68	132	0	34.00%
MiniMax M3	44	156	0	22.00%

All 1,000 model-task evaluations resolved with zero unresolved infrastructure errors, which the authors present as evidence that infrastructure-aware classification keeps runtime failures from silently becoming incorrect patches. The headline result is a 68.50-percentage-point Pass@1 spread under identical tasks, denominator policy, and dynamic oracle, attributable to end-to-end system differences rather than evaluation-policy differences. The micro-average across all configurations is 54.90%.

A more striking finding concerns agreement: Fleiss' $\kappa$ across the five binary outcome vectors is $p_f$0 ($p_f$1 versus chance agreement $p_f$2). Despite large marginal success-rate differences, the five systems show essentially no agreement beyond chance about which issues they solve. The authors interpret this as low overlap among solution sets rather than a standalone ranking, but it implies that aggregate Pass@1 conceals highly heterogeneous per-task competence — a result with direct consequences for anyone selecting or comparing mobile-repair agents.

The high absolute scores of the leading systems are attributed in part to the partition's many small controlled defects; the authors explicitly state this does not establish that mobile repair is solved, noting 19 unsolved tasks even for the strongest configuration.

Limitations and open questions

The paper is unusually explicit about its evidentiary boundaries. All quantitative findings are Android-specific; the iOS (160 records, 18 repositories) and HarmonyOS (180 records, 20 repositories) partitions exist structurally but their cross-platform results are synthetic placeholders that "must be replaced, not cited." Consequently, no cross-platform generalization claim is made, and the planned matched analysis over shared behavior categories (visibility, navigation, persistence, validation, callback handling) remains future work.

Localization results are similarly deferred: the Android experiment lacks a complete, separately audited localization-output matrix, so Hit@$p_f$3/Recall@$p_f$4 claims await reconciliation of predictions with gold defect-file sets. The failure analysis assigning unsuccessful patches to mutually exclusive primary causes (wrong localization, incomplete semantic fix, build regression, etc.) is likewise planned rather than reported, leaving open which component — localization, edit selection, or build-feedback use — drives the between-system gap.

Threats to validity include: passing one target test does not prove global correctness or regression-freedom; device state, cached dependencies, asynchronous callbacks, and flaky UI tests can affect outcomes despite serial binding, fresh installs, settle intervals, and locking; audited statistics are dominated by sample-project build roots and may not transfer to large commercial applications; and some issue-derived tasks may appear in model training data, motivating temporal splits and exact-patch overlap analyses. Public release additionally requires license review, provenance validation, leakage audit, and screening of device logs — gates the manuscript does not claim are satisfied.

Conclusion

AppEval contributes a unified, platform-native evaluation contract for mobile application repair whose defining requirement is that the same installed-app target must reach a behavioral assertion failure on the defective revision and pass after the fix, with infrastructure failures kept as a distinct outcome class. On the audited Android partition of 200 tasks, five agents span 22.00%–90.50% Pass@1 with near-zero inter-system agreement on solved tasks, demonstrating that agent choice materially affects end-to-end repair success and that runtime-aware acceptance is necessary for meaningful comparison. Whether these findings generalize across toolchains awaits audited iOS and HarmonyOS results, matched agent protocols, and the planned stratified localization and failure analyses.