- The paper introduces a 153-task benchmark spanning new features, specification-driven development, and bug fixes, with applications installed and tested on an OpenHarmony emulator.
- The evaluation finds that GLM-5.2 leads with 58.39% Task Completion, while Final Build Success reaches 94.77–100%, showing that compilation does not guarantee correct app behavior.
- The benchmark shows specification-driven tasks are hardest, with no model exceeding 35% completion, highlighting the need for executable UI, state, persistence, and platform-level testing.
OpenHarmony Bench is an app-level coding benchmark that evaluates LLM-based coding agents on OpenHarmony ArkTS application development (2608.16022). Unlike function-level synthesis benchmarks or issue-resolution benchmarks such as SWE-bench, each task requires an agent to modify a buildable ArkTS project so that a requested application behavior works end to end, spanning project structure, UI state, local persistence, build configuration, and platform APIs. The delivered application is then installed and driven on an emulator to verify that the requested behavior is actually observable.
Benchmark design and task sources
The benchmark is organized around three input sources that mirror common development workflows:
- new-feature: 32 requirement-style feature implementation tasks, built primarily from Huawei official template projects, each paired with executable UI tests.
- spec-driven: 50 structured tasks derived from open-source OpenHarmony projects, decomposed into 139 Feature points (F-points), where an F-point is one executable behavior check implemented as a Hypium test.
- bug-fix: 71 tasks derived from bug-fix pull requests in real open-source OpenHarmony repositories, screened so the test fails on the buggy revision and passes on the fixed revision.
In total, the v1.0 snapshot (dated 2026.7.31) contains 153 top-level inputs and 242 F-points drawn from 42 unique repositories. The scoring unit is the top-level task; F-points serve as fine-grained diagnostic checks within spec-driven tasks (2.78 per task on average). A spec-driven task is scored all-or-nothing: it counts as complete only if the project builds and every F-point passes. This top-level rule is stricter than independently weighting F-points and aligns with the all-tests success view used in repository-level benchmarks.
Two construction properties are notable. First, executable checks are never exposed to the agent, preventing optimization against the oracle. Second, every retained check must satisfy a fail-to-pass criterion validated by at least two curators across two runs, which gives the metric an explicit floor (a no-op submission completes 0 of 153 tasks) and ceiling (reference solutions achieve 100%).
Kit/API analysis over the 153 gold patches shows the benchmark covers 29 of 39 high-frequency OpenHarmony Kits (74.4%). ArkUI dominates, appearing in 94.1% of tasks, followed by Basic Services Kit (35.9%) and Ability Kit (32.7%). However, coverage is uneven: only one Kit is "very frequent" (present in ≥50% of tasks) while fifteen are rare and ten uncovered, so most covered platform surfaces appear in relatively few tasks.
Evaluation pipeline
Each configuration is evaluated through a six-stage pipeline: workspace initialization, agent execution, diff recording, first build, bounded compile repair (at most five iterations receiving only compiler diagnostics), and dynamic behavior validation via Hypium UI tests on an emulator running DevEco 6.1 with API 23. The primary metric is Task Completion (TC): the fraction of top-level tasks whose final project builds and passes all required checks. Complementary metrics include First Build Success Rate (IBR), Final Build Success Rate (FBR), token use, and agent wall-clock time. All results use DevEco Code as a fixed agent framework with eight LLMs — GLM-5.1/5.2, Qwen3.7-Max/3.8-Max-Preview, Kimi-K2.7-Code/K3, MiniMax-M3, and DeepSeek-V4-Pro — each run three times over the full suite, reporting arithmetic means without best-of-three selection.
Main results
The leading configuration, DevEco Code with GLM-5.2, achieves a mean Task Completion of 58.39% (run range [56.21%, 60.78%]), followed by Kimi-K3 at 57.30% and Qwen3.8-Max-Preview at 55.99%. The full spread across configurations is narrow, from 48.36% to 58.39%.
| Rank |
Model |
TC mean min, max |
First Build (%) |
Final Build (%) |
Tokens/Task (M) |
Time/Task (s) |
| 1 |
GLM-5.2 |
58.39 [56.21, 60.78] |
67.10 |
98.04 |
2.57 |
1936.4 |
| 2 |
Kimi-K3 |
57.30 [53.59, 62.09] |
87.58 |
98.91 |
1.02 |
652.3 |
| 3 |
Qwen3.8-Max-Preview |
55.99 [54.90, 57.52] |
80.61 |
99.56 |
0.98 |
394.3 |
| 4 |
Qwen3.7-Max |
53.38 [51.63, 55.56] |
69.06 |
98.47 |
1.27 |
738.3 |
| 5 |
GLM-5.1 |
52.29 [50.33, 54.90] |
73.20 |
99.57 |
1.60 |
1097.8 |
| 6 |
Kimi-K2.7-Code |
52.07 [50.98, 52.94] |
62.53 |
99.35 |
1.88 |
767.5 |
| 7 |
DeepSeek-V4-Pro |
52.07 [50.33, 54.25] |
64.71 |
100.00 |
1.33 |
1048.6 |
| 8 |
MiniMax-M3 |
48.36 [45.75, 49.67] |
59.26 |
94.77 |
3.69 |
870.8 |
Three findings stand out. First, buildability is nearly saturated while behavioral correctness is not: mean Final Build Success Rate lies between 94.77% and 100.00%, yet Task Completion tops out below 60%. DeepSeek-V4-Pro attains the highest final build rate (100.00%) but ranks seventh overall, demonstrating directly that compilation success does not imply correct interaction logic, state transitions, or persistence. The implication is that app-level evaluation must include executable behavior checks; build-based or patch-similarity signals would substantially overestimate agent capability here.
Second, spec-driven tasks are hardest: no configuration exceeds 35% Task Completion on this source (best: Kimi-K3 at 34.67%), despite several configurations reaching 100% final build rate. The gap reflects both the stricter all-F-points-passing rule and the demands of precise specification following across multi-behavior scenarios. Bug-fix tasks are easiest (Kimi-K3 leads at 75.12%), consistent with their smaller reference patches (median two files, 83 lines) and single-check structure.
Third, newer model generations outperform predecessors within evaluated family pairs (e.g., GLM-5.2 > GLM-5.1, Kimi-K3 > K2.7-Code, Qwen3.8-Max-Preview > Qwen3.7-Max), though this observation is restricted to the configurations tested under the fixed DevEco Code scaffold.
Resource use varies widely and does not track effectiveness monotonically. GLM-5.2 consumes 2.57M tokens and 1936.4 seconds per task, whereas Qwen3.8-Max-Preview achieves third place with 0.98M tokens and 394.3 seconds per task — roughly five times faster than the leader. Token counts are not strictly comparable across providers due to differing tokenizer and caching conventions, and wall-clock time is treated as a within-environment diagnostic only.
Limitations
The paper concedes three limitations explicitly. All results use a single agent framework (DevEco Code), so findings characterize DevEco Code–model configurations rather than comparing independent agent scaffolds. Three runs per configuration yield descriptive min–max ranges but no inferential confidence intervals; adjacent leaderboard ranks overlap (Kimi-K3's range contains GLM-5.2's mean), so fine-grained rankings should be read cautiously. Finally, because the bug-fix source derives from historical public pull requests, pre-training exposure cannot be excluded for those tasks, and public release of tests introduces contamination risk over time; the authors accordingly frame reported numbers as a timestamped snapshot and recommend trajectory audits against reward hacking.
Conclusion
OpenHarmony Bench provides an executable, app-level evaluation setting for OpenHarmony ArkTS development, combining requirement-style features, specification-driven multi-check tasks, and repository-grounded bug fixes into a 153-task suite verified through build-and-test pipelines on emulated devices. Its central empirical result — near-perfect buildability alongside sub-60% behavioral completion, with spec-driven tasks below 35% — establishes that end-to-end application correctness, not compilation, is the binding constraint for current coding agents in this ecosystem. An open question left by the report is whether alternative agent frameworks, rather than models alone, can close the build-to-behavior gap, since the fixed-scaffold design deliberately leaves that variable unexplored.