KG-Harmony-Bench for HarmonyOS GUI Agents

• KG-Harmony-Bench is a benchmark that uses UTG-derived knowledge graphs to evaluate retrieval-augmented GUI agents on HarmonyOS apps.
• It comprises 150 multi-step tasks across 15 popular apps, focusing on complex user flows and cross-platform decision-making.
• Evaluation metrics such as success rate, decision accuracy, and average steps demonstrate the benefits of structured retrieval in GUI automation.

Searching arXiv for the cited KG-RAG paper and closely related GUI-agent benchmark work. arxiv_search(query="KG-RAG Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation", max_results=5, sort_by="relevance") arxiv_search(query="KG-RAG Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation", max_results=5, sort_by="relevance") is not available in this environment. I’ll proceed by relying on the provided arXiv metadata and the cited arXiv id (Guan et al., 30 Aug 2025). KG-Harmony-Bench is a GUI-agent benchmark introduced in the KG-RAG paper to evaluate retrieval-augmented decision-making on HarmonyOS applications in the Chinese mobile ecosystem (Guan et al., 30 Aug 2025). It was released alongside KG-Android-Bench to address a gap in prior evaluation settings that focus on English-language Android apps and do not test cross-platform robustness. The benchmark comprises 150 realistic, multi-step tasks over 15 mainstream HarmonyOS apps, with each app paired to a UI Transition Graph-derived knowledge graph and each task paired to a high-level user intent and a ground-truth action sequence. In the KG-RAG framework, this design supports systematic study of how app-specific structured retrieval affects agent behavior under heterogeneous UI styles and operating-system-specific interaction patterns.

1. Motivation and problem setting

KG-Harmony-Bench was motivated by the observation that LLM-based GUI agents struggle with complex mobile tasks when app-specific knowledge is limited, and that existing evaluation collections do not adequately probe cross-platform generalization (Guan et al., 30 Aug 2025). The benchmark therefore targets HarmonyOS, which the paper describes as rapidly gaining share in China and as differing from standard Android in UI paradigms and system frameworks in subtle but critical ways.

The benchmark scope is explicitly cross-platform and ecosystem-specific. It focuses on 15 mainstream HarmonyOS apps, including Amap, Ctrip, and Zhixing Train Tickets, and emphasizes deeply nested user flows rather than shallow command execution. By supplying a UTG-derived knowledge graph for each app and aligning it with user intents and oracle action sequences, the benchmark is structured to test whether retrieval-augmented GUI agents can exploit structured prior knowledge rather than relying only on immediate screen interpretation.

A common simplification is to treat HarmonyOS evaluation as a near-duplicate of Android evaluation. KG-Harmony-Bench is designed against that interpretation: its tasks are expressly chosen to exercise cross-platform idiosyncrasies, including system-level route widgets and floating-window dialogs unique to HarmonyOS. This suggests that the benchmark is intended not merely as a dataset extension, but as a stress test for OS-transfer robustness.

2. Corpus composition and annotation design

KG-Harmony-Bench contains 150 end-to-end user instructions distributed as 10 tasks per app across 15 popular HarmonyOS applications (Guan et al., 30 Aug 2025). The apps cover mapping, travel, e-commerce, social, and related domains. The benchmark structure includes 15 task categories aligned with app domains, such as “Navigation” for Amap, “Travel Booking” for Ctrip, and “Ticket Inquiry” for Zhixing.

Task complexity is heterogeneous by construction. For each app, the 10 tasks range from 2-step simple actions, such as opening “Nearby Gas Stations,” to 5–7-step nested workflows, such as searching for and purchasing a train ticket. The average task length reported for the benchmark is 4.8 steps.

The annotation format is defined at the trajectory level. For each task, the benchmark provides the initial UI state as a screen ID in the UTG, a natural-language goal description in Chinese, the oracle action sequence as a list of $(\text{widget-ID}, \text{action-type})$ pairs, and end-state verification criteria such as the presence of a train ticket confirmation page. The supported action types are Tap, swipe, long press, and text input, matching the four fundamental action primitives used in KG-RAG.

This annotation design makes the benchmark suitable for both whole-task evaluation and per-action analysis. Because the oracle is expressed as a typed action sequence anchored in the UTG, the benchmark can support fine-grained trajectory comparison rather than only terminal success checking.

3. UTG extraction and knowledge-graph integration

Each HarmonyOS app in KG-Harmony-Bench is processed with the xTester tool through an 8-hour per-app automated exploration procedure for Chinese-language apps (Guan et al., 30 Aug 2025). The resulting UI Transition Graph is stored in hierarchical JSON. The paper reports that each app yields on the order of 100–300 distinct screens and 1,000–2,000 widget-action edges.

In the KG-RAG workflow, these UTGs are not used only as raw navigation graphs. Each app’s UTG is preprocessed by an offline intent-guided LLM search to produce intent–trajectory pairs, and those pairs become entries in a vector database indexed by text embeddings of the intent together with a concise summary of the action path. The benchmark therefore couples symbolic navigation structure with embedding-based retrieval.

The integration points described for loading KG-Harmony-Bench into a retrieval-augmented pipeline are operationally specific. The vector database can be FAISS or another ANN backend with 768-dim embeddings, exemplified by doubao-embedding-text. Retrieval uses cosine similarity over embedded intent summaries. The illustrated online loop encodes the task goal description as a query vector, retrieves top-$K$ similar intents with top_k=3, fuses the retrieved trajectories as context for the LLM, and executes the generated plan.

This organization is important methodologically because it defines KG-Harmony-Bench as more than a passive test set. The benchmark is built to be consumed by a KG-RAG-style architecture in which UTG-derived knowledge is compiled offline and queried at runtime.

4. Evaluation metrics and protocol

KG-Harmony-Bench adopts the same three core metrics as KG-Android-Bench and prior GUI benchmarks: Success Rate (SR), Decision Accuracy (DA), and Average Steps (AS) (Guan et al., 30 Aug 2025). SR measures the percentage of tasks completed successfully. DA is defined as

$$DA = \frac{\sum_{i=1}^{N}\sum_{j=1}^{L_i}\mathbf{1}[a_{i,j}=a^{*}_{i,j}]}{\sum_{i=1}^{N}L_i}\times100\%,$$

where $N$ is the total number of tasks, $L_i$ is the action-sequence length for task $i$, $a_{i,j}$ is the agent’s $j$th action, and $a^*_{i,j}$ is the ground-truth action. AS is defined as

$AS = \frac{1}{N}\sum_{i=1}^{N}L_i,$

with lower values indicating greater step efficiency.

The evaluation protocol is fixed and single-pass. There is no train/validation split; all 150 tasks are held out as a single test set. For each agent, one trial is launched per task. A per-task timeout of 120 seconds or a maximum of 20 UI actions is enforced, and tasks unresolved within these limits are marked as failures. Reported SR, DA, and AS are averaged across all 150 tasks.

The benchmark also includes an ablation on UTG extraction time at 1 h, 2 h, 4 h, and 8 h per app for a representative app, Weibo, showing that performance saturates at approximately 4 h. This suggests a practical deployment trade-off: longer exploration can improve graph coverage up to a point, after which additional extraction time yields diminishing returns.

5. Reported performance and baseline comparisons

The paper reports results for MobileAgent-v2 and UI-TARS, each evaluated both in its baseline form and with KG-RAG augmentation on KG-Harmony-Bench (Guan et al., 30 Aug 2025). The reported gains are substantial on both success and action-level correctness, with modest reductions in average steps.

Agent setting	SR / DA / AS	Brief note
MobileAgent-v2	41.9% / 64.2% / 4.6	Baseline
MobileAgent-v2 + KG-RAG	55.4% / 77.4% / 4.2	Retrieval-augmented
UI-TARS	61.4% / 71.0% / 4.4	Baseline
UI-TARS + KG-RAG	70.9% / 85.5% / 4.0	Retrieval-augmented

These numbers indicate that KG-RAG improves both trajectory correctness and task completion relative to the corresponding non-retrieval baselines. The strongest reported configuration on KG-Harmony-Bench is UI-TARS + KG-RAG at 70.9% SR, 85.5% DA, and 4.0 AS.

The paper also situates these results relative to KG-Android-Bench. Android-Bench has 30 apps and 300 tasks, whereas Harmony-Bench has 15 apps and 150 tasks. Android-Bench spans 10 categories, while Harmony-Bench spans 15 and emphasizes travel and mapping applications such as Amap, Ctrip, and Zhixing. Although both benchmarks share the same multi-step UTG-based annotation style, the HarmonyOS benchmark yields lower absolute SR and DA for KG-RAG, approximately 55% versus 75% SR, which the paper interprets as evidence of increased cross-platform difficulty.

6. Representative tasks and research significance

The benchmark includes concrete, fully specified end-to-end tasks that illustrate its emphasis on deeply nested application behavior (Guan et al., 30 Aug 2025). In Amap, one representative task is “从当前位置导航步行到故宫” (“Navigate on foot from here to the Forbidden City”). The action sequence begins from the Amap home screen, taps the search bar, inputs “故宫,” selects the first suggestion, taps “路线,” switches to “步行,” and taps “开始导航.” The expected outcome is an on-screen route map with turn-by-turn walking instructions and a visible “停止” button.

A second representative task is drawn from Ctrip: “预定明天从北京到上海的高铁票” (“Book a high-speed train ticket for tomorrow from Beijing to Shanghai”). Starting from the Ctrip home screen, the oracle sequence taps “火车票,” fills the departure and destination fields with “北京” and “上海,” selects tomorrow’s date, taps “查询,” selects the first available high-speed train, and taps “提交订单.” The expected outcome is an order confirmation page showing departure and arrival times and a visible “支付” button.

These examples clarify the benchmark’s intended difficulty profile. The target behavior is not simple widget grounding, but multi-step planning over app-specific state transitions, typed UI actions, and verifiable terminal conditions. This suggests that KG-Harmony-Bench is particularly useful for studying whether a GUI agent can combine current-screen perception with retrieved structural priors from UTGs.

7. Position within GUI-agent evaluation

Within the KG-RAG framework, KG-Harmony-Bench occupies the role of a HarmonyOS-specific counterpart to KG-Android-Bench, extending UTG-driven evaluation into a platform that is both linguistically and structurally distinct from the English-language Android settings emphasized by earlier benchmarks (Guan et al., 30 Aug 2025). Its defining elements are the combination of real-world HarmonyOS apps, multi-step Chinese-language tasks, UTG-based annotation, and a retrieval-ready knowledge representation derived from offline intent-guided LLM search.

Its main methodological contribution is therefore not a new metric, but a benchmark configuration that makes cross-platform retrieval-augmented GUI-agent evaluation concrete. The benchmark can be read as an attempt to operationalize a specific research question: whether structured app knowledge, compiled from UTGs into vector-searchable intent–trajectory pairs, materially improves decision-making in heterogeneous mobile environments.

The benchmark also imposes a clear interpretive boundary. Because all 150 tasks are used as a single held-out test set and evaluation is single-trial with fixed timeout and action budgets, it is primarily an assessment instrument rather than a training corpus. A plausible implication is that future work using KG-Harmony-Bench will treat it chiefly as a comparative testbed for agent architectures, retrieval mechanisms, and OS-transfer robustness, rather than as a source of supervised optimization targets.