---
title: 'KG-Android-Bench: Mobile GUI Benchmark'
url: https://www.emergentmind.com/topics/kg-android-bench
type: topic
---

# KG-Android-Bench: Mobile GUI Benchmark

Searching arXiv for KG-Android-Bench and closely related mobile GUI agent benchmark papers.
KG-Android-Bench is a benchmark for evaluating GUI agents in mobile environments, introduced alongside the KG-RAG framework as part of a broader effort to improve decision-making with knowledge graph-driven retrieval-augmented generation. It is presented as a large-scale benchmark centered on the Chinese mobile ecosystem, with **300 tasks across 30 mainstream Chinese Android applications**, and is designed to assess real-world, multi-step task completion, deep navigation, and cross-domain competence rather than only shallow UI interaction traces [2509.00366]. Its defining characteristic is the integration of **UTG-derived knowledge graphs**, **intent-action mappings**, and full navigation trajectories, making it a benchmark not only for interface manipulation but also for structured knowledge use in mobile agents.

## 1. Definition and scope

KG-Android-Bench was introduced to provide **robust evaluation** of GUI agents on **real-world, multi-step tasks** spanning a broad app ecosystem, with particular emphasis on capabilities that prior benchmarks assess only minimally: **deep navigation structures**, **cross-domain tasks**, and the use of structured knowledge such as **UI Transition Graphs (UTGs)**, intent-action pairs, and retrieval-augmented architectures [2509.00366].

The benchmark is explicitly framed as part of the Chinese mobile ecosystem. Its app set spans **10 broad functional categories**: **Music/Audio, Video/Entertainment, Social/Communication, Navigation/Travel, E-commerce/Retail, Food Services, Health/Lifestyle, News/Reading, Productivity, and Photo/Editing**. Examples listed in the source description include **QQ Music, Douyin, Weibo, Taobao, Meituan Takeaway, Tomato Novel,** and **Keep**. This category coverage is intended to stress both breadth and compositional task structure rather than isolated app-specific actions.

A central design premise is that KG-Android-Bench evaluates more than action selection on a single screen. For each app, the benchmark includes a **UTG-derived knowledge graph** that maps user intents to action sequences or trajectories. Tasks therefore encode **full navigation flows** and **multi-step task execution sequences**, not merely flat UI states or click traces. This suggests that the benchmark is structured to evaluate planning, retrieval, and long-horizon control jointly, rather than benchmarking only low-level grounding.

## 2. Corpus design and benchmark construction

Several design choices distinguish KG-Android-Bench from earlier mobile-agent datasets. The most immediate is scale: **300 tasks across 30 mainstream Chinese Android applications**, compared in the source material with **DroidTask: 162 tasks/12 apps** and **AndroidLab: 138 tasks/9 apps** [2509.00366]. The benchmark is therefore positioned as substantially larger in both task count and application coverage.

A second design choice is the emphasis on **realistic, high-frequency task selection**. The benchmark includes tasks modeled on frequent user operations such as **“View Privacy Policy”**, **“Redeem Coupon”**, and **deep settings navigation**. These are intended to represent routine but nontrivial mobile interactions, especially those requiring access to nested screens or latent app states.

A third design choice is the benchmark’s representational structure. Each task includes **required action steps as a logical sequence**, with links among **actions, intents, and associated UI elements**. This allows evaluation of **multi-turn** and **multi-goal** task completion. In that sense, KG-Android-Bench is designed around structured trajectories rather than isolated actions, with the knowledge graph serving as an organizing layer over the extracted UTG.

The benchmark also has explicit **cross-platform support** at the dataset-family level. It was introduced together with **KG-Harmony-Bench** for **HarmonyOS**, enabling **cross-platform evaluation**. While KG-Android-Bench itself is Android-focused, the paired release indicates that the benchmark program was conceived with OS variation in mind rather than as a single-platform artifact.

## 3. Data acquisition and annotation pipeline

The benchmark construction pipeline begins with **automated systematic app exploration** using **xTester**, described as adapted from **DroidBot** [2509.00366]. This exploration is used to extract UTGs from the target applications. The source description distinguishes between app regimes: for **simpler (English) apps**, **1-hour automated exploration** captures most UI structures, whereas **complex Chinese apps** may require **up to 8 hours of in-depth exploration per app** to cover intricate UIs and latent states.

The output of this exploration is a set of **hierarchical JSON-structured UTGs** containing **detailed metadata, screen layouts, widget descriptions, and mapped actions**. These UTGs are subsequently transformed into benchmark assets that support knowledge-guided retrieval and navigation. The abstract further reports a **UTG cost ablation** showing that **accuracy saturates at approximately 4 hours per complex app**, which introduces an explicit cost-quality trade-off into benchmark construction and practical deployment [2509.00366].

Trajectory annotation is layered on top of UTG extraction. **Vision-Language Models (VLMs)** are used to infer plausible user intents from UI screenshots. **Instruction-tuned LLMs** then decompose high-level intents into a series of **“milestones”**, that is, intermediate subgoals. A combination of **offline pathfinding** and **LLM-based scoring** is used to generate and summarize valid **multi-step trajectories** for each intent. The resulting benchmark representation therefore couples static UI graph structure with intent decomposition and path-level supervision.

The data collection process is also described as subject to **strict ethical guidelines**, with all data **anonymized and privacy-masked**. This is relevant because benchmarks built from commercial mobile applications can otherwise expose user-facing text or content traces that are not suitable for open evaluation settings.

## 4. Evaluation protocol and benchmark usage

KG-Android-Bench defines **three primary metrics** for agent evaluation: **Success Rate (SR)**, **Decision Accuracy (DA)**, and **Average Steps (AS)** [2509.00366]. SR is the percentage of tasks in which the agent completes all required steps for the user instruction. DA is the proportion of correct actions made during execution. AS is the mean number of steps needed to complete the task and is used as an efficiency measure.

These metrics support several distinct uses. First, the benchmark is intended to **benchmark existing and new GUI agents**, including **AutoDroid, MobileAgent-v2, UI-TARS,** and **KG-RAG** itself. Second, it supports **plug-and-play enhancement studies**, where a knowledge-augmented module is added to an existing agent architecture and its effect measured. Third, it is used to study **generalization**, including **cross-device**, **cross-OS**, and even **non-mobile transfer** settings.

The benchmark’s structure is especially aligned with research on knowledge integration. The source description characterizes it as the **first major benchmark where knowledge-driven retrieval**, based on the mutual context of **UTG, intents, and actions**, is central to successful completion of **deep, multi-turn tasks**. This implies that models which treat the environment as a sequence of local screenshots alone may be systematically disadvantaged relative to architectures that exploit graph-structured memory or retrieval over navigation knowledge.

A common misunderstanding would be to treat KG-Android-Bench as merely a larger action-trace dataset. The benchmark description explicitly contradicts this: it includes **full navigation flows**, **intent-action mappings**, and **UTG-derived knowledge graphs**, and therefore evaluates whether an agent can exploit structured app knowledge during task execution rather than only mimic click sequences.

## 5. Empirical results and comparative position

The benchmark is reported as larger than previously used Chinese mobile-agent datasets, and the source provides a direct comparison with prior benchmarks:

| Benchmark | Tasks / Apps | Stated property |
|---|---:|---|
| KG-Android-Bench | 300 / 30 | UTG/KG integration; intent-action mapping |
| DroidTask | 162 / 12 | Prior benchmark |
| AndroidLab | 138 / 9 | Prior benchmark |

In large-scale experiments, **KG-RAG** is reported to outperform existing methods across diverse mobile apps, achieving a **75.8% success rate**, **84.6% decision accuracy**, and reducing **average task steps from 4.5 to 4.1**, with the abstract stating an **8.9% improvement over AutoDroid** in success rate and an **8.1% improvement** in decision accuracy [2509.00366]. The benchmark-specific summary further states that on **KG-Android-Bench**, agents using KG-RAG achieve **up to 78.3%** SR, **82.0%** DA, and **as low as 4.9** AS, with **8–9% higher SR and DA** than previous state of the art.

The same source states that KG-RAG functions as a **plug-and-play** module: integrating it into **MobileAgent-v2** and **UI-TARS** yields **10–20% absolute SR/DA gains** while reducing errors and redundant navigation. The benchmark therefore serves not only as a leaderboard substrate but also as an ablation environment for studying how structured retrieval alters agent behavior.

Generalization results are also emphasized. The benchmark program includes **cross-device** and **HarmonyOS** tests, and the abstract reports transfer to **web/desktop** interfaces with **+40% SR on Weibo-web** and **+20% on QQ Music-desktop** without retraining [2509.00366]. Strictly speaking, these numbers describe transfer performance of the framework evaluated with the benchmark ecosystem rather than a property intrinsic to KG-Android-Bench alone. A plausible implication is that the benchmark was designed to expose reusable structural regularities rather than only app-specific patterns.

## 6. Research significance, boundaries, and relation to adjacent benchmarks

KG-Android-Bench is significant because it operationalizes a particular view of mobile-agent evaluation: competence is not reduced to screen parsing or one-step grounding, but includes navigation over latent app structure, retrieval of prior interaction knowledge, and execution of **deep, real-life, multi-step tasks**. The source description presents it as **purpose-built for knowledge-driven agents**, with **structural richness** and **cross-task compositionality** explicitly built into the benchmark design [2509.00366].

That same specialization also defines its boundary conditions. Because the benchmark is centered on **UTG-derived knowledge graphs** and **intent-action mappings**, it is especially suitable for evaluating **retrieval-augmented**, **graph-guided**, and **hierarchical planning** systems. This suggests that it may not be a neutral probe of all possible agent paradigms; rather, it foregrounds one increasingly important class of architectures in mobile automation research.

Another boundary is ecosystem specificity. KG-Android-Bench is tailored to the **Chinese mobile ecosystem**, including mainstream Chinese applications and high-frequency tasks characteristic of that setting. This is a strength for realism within that domain, but it also means that cross-lingual or cross-market generalization should be established empirically rather than assumed.

In the broader benchmark landscape, KG-Android-Bench can be contrasted with **KnowU-Bench**, which evaluates **interactive, proactive, and personalized mobile agents** in a reproducible Android emulation environment and focuses on hidden user profiles, behavioral logs, preference elicitation, and intervention calibration rather than primarily on knowledge-graph-guided navigation [2604.08455]. The comparison indicates an emerging division of labor in mobile-agent evaluation: KG-Android-Bench emphasizes **structured app knowledge and deep navigation**, whereas KnowU-Bench emphasizes **personalization, clarification dialogue, and proactive behavior**. This suggests that the field is moving from generic GUI execution benchmarks toward more specialized evaluations of distinct agent competencies.

Overall, KG-Android-Bench occupies a specific place in that progression. It is a benchmark for **knowledge-driven**, **multi-domain**, and **multi-step** mobile GUI evaluation, with explicit support for UTG/KG-based reasoning and with construction procedures designed to capture the latent navigational structure of complex real-world apps [2509.00366].

Source: https://www.emergentmind.com/topics/kg-android-bench