KG-Android-Bench: Mobile GUI Benchmark
- KG-Android-Bench is a benchmark that evaluates mobile GUI agents through deep, multi-step tasks using UTG-derived knowledge graphs and structured intent-action mappings.
- It covers 300 tasks across 30 top Chinese Android apps, emphasizing realistic user interactions like deep navigation and cross-domain task execution.
- The benchmark leverages automated exploration and LLM-based trajectory annotation to assess decision accuracy, success rate, and efficient multi-turn task planning.
Searching arXiv for KG-Android-Bench and closely related mobile GUI agent benchmark papers. KG-Android-Bench is a benchmark for evaluating GUI agents in mobile environments, introduced alongside the KG-RAG framework as part of a broader effort to improve decision-making with knowledge graph-driven retrieval-augmented generation. It is presented as a large-scale benchmark centered on the Chinese mobile ecosystem, with 300 tasks across 30 mainstream Chinese Android applications, and is designed to assess real-world, multi-step task completion, deep navigation, and cross-domain competence rather than only shallow UI interaction traces (Guan et al., 30 Aug 2025). Its defining characteristic is the integration of UTG-derived knowledge graphs, intent-action mappings, and full navigation trajectories, making it a benchmark not only for interface manipulation but also for structured knowledge use in mobile agents.
1. Definition and scope
KG-Android-Bench was introduced to provide robust evaluation of GUI agents on real-world, multi-step tasks spanning a broad app ecosystem, with particular emphasis on capabilities that prior benchmarks assess only minimally: deep navigation structures, cross-domain tasks, and the use of structured knowledge such as UI Transition Graphs (UTGs), intent-action pairs, and retrieval-augmented architectures (Guan et al., 30 Aug 2025).
The benchmark is explicitly framed as part of the Chinese mobile ecosystem. Its app set spans 10 broad functional categories: Music/Audio, Video/Entertainment, Social/Communication, Navigation/Travel, E-commerce/Retail, Food Services, Health/Lifestyle, News/Reading, Productivity, and Photo/Editing. Examples listed in the source description include QQ Music, Douyin, Weibo, Taobao, Meituan Takeaway, Tomato Novel, and Keep. This category coverage is intended to stress both breadth and compositional task structure rather than isolated app-specific actions.
A central design premise is that KG-Android-Bench evaluates more than action selection on a single screen. For each app, the benchmark includes a UTG-derived knowledge graph that maps user intents to action sequences or trajectories. Tasks therefore encode full navigation flows and multi-step task execution sequences, not merely flat UI states or click traces. This suggests that the benchmark is structured to evaluate planning, retrieval, and long-horizon control jointly, rather than benchmarking only low-level grounding.
2. Corpus design and benchmark construction
Several design choices distinguish KG-Android-Bench from earlier mobile-agent datasets. The most immediate is scale: 300 tasks across 30 mainstream Chinese Android applications, compared in the source material with DroidTask: 162 tasks/12 apps and AndroidLab: 138 tasks/9 apps (Guan et al., 30 Aug 2025). The benchmark is therefore positioned as substantially larger in both task count and application coverage.
A second design choice is the emphasis on realistic, high-frequency task selection. The benchmark includes tasks modeled on frequent user operations such as “View Privacy Policy”, “Redeem Coupon”, and deep settings navigation. These are intended to represent routine but nontrivial mobile interactions, especially those requiring access to nested screens or latent app states.
A third design choice is the benchmark’s representational structure. Each task includes required action steps as a logical sequence, with links among actions, intents, and associated UI elements. This allows evaluation of multi-turn and multi-goal task completion. In that sense, KG-Android-Bench is designed around structured trajectories rather than isolated actions, with the knowledge graph serving as an organizing layer over the extracted UTG.
The benchmark also has explicit cross-platform support at the dataset-family level. It was introduced together with KG-Harmony-Bench for HarmonyOS, enabling cross-platform evaluation. While KG-Android-Bench itself is Android-focused, the paired release indicates that the benchmark program was conceived with OS variation in mind rather than as a single-platform artifact.
3. Data acquisition and annotation pipeline
The benchmark construction pipeline begins with automated systematic app exploration using xTester, described as adapted from DroidBot (Guan et al., 30 Aug 2025). This exploration is used to extract UTGs from the target applications. The source description distinguishes between app regimes: for simpler (English) apps, 1-hour automated exploration captures most UI structures, whereas complex Chinese apps may require up to 8 hours of in-depth exploration per app to cover intricate UIs and latent states.
The output of this exploration is a set of hierarchical JSON-structured UTGs containing detailed metadata, screen layouts, widget descriptions, and mapped actions. These UTGs are subsequently transformed into benchmark assets that support knowledge-guided retrieval and navigation. The abstract further reports a UTG cost ablation showing that accuracy saturates at approximately 4 hours per complex app, which introduces an explicit cost-quality trade-off into benchmark construction and practical deployment (Guan et al., 30 Aug 2025).
Trajectory annotation is layered on top of UTG extraction. Vision-LLMs (VLMs) are used to infer plausible user intents from UI screenshots. Instruction-tuned LLMs then decompose high-level intents into a series of “milestones”, that is, intermediate subgoals. A combination of offline pathfinding and LLM-based scoring is used to generate and summarize valid multi-step trajectories for each intent. The resulting benchmark representation therefore couples static UI graph structure with intent decomposition and path-level supervision.
The data collection process is also described as subject to strict ethical guidelines, with all data anonymized and privacy-masked. This is relevant because benchmarks built from commercial mobile applications can otherwise expose user-facing text or content traces that are not suitable for open evaluation settings.
4. Evaluation protocol and benchmark usage
KG-Android-Bench defines three primary metrics for agent evaluation: Success Rate (SR), Decision Accuracy (DA), and Average Steps (AS) (Guan et al., 30 Aug 2025). SR is the percentage of tasks in which the agent completes all required steps for the user instruction. DA is the proportion of correct actions made during execution. AS is the mean number of steps needed to complete the task and is used as an efficiency measure.
These metrics support several distinct uses. First, the benchmark is intended to benchmark existing and new GUI agents, including AutoDroid, MobileAgent-v2, UI-TARS, and KG-RAG itself. Second, it supports plug-and-play enhancement studies, where a knowledge-augmented module is added to an existing agent architecture and its effect measured. Third, it is used to study generalization, including cross-device, cross-OS, and even non-mobile transfer settings.
The benchmark’s structure is especially aligned with research on knowledge integration. The source description characterizes it as the first major benchmark where knowledge-driven retrieval, based on the mutual context of UTG, intents, and actions, is central to successful completion of deep, multi-turn tasks. This implies that models which treat the environment as a sequence of local screenshots alone may be systematically disadvantaged relative to architectures that exploit graph-structured memory or retrieval over navigation knowledge.
A common misunderstanding would be to treat KG-Android-Bench as merely a larger action-trace dataset. The benchmark description explicitly contradicts this: it includes full navigation flows, intent-action mappings, and UTG-derived knowledge graphs, and therefore evaluates whether an agent can exploit structured app knowledge during task execution rather than only mimic click sequences.
5. Empirical results and comparative position
The benchmark is reported as larger than previously used Chinese mobile-agent datasets, and the source provides a direct comparison with prior benchmarks:
| Benchmark | Tasks / Apps | Stated property |
|---|---|---|
| KG-Android-Bench | 300 / 30 | UTG/KG integration; intent-action mapping |
| DroidTask | 162 / 12 | Prior benchmark |
| AndroidLab | 138 / 9 | Prior benchmark |
In large-scale experiments, KG-RAG is reported to outperform existing methods across diverse mobile apps, achieving a 75.8% success rate, 84.6% decision accuracy, and reducing average task steps from 4.5 to 4.1, with the abstract stating an 8.9% improvement over AutoDroid in success rate and an 8.1% improvement in decision accuracy (Guan et al., 30 Aug 2025). The benchmark-specific summary further states that on KG-Android-Bench, agents using KG-RAG achieve up to 78.3% SR, 82.0% DA, and as low as 4.9 AS, with 8–9% higher SR and DA than previous state of the art.
The same source states that KG-RAG functions as a plug-and-play module: integrating it into MobileAgent-v2 and UI-TARS yields 10–20% absolute SR/DA gains while reducing errors and redundant navigation. The benchmark therefore serves not only as a leaderboard substrate but also as an ablation environment for studying how structured retrieval alters agent behavior.
Generalization results are also emphasized. The benchmark program includes cross-device and HarmonyOS tests, and the abstract reports transfer to web/desktop interfaces with +40% SR on Weibo-web and +20% on QQ Music-desktop without retraining (Guan et al., 30 Aug 2025). Strictly speaking, these numbers describe transfer performance of the framework evaluated with the benchmark ecosystem rather than a property intrinsic to KG-Android-Bench alone. A plausible implication is that the benchmark was designed to expose reusable structural regularities rather than only app-specific patterns.
6. Research significance, boundaries, and relation to adjacent benchmarks
KG-Android-Bench is significant because it operationalizes a particular view of mobile-agent evaluation: competence is not reduced to screen parsing or one-step grounding, but includes navigation over latent app structure, retrieval of prior interaction knowledge, and execution of deep, real-life, multi-step tasks. The source description presents it as purpose-built for knowledge-driven agents, with structural richness and cross-task compositionality explicitly built into the benchmark design (Guan et al., 30 Aug 2025).
That same specialization also defines its boundary conditions. Because the benchmark is centered on UTG-derived knowledge graphs and intent-action mappings, it is especially suitable for evaluating retrieval-augmented, graph-guided, and hierarchical planning systems. This suggests that it may not be a neutral probe of all possible agent paradigms; rather, it foregrounds one increasingly important class of architectures in mobile automation research.
Another boundary is ecosystem specificity. KG-Android-Bench is tailored to the Chinese mobile ecosystem, including mainstream Chinese applications and high-frequency tasks characteristic of that setting. This is a strength for realism within that domain, but it also means that cross-lingual or cross-market generalization should be established empirically rather than assumed.
In the broader benchmark landscape, KG-Android-Bench can be contrasted with KnowU-Bench, which evaluates interactive, proactive, and personalized mobile agents in a reproducible Android emulation environment and focuses on hidden user profiles, behavioral logs, preference elicitation, and intervention calibration rather than primarily on knowledge-graph-guided navigation (Chen et al., 9 Apr 2026). The comparison indicates an emerging division of labor in mobile-agent evaluation: KG-Android-Bench emphasizes structured app knowledge and deep navigation, whereas KnowU-Bench emphasizes personalization, clarification dialogue, and proactive behavior. This suggests that the field is moving from generic GUI execution benchmarks toward more specialized evaluations of distinct agent competencies.
Overall, KG-Android-Bench occupies a specific place in that progression. It is a benchmark for knowledge-driven, multi-domain, and multi-step mobile GUI evaluation, with explicit support for UTG/KG-based reasoning and with construction procedures designed to capture the latent navigational structure of complex real-world apps (Guan et al., 30 Aug 2025).