GUITestBench: GUI Testing Benchmark

• GUITestBench is a comprehensive benchmark framework that evaluates GUI agents on compatibility, defect discovery, and cross-platform performance.
• It employs a three-layer architecture (dataset, execution, analysis) to ensure reproducibility, fine-grained fault diagnosis, and realistic user flow simulations.
• Empirical evaluations highlight distinct tool-specific failure modes and improvements in defect detection using LLM-based planning modules.

GUITestBench is a framework and dataset designed for rigorous, systematic evaluation of graphical user interface (GUI) agents, with specific emphasis on compatibility, defect discovery, and cross-platform robustness in automated GUI testing. It serves as a reference benchmark for developing and measuring the effectiveness of tool-assisted approaches in GUI compatibility testing, autonomous exploratory testing agents, and transfer learning systems across different device and software configurations.

1. Concept and Motivation

GUITestBench was introduced as the first dedicated benchmark targeting GUI compatibility in mobile applications, aiming to answer whether existing GUI test-case replay techniques—both intrusive and non-intrusive—provide reliable cross-device, cross-version assistance. The benchmark was motivated by the recognition that standard functionality testing overlooks the unique challenges posed by device heterogeneity, interface drift, and ambiguous GUI element matching in real-world settings (Ye et al., 2022). Expanding on this, recent work established GUITestBench as the first interactive benchmark for autonomous exploratory defect discovery, explicitly addressing the inability of conventional GUI agents to autonomously detect anomalies due to phenomena such as goal-oriented masking and execution-bias attribution (Gao et al., 8 Jan 2026).

2. Architecture and Design Principles

GUITestBench is architected as a three-layer system:

• Dataset Layer: Contains curated test-case scripts, each representing real-world user flows in widely used Android applications (e.g., Booking.com, Chrome). Scripts are recorded manually to maximize ecological validity (Ye et al., 2022).
• Execution Layer: Provides a platform-agnostic harness that deploys apps on emulators or real devices, invokes tool-specific replay engines, and collects detailed per-step execution logs.
• Analysis Layer: Computes step-wise replay success rates, categorizes failure modes (e.g., incorrect localization, filtering-failure), and generates summary statistics and reports.

This architecture is designed for reproducibility (open datasets, platform independence), extensibility (plug-in support for new tools), step-level granularity (fine fault diagnosis), and realism (diverse, end-to-end GUI flows).

In its interactive exploratory variant, GUITestBench adds a server-agent loop for Android emulators, formal task specifications (instruction, screenshots, defect descriptors in JSON), and LLM-based judging for complex, multi-action defects (Gao et al., 8 Jan 2026).

3. Task Types, Defect Taxonomy, and Dataset Composition

GUITestBench provides two principal categories of tasks, each addressing a distinct evaluation goal:

• Defect-Oriented Tasks: Explicit, step-by-step reproduction flows enabling evaluation of defect recognition and reporting accuracy.
• Exploration-Oriented Tasks: Open-ended high-level intents, where successful exploration entails traversing one or more defect-relevant states, thus measuring the end-to-end discovery capability.

The benchmark’s defect taxonomy draws from paradigms established in GTArena and other GUI testing suites (Zhao et al., 2024), including:

Category	Subtype	Description
UI Functional Defects	ONR	Operation No Response (no feedback after action)
	UTR	Unexpected Task Result (outcome deviates from expectation)
	NLE	Navigation Logic Error (misleading navigation behavior)
UX Defects	UX-UTR	Valid UI operations yield an aggregate incorrect outcome
	UX-NLE	Dialog/back-navigation flows deviate from intended interaction sequence

Current GUITestBench datasets include 26 real defects harvested from GitHub repositories of 12 Android apps, expanded into 143 tasks distributed across five domains (productivity, news, e-commerce, travel, utilities), with both single-action (62.24%) and multi-action (37.76%) defects (Gao et al., 8 Jan 2026).

4. Evaluation Protocols and Metrics

The evaluation protocol mandates multiple independent executions on a common Android emulator configuration, using both intrusive (MAPIT) and non-intrusive (Roscript) replay tools, as well as LLM-based multi-agent frameworks (e.g., GUITester):

• Replay Success Rate: Step-wise metric: $$\mathrm{SuccessRate} = \frac{S_{\mathrm{succ}}}{S_{\mathrm{tot}}} \times 100\%$$
• Recall (Defect Discovery): $$\mathrm{Recall} = \frac{|T_{\mathrm{detected}}|}{|T_{\mathrm{total}}|}$$
• Precision: $$\mathrm{Precision} = \frac{|T_{\mathrm{detected}}|}{|T_{\mathrm{declared}}|}$$
• F₁-Score: $$F_1 = \frac{2 \times \mathrm{Precision} \times \mathrm{Recall}}{\mathrm{Precision + Recall}}$$
• Pass@k: Fraction of tasks solved successfully in at least one of $k$ independent runs.

For baseline compatibility evaluation, GUITestBench also tracks failure mode breakdowns (Incorrect Localization, Filtering Failure, Matching Failure for non-intrusive tools). In the interactive defect-discovery setting, multi-step outcomes are LLM-judged, accounting for ambiguous trajectories and enabling nuanced attributions.

5. Quantitative Results and Failure Analysis

Empirical studies using GUITestBench have revealed:

• Existing replay tools (MAPIT, Roscript) achieve approximately 50% step-wise success rates, with no statistically significant difference between paradigms (paired t-test $p=0.72$) (Ye et al., 2022).
• Failure modes differ by approach: intrusive tools are dominated by Incorrect Localization (75.4% of MAPIT failures), whereas non-intrusive template matching is prone to Matching Failure and sensitivity to overlays or custom GUI components.
• The GUITester framework, leveraging a planning-execution module with proactive defect probing and hierarchical reflection for execution-bias disambiguation, outperforms SOTA baselines by a margin of +15.55 percentage points in overall F₁-score (48.90% vs. 33.35% at Pass@3 with UI-TARS-1.5-7B executor) (Gao et al., 8 Jan 2026).

6. Benchmark Implementation and Use

GUITestBench is released as an open-source Python package, supporting emulator-based execution and standardized result logging. The typical workflow is as follows:

1. Installation: Python 3.8+, Android SDK/AVD, prerequisite packages via pip.
2. Execution: Launch emulator, prepare task JSONs, run agents using provided scripts (supporting multiple baselines and custom models).
3. Analysis: CSV logs per task, with auto-computed aggregate metrics (precision, recall, F₁, Pass@k).

This reproducibility protocol enables fair, side-by-side assessment of new LLM-based GUI agents, defect detectors, or compatibility runners.

7. Research Impact, Relation to Other Benchmarks, and Future Directions

GUITestBench is distinct from GUI agent transferability and general automation testbeds such as TransBench (Lu et al., 23 May 2025), GTArena (Zhao et al., 2024), and MMBench-GUI (Wang et al., 25 Jul 2025) by its focus on compatibility and defect-centric analysis. Whereas benchmarks like TransBench evaluate cross-version/platform/app transfer of elementary grounding and planning, GUITestBench isolates and quantifies the challenge of test-case replay and autonomous fault detection under real-world, heterogeneously sourced defects.

Key recommendations and research avenues drawn from GUITestBench studies include:

• Exploration of hybrid semantic-visual matching (content-descriptor plus deep detection) to counteract localization and matching failures.
• Expansion of the dataset to cover additional app categories (games, finance), richer input modalities (gestures, text), and cross-OS scenarios (iOS, desktop).
• Improved statistical evaluation protocols (power analysis, effect sizes, nonparametric tests) for robust comparison.
• Integration of planning modules capable of proactive defect discovery and long-horizon reflection.
• Modular benchmarks and agent architectures, as highlighted by MMBench-GUI, enabling swap-in upgrades of perception, planning, and memory components.

A plausible implication is that GUITestBench, by enabling precise, transparent evaluation of GUI compatibility and defect identification, will catalyze broader adoption of robust, tool-assisted methods for mobile and multi-platform software quality assurance. Its rigorous protocols and extensible design position it as a keystone resource for both benchmark authors and next-generation GUI testing agents.