AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark

Abstract: Autonomous agents capable of navigating Graphical User Interfaces (GUIs) hold the potential to revolutionize digital productivity. However, achieving true digital autonomy extends beyond reactive element matching; it necessitates a predictive mental model of interface dynamics and the ability to foresee the "digital world state" resulting from interactions. Despite the perceptual capabilities of modern Vision-LLMs (VLMs), existing benchmarks remain bifurcated (focusing either on black-box task completion or static, shallow grounding), thereby failing to assess whether agents truly comprehend the implicit functionality and transition logic of GUIs. To bridge this gap, we introduce AutoGUI-v2, a comprehensive benchmark designed to evaluate deep GUI functionality understanding and interaction outcome prediction. We construct the benchmark using a novel VLM-human collaborative pipeline that recursively parses multi-platform screenshots into hierarchical functional regions to generate diverse evaluation tasks. Providing 2,753 tasks across six operating systems, AutoGUI-v2 rigorously tests agents on region and element-level semantics, grounding, and dynamic state prediction. Our evaluation reveals a striking dichotomy in VLMs: while open-source models fine-tuned on agent data (e.g., Qwen3-VL) excel at functional grounding, commercial models (e.g., Gemini-2.5-Pro-Thinking) dominate in functionality captioning. Crucially, all models struggle with complex interaction logic of uncommon actions, highlighting that deep functional understanding remains a significant hurdle. By systematically measuring these foundational capabilities, AutoGUI-v2 offers a new lens for advancing the next generation of GUI agents.

• The paper advances GUI functionality understanding by introducing a hierarchical benchmark with 3,710 annotated regions and 2,753 evaluation tasks.
• It employs a recursive divide-and-verify annotation pipeline that combines automated VLM proposals with human refinement for precise segmentation.
• The benchmark highlights critical gaps between functionality-based grounding and captioning, underlining challenges in handling complex interactive GUI actions.

AutoGUI-v2: Advancing Contextual GUI Functionality Understanding in Vision-Language Agents

Motivation and Benchmark Scope

AutoGUI-v2 addresses the deficiencies of existing GUI benchmarks, which either focus on black-box task completion or shallow element grounding without probing functionality comprehension or interaction outcomes. The benchmark tests VLM-based agents on their ability to construct a predictive mental model of interface dynamics—a prerequisite for robust automation across diverse digital platforms. By offering rich functional semantics for both GUI elements and regions, AutoGUI-v2 provides a rigorous foundation for evaluating whether agents can interpret implicit functionality and reason about interface state transitions beyond reactive matching.

Figure 1: AutoGUI-v2 provides rich functional semantics for both GUI elements and regions compared with existing benchmarks.

Annotation Pipeline and Dataset Structure

The dataset is created through a VLM-human collaborative pipeline, wherein Gemini-2.5-Pro-Thinking initially decomposes screenshots from six operating systems into hierarchical functional regions, automatically scoring each stage on region divisibility and bounding box quality. This recursive "divide-and-verify" methodology yields granular regions describing high-level purpose, subsequently refined by human annotators for pixel-perfect bounding box precision. The final annotation incorporates VLM-generated high-fidelity descriptions, yielding 3,710 functional regions and 2,753 evaluation tasks spanning region and element levels.

Figure 2: Overview of the AutoGUI-v2 annotation pipeline, integrating automated VLM region proposal, recursive division, scoring, and human refinement for precise geometry and semantic description.

Task generation leverages semantic clustering and VLM verification to group visually similar but functionally distinct regions, ensuring distractors in evaluation. Element detection utilizes OmniParser-v2 and DINO-v3 embeddings to construct similarity groups, again refined via VLM-based verification and manual checks. Tasks include functionality-oriented grounding (localize by functional description) and captioning (predict interaction outcomes), both at region and element granularity.

Figure 3: The pipeline for generating AutoGUI-v2 benchmark tasks, establishing hierarchy-aware, context-driven queries for both region and element tasks.

Evaluation and Model Analysis

Region-Level Functionality Understanding

The benchmark reveals a marked dichotomy between model families. GUI-specialized open-source models (e.g., Qwen3-VL-32B-Instruct, GLM-4.5V) dominate region-level functionality grounding, achieving up to 84.4% and 84.6% accuracy, respectively. These models consistently outperform commercial VLMs (Gemini-2.5-Pro-Thinking, GPT-5, O3) by margins exceeding 40 percentage points in grounding tasks. However, when the benchmark shifts from appearance-based description to functionality-based queries, all models present a performance drop, underlining the semantic complexity inherent in functional localization.

Region typology analysis shows that standardized containers and interaction controls are easier to localize, while irregular or context overlays (Others) reduce model accuracy by 5% or more. Error rates against hard distractors (visually similar but functionally distinct regions) consistently surpass easy negatives, demonstrating persistent gaps in robust, context-aware functional reasoning.

Element-Level Functionality Understanding

Open-source VLMs, especially those fine-tuned with GUI-centric datasets (Qwen3-VL-32B-Instruct), again dominate functionality-based element grounding, with scores over 70%. Commercial VLMs, despite strong performance on appearance-based benchmarks, demonstrate substantially lower accuracies (e.g., Gemini-2.5-Pro-Thinking at 67.7%). Functionality-based grounding is universally harder than appearance or intent-based tasks, confirming that context-driven reasoning is the principal challenge for state-of-the-art VLMs.

Action-type decomposition exposes model struggles with implicit or complex actions (e.g., right-click, drag), where accuracy plummets relative to overt actions (e.g., type, long-press). The Normalized Interference Density (NID) metric trends upward for almost all models: element-level functionality grounding becomes easier in visually dense contexts, implying that spatial cues and group context are critical for correct semantic inference.

Functionality Captioning Divergence

A salient claim is the grounding-reasoning divide: commercial VLMs excel at captioning/prediction tasks (“what does it do?”) with Gemini-2.5-Pro-Thinking scoring 70.3% on element outcome prediction, well above most open-source baselines. Captioning results correlate strongly with action types: models succeed on overt interactions but systematically fail on subtle feedback (e.g., hover). Hard negatives again drive error rates higher, reinforcing the notion that distractors exploit the models’ weak contextual discrimination rather than mere randomness.

Figure 4: Functionality understanding cases of Gemini-2.5-Pro-Thinking, highlighting its strengths with common targets and failures on complex or stateful components.

Failure Pattern Analysis

In grounding tasks, Gemini-2.5-Pro-Thinking often predicts the functional region correctly in reasoning content but fails spatial regression, with bounding boxes shifted or loose. Open-source models show strong pixel accuracy but are prone to misidentification. Captioning failures center on distinctions between system vs. application layers, and confusion among similar icons within apps, validating that functional abstraction and hierarchical reasoning are unresolved for current architectures.

Practical and Theoretical Implications

AutoGUI-v2 establishes a methodology and dataset that enables precise diagnostics of functionality understanding, moving beyond success rates to detailed reasoning about agent capabilities. The strong divergence between localization and captioning proficiencies suggests that future architectures should explicitly couple spatial grounding and semantic model-building, possibly via integrated world models or hierarchical context modeling. The challenge in complex or less common actions points to gaps in training data distribution and reward modeling, and implies the need for richer multimodal feedback loops and sequence modeling approaches in GUI agent learning.

On practical grounds, robust agents that predict interaction outcomes can underpin advanced automation, accessibility, and intelligent digital copilot applications. However, persistent error modes on complex layouts, stateful controls, and plausible distractors highlight risks for reliability, safety, and usability.

Outlook and Future Directions

AutoGUI-v2 lays the foundation for benchmarking not only basic UI element recognition but also high-level functional generalization and digital state prediction, both critical for scalable and trustworthy GUI agents. Full automation in annotation and hierarchical benchmarking scalability are open directions, as is the linkage between functionality understanding and chain-of-thought planning in multi-step interaction. To realize practical and generalizable digital agents, further examination of how deep functionality understanding impacts planning, strategic reasoning, and safety in cross-platform contexts is essential.

Conclusion

AutoGUI-v2 presents a rigorous, hierarchical benchmark for vision-language GUI agents, systematically diagnosing functional grounding and reasoning at region and element levels. Strong numerical results and contradictory claims regarding grounding versus captioning proficiency elucidate the limitations of current architectures, underscoring the challenge of deep GUI functionality understanding. The benchmark prompts future work on world-model-based reasoning, scalable annotation pipelines, and context-rich planning in multimodal, cross-platform environments (2604.24441).