Scene-Aware and GUI-Grounded Datasets

• Scene-aware and GUI-grounded datasets are tailored collections that integrate visual UI states with detailed semantic and procedural annotations for robust AI evaluation.
• They employ advanced methodologies like LLM-driven automated annotations, synthetic data generation, and human-in-loop processes to capture GUI interactions accurately.
• Benchmark protocols and metrics from these datasets guide model development by assessing sequential reasoning, spatial localization, and error taxonomy in complex workflows.

Scene-aware and GUI-grounded datasets are foundational resources designed to benchmark and advance artificial intelligence systems—especially vision-LLMs—in understanding, localizing, and acting within complex digital environments. These datasets provide explicit mappings between GUI scenes (screenshots, view hierarchies, or videos) and granular semantic or procedural annotations, enabling quantitative assessment of grounding, sequential reasoning, action prediction, and robustness in the presence of real-world and workflow-driven context.

1. Core Definitions and Dataset Structures

Scene-aware datasets encode not only the visual state of a GUI (such as screenshots or window layouts) but also the evolving structure of the scene across interactions or workflows. GUI-grounded datasets provide explicit spatial/semantic correspondence between regions of the screen and actions (clicks, typing, navigation), often through bounding boxes, element trees, or multi-level descriptions.

Key components across recent benchmarks include:

• GUI states: Images, accessibility trees (AXTree), or DOM snapshots capturing all interactable elements and their properties (Li et al., 4 Feb 2025, Mu et al., 6 Nov 2025).
• Action records: Precise logs of user or agent actions (e.g., click on $(x, y)$, select menu item, enter text), potentially with pre- and post-interaction state diffs (Li et al., 4 Feb 2025, Lin et al., 19 Sep 2025).
• Scene sequences: Temporally ordered keyframes or videos, reflecting interface transitions in response to actions—critical for realistic, sequential task modeling (Shakeel et al., 20 Mar 2026, Sun et al., 22 Mar 2025).
• Annotations: Bounding boxes $(x, y, w, h)$, region captions, semantic descriptions, and chain-of-thought (CoT) rationales allow task, intent, and context to be encoded for each interaction (Chawla et al., 2024, Xu et al., 14 Mar 2025).
• Workflow/task definitions: Ordered sequences of intent-driven steps mimicking real-world operations, particularly in high-stakes or professional GUI environments (Shakeel et al., 20 Mar 2026, Mu et al., 6 Nov 2025).

2. Data Collection and Annotation Methodologies

Scene-aware and GUI-grounded datasets leverage diverse annotation strategies to achieve scale, precision, and contextual richness:

• Automated annotation with LLMs: LLMs are used to infer functional intent of GUI elements by analyzing state transitions before and after interactions, producing detailed natural language descriptions at scale (Li et al., 4 Feb 2025). This includes multi-stage LLM prompts for rejection, verification, and annotation as in AutoGUI.
• Domain-expert task execution: In sensitive domains (e.g., clinical GUIs), workflows are enacted by experts and video-recorded, with key decision frames extracted for annotation (Shakeel et al., 20 Mar 2026).
• Synthetic data generation: Frameworks such as EDGE synthesize large quantities of multi-granularity data via scripted rendering and annotation of public web pages, pairing screenshot regions with semantic queries or referential tasks (Chen et al., 2024).
• Semi-automated pipelines: Mixed pipelines combine RPA tools for logging interactions (screenshots, element locations) with LLM-generated step summaries, followed by human post-editing (Yang et al., 17 Jun 2025).
• Trajectory synthesis: GUI-ReWalk implements a two-phase process—Markov-chain-based random walks over the GUI state space for exploration, then LLM-driven, intent-aware reasoning for goal completion—producing realistic, diverse, executable trajectories (Lin et al., 19 Sep 2025).
• Structured annotation tools: Instruments like NEXTAG (Next Action Grounding and Annotation Tool) provide real-time capture of user actions and bounding box groundings, facilitating high-fidelity action-grounded logging (Chawla et al., 2024).

3. Benchmarking Protocols and Evaluation Metrics

Quantitative evaluation in scene-aware and GUI-grounded settings demands precise protocols that capture not only per-step localization but also sequential robustness and error propagation. Notable methodologies include:

• Step-wise and sequential evaluation: Measures such as Step Hit Rate (SHR), Task Completion Accuracy (TCA) or sequential accuracy $\mathrm{Acc}_{\mathrm{seq}}$, and Weighted Prefix Score (WPS) reflect the fraction and correctness of multi-step workflow execution, especially under strict early-termination protocols (Shakeel et al., 20 Mar 2026).
• Error taxonomy: Fine-grained failure analyses are used for diagnostic purposes, categorizing errors as no prediction, small target, near miss, edge bias, toolbar confusion, or outright semantic failure (Shakeel et al., 20 Mar 2026).
• Grounding accuracy: For single-step grounding tasks, accuracy is computed as the fraction of predictions $\hat{p}_t$ falling within the ground-truth bounding box $B_t$ (Mu et al., 6 Nov 2025, Li et al., 4 Feb 2025).
• Screen parsing and action prediction: In complex benchmarks, joint evaluation of object detection (F1, IoU), semantic matching (cosine similarity of element names), and action triple correctness ($\hat{f}, \hat{a}, \hat{s}$) is performed (Mu et al., 6 Nov 2025).
• Robustness to anomalies: Datasets such as GUI-Robust explicitly benchmark agents under seven anomaly types (e.g., modal popups, network failures), with metrics stratified by scenario (Yang et al., 17 Jun 2025).
• Diversity/entropy metrics: Trajectory entropy, defined as $H=-\sum p_i \log p_i$ over state/action visitations, quantifies coverage and stochasticity in exploration-based datasets (Lin et al., 19 Sep 2025).

4. Dataset Landscape and Comparative Features

Table summarizing distinctive features among prominent scene-aware and GUI-grounded datasets:

Dataset	Key Modalities	Task Granularity	Sequential/Workflow	Anomalies	Workflow Domain
MedSPOT (Shakeel et al., 20 Mar 2026)	Video, keyframes, semantics	Multi-step GUI, click	Yes (strict protocol)	No	Clinical
AutoGUI (Li et al., 4 Feb 2025)	Screenshots, AXTree, LLM fx	Single-step + context	No	No	Web, Mobile
GUI-360° (Mu et al., 6 Nov 2025)	Screens, a11y, reasoning	Multi-modal, 3 tasks	Yes	No	Desktop/Office
GUIDE (Chawla et al., 2024)	Screenshots, CoT, bounding	Stepwise, CoT-linked	No	No	Web
GUI-Xplore (Sun et al., 22 Mar 2025)	Video, VHs, logs	Multi-choice, temporal	Indirectly	No	Android apps
GUI-Robust (Yang et al., 17 Jun 2025)	Screens, step desc, action	Robustness, anomaly	Yes	Yes	Web, Desktop
DeskVision (Xu et al., 14 Mar 2025)	Screens, region captions	Region, caption	No	No	Desktop/Multi-OS
OSWorld-G/Jedi (Xie et al., 19 May 2025)	Screens, instructions	Component, layout, icon	No	No	Desktop
GUI-ReWalk (Lin et al., 19 Sep 2025)	Screenshots, UI trees	Multi-stride, goal	Yes	No	Desktop, Mobile
EDGE (Chen et al., 2024)	Synthetic screens, QA	Multi-level, refer/synth	No	No	Web-derived

Distinctive findings include:

• MedSPOT is unique in workflow-driven, step-dependent evaluation and explicit error taxonomy for clinical GUI environments (Shakeel et al., 20 Mar 2026).
• GUI-360° and OSWorld-G/Jedi span large-scale, real-world office/desktop environments, linking grounding, parsing, and GUI/API action semantics (Mu et al., 6 Nov 2025, Xie et al., 19 May 2025).
• GUI-Robust prioritizes anomaly diversity, uniquely measuring model degradation in non-ideal GUI scenes (Yang et al., 17 Jun 2025).
• GUIDE and AutoGUI emphasize detailed action planning and functional intent, leveraging CoT and LLM-based annotation to support robust agent training (Chawla et al., 2024, Li et al., 4 Feb 2025).
• EDGE exemplifies fully synthetic, multi-granularity supervision with massive web-scale QA, facilitating transfer to desktop/mobile GUIs (Chen et al., 2024).
• DeskVision provides dense scene-level region-captioned annotations from real multi-OS environments, supporting both spatial and relational reasoning (Xu et al., 14 Mar 2025).
• GUI-ReWalk formalizes a two-phase stochastic/reasoning generation paradigm, with scene diversity metrics and long-horizon cross-app task composition (Lin et al., 19 Sep 2025).

5. Implications for Model Development and Error Analysis

The integration of scene-awareness and GUI grounding within large, richly annotated datasets has catalyzed advances in robust vision-LLMs (VLMs) and agentic systems. Key practical insights include:

• Compositional generalization: Multi-perspective data (icon-level, component-level, layout-level) fosters the acquisition of reusable primitives, supporting generalization to unseen GUI designs (Xie et al., 19 May 2025).
• Layout and context modeling: Explicit annotation of interface hierarchies, region captions, and inter-element spatial relations reduces ambiguity and enhances reasoning over complex scenes (Xu et al., 14 Mar 2025, Mu et al., 6 Nov 2025).
• Sequential robustness: Early-termination protocols and compounded error evaluation in benchmarks like MedSPOT directly measure the brittleness of current models as task sequences lengthen; even small per-step error rates result in exponential failure rates over multi-step workflows (Shakeel et al., 20 Mar 2026).
• Error taxonomy and targeted improvement: Category-wise failure analysis (e.g., near miss, edge bias, toolbar confusion) directs module-level diagnostics and motivates spatial or semantic policy enhancements (Shakeel et al., 20 Mar 2026, Yang et al., 17 Jun 2025).
• Robustness to disturbance: Data covering anomalies (popups, network issues) is critical—models often excel in action-type prediction but fail in spatial localization or in adapting to disrupted scenes (Yang et al., 17 Jun 2025).
• Scaling effects: Empirical studies confirm that larger, more diverse datasets (e.g., scaling from 25K to 700K in AutoGUI) can increase grounding accuracy from single-digit to over 60% success rates, with diminishing returns at the upper end (Li et al., 4 Feb 2025, Xie et al., 19 May 2025).
• Hybrid action semantics: Unifying GUI primitives (e.g., click, scroll) with semantic API-level calls enables agents to flexibly exploit semantic knowledge when available (Mu et al., 6 Nov 2025).

6. Current Limitations and Future Directions

Despite the rapid progress, several challenges persist in the construction and deployment of scene-aware, GUI-grounded datasets:

• Modality gaps: Some datasets lack sequential trajectories, region-level hierarchy metadata, or executable action labels, limiting their utility for sophisticated agent training (Xu et al., 14 Mar 2025, Sun et al., 22 Mar 2025).
• Workflow depth: Many benchmarks remain focused on single-step or shallow multi-step tasks, with few matching the full procedural complexity of domain-critical workflows found in clinical or enterprise software (Shakeel et al., 20 Mar 2026, Lin et al., 19 Sep 2025).
• Human-in-the-loop constraints: Annotation at large scale is still resource-intensive unless automated pipelines are used; LLM-based verification and filtering mitigate, but do not eliminate, noise (Li et al., 4 Feb 2025, Yang et al., 17 Jun 2025).
• Semantic richness: Most grounding tasks remain spatial or referential, with less emphasis on deep semantic intent or functional correctness (e.g., models may localize a “Save” button without understanding its contextual significance).
• Evaluation stratification: Current protocols rarely weight errors by risk or workflow criticality—misclicks on benign controls and catastrophic mistakes in high-risk dialogs are currently scored equally (Shakeel et al., 20 Mar 2026).
• Cross-platform generalization: OS, device, and browser diversity is partially addressed in some datasets, but coverage remains incomplete for agents deployed in global or BYOD (bring-your-own-device) contexts (Chawla et al., 2024).
• Privacy and licensing: App recording and annotation at scale, particularly in proprietary or privacy-sensitive environments, introduces obstacles (Sun et al., 22 Mar 2025).

A plausible implication is that continued progress will require not only larger and more diverse datasets but also advances in multi-modal scene graph construction, hierarchical workflow annotation, and dynamic scenario simulation (e.g., generating “clinically critical” tasks or “failure recovery” episodes). Integrating scene-aware benchmarks with real-world deployment data—coupled with open, reproducible annotation standards—will further accelerate the development of reliable, human-aligned GUI agents.