GUIrilla-Task: MacOS GUI Task Dataset

• GUIrilla-Task is a dataset of functionally grounded tasks derived from automated macOS UI exploration, featuring 27,171 tasks across 1,108 applications.
• It systematically captures full-desktop screenshots, hierarchical accessibility trees, and semantic action traces to support realistic GUI grounding and autonomous agent research.
• The dataset addresses scalability issues of manual annotation and limitations of single-window datasets, offering a comprehensive benchmark for desktop GUI automation.

GUIrilla-Task is a large-scale, function-oriented macOS desktop GUI task dataset generated from the GUIrilla framework for automated desktop UI exploration. It is built by systematically crawling real macOS applications through the native accessibility layer, recording UI states and transitions as hierarchical GUI graphs, and converting those transitions into “functionally grounded tasks” linked to full-desktop and window-level screenshots, accessibility metadata, and semantic action traces. The dataset is explicitly positioned as a response to the limited scalability of manual annotation, the mismatch between single-window datasets and real desktop usage, and the lack of open, reproducible collection pipelines for desktop GUI automation (Garkot et al., 16 Oct 2025).

1. Definition and scope

GUIrilla-Task is defined as a dataset release derived from the broader GUIrilla framework for automated desktop UI exploration. Its central unit is a task instance that pairs a full-desktop screenshot, the corresponding accessibility tree or metadata, a natural-language instruction, and a ground-truth action such as a click or text input (Garkot et al., 16 Oct 2025). In the abstract, the release is described as “a large-scale dataset of 27,171 functionally grounded tasks across 1,108 macOS applications, each annotated with full-desktop and window-level screenshots, accessibility metadata, and semantic action traces” (Garkot et al., 16 Oct 2025).

The dataset is intended for at least two closely related settings. The first is desktop GUI grounding, in which a model must map a natural-language instruction to the correct UI element in a screenshot. The second is autonomous agent research, in which a model must output the correct low-level action, such as clicking at the correct coordinates or typing the correct text into the correct field (Garkot et al., 16 Oct 2025). The paper also emphasizes full-desktop and multi-window realism rather than isolated widget crops, making the dataset relevant to desktop autonomy settings that operate over overlapping windows, dialogs, and system widgets (Garkot et al., 16 Oct 2025).

A concise summary of the reported dataset scale is given below.

Property	Value
Tasks	27,171
Applications	1,108
Unique UIs	6,835

The train/test split reported in the appendix is application-disjoint: 881 applications with 25,606 task entries for training, and 227 applications with 1,565 task entries for testing. The test applications were selected to be more challenging because they contained larger, more complicated accessibility trees (Garkot et al., 16 Oct 2025).

2. Collection framework and exploration pipeline

GUIrilla-Task is inseparable from the GUIrilla collection framework. GUIrilla is built on the macOS Accessibility API, which exposes UI elements in a hierarchical tree structure with properties such as name, role, description, position, and size (Garkot et al., 16 Oct 2025). The crawler uses this layer as the primary structural representation of application state, but the paper is explicit that accessibility metadata is often noisy or incomplete because app developers manually annotate it. Reported issues include stale elements that remain in the tree after disappearing, incorrect role classifications, inaccurate positions, and generic or missing labels (Garkot et al., 16 Oct 2025).

The collection pipeline is described as a single-app processing pipeline with the following stages: application installation, automated application exploration, state capture through the macOS Accessibility API, construction of hierarchical application GUI graphs, postprocessing to remove redundant transitions, conversion of graph edges into task instances, and optional GPT-4-based rewriting into natural, function-oriented task text (Garkot et al., 16 Oct 2025). The crawler can operate deterministically or with GPT-4 assistance. Deterministic collection uses fixed element processing order and default input strings, whereas GPT-4 is used to improve safety and contextual relevance (Garkot et al., 16 Oct 2025).

The crawler executes four interaction types: click, cursor move, type, and press Enter, using pyautogui (Garkot et al., 16 Oct 2025). User-configurable parameters include maximum parsing duration, maximum graph depth, whether to use deterministic text input, whether to use agents, whether to collect tasks, and whether to move the cursor before clicking (Garkot et al., 16 Oct 2025). The appendix reports a default maximum parsing duration of 2 hours, a default maximum parsing tree depth of 25, deterministic text input equal to 'DEFAULT', and task collection enabled by default (Garkot et al., 16 Oct 2025).

Three GPT-4-based agents are described. The Input Agent generates contextually appropriate text for input fields based on the accessibility tree. The Order and Login Agent chooses a safe interaction order, prioritizing low-risk actions and flagging login pages or system-access-required states. The Task Agents run in postprocessing to clean duplicates, refine graph-derived task strings, and generate natural-language tasks (Garkot et al., 16 Oct 2025). The paper also introduces specialized handlers for pop-ups, invisible elements, dynamically unrolled menu items, and empty elements with missing metadata. These handlers materially affect coverage: the ablation reports that handlers increased task discovery by 5× in Stocks and 3× in Maps while reducing duplicates and processing time (Garkot et al., 16 Oct 2025).

3. Graph representation and task derivation

GUIrilla represents each application as an interaction graph whose nodes are UI states and whose edges are interactions that transform one state into another (Garkot et al., 16 Oct 2025). A node stores the accessibility tree of the application window at that state, the filename of the full desktop screenshot, and a list of actions executable without causing significant UI changes. The paper defines significant change as the addition or removal of more than 10 UI elements following an interaction (Garkot et al., 16 Oct 2025). An edge stores the triggering UI element, a human-readable action description, a structured dictionary representation with a 1-to-1 mapping to pyautogui commands, and the resulting UI state (Garkot et al., 16 Oct 2025).

The paper does not give a formal graph equation, but its prose description supports the following structural reconstruction:

$G = (V, E)$

with a state node

$v = (\text{AX}_v, I_v, A_v)$

and an edge

$e = (v_i, a, v_j).$

This suggests a state-transition dataset in which graph edges are the immediate source of training examples. A plausible implication is that GUIrilla-Task can be treated as a graph-derived grounding corpus rather than a corpus of manually authored demonstrations.

Task generation begins by filtering out actions that did not produce observable UI changes. Raw graph-edge strings are then converted into direct descriptions of what was clicked or typed. A Task Agent subsequently rewrites these raw strings into more natural, function-oriented language suitable for grounding and instruction following. The postprocessor uses both the target element’s accessibility representation and its visual appearance in the screenshot. Task generation is divided into click-based task generation for visible UI element interactions and text-input task generation for keyboard-input tasks in appropriate fields (Garkot et al., 16 Oct 2025).

The graph is sequential and hierarchical, but the released task instances appear primarily to be single interaction tasks. This is reflected in the evaluation protocol, where click success is determined by whether predicted coordinates fall inside a target bounding box and input success requires an exact text match (Garkot et al., 16 Oct 2025). This suggests that GUIrilla-Task is directly optimized for grounded single-step interaction, even though the upstream graph structure is multi-step.

4. Annotation schema and functional grounding

The appendix gives a detailed per-example schema. Each task entry includes Screen ID, App Name, Task, Raw Action, Action, Element Data, Scaling Factor, Original Task, A11y Path, Image, Cropped Image, Segmented Image, Task Category, and Element Category (Garkot et al., 16 Oct 2025). Visual annotations therefore include full-desktop screenshots, application-window crops, and segmented images. Accessibility annotations include the full accessibility tree and target-element metadata in JSON. Action annotations include both human-readable action descriptions and structured actions. Semantic labels include the rewritten task text plus category assignments (Garkot et al., 16 Oct 2025).

A faithful abstraction of an example can be written as

$\tau = (s, t, y),$

where

$$s = (I_{\text{full}}, I_{\text{crop}}, \text{AX}, m_{\text{app}})$$

is the UI state representation, $t$ is the natural-language task, and $y$ is the target action (Garkot et al., 16 Oct 2025). For click tasks, the action includes a coordinate and target element grounding; for input tasks, it includes the exact string and the associated input field (Garkot et al., 16 Oct 2025).

The paper’s central semantic claim is that GUIrilla-Task consists of “functionally grounded tasks.” This means the task description should express what the UI element does in the application context rather than merely its appearance or literal raw accessibility label (Garkot et al., 16 Oct 2025). The click-task prompt explicitly instructs that “The task must describe the function, not the appearance of the element,” preferring “Create a new document” over “Click the grey + button” (Garkot et al., 16 Oct 2025). Examples in the paper contrast simple actions such as “open settings” with more contextual tasks such as “change your working hours to end at 18:00” (Garkot et al., 16 Oct 2025).

Functional grounding is produced by combining actual executed interactions, screen context, accessibility metadata, screenshot evidence, and GPT-4 rewriting into purpose-oriented language (Garkot et al., 16 Oct 2025). The paper argues that this is substantially better than deterministic accessibility-only strings. In an ablation with Florence-0.7B, GPT-generated tasks yield 53.55% accuracy, while accessibility-based tasks yield 40.35%, a 13-point gap (Garkot et al., 16 Oct 2025). The paper interprets this as evidence that functional supervision outperforms surface-level properties.

5. Empirical use as training data and benchmark

GUIrilla-Task is used both as a training corpus and as an evaluation benchmark. Fine-tuning on GUIrilla-Task substantially improves GUI grounding performance for several backbones. Florence Large (0.7B) improves from 8.31% to 53.55%; Qwen 2.5 VL (3B) improves from 18.40% to 73.48%; and Qwen 2.5 VL (7B) improves from 35.78% to 75.59% (Garkot et al., 16 Oct 2025). The resulting models are referred to as GUIrilla-See in the paper’s evaluation (Garkot et al., 16 Oct 2025).

On the GUIrilla-Task test set, overall grounding accuracy is reported as 47.53% for UI-TARS 2B, 69.07% for UI-TARS 1.5 7B, 64.86% for OS-Atlas 7B, 64.03% for UGround 2B, 69.46% for UGround 7B, 73.48% for GUIrilla-See 3B, and 75.59% for GUIrilla-See 7B (Garkot et al., 16 Oct 2025). The paper highlights category-level gains in Settings, Connectivity, Files, and Input, and reports particularly strong element-level results for buttons, menus, and checkboxes (Garkot et al., 16 Oct 2025).

The paper also evaluates transfer to external grounding benchmarks. GUIrilla-See 7B reaches 90.33% on ScreenSpot-v2 and 23.66% overall on ScreenSpot-Pro, including 27.81% on the macOS subset (Garkot et al., 16 Oct 2025). The paper argues that this performance is notable given the much smaller training corpus relative to baselines such as OS-Atlas (Garkot et al., 16 Oct 2025). It also reports cross-OS transfer: GUIrilla-See 7B, trained only on macOS, achieves 21.7% on Windows ScreenSpot-Pro and 27.8% on macOS, outperforming OS-Atlas 7B and UGround 7B on those compared settings (Garkot et al., 16 Oct 2025).

In an agentic evaluation setting, the task success criterion is action correctness: click inside the target bounding box for click tasks and exact text match for input tasks (Garkot et al., 16 Oct 2025). Reported overall success rates include 64.41% for OpenAI Computer Use, 61.53% for Claude Computer Use, 58.85% for OS-Atlas-Pro-7B, 50.86% for UI-TARS 1.5 7B, and 36.55% for Qwen 2.5 VL 7B (Garkot et al., 16 Oct 2025). The paper notes that all models struggled with input tasks, with a maximum of 12.5% success (Garkot et al., 16 Oct 2025).

GUIrilla-Gold serves as the manually verified subset. The abstract reports 1,283 human-verified tasks, while the results section refers to 1,319 original English language-based tasks in the annotation effort (Garkot et al., 16 Oct 2025). Five annotators reviewed task feasibility, instruction clarity, manual execution, accessibility quality, and element-level semantics and bounding boxes. Of the 1,319 original English tasks, 84.3% were marked DOABLE, 91% of GPT task strings required no change, and edited tasks remained 97% similar to the originals (Garkot et al., 16 Oct 2025). The abstract also reports a 90.26% human baseline for GUIrilla-Gold (Garkot et al., 16 Oct 2025).

6. Position within GUI-agent research and limitations

GUIrilla-Task occupies a distinct place in GUI-agent research because it is desktop-focused, macOS-specific, graph-derived, and function-oriented. This differentiates it from mobile conversational datasets such as META-GUI, which pairs multi-turn dialogue with real Android GUI traces (Sun et al., 2022), and from Windows productivity benchmarks such as AssistGUI, which contains 100 tasks from nine applications and evaluates long-horizon desktop automation through mouse and keyboard control (Gao et al., 2023). It also differs from GUI Testing Arena, which targets automated GUI testing through test intention generation, test task execution, and GUI defect detection in mobile applications (Zhao et al., 2024). A plausible implication is that GUIrilla-Task is best understood not as a full long-horizon execution benchmark, but as a large-scale desktop grounding and single-step action corpus with unusually strong application coverage.

The strengths claimed in the paper are openness, scalability, functional rather than superficial supervision, full-desktop realism, and macOS coverage (Garkot et al., 16 Oct 2025). The release includes the full GUIrilla-Task dataset, the GUIrilla-Gold benchmark, the GUIrilla framework code, training and evaluation code, and the open-source macapptree library for reproducible collection of structured accessibility metadata (Garkot et al., 16 Oct 2025).

The limitations are also explicit. The release is fundamentally macOS-specific (Garkot et al., 16 Oct 2025). It depends heavily on accessibility APIs, even though the paper documents substantial metadata noise: 64% GOOD screen-level accessibility quality, 24% MEDIUM, and 12% BAD; at the element level, only 40% have correct role-description pairs, 49% contain role only, and 11% are mislabeled; for bounding boxes, 80% are accurate and 10% extend outside the visible window (Garkot et al., 16 Oct 2025). Coverage is incomplete: of 12,298 attempted apps, only 1,108 were retained because the rest could not be installed, did not support Accessibility, or did not yield interaction graphs (Garkot et al., 16 Oct 2025). The failure analysis highlights weak coverage for icon-dense engineering tools and creative software such as Vivado, Illustrator, and DaVinci Resolve (Garkot et al., 16 Oct 2025). The task instances are also primarily single-step click or type actions, even though the underlying graph is multi-step (Garkot et al., 16 Oct 2025).

Within the broader literature on GUI agents, this suggests a specific tradeoff. GUIrilla-Task provides much broader desktop application coverage than manually curated desktop benchmarks, but it inherits the structural noise and application-selection bias of accessibility-driven crawling. The paper’s strongest empirical claim is therefore not that the dataset eliminates these problems, but that realistic function-level supervision and full-desktop structure can matter more than sheer dataset size for desktop GUI grounding (Garkot et al., 16 Oct 2025).