---
title: 'GUIrilla-Task: MacOS GUI Task Dataset'
url: https://www.emergentmind.com/topics/guirilla-task
type: topic
---

# GUIrilla-Task: MacOS GUI Task Dataset

GUIrilla-Task is a large-scale, function-oriented macOS desktop GUI task dataset generated from the GUIrilla framework for automated desktop UI exploration. It is built by systematically crawling real macOS applications through the native accessibility layer, recording UI states and transitions as hierarchical GUI graphs, and converting those transitions into “functionally grounded tasks” linked to full-desktop and window-level screenshots, accessibility metadata, and semantic action traces. The dataset is explicitly positioned as a response to the limited scalability of manual annotation, the mismatch between single-window datasets and real desktop usage, and the lack of open, reproducible collection pipelines for desktop GUI automation [2510.16051].

## 1. Definition and scope

GUIrilla-Task is defined as a dataset release derived from the broader GUIrilla framework for automated desktop UI exploration. Its central unit is a task instance that pairs a full-desktop screenshot, the corresponding accessibility tree or metadata, a natural-language instruction, and a ground-truth action such as a click or text input [2510.16051]. In the abstract, the release is described as “a large-scale dataset of 27,171 functionally grounded tasks across 1,108 macOS applications, each annotated with full-desktop and window-level screenshots, accessibility metadata, and semantic action traces” [2510.16051].

The dataset is intended for at least two closely related settings. The first is desktop GUI grounding, in which a model must map a natural-language instruction to the correct UI element in a screenshot. The second is autonomous agent research, in which a model must output the correct low-level action, such as clicking at the correct coordinates or typing the correct text into the correct field [2510.16051]. The paper also emphasizes full-desktop and multi-window realism rather than isolated widget crops, making the dataset relevant to desktop autonomy settings that operate over overlapping windows, dialogs, and system widgets [2510.16051].

A concise summary of the reported dataset scale is given below.

| Property | Value |
|---|---:|
| Tasks | 27,171 |
| Applications | 1,108 |
| Unique UIs | 6,835 |

The train/test split reported in the appendix is application-disjoint: 881 applications with 25,606 task entries for training, and 227 applications with 1,565 task entries for testing. The test applications were selected to be more challenging because they contained larger, more complicated accessibility trees [2510.16051].

## 2. Collection framework and exploration pipeline

GUIrilla-Task is inseparable from the GUIrilla collection framework. GUIrilla is built on the macOS Accessibility API, which exposes UI elements in a hierarchical tree structure with properties such as name, role, description, position, and size [2510.16051]. The crawler uses this layer as the primary structural representation of application state, but the paper is explicit that accessibility metadata is often noisy or incomplete because app developers manually annotate it. Reported issues include stale elements that remain in the tree after disappearing, incorrect role classifications, inaccurate positions, and generic or missing labels [2510.16051].

The collection pipeline is described as a single-app processing pipeline with the following stages: application installation, automated application exploration, state capture through the macOS Accessibility API, construction of hierarchical application GUI graphs, postprocessing to remove redundant transitions, conversion of graph edges into task instances, and optional GPT-4-based rewriting into natural, function-oriented task text [2510.16051]. The crawler can operate deterministically or with GPT-4 assistance. Deterministic collection uses fixed element processing order and default input strings, whereas GPT-4 is used to improve safety and contextual relevance [2510.16051].

The crawler executes four interaction types: click, cursor move, type, and press Enter, using `pyautogui` [2510.16051]. User-configurable parameters include maximum parsing duration, maximum graph depth, whether to use deterministic text input, whether to use agents, whether to collect tasks, and whether to move the cursor before clicking [2510.16051]. The appendix reports a default maximum parsing duration of 2 hours, a default maximum parsing tree depth of 25, deterministic text input equal to `'DEFAULT'`, and task collection enabled by default [2510.16051].

Three GPT-4-based agents are described. The Input Agent generates contextually appropriate text for input fields based on the accessibility tree. The Order and Login Agent chooses a safe interaction order, prioritizing low-risk actions and flagging login pages or system-access-required states. The Task Agents run in postprocessing to clean duplicates, refine graph-derived task strings, and generate natural-language tasks [2510.16051]. The paper also introduces specialized handlers for pop-ups, invisible elements, dynamically unrolled menu items, and empty elements with missing metadata. These handlers materially affect coverage: the ablation reports that handlers increased task discovery by 5× in Stocks and 3× in Maps while reducing duplicates and processing time [2510.16051].

## 3. Graph representation and task derivation

GUIrilla represents each application as an interaction graph whose nodes are UI states and whose edges are interactions that transform one state into another [2510.16051]. A node stores the accessibility tree of the application window at that state, the filename of the full desktop screenshot, and a list of actions executable without causing significant UI changes. The paper defines significant change as the addition or removal of more than 10 UI elements following an interaction [2510.16051]. An edge stores the triggering UI element, a human-readable action description, a structured dictionary representation with a 1-to-1 mapping to `pyautogui` commands, and the resulting UI state [2510.16051].

The paper does not give a formal graph equation, but its prose description supports the following structural reconstruction:
$$
G = (V, E)
$$
with a state node
$$
v = (\text{AX}_v, I_v, A_v)
$$
and an edge
$$
e = (v_i, a, v_j).
$$
This suggests a state-transition dataset in which graph edges are the immediate source of training examples. A plausible implication is that GUIrilla-Task can be treated as a graph-derived grounding corpus rather than a corpus of manually authored demonstrations.

Task generation begins by filtering out actions that did not produce observable UI changes. Raw graph-edge strings are then converted into direct descriptions of what was clicked or typed. A Task Agent subsequently rewrites these raw strings into more natural, function-oriented language suitable for grounding and instruction following. The postprocessor uses both the target element’s accessibility representation and its visual appearance in the screenshot. Task generation is divided into click-based task generation for visible UI element interactions and text-input task generation for keyboard-input tasks in appropriate fields [2510.16051].

The graph is sequential and hierarchical, but the released task instances appear primarily to be single interaction tasks. This is reflected in the evaluation protocol, where click success is determined by whether predicted coordinates fall inside a target bounding box and input success requires an exact text match [2510.16051]. This suggests that GUIrilla-Task is directly optimized for grounded single-step interaction, even though the upstream graph structure is multi-step.

## 4. Annotation schema and functional grounding

The appendix gives a detailed per-example schema. Each task entry includes Screen ID, App Name, Task, Raw Action, Action, Element Data, Scaling Factor, Original Task, A11y Path, Image, Cropped Image, Segmented Image, Task Category, and Element Category [2510.16051]. Visual annotations therefore include full-desktop screenshots, application-window crops, and segmented images. Accessibility annotations include the full accessibility tree and target-element metadata in JSON. Action annotations include both human-readable action descriptions and structured actions. Semantic labels include the rewritten task text plus category assignments [2510.16051].

A faithful abstraction of an example can be written as
$$
\tau = (s, t, y),
$$
where
$$
s = (I_{\text{full}}, I_{\text{crop}}, \text{AX}, m_{\text{app}})
$$
is the UI state representation, $t$ is the natural-language task, and $y$ is the target action [2510.16051]. For click tasks, the action includes a coordinate and target element grounding; for input tasks, it includes the exact string and the associated input field [2510.16051].

The paper’s central semantic claim is that GUIrilla-Task consists of “functionally grounded tasks.” This means the task description should express what the UI element does in the application context rather than merely its appearance or literal raw accessibility label [2510.16051]. The click-task prompt explicitly instructs that “The task must describe the function, not the appearance of the element,” preferring “Create a new document” over “Click the grey + button” [2510.16051]. Examples in the paper contrast simple actions such as “open settings” with more contextual tasks such as “change your working hours to end at 18:00” [2510.16051].

Functional grounding is produced by combining actual executed interactions, screen context, accessibility metadata, screenshot evidence, and GPT-4 rewriting into purpose-oriented language [2510.16051]. The paper argues that this is substantially better than deterministic accessibility-only strings. In an ablation with Florence-0.7B, GPT-generated tasks yield 53.55% accuracy, while accessibility-based tasks yield 40.35%, a 13-point gap [2510.16051]. The paper interprets this as evidence that functional supervision outperforms surface-level properties.

## 5. Empirical use as training data and benchmark

GUIrilla-Task is used both as a training corpus and as an evaluation benchmark. Fine-tuning on GUIrilla-Task substantially improves GUI grounding performance for several backbones. Florence Large (0.7B) improves from 8.31% to 53.55%; Qwen 2.5 VL (3B) improves from 18.40% to 73.48%; and Qwen 2.5 VL (7B) improves from 35.78% to 75.59% [2510.16051]. The resulting models are referred to as GUIrilla-See in the paper’s evaluation [2510.16051].

On the GUIrilla-Task test set, overall grounding accuracy is reported as 47.53% for UI-TARS 2B, 69.07% for UI-TARS 1.5 7B, 64.86% for OS-Atlas 7B, 64.03% for UGround 2B, 69.46% for UGround 7B, 73.48% for GUIrilla-See 3B, and 75.59% for GUIrilla-See 7B [2510.16051]. The paper highlights category-level gains in Settings, Connectivity, Files, and Input, and reports particularly strong element-level results for buttons, menus, and checkboxes [2510.16051].

The paper also evaluates transfer to external grounding benchmarks. GUIrilla-See 7B reaches 90.33% on ScreenSpot-v2 and 23.66% overall on ScreenSpot-Pro, including 27.81% on the macOS subset [2510.16051]. The paper argues that this performance is notable given the much smaller training corpus relative to baselines such as OS-Atlas [2510.16051]. It also reports cross-OS transfer: GUIrilla-See 7B, trained only on macOS, achieves 21.7% on Windows ScreenSpot-Pro and 27.8% on macOS, outperforming OS-Atlas 7B and UGround 7B on those compared settings [2510.16051].

In an agentic evaluation setting, the task success criterion is action correctness: click inside the target bounding box for click tasks and exact text match for input tasks [2510.16051]. Reported overall success rates include 64.41% for OpenAI Computer Use, 61.53% for Claude Computer Use, 58.85% for OS-Atlas-Pro-7B, 50.86% for UI-TARS 1.5 7B, and 36.55% for Qwen 2.5 VL 7B [2510.16051]. The paper notes that all models struggled with input tasks, with a maximum of 12.5% success [2510.16051].

GUIrilla-Gold serves as the manually verified subset. The abstract reports 1,283 human-verified tasks, while the results section refers to 1,319 original English language-based tasks in the annotation effort [2510.16051]. Five annotators reviewed task feasibility, instruction clarity, manual execution, accessibility quality, and element-level semantics and bounding boxes. Of the 1,319 original English tasks, 84.3% were marked DOABLE, 91% of GPT task strings required no change, and edited tasks remained 97% similar to the originals [2510.16051]. The abstract also reports a 90.26% human baseline for GUIrilla-Gold [2510.16051].

## 6. Position within GUI-agent research and limitations

GUIrilla-Task occupies a distinct place in GUI-agent research because it is desktop-focused, macOS-specific, graph-derived, and function-oriented. This differentiates it from mobile conversational datasets such as META-GUI, which pairs multi-turn dialogue with real Android GUI traces [2205.11029], and from Windows productivity benchmarks such as AssistGUI, which contains 100 tasks from nine applications and evaluates long-horizon desktop automation through mouse and keyboard control [2312.13108]. It also differs from GUI Testing Arena, which targets automated GUI testing through test intention generation, test task execution, and GUI defect detection in mobile applications [2412.18426]. A plausible implication is that GUIrilla-Task is best understood not as a full long-horizon execution benchmark, but as a large-scale desktop grounding and single-step action corpus with unusually strong application coverage.

The strengths claimed in the paper are openness, scalability, functional rather than superficial supervision, full-desktop realism, and macOS coverage [2510.16051]. The release includes the full GUIrilla-Task dataset, the GUIrilla-Gold benchmark, the GUIrilla framework code, training and evaluation code, and the open-source macapptree library for reproducible collection of structured accessibility metadata [2510.16051].

The limitations are also explicit. The release is fundamentally macOS-specific [2510.16051]. It depends heavily on accessibility APIs, even though the paper documents substantial metadata noise: 64% GOOD screen-level accessibility quality, 24% MEDIUM, and 12% BAD; at the element level, only 40% have correct role-description pairs, 49% contain role only, and 11% are mislabeled; for bounding boxes, 80% are accurate and 10% extend outside the visible window [2510.16051]. Coverage is incomplete: of 12,298 attempted apps, only 1,108 were retained because the rest could not be installed, did not support Accessibility, or did not yield interaction graphs [2510.16051]. The failure analysis highlights weak coverage for icon-dense engineering tools and creative software such as Vivado, Illustrator, and DaVinci Resolve [2510.16051]. The task instances are also primarily single-step click or type actions, even though the underlying graph is multi-step [2510.16051].

Within the broader literature on GUI agents, this suggests a specific tradeoff. GUIrilla-Task provides much broader desktop application coverage than manually curated desktop benchmarks, but it inherits the structural noise and application-selection bias of accessibility-driven crawling. The paper’s strongest empirical claim is therefore not that the dataset eliminates these problems, but that realistic function-level supervision and full-desktop structure can matter more than sheer dataset size for desktop GUI grounding [2510.16051].

Source: https://www.emergentmind.com/topics/guirilla-task