GUIrilla-Gold: Verified macOS UI Benchmark

• GUIrilla-Gold is a human-verified benchmark subset derived from GUIrilla-Task, providing a clear standard for desktop UI automation on macOS.
• It is constructed from a non-overlapping test split with 1,283 tasks, ensuring rigorous manual correction of task feasibility and accessibility metadata.
• Quantitative metrics, including a 90.26% human baseline and high grounding accuracy, highlight its effectiveness in validating synthetic task generation.

GUIrilla-Gold is the manually verified, human-curated benchmark subset released alongside the larger automatically generated GUIrilla-Task dataset within the GUIrilla framework for desktop UI exploration on macOS. In the paper’s framing, GUIrilla is the automated crawling framework, GUIrilla-Task is the large-scale synthetic dataset produced by that crawler, and GUIrilla-Gold is the quality-checked benchmark derived from a portion of that data to measure how reliable the automatically generated tasks and accessibility metadata are (Garkot et al., 16 Oct 2025). The benchmark is introduced as “GUIrilla-Gold (1,283 human-verified tasks) with a 90.26\% human baseline,” and is described elsewhere as a “manully edited dataset,” establishing it as a gold-standard evaluation set rather than a full crawl or a purely synthetic corpus.

1. Definition and position within the GUIrilla project

GUIrilla-Gold is a human-edited, human-verified benchmark subset intended to serve as a gold-standard evaluation set for desktop UI tasks on macOS (Garkot et al., 16 Oct 2025). Its defining role is comparative: GUIrilla-Task is the scalable resource generated automatically from the crawler, whereas GUIrilla-Gold is the benchmark-grade reference subset used to assess task validity, annotation quality, and the reliability of accessibility-derived supervision.

The distinction is central to the project’s architecture. GUIrilla-Task contains 27,171 tasks across 1,108 applications, with about 6.8K unique screens, and is positioned as the large-scale training resource. GUIrilla-Gold, by contrast, contains 1,283 human-verified tasks and is used to establish trustworthy evaluation. This separation implies a methodological division between scalable synthetic generation and benchmark curation. A plausible implication is that GUIrilla-Gold functions as the primary mechanism for testing whether the crawler’s output is sufficiently accurate for downstream desktop automation research.

The benchmark is specifically framed around full-desktop, multi-application UI interaction on macOS. That focus matters because the larger paper situates macOS as “an ecosystem with limited representation in current UI datasets,” and GUIrilla-Gold therefore inherits both the desktop-level scope and the platform specificity of the broader framework (Garkot et al., 16 Oct 2025).

2. Construction from the test split

GUIrilla-Gold is constructed from the test split of GUIrilla-Task (Garkot et al., 16 Oct 2025). The appendix states that the collected tasks were split so that test applications do not overlap with train applications, and that the test applications were larger and more complex. The split is reported as 881 applications with 25,606 entries in train and 227 applications with 1,565 task entries in test.

The gold subset is created by giving annotators the test split data for inspection and correction. The paper states that 5 annotators were hired, and that annotators with accessibility expertise reviewed each data entry along five dimensions:

1. task feasibility
2. task instruction clarity and editing for ambiguity
3. manual task execution
4. accessibility tree quality rating (Good/Medium/Bad scale)
5. element-level verification of semantic properties (role, description, value) and bounding-box accuracy

The verification criteria are deliberately restrictive. Annotators first assess whether the task is clear and executable; they mark it DOABLE only if the element is visible and the action can actually be performed; they attempt execution exactly once; they verify semantic labels, roles, and bounding boxes; and they rate accessibility-tree quality on a 1–3 scale. The guidelines explicitly state, “Attempt to execute the task exactly once to verify correctness” and “Multi-step Tasks: Mark as NOT DOABLE if requiring multiple distinct actions” (Garkot et al., 16 Oct 2025).

These constraints define GUIrilla-Gold as a benchmark of human-verified task feasibility and annotation quality, not merely a relabeled subset. This suggests that the benchmark is designed to filter out two distinct failure modes in synthetic desktop data: instruction-level ambiguity and grounding-level metadata errors.

3. Dataset structure and sample representation

GUIrilla-Gold inherits the multimodal structure of GUIrilla-Task. Its samples are grounded in the same desktop observations and accessibility-derived representations used throughout the GUIrilla pipeline (Garkot et al., 16 Oct 2025). The paper’s representative sample list gives the following fields:

Field category	Reported fields
Identification and tasking	Screen ID; App Name; Task; Original Task
Action and grounding	Raw Action; Action; Element Data; A11y Path
Visual and metadata context	Scaling Factor; Image; Cropped Image; Segmented Image; Task Category; Element Category

The dataset format therefore couples task text, structured actions, accessibility information, and image observations. The paper’s wording specifies: “Screen ID … App Name … Task … Raw Action … Action … Element Data … Scaling Factor … Original Task … A11y Path … Image … Cropped Image … Segmented Image … Task Category … Element Category” (Garkot et al., 16 Oct 2025).

For GUIrilla-Gold, the critical distinction is not a unique schema but the fact that the included tasks are manually corrected and validated. The benchmark is therefore intended to reflect the true task an agent should perform rather than a noisy synthetic instruction. In practical terms, this makes the benchmark suitable for evaluating whether a model can map a full-desktop screenshot and accessibility trace to a functionally correct action grounded in the intended element.

4. Quantitative properties and annotation outcomes

Several reported quantities define GUIrilla-Gold’s scale and its empirical role within the paper (Garkot et al., 16 Oct 2025). The benchmark contains 1,283 human-verified tasks and is associated with a 90.26\% human baseline. Because it is derived from the test split of 1,565 task entries, the benchmark represents a curated subset rather than the entire held-out partition.

Within the original English-language tasks in the verified subset, the paper reports that:

• 84.3\% were marked DOABLE
• 91\% of GPT-generated task strings required no change
• the 109 edited cases had 97\% similarity to the originals by Ratcliff/Obershelp

These figures indicate that the synthetic instruction generation process is often close to acceptable as-is, but not uniformly reliable. The fact that only 84.3\% were marked DOABLE shows that feasibility is a stricter criterion than mere textual plausibility. The fact that 91\% of task strings required no change, while a nontrivial subset still needed editing, further distinguishes instruction quality from execution validity.

A plausible implication is that GUIrilla-Gold is valuable precisely because it quantifies the residual gap between high-volume synthetic data generation and benchmark-grade desktop supervision. The benchmark does not merely certify correctness; it exposes the error profile of the automatic pipeline.

5. Accessibility metadata quality and the rationale for manual verification

GUIrilla-Gold is motivated in part by the paper’s finding that accessibility metadata is informative but noisy (Garkot et al., 16 Oct 2025). At the screen level, accessibility-tree quality is distributed as:

• 64\% GOOD
• 24\% MEDIUM
• 12\% BAD

At the element level, the paper reports that:

• 40\% have correct role and description pairs
• 49\% contain role information only
• 11\% are mislabeled

For spatial grounding, the benchmark analysis reports that bounding boxes are accurate for 80\% of elements, while 10\% extend outside the visible window.

These figures explain why a manually verified subset is necessary. If only 40\% of elements have correct role-description pairs and 11\% are mislabeled, then accessibility-only task generation cannot be assumed to yield benchmark-grade supervision. Likewise, imperfect bounding boxes can corrupt grounding evaluation even when task text is reasonable. The paper uses these findings to argue that accessibility-only generation is unreliable and recommends combining accessibility trees with screenshots and vision-based adjustment.

A common misconception would be to treat GUIrilla-Gold as merely a cleaner version of GUIrilla-Task. The reported quality analysis suggests a more specific function: it is a diagnostic benchmark for measuring how synthetic task generation interacts with imperfect accessibility structures. Its role is therefore epistemic as well as evaluative—it helps determine what parts of the automatic pipeline are dependable.

6. Relation to the crawler, hierarchical GUI graphs, and task synthesis

GUIrilla-Gold is the verified output of a broader automated system whose core mechanism is graph-based exploration of desktop interfaces (Garkot et al., 16 Oct 2025). The GUIrilla crawler uses the macOS Accessibility API to extract interface state and traverse applications by simulated interactions. It builds a hierarchical graph in which:

• nodes are UI states
• edges are actions leading from one state to another
• each node stores the accessibility tree, a screenshot, and available actions
• each edge stores the action description and the resulting state

The graph fields are defined explicitly. For nodes:

• “Element: The accessibility tree of the application window at a state.”
• “Image name: The filename of the full desktop screenshot associated with a state.”
• “Actions: A list of actions that can be executed without causing significant changes to the UI.”

For edges:

• “Action: Information about the UI element that triggered the interaction, along with a human-readable action description and a structured dictionary representation that has a 1-to-1 map to pyautogui commands.”
• “Out vertex: The resulting UI state after the interaction of the crawler with the GUI.”

The crawler also uses specialized handlers for pop-ups, invisible elements, unrolled menu items, and empty elements, as well as GPT-4-based agents for input generation, safe ordering, login handling, and task postprocessing (Garkot et al., 16 Oct 2025). GUIrilla-Gold matters in this context because it is the subset used to test whether this entire graph-construction and task-synthesis pipeline produces valid desktop tasks under realistic conditions.

This suggests that the benchmark is not independent of the crawler design; rather, it is the empirical check on that design. If the crawler’s graph states, action representations, or accessibility extractions are systematically flawed, GUIrilla-Gold is the instrument intended to reveal those flaws.

7. Benchmark role in desktop UI automation research

GUIrilla-Gold is intended as a gold benchmark for assessing desktop UI task feasibility, grounding quality, and instruction clarity on macOS (Garkot et al., 16 Oct 2025). In that role, it helps answer whether a model can understand a full-desktop screenshot, identify the correct interactive element, infer the function of the element rather than merely its appearance, and execute a task that is genuinely doable.

Its benchmark function is aligned with the broader evaluation logic of the paper. GUIrilla-Task is used to fine-tune GUIrilla-See models, and those models improve grounding performance on downstream benchmarks. Reported downstream results include:

• 90.33\% on ScreenSpot-v2
• 27.81\% on the macOS subset of ScreenSpot-Pro
• 75.59\% grounding accuracy on GUIrilla-Task test grounding
• 64.41\% overall on agentic tasks for OpenAI Computer Use

These are not reported as direct GUIrilla-Gold scores. Their significance is indirect: they contextualize GUIrilla-Gold as the benchmark that validates the quality of the synthetic supervision underlying those downstream gains. A plausible implication is that the benchmark’s value lies less in scale than in its function as a trusted calibration set for a much larger automatically collected corpus.

In summary, GUIrilla-Gold is the benchmark-grade core of the GUIrilla project: a 1,283-task macOS evaluation set built from the crawler’s test split, manually audited for feasibility and semantic correctness, and designed to measure desktop UI automation under full-desktop conditions. It formalizes the distinction between scalable automated collection and trustworthy evaluation, and it operationalizes the paper’s central claim that large-scale desktop UI data can be collected automatically but must still be checked for quality (Garkot et al., 16 Oct 2025).