GUIrilla: Scalable macOS GUI Exploration
- GUIrilla is a scalable framework for automated macOS UI exploration that systematically collects accessibility data and functionally grounded tasks.
- The system organizes native UI elements and interactions into hierarchical graphs, enabling reproducible task synthesis and effective agent training.
- It demonstrates practical benefits in grounding accuracy and data efficiency through detailed accessibility metadata capture and graph-based exploration.
GUIrilla is a scalable framework for automated desktop UI exploration that targets the data collection bottleneck in GUI automation by systematically exploring macOS applications via native accessibility APIs, organizing discovered interface elements and crawler actions into hierarchical GUI graphs, and deriving functionally grounded tasks for training and evaluation of LLM-based GUI agents (Garkot et al., 16 Oct 2025). It is centered on full-desktop, multi-application environments rather than isolated single-window scenes, and the released artifacts include GUIrilla-Task, a dataset of 27,171 tasks across 1,108 macOS applications, the manually verified GUIrilla-Gold benchmark, the macapptree accessibility metadata library, and GUIrilla-See grounding models tuned on GUIrilla-Task (Garkot et al., 16 Oct 2025).
1. Scope, motivation, and problem setting
GUIrilla addresses a specific deficit in contemporary GUI-agent research: the lack of large, open, realistic, functionally grounded desktop data, especially on macOS (Garkot et al., 16 Oct 2025). The underlying motivation is that desktop autonomy differs materially from mobile and web settings. The paper emphasizes overlapping windows, transient dialogs, small icon-only controls, system widgets, non-standard layouts, and platform-specific quirks such as macOS coordinate systems, menu bars, and permissions. It also argues that many existing datasets focus on single, clean windows rather than full desktops, are manually curated and therefore hard to scale, or do not expose the full collection pipeline (Garkot et al., 16 Oct 2025).
The framework is explicitly macOS-focused because macOS is underrepresented in existing UI corpora. The paper states that OS-Atlas contains only 0.06% macOS UIs overall and 2.45% of automatically collected desktop samples. This underrepresentation matters because the paper also reports that agents trained on simplified views can reach ~83% success on single-window benchmarks but collapse to ~38% on full-desktop scenes, as reported for ScreenSpot-Pro (Garkot et al., 16 Oct 2025). GUIrilla is therefore positioned as an infrastructure project for realistic desktop autonomy rather than a narrow grounding benchmark.
The released artifacts form a layered research stack rather than a single model.
| Artifact | Role |
|---|---|
| GUIrilla framework | Automated macOS crawler via the Accessibility API |
| GUIrilla-Task | 27,171 functionally grounded tasks across 1,108 macOS applications |
| GUIrilla-Gold | Human-verified benchmark derived from the test split |
macapptree |
Open-source library for reproducible accessibility metadata collection |
| GUIrilla-See | 0.7B, 3B, and 7B grounding models tuned on GUIrilla-Task |
This composition suggests a methodological emphasis on open data generation, reproducible metadata capture, and downstream evaluation, rather than on a closed agentic system alone.
2. Crawler architecture and graph-based exploration
GUIrilla’s collection pipeline begins with an application bundle plus configuration parameters such as max depth, max time, and whether exploration is deterministic or GPT-assisted (Garkot et al., 16 Oct 2025). After installation and launch, the framework captures accessibility trees for each window through the macOS Accessibility API, including attributes such as AXRole, AXTitle or AXName, description, position, and size. Interactions are executed through pyautogui, specifically clicks, cursor moves, typing text, and pressing Enter (Garkot et al., 16 Oct 2025).
The crawler core is handler-augmented. The paper lists a pop-up handler, an invisible elements handler, an unrolling menu items handler, and an empty elements handler. These handlers address recurrent AX-side pathologies: transient modals, hidden but still enumerated elements, dynamic menu expansion, and nodes with missing or placeholder metadata. GPT-4 agents are optional rather than mandatory. The Input Agent proposes realistic text inputs for AXTextField nodes, while the Order & Login Agent outputs a safe interaction sequence, groups elements, and flags login pages or system-access requirements. The crawl can also run deterministically, using fixed element order and default texts such as DEFAULT (Garkot et al., 16 Oct 2025).
The central representation is the application graph. Nodes correspond to UI states and include the full accessibility tree, a screenshot filename, and a list of local actions that do not produce a significant UI change. Edges correspond to interactions that do change state and store the target element information, a human-readable description, a structured action representation that maps one-to-one to pyautogui commands, and the resulting state node identifier (Garkot et al., 16 Oct 2025). GUIrilla treats a state transition as significant when it induces “the addition or removal of more than 10 UI elements following an interaction.” Actions below that threshold are stored locally in the node rather than producing new nodes (Garkot et al., 16 Oct 2025).
Exploration is graph-based and hierarchical. The crawler maintains a frontier akin to BFS or DFS, bounded by a default max depth of 25 and a default max parsing duration of 2 hours. At each state it captures the accessibility tree and screenshot, identifies actionable elements, applies handler logic, optionally invokes the Order Agent for ranking, and executes actions in sequence. The resulting graph supports trajectory extraction, task synthesis, visualization, and debugging. Figure 1 in the paper illustrates such a graph for the Session app, and the authors also save node trees as SVG for inspection (Garkot et al., 16 Oct 2025).
A plausible implication is that GUIrilla’s graph abstraction occupies an intermediate position between DOM-native web pipelines and trajectory-only GUI datasets: it preserves state structure densely enough for task synthesis while remaining executable enough for agent training and replay.
3. GUIrilla-Task and GUIrilla-Gold
GUIrilla-Task is produced by postprocessing the application graphs into tasks that describe what an interaction accomplishes rather than merely which pixel region is clicked (Garkot et al., 16 Oct 2025). The pipeline first prunes redundant states by skipping actions that do not satisfy the 10-element change heuristic. It then extracts deterministic raw action descriptions from edges and local actions, and finally uses a GPT-4 Task Agent to rewrite these descriptions into function-oriented instructions. The paper gives examples such as turning “button Open_Media…” into “Open a media file” and “ADD FILES OR IMAGES button” into “Add files or images to the fax” (Garkot et al., 16 Oct 2025).
The paper describes these as “functionally grounded tasks.” The defining property is that they specify what the action does in app context rather than describing color, shape, or coordinate. This is important because the target audience for GUI agents is not a detector of static widgets alone, but a controller that must connect interface affordances to user-centric function (Garkot et al., 16 Oct 2025).
The collection scale is stated precisely. From 12,298 macOS apps, 1,108 were successfully installed and exposed meaningful accessibility trees. The resulting dataset contains 27,171 tasks and about 6,835 unique screens. The split is by app: 881 apps and 25,606 tasks for training, 227 apps and 1,565 tasks for testing, with test apps chosen to have more complex AX trees. Tasks span 23 app genres, and the annotations include 22 task categories and 16 element categories (Garkot et al., 16 Oct 2025).
Each entry contains a screen ID, app bundle ID, natural-language task, raw action string, structured action, element data from the AX tree, the full accessibility tree path before the action, full-desktop screenshot, app crop, segmented app screenshot, scaling factor for coordinates, and flags such as original_task, task category, and element category (Garkot et al., 16 Oct 2025). The paper also notes that spatial statistics show many targets are small and located in peripheral regions such as edges, toolbars, and corners, which is consistent with realistic desktop layouts (Garkot et al., 16 Oct 2025).
GUIrilla-Gold is the human-verified evaluation subset. Annotators with accessibility expertise reviewed 1,319 English tasks from the test split and the released benchmark contains 1,283 tasks after edits and filtering (Garkot et al., 16 Oct 2025). The review protocol covers feasibility, task clarity and editing, manual execution, screen-level AX quality, and element-level correctness of role, description, and bounding box. Reported outcomes are 84.3% DOABLE, 91% of GPT task strings requiring no edits, 97% string similarity for edited cases, 64% GOOD / 24% MEDIUM / 12% BAD AX quality at screen level, about 40% of elements with fully correct role and description, 11% mislabeled elements, and bounding boxes accurate for about 80% of elements (Garkot et al., 16 Oct 2025). The paper reports a human baseline of 90.26% success rate on GUIrilla-Gold (Garkot et al., 16 Oct 2025).
These statistics clarify both the value and the noise profile of the benchmark. GUIrilla-Gold is not a claim of perfect accessibility metadata; rather, it is a curated subset that exposes the practical quality ceiling of AX-derived supervision on macOS.
4. Training protocol, model variants, and empirical performance
GUIrilla-See denotes the grounding models trained on GUIrilla-Task. The paper trains three variants: GUIrilla-See-0.7B based on Florence-2-large, GUIrilla-See-3B based on Qwen-2.5-VL-3B-Instruct, and GUIrilla-See-7B based on Qwen-2.5-VL-7B-Instruct (Garkot et al., 16 Oct 2025). All are trained exclusively on GUIrilla-Task images, approximately 6.8K images (Garkot et al., 16 Oct 2025).
The 0.7B model is cast as open-vocabulary detection on GUI screenshots, with output as a bounding box or polygon mask. Its LoRA with RSLoRA configuration uses rank , scaling , dropout $0.05$, and target modules {q_proj, o_proj, k_proj, v_proj, linear, Conv2d, lm_head, fc2}. Training uses AdamW with learning rate , cosine decay, 5% warm-up, batch size 8, 4 epochs, and 1 × A100 40GB for about 10 hours (Garkot et al., 16 Oct 2025).
The 3B and 7B models are framed as localization on full-desktop screenshots. The input is a screenshot plus natural-language instruction, and the output is a single coordinate near the center of the target element. Their LoRA configuration uses rank , , dropout $0.1$, and target modules {down_proj, o_proj, k_proj, q_proj, gate_proj, up_proj, v_proj}. Optimization uses AdamW with , , learning rate 0, cosine decay, warm-up ratio 1, and FlashAttention-2. The 3B model trains on 2 × H100 80GB with global batch 16 for 2 epochs in about 5 hours; the 7B model uses the same hardware in about 6–7 hours (Garkot et al., 16 Oct 2025).
The paper summarizes GUI grounding as supervised learning. For coordinate prediction, with screenshot 2, instruction 3, and true coordinate 4, the training objective is written as
5
For LoRA, the parameterization is
6
where only the low-rank matrices are trained (Garkot et al., 16 Oct 2025).
On GUIrilla-Task itself, GUIrilla-See-7B achieves 75.59% overall grounding accuracy and GUIrilla-See-3B reaches 73.48%, outperforming all baselines reported in the paper (Garkot et al., 16 Oct 2025). The 7B model records 74.9% on Files, 87.3% on Settings, 78.9% on Connectivity, and 65.2% on Input. The paper also reports large gains over untuned base models, for example Qwen 7B base improving from 35.78% to 75.59% after tuning (Garkot et al., 16 Oct 2025).
On independent benchmarks, GUIrilla-See-7B achieves 90.33% on ScreenSpot-v2, 23.66% on ScreenSpot-Pro overall, and 27.81% on the ScreenSpot-Pro macOS subset (Garkot et al., 16 Oct 2025). The paper states that this macOS-subset result is essentially equal to UI-TARS-7B’s 27.7% and exceeds OS-Atlas 7B’s 20.0%. It further emphasizes data efficiency: GUIrilla-See uses 6.8K synthetic macOS images, whereas UI-TARS uses an estimated ~20M images, corresponding to roughly 97% less data than some baselines and about 300× fewer images than OS-Atlas (Garkot et al., 16 Oct 2025).
The paper also presents an ablation on task wording. Training Florence-0.7B on pure accessibility metadata yields 40.35% accuracy, whereas training it on GPT-generated functional tasks yields 53.55%, a 13-point gap (Garkot et al., 16 Oct 2025). This indicates that functional task descriptions are materially better supervision signals than raw AX roles or names.
5. Position within the GUI-agent literature
GUIrilla is best understood as a data-centric desktop collection framework, and this distinguishes it from adjacent work that emphasizes synthetic trajectory generation, weak supervision, RL-based grounding, lightweight agent training, or mobile GUI mining. The closest contrast in spirit is GUI-ReWalk, which addresses the scarcity of GUI trajectories through a reasoning-enhanced, multi-stage synthesis pipeline that combines stochastic exploration, task-guided completion, retrospective annotation, task recovery, and cross-application multi-stride workflows. GUI-ReWalk reports 50k+ tasks with average 22.5 steps and evaluates on Screenspot-Pro, OSWorld-G, UI-Vision, AndroidControl, and GUI-Odyssey (Lin et al., 19 Sep 2025). GUIrilla differs in that its primary asset is accessibility-driven exploration of real macOS applications and graph-derived function-level tasks, rather than stochastic and intent-aware synthesis.
GUICrafter occupies another complementary position. It uses large-scale unannotated screenshots and webpages for weakly supervised GUI pretraining, followed by a small RLVR fine-tuning stage. The paper reports that GUICrafter achieves competitive or superior performance to UI-TARS while using about 0.1% of UI-TARS’s annotated data volume, and frames its contribution as a two-stage curriculum: weakly supervised GUI pretraining plus high-quality RL fine-tuning (Fan et al., 29 Jun 2026). Relative to GUICrafter, GUIrilla supplies a macOS-specific, accessibility-structured corpus and a full-desktop benchmark substrate rather than a weak-supervision training regime.
GuirlVG focuses narrowly on GUI visual grounding and studies rule-based reinforcement fine-tuning for that subproblem. Using only 5.2K training samples, it reports 88.7% on ScreenSpot, 91.9% on ScreenSpotV2, and 36.1% on ScreenSpot-Pro, with a Soft Reward Function and an Adversarial KL Factor to stabilize GRPO-based training (Kang et al., 6 Aug 2025). A plausible implication is that GUIrilla-Task could serve as a compact macOS-oriented grounding corpus for similar RL-based post-training, especially where data volume rather than environment access is the limiting factor.
LiteGUI addresses lightweight desktop agents rather than collection itself. It introduces Guided On-policy Distillation and a Multi-solution Dual-level GRPO framework, and evaluates on ScreenSpot-Pro, OS-World, and Lite-Bench. LiteGUI-2B reaches 46.86 on ScreenSpot-Pro and 13.24 on OS-World, while LiteGUI-30B-A3B reaches 58.95 and 22.7 respectively (Wu et al., 8 May 2026). Its explicit reasoning and subtask outputs contrast with GUIrilla’s current emphasis on single-step grounding and graph-derived local tasks.
M7-Miner is the closest analogue on mobile. It formulates automated data mining as MCTS-guided exploration with InferAgent, OrchestraAgent, and JudgeAgent, adds intent recycling, and constructs a 20k-image, 2,565-trajectory dataset. The paper states that its estimated cost per image is \$\alpha = 16$80.36 in manual sets, and that GUI agents fine-tuned on the mined data achieve state-of-the-art results on AndroidControl, AITZ, GUI-Odyssey, and CAGUI (Lv et al., 5 Feb 2026). Compared with M$\alpha = 16$9-Miner, GUIrilla’s distinctive contribution is desktop-specific accessibility instrumentation and graph extraction on macOS rather than search-driven mobile mining.
Taken together, these works suggest a broader decomposition of the GUI-agent pipeline. GUIrilla contributes structured macOS exploration and function-level supervision; GUI-ReWalk and M$0.05$0-Miner contribute scalable trajectory generation; GUICrafter and GuirlVG contribute data-efficient training paradigms; LiteGUI contributes compact long-horizon control. This suggests that GUIrilla is most naturally used as an infrastructure layer within a larger desktop-agent stack.
6. Limitations, extensibility, and research significance
The paper identifies several limitations. First, GUIrilla depends on accessibility metadata quality. Accessibility trees may be stale, misaligned, or mislabeled; only about 40% of elements have fully correct role and description, 11% are mislabeled, bounding boxes are accurate for about 80% of elements, and 10% extend outside the visible window (Garkot et al., 16 Oct 2025). This affects both graph reliability and downstream supervision.
Second, coverage is uneven across application domains. The error analysis highlights weaknesses on icon-dense engineering tools such as Vivado, creative tools such as Illustrator and DaVinci Resolve, and canvas-centric actions such as drawing or brush selection. The paper attributes this to underrepresentation of creative and engineering applications in the crawled set and to limited user-generated content (Garkot et al., 16 Oct 2025).
Third, the training setup emphasizes mostly single-step grounding. The framework does build graphs and trajectories, but the paper states that model training is primarily at the single-step grounding level (Garkot et al., 16 Oct 2025). This means GUIrilla provides rich state structure without yet fully exploiting it for long-horizon policy learning.
Fourth, the implementation in the paper is macOS-only. The authors argue that the same accessibility-driven methodology is broadly applicable to Windows through Microsoft UI Automation, Linux through AT-SPI, and Android through AccessibilityNodeInfo, but also note that porting requires platform-specific engineering around permissions, event handling, coordinate systems, and display scaling (Garkot et al., 16 Oct 2025). This suggests cross-platform transfer is architectural rather than immediate.
Finally, the paper flags ethical and security concerns. Automated GUI agents could be misused for sensitive applications, and the mitigation strategy includes sandboxed environments, local-only operation, deterministic handlers, avoiding authentication, payment, and CAPTCHA UIs, and explicit acceptable-use guidelines (Garkot et al., 16 Oct 2025).
Despite these constraints, GUIrilla’s significance is methodological. It makes macOS desktop autonomy more experimentally accessible by releasing not only a dataset and benchmark but also the acquisition substrate. For researchers, it supports training grounding models, evaluating agents on GUIrilla-Gold, studying accessibility metadata quality, extending exploration to internal or enterprise applications, and building testing or accessibility-analysis tools on top of the same graph representation (Garkot et al., 16 Oct 2025). This suggests that GUIrilla is less a terminal solution to desktop autonomy than a reusable collection and benchmarking framework that helps close the gap between isolated grounding benchmarks and real full-desktop agent behavior.