macapptree: macOS Accessibility Data Collector
- macapptree is an open-source macOS library that collects structured accessibility metadata and screenshots using the native Accessibility API.
- It underpins the GUIrilla ecosystem by enabling automated UI crawling, graph construction, and reproducible task dataset creation.
- The toolkit captures detailed UI element properties, facilitating rigorous desktop UI exploration and empirical evaluations.
Searching arXiv for “macapptree” and closely related macOS accessibility-tree tooling to ground the article in the relevant papers. macapptree is an open-source macOS library for the reproducible collection of structured accessibility metadata and screenshots from desktop applications. Within the GUIrilla framework, it functions as the low-level accessibility-data collection toolkit that enables inspection of applications through native accessibility interfaces, capture of hierarchical UI structure, and support for automated GUI crawling, graph construction, and downstream task-dataset creation (Garkot et al., 16 Oct 2025).
1. Definition and scope
macapptree is introduced not as a separate research framework with its own standalone methodology section, but as the released library component that underlies or supports the GUIrilla data-collection pipeline. The paper describes it both as an independently released open-source library and as a component of the broader GUIrilla ecosystem. Its stated purpose is the reproducible collection of structured accessibility metadata and screenshots from macOS applications, addressing the difficulty of obtaining structured, machine-readable UI metadata in a desktop environment that lacks a web-like DOM and is underrepresented in open UI datasets (Garkot et al., 16 Oct 2025).
The surrounding artifact structure is narrow and specific. macapptree is the collection layer; GUIrilla is the exploration framework; GUIrilla-Task is the large-scale task dataset constructed from application graphs; and GUIrilla-Gold is the manually verified benchmark derived from those graphs.
| Artifact | Role |
|---|---|
| macapptree | Library for collecting accessibility metadata and screenshots |
| GUIrilla | Framework for automated desktop UI exploration |
| GUIrilla-Task | Dataset of 27,171 functionally grounded tasks |
| GUIrilla-Gold | Manually verified benchmark |
A common misconception is to treat macapptree as synonymous with GUIrilla itself. The paper does not support that reading. The stronger interpretation is that macapptree provides the structured accessibility-tree and screenshot acquisition on which GUIrilla’s crawler, graph construction, and task generation depend.
2. Native interfaces and collected metadata
The technical basis of macapptree is the macOS Accessibility API. GUIrilla is described as being built on macOS’s accessibility API, and the crawler first attempts to extract an application’s accessibility tree according to the macOS accessibility framework. That framework organizes UI elements into a hierarchical tree structure in which each element contains properties such as name, role, description, position, and size (Garkot et al., 16 Oct 2025).
For interaction, the crawler performs click, cursor move, type, and press Enter key actions using pyautogui. The paper does not name lower-level Cocoa or AX classes, so the explicit claim is limited to dependence on the macOS Accessibility API for inspection and pyautogui for action execution. The collected representation is therefore not merely screenshot-based; it is anchored in native system accessibility state.
The paper also identifies richer metadata fields present in the resulting ecosystem. In dataset examples and verification procedures, element-level data include name, role, role_description, description, value, position, size, and bounding-box information. The appendix schema for generated samples further records JSON metadata of the target UI element and an “A11y Path,” defined as the full accessibility tree before the action was taken. This establishes macapptree’s output as structured, hierarchical, and suitable for downstream serialization and replay.
3. Representation inside GUIrilla
Within GUIrilla, the data collected through macapptree are embedded in a graph-centric exploration pipeline. At the state level, the raw object is the full accessibility tree of an application window. At the application level, GUIrilla organizes discovered states and actions into application interaction graphs. A node contains the accessibility tree of the application window at a state, the corresponding full-desktop screenshot filename, and the actions executable without causing significant UI changes. An edge contains UI element information that triggered the interaction, a human-readable action description, a structured dictionary with a 1-to-1 map to pyautogui commands, and the resulting UI state (Garkot et al., 16 Oct 2025).
The crawler defines a significant change as the addition or removal of more than 10 UI elements following an interaction. This operational threshold is part of graph construction rather than of macapptree alone, but it shows how the collected accessibility trees are consumed: they are compared structurally across successive states to determine whether an interaction should branch the graph.
The same section of the paper describes specialized handlers that operate over accessibility-derived UI state: a Pop-up handler for transient modal content, an Invisible elements handler for off-screen components present in accessibility, an Unrolling menu items handler for dynamically generated navigation elements, and an Empty elements handler for placeholder elements with missing metadata. These handlers are significant because they reveal the practical conditions under which raw accessibility-tree collection is insufficient and must be supplemented by application-specific exploration logic.
4. Usage, outputs, and reproducibility
The paper frames macapptree as a reproducible research tool rather than a private internal dependency. The released repository is https://github.com/MacPaw/macapptree, and the broader project release page is https://huggingface.co/GUIrilla. The intended use is local collection of structured accessibility metadata and screenshots, feeding application crawling, graph generation, task synthesis, and benchmark construction (Garkot et al., 16 Oct 2025).
The output formats are concrete. The ecosystem includes JSON metadata for target UI elements, structured action dictionaries with a 1-to-1 correspondence to pyautogui commands, SVG graph visualizations, full-desktop screenshots, and auxiliary image crops or segmentations in dataset releases. The appendix schema records fields such as Screen ID, App Name, Task, Raw Action, Action, Element Data, Scaling Factor, Original Task, A11y Path, Image, Cropped Image, Segmented Image, Task Category, and Element Category.
The paper also supplies practical configuration parameters for the crawler: maximum parsing duration in minutes with default 2 hours, deterministic text input with default string 'DEFAULT', maximum parsing tree depth with default 25, cursor move before click with default False, agent usage enabled by default when an OpenAI API key is provided in a separate file, and task collection enabled by default. The runtime environment reported for large-scale collection consisted of four 16 GB RAM M1 Mac Mini machines running macOS 14.7.5 Sonoma, plus two MacBook Pros.
A crucial operational condition is that target applications must support the macOS Accessibility framework. Out of 12,298 macOS applications processed in the larger pipeline, 1,108 applications were installed, supported the macOS Accessibility framework, and yielded interaction graphs. The paper also emphasizes a privacy-preserving workflow: all collection, replay, and annotation occur entirely locally, and recommended use is in sandboxed environments with anonymized profiles.
5. Empirical evidence and limitations
There is no benchmark in which macapptree alone is isolated as a standalone system. The empirical evidence is instead indirect: it is the scale and downstream utility of the GUIrilla collection pipeline that macapptree enables. That pipeline was deployed on 12,298 macOS applications and produced interaction graphs for 1,108 applications, from which GUIrilla-Task was constructed with 27,171 tasks across 6,835 unique screens and over 23 app genres (Garkot et al., 16 Oct 2025).
The paper also provides explicit quality measurements for the native accessibility metadata being collected. In GUIrilla-Gold, 64% of screens were rated GOOD, 24% MEDIUM, and 12% BAD. At the element level, 40% had correct role and description pairs, 49% contained role information only, and 11% were mislabeled. Bounding boxes were 80% accurate, while 10% extended outside the visible window. These numbers are central to understanding macapptree’s status: it provides structured access to native accessibility data at scale, but the fidelity of that data is bounded by what applications expose.
Handler-assisted crawling further shows the practical value of accessibility-tree collection when combined with targeted exploration logic. On Stocks, handler-supported crawling produced 162 tasks versus 32 for a random crawler; on Maps it produced 107 versus 36. The paper also reports reduced duplicate rates and shorter processing times in those ablations. These are GUIrilla-level outcomes, but they document the value of accessibility-tree-centric collection as a substrate for automated desktop exploration.
The limitations are equally explicit. The dominant constraint is dependence on developer-provided accessibility metadata, which can be sparse, stale, noisy, mislabeled, or geometrically inaccurate. The paper notes stale elements remaining in the tree after disappearing, incorrect role classifications, and inaccurate positioning information. It also identifies recurrent failure cases involving invisible elements, transient popups, unrolled menus, and empty placeholders. Accessibility-only task generation is therefore described as unreliable; the authors conclude that screenshots, handler logic, and semantic refinement remain necessary companions to the collected tree structure.
6. Relation to Screen2AX and broader significance
macapptree occupies a distinct position within recent macOS UI-structure research. In Screen2AX, the authors state that screenshot-plus-accessibility-tree pairs were collected together with built-in accessibility metadata using the open-source macapptree library. Screen2AX then addresses the complementary problem: automatically reconstructing real-time, tree-structured accessibility metadata from a single screenshot when native accessibility is absent or degraded. Its motivation is the finding that only 33% of applications on macOS offer full accessibility support, and it reports a 77% F1 score in reconstructing a complete accessibility tree as well as a 2.2x performance improvement over native accessibility representations on Screen2AX-Task (Muryn et al., 22 Jul 2025).
This relation clarifies what macapptree is and is not. It is not a vision model, not an automatic accessibility generator, and not a benchmark in itself. It is a library for collecting the native structured representation that macOS applications expose through accessibility APIs. Screen2AX, by contrast, is a screenshot-to-tree reconstruction system intended to substitute for or augment missing native metadata. The two are therefore complementary rather than interchangeable: macapptree acquires accessibility trees and screenshots from the operating system and running application; Screen2AX infers a macOS-style accessibility tree from pixels when the native tree is incomplete.
The broader significance of macapptree lies in this infrastructural role. It makes macOS accessibility-tree collection open, structured, and reproducible; it supports graph-based exploration of full-window, multi-application desktop environments; and it provides the raw material from which larger artifacts such as GUIrilla-Task, GUIrilla-Gold, and Screen2AX-Tree can be constructed. For desktop autonomy research, its importance is less algorithmic than epistemic: it supplies a reproducible path to the hierarchical UI metadata that many downstream methods presuppose but that the macOS ecosystem often exposes inconsistently.