Screen2AX-Task: macOS Autonomy Benchmark

• Screen2AX-Task is a benchmark that evaluates autonomous agent performance in macOS by leveraging hierarchical accessibility metadata from vision-based pipelines.
• It employs a multi-stage methodology using YOLOv11 for UI element detection, OCR for text extraction, and tree reconstruction for grouping elements.
• Key metrics show a 33.7% success rate with the hierarchical model outperforming native macOS trees and existing paradigms in grounding natural language commands.

Screen2AX-Task is a benchmark specifically constructed to evaluate the performance of autonomous agents in executing one-step commands within macOS desktop environments by leveraging automatically generated, hierarchical accessibility metadata. Its primary function is to measure how effectively an agent, such as a large vision-LLM, utilizes structured screen representations derived from vision-based pipelines to ground natural language instructions to specific interactive UI elements (Muryn et al., 22 Jul 2025).

1. Definition and Benchmark Objectives

Screen2AX-Task is designed as the first task-execution benchmark focused on desktop macOS, targeting the task grounding and execution challenge for AI agents. The core objective is to quantify the utility of real-time, vision-derived tree-structured accessibility representations (AX trees) versus both native accessibility APIs and existing state-of-the-art parsers. For each evaluation instance, the benchmark pairs a screenshot from a real macOS application with (i) human-annotated bounding boxes for UI elements, (ii) a natural language command (e.g., “select the red minimize button”), and (iii) an element-level visual description. During evaluation, the agent receives only the JSON-encoded accessibility tree and the command—the screenshot itself is withheld—and must identify (by numeric ID) which UI element should be selected. This setup strictly measures screen interpretation and instruction grounding, independent of visual feature extraction from pixels.

2. Dataset Composition and Evaluation Environment

Screen2AX-Task is constructed atop three foundational datasets, each capturing different annotation granularities from 112 macOS applications (containing 1,127 screenshots):

Dataset Name	Scope	Key Statistics/Annotations
Screen2AX-Element	UI element detection	986 images, 44,150 bounding boxes (5 classes)
Screen2AX-Group	Logical grouping of elements	808 images, 30,774 AXGroup annotations
Screen2AX-Tree	Hierarchical AX metadata	1,127 screenshots, 52 raw roles → 7 consolidated

Screen2AX-Task itself is a held-out subset of 166 screenshots, not used in model training. Each screenshot features full bounding-box annotation of interactive elements, a human-generated natural-language instruction, and a concise description. There are 5,934 (element, command) pairs in total, ensuring robust statistical significance across applications and UI contexts.

During evaluation, only the metadata (AX tree) and the command are available—the agent must output which element to select, testing not pixel-matching but the fidelity of semantic grounding to real UI structures.

3. Hierarchical Accessibility Metadata Generation Pipeline

The Screen2AX metadata pipeline processes raw screenshots to yield an accessibility tree closely mirroring the true macOS API structure via three sequential vision-based stages:

1. UI Element Detection: A fine-tuned YOLOv11 model detects UI elements. Training optimizes the loss

$L_{det} = L_{cls} + L_{box} + L_{obj}$

accounting for classification, bounding box regression, and objectness.

1. UI Element Description: Text inside detected elements is extracted using platform-specific OCR (ocrmac). For graphical icons without text, the pipeline employs a BLIP model, fine-tuned on a mobile UI-icon dataset, generating natural-language captions for the icons.
2. Hierarchy (AX Tree) Reconstruction: A second YOLOv11 predicts bounding boxes for AXGroups, interpreted as tree internal nodes. Each UI element (leaf) is assigned to the smallest enclosing group, and groups are recursively merged up to the root (the window), yielding a tree representing the UI’s hierarchical structure.

Accuracy assessment employs:

• Edge-based F1:

Let $E_{pred}$ and $E_{gt}$ denote edge sets from predicted and ground-truth trees.

$$\text{Precision} = \frac{|E_{pred} \cap E_{gt}|}{|E_{pred}|} \qquad \text{Recall} = \frac{|E_{pred} \cap E_{gt}|}{|E_{gt}|}$$

$$F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

• Graph Edit Distance (GED):

$$\text{GED}(G_1, G_2) = \min_{\text{operations}} \sum_k c(e_k)$$

using NetworkX’s approximate algorithm; lower values indicate closer structural matching.

• Group-based Mean IoU:

For each matched AXGroup, the mean intersection-over-union (IoU) is computed between predicted and ground-truth bounding boxes.

4. Evaluation Metrics and Quantitative Performance

Screen2AX-Task evaluates both the quality of reconstructed AX trees and the downstream agent task success rate (SR). Key results are summarized below:

Representation	Edge-F1 (AX Tree)	Success Rate (SR, %)
Heuristic baseline	67	—
Screen2AX (YOLOv11 grouping)	77	33.7 (Screen2AX-Task)
Built-in macOS AX tree	—	16.9
OmniParser V2	—	28.0
Screen2AX (flat list)	—	30.7

• On Screen2AX-Task, the Screen2AX hierarchy yields an SR of 33.7%, a 2.2× improvement over the built-in macOS tree and 5–7 percentage points above OmniParser V2.
• On the public ScreenSpot benchmark (macOS subset), SR using only the screenshot image is 6.9%, OmniParser V2 achieves 31.9%, Screen2AX (flat) yields 34.3%, and Screen2AX (hierarchy) reaches 36.6%.

The gain from hierarchy (+3 percentage points over flat Screen2AX output) confirms the importance of explicit structural grouping.

5. Model Capabilities, Insights, and Agent Interaction

Screen2AX-Task directly demonstrates that higher-fidelity, structured representations of UI hierarchies—specifically, tree-structured accessibility metadata—improves LLM-based agent task execution in desktop environments. The 77% Edge-F1 score in reconstructing the AX tree translates into substantive improvements in element grounding accuracy, as a single missing or mis-grouped node can lead to misinterpretation by the agent. By removing ambiguities inherent in flat or incomplete accessibility data, the benchmark makes it possible to evaluate agent reasoning in a much more granular fashion.

A key insight is that a robust, vision-only pipeline enables coverage extension to the 64% of macOS applications without sufficient native accessibility metadata and, in many cases, outperforms built-in trees in precision and completeness. This suggests vision-based accessibility pipelines are viable for large-scale screen interpretation and agent-based interaction, especially in heterogeneous software environments.

6. Limitations and Future Directions

Screen2AX-Task identifies several current constraints:

• Dataset noise: Developer-provided annotations can vary, especially for non-standard UI controls.
• Icon captioning domain gap: The BLIP model’s icon captioning is fine-tuned on mobile icons, thus struggles with complex desktop iconography.
• Inference throughput: The three-stage pipeline (two YOLOv11 models plus BLIP) requires ~0.7s per screenshot, which might not satisfy applications with stringent real-time requirements.

Planned avenues for improvement include:

• Enriching icon-caption training with desktop-specific icon datasets.
• Incorporation of semantic group labels (e.g., “toolbar,” “sidebar”) to supply richer context for agent guidance.
• Research into lightweight vision models, pruning, and quantization for reduced latency.
• Extension of the paradigm to additional desktop platforms (Windows, Linux) and multi-step task sequences.

This suggests that, as hierarchical vision-based accessibility pipelines mature and datasets grow, autonomous agents will achieve even greater proficiency at high-level screen interpretation and interaction across a wider array of software ecosystems (Muryn et al., 22 Jul 2025).