Screen2AX: Vision-Based macOS Accessibility

• Screen2AX is a vision-based framework that automatically generates hierarchical accessibility metadata from single screenshots of macOS desktop applications.
• It employs fine-tuned modules like YOLOv11 and BLIP to accurately detect, describe, and hierarchically organize UI elements, achieving marked improvements in tree reconstruction and accessibility support.
• The framework is underpinned by extensive annotated datasets and benchmarks, providing a robust resource for advancing assistive technology and autonomous GUI agents.

Screen2AX is a vision-based framework for the automatic generation of hierarchical accessibility metadata from single screenshots of macOS desktop applications. Designed to address the substantial gap in accessibility metadata coverage for desktop GUIs, Screen2AX reconstructs the full tree-structured interface representation required by both screen-reader assistive technologies and AI-driven autonomous agents. It leverages advances in object detection and vision–language modeling to accurately detect, describe, and hierarchically organize visual interface elements, mirroring macOS’s native system-level accessibility trees (Muryn et al., 22 Jul 2025).

1. Motivation and Background

macOS accessibility infrastructure is predicated on complete, high-fidelity, tree-structured metadata for all interface elements. This metadata underpins VoiceOver (Apple’s screen reader) and is essential for autonomous agents that automate interaction and testing of GUIs. However, empirical analysis of 452 randomly sampled macOS applications (including the top-99 by popularity) revealed a pervasive lack of coverage:

• 29.4% (random subset) and 36.5% (popular apps) provided full accessibility support.
• 37.8% (random) and 45.9% (popular) provided partial support.
• 32.7% (random) and 17.7% (popular) had no accessibility metadata.

The absence or incompleteness of metadata leads to failures in context preservation, element identification, and manipulation for both accessibility users and autonomous agents. A significant accessibility gap remains: only one-third of applications provide full support, leaving the majority of desktop apps inaccessible or partially accessible (Muryn et al., 22 Jul 2025).

2. Formalization of macOS Accessibility Trees

Screen2AX adheres to the native representation of desktop interface accessibility as a rooted, ordered tree $T = (V, E)$, where $V$ is the set of element nodes and $E$ encodes containment relationships. Each node $v \in V$ comprises

$v = (N, R, D, PD, VL, B)$

where:

• $N \in \Sigma^*$: Human-readable name
• $R \in$ Roles: Element role (e.g., AXButton, AXStaticText)
• $D \in \Sigma^*$: Description or alt-text
• $PD \in \Sigma^*$: Role description (user-friendly label)
• $VL \in$ Values: Current value or state (e.g., "checked", field contents)
• $V$0: Bounding box
• Children listed as $V$1

The root node often represents the application window (AXWindow), with children as semantic groups (AXGroup), and so on. Evaluation of accuracy is performed by matching parent-child edges between the predicted tree $V$2 and ground truth $V$3.

3. Screen2AX System Architecture

Screen2AX is a modular pipeline comprising three major components:

3.1 UI Element Detection

Raw screenshots $V$4 are processed using a fine-tuned YOLOv11 detector. The system predicts element detections $V$5 where $V$6 is the semantic class (e.g., AXButton, AXDisclosureTriangle, AXImage, AXLink, AXTextArea), $V$7 the bounding box, and $V$8 the confidence score. Training employs a multi-task loss:

$V$9

• $E$0: Binary cross-entropy on class labels
• $E$1: Smooth-$E$2 loss for bounding box regression
• $E$3: Objectness confidence

3.2 UI Element Description

Detected elements are assigned semantic descriptions. Text-bearing elements are processed with OCRmac; non-textual (e.g., icon-only) elements are cropped and described using a fine-tuned BLIP vision–LLM, employing standard cross-entropy loss for caption sequence prediction.

3.3 Hierarchy Generation

Hierarchy inference is treated as a secondary object detection task. YOLOv11 is trained using "AXGroup" annotations (from the Screen2AX-Group dataset) to locate higher-level containers such as toolbars, menus, and panels. Each element is assigned to the smallest enclosing group (determined by bounding box containment). Group nesting recursively constructs the final tree $E$4. Edges $E$5 are added if $E$6 and there is no $E$7 with $E$8.

A formal objective function penalizes grouping ambiguity:

$E$9

4. Datasets and Benchmarks

Screen2AX introduces and publicly releases several annotated datasets for macOS desktop interfaces:

Dataset	#Screenshots	#Annotations	#Classes	#Apps
Screen2AX-Tree	1127	≈50,000	52	112
Screen2AX-Element	986	44,150	5	92
Screen2AX-Group	808	30,774	1	75
Screen2AX-Task	435	5,934	—	112

• Screen2AX-Tree: Full screenshots with manually corrected accessibility tree annotations.
• Screen2AX-Element: Per-image UI element classes and bounding boxes.
• Screen2AX-Group: Pixel-level segmentation of semantic groups.
• Screen2AX-Task: Task-oriented agent benchmark with commands and goal localizations sourced from GPT-4 and human annotators (Muryn et al., 22 Jul 2025).

5. Hierarchical Inference and Evaluation Protocols

The hierarchy generation process is distilled in the following pseudocode:

$v \in V$0

Element and container detection accuracy, edge match scores, leaf node precision/recall, and task success rates are the principal evaluation metrics.

6. Empirical Results

6.1 UI Element Detection

• YOLOv11 achieves 65.4% [email protected] vs. 37.8% for MSER.
• All-YOLOv11: 43.8% F1, 46.6% mAP50, 32.0% mAP50-95 at 0.20 s per screen.
• MSER+GPT-4: 8.1% F1 at 13.3 s per screen.

6.2 Icon Captioning

• BLIP (Screen2AX): CIDEr 0.68, GPT-measured accuracy 0.76
• Florence (OmniParser V2): CIDEr 0.39, GPT-measured accuracy 0.44

6.3 Hierarchy Generation

• YOLOv11-Group improves Container Match (CM) from 24% (heuristics) to 55%.
• Edge-F1 increases from 67% to 77%.
• Graph Edit Distance decreases from 218.71 to 200.28.
• Inference time: 0.76 s per screen.

6.4 Autonomous Agent Benchmarks

ScreenSpot Success Rate:

Method	Success Rate
Only image	6.9%
OmniParser V2	31.9%
Screen2AX (no hierarchy)	34.3%
Screen2AX (with hierarchy)	36.6%

Screen2AX-Task Success Rate:

Representation	Success Rate
Built-in AX metadata	16.9%
OmniParser V2	28.0%
Screen2AX (flat)	30.7%
Screen2AX (hierarchy)	33.7%

Screen2AX (with hierarchy) therefore yields a 2.2× improvement over native macOS metadata and outperforms OmniParser V2 on both benchmarks (Muryn et al., 22 Jul 2025).

7. Impact, Limitations, and Future Directions

By transforming pixel-level interface captures into semantically and structurally rich AX trees, Screen2AX directly addresses the 67% accessibility deficit observed on macOS, with immediate benefit for screen-reader users and enhanced performance for AI-based agents performing GUI navigation or automation. The public release of datasets, models, and benchmarks fosters further research in computer vision-driven accessibility and inclusive computing.

Limitations include modality restriction (macOS only), group type granularity (currently merged into AXGroup), and real-time performance on low-powered hardware. Proposed future work involves:

• Integration into development pipelines for live AX-validation.
• Extension of the pipeline to other OSes (Windows, Linux) and frameworks (Electron, Qt).
• Enhanced semantic group classification to refine agent planning.
• Model compression via pruning or quantization for sub-second inference.

Screen2AX is open-source at https://github.com/MacPaw/Screen2AX, supporting continued advancement in automated accessibility and agentic desktop interaction (Muryn et al., 22 Jul 2025).