A11y-Compressor: A Framework for Enhancing the Efficiency of GUI Agent Observations through Visual Context Reconstruction and Redundancy Reduction

Abstract: AI agents that interact with graphical user interfaces (GUIs) require effective observation representations for reliable grounding. The accessibility tree is a commonly used text-based format that encodes UI element attributes, but it suffers from redundancy and lacks structural information such as spatial relationships among elements. We propose A11y-Compressor, a framework that transforms linearized accessibility trees into compact and structured representations. Our implementation, Compressed-a11y, applies a lightweight and structured transformation pipeline with modal detection, redundancy reduction, and semantic structuring. Experiments on the OSWorld benchmark show that Compressed-a11y reduces input tokens to 22% of the original while improving task success rates by 5.1 percentage points on average.

• The paper introduces the A11y-Compressor framework that transforms redundant a11y trees using a three-phase approach including modal detection, redundancy reduction, and semantic structuring.
• It demonstrates a reduction in token usage to approximately 22% and a 5.1 percentage point improvement in task success rates compared to traditional methods.
• The modular pipeline provides adaptable, semantically structured observations that enhance GUI agent performance in complex, multi-modal desktop environments.

Structured Compression for GUI Agent Observations: The A11y-Compressor Framework

Motivation and Challenges in GUI Observation Representations

Agents operating on GUIs with LLM- and MLLM-based reasoning require structured, task-relevant observation spaces for effective action grounding. While linearized accessibility (a11y) trees are widely adopted as a textual proxy of UI structure, they suffer from significant redundancy and an inability to adequately capture visual, spatial, and hierarchical constraints. Key issues in linearized a11y trees include unnecessary coordinate-level reasoning, parallelization of modal and background elements, ambiguous tagging for shared visual elements, inclusion of hidden elements, and token proliferation from long text spans.

Figure 1: Examples of representation deficiencies in linearized a11y trees, including ambiguous element selection, lack of modal hierarchy, and context bloat from redundant or hidden elements.

Such inefficiencies impose substantial demands on model attention, increase token consumption, and degrade interaction precision, particularly in complex, multi-modal desktop environments. Existing compression attempts, such as LineRetriever, reduce irrelevant content but often lose global structural context and fail in scenarios demanding explicit modeling of modal dependencies or region-level UI semantics.

The A11y-Compressor Framework: Pipeline and Methodology

A11y-Compressor introduces a three-phase transformation framework targeting structural alignment, redundancy minimization, and semantic enrichment of GUI agent observations. The pipeline processes linearized a11y trees in the following stages:

1. Modal Detection: Foreground modal UI elements are extracted and separated from background elements, reconstructing the interaction-valid layering and explicitly encoding transient interaction constraints.
2. Redundancy Reduction: Rule-based heuristics merge redundant UI elements (priority-based on tag interactivity), remove hidden/irrelevant elements, compress bounding box representations to center coordinates, and dynamically filter or summarize long text spans using instruction relevance.
3. Semantic Structuring: Elements are reorganized into application- and domain-specific semantic regions (e.g., navigation bars, sidebars, content panes), leveraging spatial clustering and contextual heuristics to reflect functional UI organization and enhance downstream grounding.

   Figure 2: A11y-Compressor pipeline overview, demonstrating transformation from a redundant linearized a11y tree to a semantically and structurally compact observation suitable for efficient GUI agent grounding.

The modularity of each phase allows domain-driven adaptation and extensibility. Instance implementations for the OSWorld benchmark were constructed using heuristics derived from screen states covering a diverse range of desktop applications.

Empirical Evaluation and Quantitative Results

Evaluation on the OSWorld benchmark encompassed 358 tasks across domains including web browsing, office productivity suites, email, media editing, code development, and system operations. Agent reasoning leveraged Qwen3-VL-32B, although the approach is model-agnostic for any MLLM with textual observation interfaces.

Token efficiency analyses reveal that Compressed-a11y reduces input token usage to approximately 22% compared to linearized a11y trees, maintaining stable representation sizes even when application UI complexity increases. Unlike retriever baselines, which can struggle in high-complexity domains, the proposed framework consistently constrains observation size.

Figure 3: Compressed-a11y achieves superior token compression and stability relative to baseline observational formats across domains, supporting efficient MLLM inference.

Task success rates are similarly improved. Compressed-a11y achieves a 5.1 percentage point increase over the baseline linearized a11y tree, with pronounced absolute gains in domains such as office productivity and email management. Notably, it outperforms both screenshot (image-only) agents and retriever-based approaches, the latter suffering from structural context loss, especially in exploratory task settings where critical elements may be omitted from early-stage observations.

Qualitative Analysis: Modal and Redundancy Handling

Qualitative analysis of tasks involving modal dialogs demonstrates the superiority of semantically structured and modal-aware observation spaces. In screenshot-based representations, the agent fails due to inaccurate coordinate predictions. Linearized a11y trees do not encode the modal-background exclusivity, resulting in futile background interactions.

Figure 4: Compressed-a11y explicitly models modal dialogs and their interaction constraints, enabling successful multi-step control sequencing in contrast to screenshot and vanilla a11y-tree representations.

LineRetriever, despite producing compact observations, misses active modals when extraction bounds are insufficient, resulting in unrelated browser actions that disrupt task execution.

Figure 5: Example failure with LineRetriever: omission of modal state leads to misaligned action sequences (e.g., unintended page reload on fallback).

Component-Wise Contributions: Ablation Study

Ablation studies demonstrate that redundancy reduction (especially coordinate compression) is the most beneficial single component, reducing click ambiguity on non-rectangular UI shapes. Modal detection alone improves dialog handling but is insufficient for general task performance, while semantic structuring yields domain-dependent benefits, especially in interfaces with spatially segmented workflows (e.g., spreadsheets).

The combined, three-phase pipeline consistently achieves the highest and most stable performance across all application types, emphasizing the necessity of integrating spatial, structural, and semantic compression.

Theoretical and Practical Implications

Compressed-a11y represents a significant advance in the principled reduction of textual observation overhead while aligning observation structure more closely with executable UI semantics and layout. Explicit modal segmentation corrects a critical deficiency in prior linearization-based approaches, which is essential for reliable operation in multi-modal, event-driven environments. Region-based structuring provides a grounding substrate that can be expected to aid higher-level agent reasoning and aspect-based attention in future LLM architectures.

Pragmatically, this framework enables local, privacy-preserving deployment of GUI LLM agents by sharply reducing hardware and runtime requirements. It provides a unified methodology for constructing domain-adaptive representations applicable across a spectrum of desktop-like and (with extension) mobile UIs.

Future Directions and Limitations

The paper notes several limitations: reliance on a11y trees omits purely visual cues (such as iconography or color), and rule-based processes may be brittle or suboptimal in unseen application paradigms or rapidly evolving UI genres. The approach has not been systematically benchmarked on mobile environments, where spatial and modal relationships often diverge significantly from desktop patterns.

Future work could involve replacing static heuristics with trainable compression or structuring modules, integrating screenshot and a11y modalities for richer grounding, and extending semantic segmentation mechanisms for finer-grained or context-sensitive UI partitions. Scaling to closed-source MLLMs with stronger reasoning is also a promising avenue, as the compression-structuring framework is agnostic to the downstream model.

Conclusion

A11y-Compressor establishes a three-phase, modular framework for transforming linearized a11y trees into compact, semantically-structured, and modally-aware observation spaces. This design achieves both superior token efficiency and improved execution success rates for GUI agents operating in diverse desktop environments. The integration of explicit modal detection, redundancy reduction, and semantic region structuring closes key gaps left by prior observation paradigms, pointing toward the necessity of structurally-aligned, task-relevant representations as agents are scaled to increasingly complex and heterogeneous user interfaces.

---

Reference:

"A11y-Compressor: A Framework for Enhancing the Efficiency of GUI Agent Observations through Visual Context Reconstruction and Redundancy Reduction" (2605.00551)