From User Interface to Agent Interface: Efficiency Optimization of UI Representations for LLM Agents

Abstract: While LLM agents show great potential for automated UI navigation such as automated UI testing and AI assistants, their efficiency has been largely overlooked. Our motivating study reveals that inefficient UI representation creates a critical performance bottleneck. However, UI representation optimization, formulated as the task of automatically generating programs that transform UI representations, faces two unique challenges. First, the lack of Boolean oracles, which traditional program synthesis uses to decisively validate semantic correctness, poses a fundamental challenge to co-optimization of token efficiency and completeness. Second, the need to process large, complex UI trees as input while generating long, compositional transformation programs, making the search space vast and error-prone. Toward addressing the preceding limitations, we present UIFormer, the first automated optimization framework that synthesizes UI transformation programs by conducting constraint-based optimization with structured decomposition of the complex synthesis task. First, UIFormer restricts the program space using a domain-specific language (DSL) that captures UI-specific operations. Second, UIFormer conducts LLM-based iterative refinement with correctness and efficiency rewards, providing guidance for achieving the efficiency-completeness co-optimization. UIFormer operates as a lightweight plugin that applies transformation programs for seamless integration with existing LLM agents, requiring minimal modifications to their core logic. Evaluations across three UI navigation benchmarks spanning Android and Web platforms with five LLMs demonstrate that UIFormer achieves 48.7% to 55.8% token reduction with minimal runtime overhead while maintaining or improving agent performance. Real-world industry deployment at WeChat further validates the practical impact of UIFormer.

• The paper introduces UIFormer, a DSL-guided, feedback-driven synthesis approach that optimizes UI representations by reducing token overhead while preserving semantics.
• It employs an iterative synthesis process with a composite reward mechanism to achieve token reductions of up to 63% and improve agent success rates by up to 7%.
• The method integrates as a plug-and-play preprocessing module across Android, web, and hybrid UI frameworks, significantly lowering latency and enhancing throughput in industrial deployments.

UIFormer: Programmatic UI Representation Compression for Efficient LLM Agents

Motivation and Problem Statement

Efficiency in LLM agent-based UI navigation is severely constrained by the token overhead of large, hierarchically structured GUI representations. Empirical study demonstrates that raw UI trees—such as those derived from Android accessibility APIs or web DOMs—contribute 80–99% of agent token usage, representing the primary computational and scalability bottleneck (Figure 1).

Figure 1: A minimal Android search UI screenshot (left) and corresponding UI tree (right), highlighting implementation-driven fragmentation that bloats token representation.

Traditional heuristic-based and LLM-driven UI transformation approaches present adverse tradeoffs: while aggressive flattening yields token savings, it systematically removes critical hierarchy and context necessary for precise agent planning; conversely, UI-specific heuristic filters risk omitting semantically relevant elements or retaining superfluous detail, degrading agent effectiveness or limiting compression. The absence of verifiable oracles for semantic equivalence, and the combinatorial explosion in the search space for programmatically compressing arbitrary UI trees, exacerbate the challenge.

UIFormer: Automated Program Synthesis for UI Representation Optimization

UIFormer is introduced as a principled, automated program synthesis framework for generating UI transformation programs that reduce representation tokens while preserving LLM-relevant semantics. The core methodology encompasses two tightly coupled phases (Figure 2).

Figure 2: Schematic illustration of UIFormer: Phase (a) offline iterative program synthesis using a restricted DSL, and Phase (b) transparent integration at agent inference time.

Offline Program Synthesis: DSL-Restricted, Feedback-Driven

UIFormer constrains the transformation program synthesis task using a domain-specific language (DSL) that admits only semantically verifiable, recursive, parent-child consolidations, and context-aware filters. This restriction eliminates the infeasible search space of general-purpose program synthesis over full UI graphs.

The iterative synthesis process leverages a code LLM that is prompted (Figure 3) to generate candidate DSL programs, which are then evaluated on curated task examples using a composite reward: strict penalties for semantic loss (e.g., incorrect association of functional components, loss of interaction affordance labels) and proportional rewards for token reduction.

Figure 3: Example DSL-based prompt for code LLM, illustrating how transformation tasks and correctness feedback are articulated for refinement.

Negative and positive transformation instances are systematically reincorporated into the prompt, catalyzing self-improving program synthesis through several iterations. This yields a library of verified transformation routines capable of modularly compressing UI trees according to both global efficiency and downstream agent performance constraints.

Runtime Integration: Lightweight, Framework-Agnostic

The verified transformation library is integrated as an agent-agnostic preprocessing plugin at agent inference time, intercepting and rewriting raw UI representations before prompt construction. No modifications are required to agent core logic or action modules, enabling straightforward deployment across Android, web, and hybrid agent frameworks.

Experimental Results

Empirical validation across multiple benchmarks and LLMs underscores UIFormer’s simultaneous achievement of strong compression and preservation or enhancement of agent effectiveness. In Android and web UI navigation, UIFormer produces 48.7–55.8% average reduction in UI representation tokens, and up to 63% on verbose web DOMs, while improving agent success rates by up to 7% absolute over the best heuristic baseline. In all cases, the agent’s downstream performance is maintained or improved, refuting the typical assumption of an inevitable performance-efficiency tradeoff.

Ablation studies demonstrate the necessity of both DSL constraints and feedback-driven iterative refinement. When unconstrained, code LLMs overfit to superficial transformations or generate programs that conflate distinct UI semantics, yielding up to 549% token inefficiency or semantic breakdown. The iterative reward loop consistently enables convergence to high-performing, compressed transformation programs (Figure 4).

Figure 4: Token reduction per synthesis iteration; red marks indicate iterations where semantic completeness is violated before program refinement.

Comparison of prompt formatting strategies reveals that major performance gains derive from semantically aware node consolidation, not merely hierarchical prompt style.

A case study (Figure 5) illustrates improvement: By consolidating fragmented label-input UI nodes into unified semantic units, UIFormer enables the LLM agent to correctly resolve field associations missed under purely leaf-based or flat representations.

Figure 5: Case demonstrating successful agent planning with a UIFormer-optimized representation that preserves label-field association.

Industrial Deployment and Practical Impact

In production WeChat agent serving, UIFormer delivers a 76.9% reduction in average tokens per request, a 26.1% reduction in average response latency, and a 35.2% increase in overall throughput at scale, significantly enhancing agent service scalability and user experience. UIFormer maintains robust, low-latency operation across concurrency regimes (Figure 6).

Figure 6: UIFormer achieves lower average agent latency across increasing concurrency levels, supporting scalable industry deployment.

The runtime costs for transformation are negligible (<6ms on average per UI), achieving plug-and-play deployment benefits for any LLM UI agent pipeline.

Implications, Limitations, and Future Perspectives

The UIFormer approach demonstrates that principled, DSL-guided, feedback-driven program synthesis is tractable and effective for consolidating complex UI trees in a manner that is verifiably lossless with respect to agent-level semantics. This disrupts prior assumptions about the necessity of hand-tuned heuristics or direct LLM-based flattening, and provides a scalable abstraction for token cost management in LLM-agent-based UI automation and testing.

Practically, UIFormer underwrites more cost-effective, higher-throughput deployment of LLM agents in real-world software engineering, test automation, and accessibility applications, aligning the interface abstraction level with LLM reasoning capacities.

Theoretically, the iterative program synthesis framework—grounded in composite reward optimization without Boolean oracles—suggests broad generalization potential for other program synthesis tasks in settings where classical input-output specification is infeasible and semantic preservation is softly defined.

Future work could extend UIFormer to multi-modal UI trees, dynamic or user-context adaptive transformations, or closed-loop joint optimization of representation and agent policy. Integration with model distillation or further prompt-level compression could yield even larger computational gains, especially as context windows continue to scale across LLM platforms.

Conclusion

UIFormer establishes a new paradigm for UI-to-agent representation optimization via automated, DSL-restricted program synthesis. It achieves substantial token compression with maintained or improved agent performance, is plug-and-play across frameworks, and translates into significant real-world efficiency and scalability gains. This work sets a technical foundation for efficient LLM agent operation in the pervasive, complex ecosystem of human-computer interfaces (2512.13438).