LLM-Brained GUI Agents

• LLM-brained GUI agents are autonomous systems that combine modular architectures, integrating perception, planning, action, and memory for sophisticated GUI manipulation.
• They leverage dual vision encoders and chain-of-thought reasoning to interpret graphical interfaces across mobile, desktop, and web platforms.
• Applications include automated testing, accessibility enhancements, and digital assistance, though challenges in cross-platform robustness and privacy persist.

LLM-brained GUI agents are autonomous systems that utilize LLMs or multimodal LLMs (MLLMs) as their core decision-making and perception engines to interpret, reason about, and interact with graphical user interfaces (GUIs) across platforms including mobile, desktop, and web environments. These agents leverage advances in foundation models for natural language understanding and vision, endowing them with capabilities such as semantic intent understanding, multimodal perception, memory management, dynamic planning, and robust action prediction—enabling them to mimic or automate human GUI operation at scale, with applications in testing, accessibility, and digital assistance.

1. Foundational Architectures and Key Module Design

LLM-brained GUI agents generally employ modular, agent-based architectures that decompose the complex loop of GUI autonomy into perception, planning, action, and memory submodules. Canonical blueprints, as detailed in DroidAgent (Yoon et al., 2023), CoCo-Agent (Ma et al., 19 Feb 2024), and comprehensive surveys (Wang et al., 7 Nov 2024, Zhang et al., 27 Nov 2024, Tang et al., 27 Mar 2025), include:

• GUI Perceiver(s): Multimodal encoders process screenshots, accessibility data, or OCR layouts. Visual-language integration is achieved via models such as EVA2-CLIP in CogAgent (Hong et al., 2023), or through three distinct perceivers (textual, graphical, spatial) as in MP-GUI (Li et al., 29 Apr 2025).
• Task Planner: LLM/MLLM modules generate high-level, semantically grounded plans or tasks, often leveraging chain-of-thought or dynamic planning-of-thoughts (D-PoT) methodologies (Zhang et al., 1 Oct 2024).
• Actor/Executor: Specialized modules (Actors, Decision Makers, or Executors) translate plans into device-specific UI events (e.g., tap, scroll, type), informed by real-time GUI states.
• Observer/Reflector: Submodules record state changes, detect failures, and provide reflection/feedback that supports both immediate correction and long-term learning.
• Memory Systems: Agents maintain explicit short-term and long-term memory, storing past explorations, results, and even widget-level knowledge as in DroidAgent’s spatial memory (Yoon et al., 2023) or the Chain-of-Memory (CoM) approach (Gao et al., 22 Jun 2025).

This modularization enables the integration of advanced reasoning, historical context, and environment feedback in both perception and action.

2. Multimodal Perception and Input Representation

MLLM-based agents surpass purely language-based models by directly consuming screenshots or GUI images, exploiting both textual and layout cues that are often lost in traditional DOM- or tree-based representations. Architectures such as CogAgent (Hong et al., 2023) employ dual visual encoders—a low-resolution global branch fused with a high-resolution local branch—cooperating through cross-attention at each layer to accurately parse fine-grained GUI elements. This enables the model to read tiny icons, spatial arrangements, and detailed text spans, providing a richer, more robust input for downstream reasoning and action generation. In parallel, supporting modules incorporate OCR-extracted layouts, accessibility meta-data, and even historical action logs to provide additional context (as in the Comprehensive Environment Perception module of CoCo-Agent (Ma et al., 19 Feb 2024)).

Table: Input Modalities and Example Models

Model	Input Sources	Visual-Text Integration
DroidAgent	JSON GUI tree, action logs	LLM (text-only)
CogAgent	Screenshot, OCR	Dual vision encoders + CLIP
CoCo-Agent	Screenshot, OCR, action log	CLIP, LLM, prompt fusion
WinClick	Screenshot	Phi3-vision, LLM-alignment

By integrating these modalities, agents achieve more robust environmental modeling, essential for tasks in open-world and dynamic GUIs.

3. Planning, Reasoning, and Decision-Making Frameworks

Contemporary GUI agents increasingly employ sophisticated reasoning frameworks, moving from simple stepwise action prediction to deliberative, multi-step planning. Methods include:

• Chain-of-Thought (CoT) and Dynamic Planning: Planners generate intermediate subgoals and update them in response to environmental feedback—see D-PoT (Zhang et al., 1 Oct 2024), which outperforms baseline React (Zhang et al., 1 Oct 2024) by maintaining succinct execution histories, enabling dynamic re-planning and reducing LLM hallucinations.
• Reasoning Injection and Deliberation: InfiGUI-R1 (Liu et al., 19 Apr 2025) introduces the Actor2Reasoner paradigm, infusing models with explicit spatial reasoning via supervised trajectories, followed by reinforcement learning refinement for robustness in planning and error recovery.
• Code Generation Paradigm: AutoDroid-V2 (Wen et al., 24 Dec 2024) reframes mobile task automation as program synthesis, leveraging the coding capabilities of SLMs to produce multi-step scripts, which are executed by dedicated interpreters. This “plan-ahead” paradigm greatly enhances efficiency, achieving higher task success rates and lower latency compared to stepwise LLM queries.

These approaches demonstrate that contemporary agents are not mere reactive, perception-driven actors but deliberative reasoners capable of decomposing long-horizon tasks and adapting plans on the fly.

4. Memory Systems and Cross-Session Generalization

Explicit memory modeling has emerged as a crucial enabler for context-aware, cross-task, and cross-application autonomy. The Chain-of-Memory (CoM) system (Gao et al., 22 Jun 2025) demonstrates an explicit bifurcation between short-term and long-term memory:

• Short-term memory (STM): Rolling buffer of immediately recent action results (textual descriptions compared between screen states), formalized as $$M_t = \{ r_1, \dots, r_k \}, \quad M_{t+1} = M_t \cup \{ r_{t+1} \}$$, with bounded buffer size.
• Long-term memory (LTM): Only information explicitly identified as critical is retained across sessions or application boundaries, following a gating and storage process capturing persistent facts or configurations.
• Memory Module Functions: At each step, $f(Q, STM_t, I_t)$ evaluates which information should be migrated to LTM, enabling the agent to recall prior searches, navigation paths, or entity attributes needed for future steps.

Empirically, training on CoM-annotated trajectories allows 7B-parameter models to achieve memory-driven task accuracy comparable to 72B-parameter counterparts (Gao et al., 22 Jun 2025), enabling both efficiency and scalability in deployment.

5. Training Regimes and Reinforcement Learning Enhancements

Beyond prompt engineering and imitation learning, reinforcement learning (RL)—especially with advanced group-based policy optimization—has become central for developing adaptive and generalized agents:

• Group Relative Policy Optimization (GRPO): Used by GUI-R1 (Luo et al., 14 Apr 2025), AgentCPM-GUI (Zhang et al., 2 Jun 2025), and ScreenExplorer (Niu et al., 25 May 2025), GRPO normalizes the reward for each candidate action trajectory within a group batch:

$$A_i = \frac{r_i - \mathrm{mean}(\{r_j\})}{\mathrm{std}(\{r_j\})}$$

with policy updates regularized by KL divergence to a reference policy. This strategy accelerates convergence, increases data efficiency (e.g., achieving superiority at just 0.02% of data compared to prior SFT baselines), and boosts OOD generalization.

• Curiosity-Driven Exploration: ScreenExplorer (Niu et al., 25 May 2025) augments external/task rewards with intrinsic curiosity, calculated as the prediction error of a learned world model (e.g., $$r^{world}_{vis}(t) = 1 - \mathrm{sim}(o'_{pred}, o')$$), promoting diversity in state visitation and sustained exploration in previously unseen GUI worlds.

Such reinforcement strategies yield agents with improved exploration, lower susceptibility to overfitting, and heightened robustness under dynamic or unfamiliar interface conditions.

6. Evaluation, Benchmarks, and Performance Metrics

The evaluation of LLM-brained GUI agents leverages a range of metrics and newly curated multi-domain benchmarks. Principal metrics include:

• Task/Step Success Rate: Proportion of correct actions either per step or at task completion; often evaluated via macro-averaging across task trajectories (Zhang et al., 27 Nov 2024, Luo et al., 14 Apr 2025, Zhang et al., 2 Jun 2025).
• Action Type/Exact Match: For fine-grained GUI action evaluation, including accuracy of action parameters (coordinates, text).
• Feature and Activity Coverage: Used in GUI testing (e.g., DroidAgent (Yoon et al., 2023)): percentage of activities, features, or use-case scenarios autonomously reached during exploration.
• Efficiency—Latency, Token Cost, Action Redundancy: Measured across benchmarks such as DroidTask, CAGUI, or AITW, especially crucial for on-device mobile agents.

Recent benchmarks include ScreenSpot (Singh et al., 12 Feb 2025, Hui et al., 27 Jan 2025), GUI-World (Chen et al., 16 Jun 2024), CAGUI (Zhang et al., 2 Jun 2025), WinSpot (Hui et al., 27 Jan 2025), and GUI Odyssey-CoM (Gao et al., 22 Jun 2025), spanning mobile, desktop, web, and cross-platform domains. Evaluation frameworks are moving toward mixed strategies incorporating both trajectory-based and graph-driven methods (allowing for equivalence classes of solutions) (Tang et al., 27 Mar 2025).

7. Open Challenges, Limitations, and Future Directions

Several open challenges persist:

• Dynamic and Cross-Platform Robustness: Generalization to unseen apps, changing layouts, or multi-window scenarios remains non-trivial, requiring advances in procedural grounding, temporal perception, and memory integration (Chen et al., 16 Jun 2024, Singh et al., 12 Feb 2025).
• Privacy and Security: LLM-GUI agents pose amplified risks for sensitive data leakage, reduced user control, and insufficient privacy guardrails due to their broad, autonomous data access (Chen et al., 24 Apr 2025). Proposals include explicit in-context consent, structured privacy prompts, and joint evaluation of performance with risk ratio metrics.
• On-Device Efficiency: Reducing model size, optimizing action spaces (as in AgentCPM-GUI (Zhang et al., 2 Jun 2025)) and migrating computation to edge devices are crucial for practical deployment in mobile or resource-constrained environments.
• Memory and Long-Horizon Reasoning: Explicit memory—short- and long-term—is essential for consistent cross-app operation and for closing the performance gap between smaller and larger models (Gao et al., 22 Jun 2025).
• Standardized, Comprehensive Benchmarking: There is a continuing need for large-scale, multilingual, video- and sequence-centric datasets that reflect the true diversity and dynamics of GUIs in the wild.

Advances in training algorithms (e.g., RL with world model curiosity), cross-modal perception, and explicit reasoning paradigms (deliberation, error recovery, reflection) are expected to further close the gap between human-like interaction and automated GUI manipulation. Agents that combine scalable learning, robust grounding, privacy-awareness, and effective memory systems will define the next generation of LLM-brained GUI autonomy.