Computer Use Agents: Autonomous GUI Control

• Computer Use Agents (CUAs) are autonomous systems that integrate vision-language models to interpret and execute tasks via GUIs across diverse platforms.
• They employ hybrid pipelines combining visual perception, structured data, and chain-of-thought reasoning to plan and execute multi-step workflows.
• CUAs achieve robust cross-platform performance while raising challenges in efficiency, security, and safe deployment in real-world environments.

Computer Use Agents (CUAs) are autonomous systems, typically built upon vision-language or multimodal LLMs, that can interpret and control software interfaces—most commonly graphical user interfaces (GUIs)—to accomplish a wide range of computer tasks on behalf of users. Unlike conventional dialogue agents, CUAs are designed to physically manipulate computing environments, execute multi-step workflows, interact with desktop, mobile, and web applications, and sometimes leverage APIs directly alongside GUI input. Recent advances have fueled rapid progress in both general-purpose and specialized CUAs, supported by large-scale cross-platform datasets, open benchmarks, and new frameworks for robust control, training efficiency, and real-world safety. However, their increasing autonomy and system access also introduce unique challenges around efficiency, security, safety, and practical deployment.

1. Architecture, Core Paradigm, and Data Modalities

CUAs integrate perception, reasoning, and action across diverse software environments. Architecturally, a canonical agent comprises:

• Perception layer: Ingests one or more modalities such as screenshots (image), accessibility trees (structural), and sometimes DOM or raw HTML for web agents.
• Reasoning/planning unit: A large vision-LLM (VLM/LLM), responsible for action planning, memory, and long-horizon decision-making, often augmented by explicit chain-of-thought (CoT) or reflective reasoning traces.
• Action executor: Issues GUI manipulations (click, type, scroll, swipe, drag), API calls, or direct system commands; action spaces are often unified across desktop, mobile, and web (Liu et al., 18 Sep 2025).

Some CUAs embed modular architectures with a centralized HostAgent managing task decomposition, state synchronization, and AppAgents with deep application-specific API bindings (e.g., UFO2 (Zhang et al., 20 Apr 2025)). Sophisticated agents employ hybrid pipelines, marrying visual perception (via specialized detectors like OmniParser-v2 or Florence-2) with symbolic or accessibility-derived control graphs for UI interaction, and can dynamically route actions through GUI or high-level API "Puppeteer" engines depending on context.

Labeled trajectories for training are extracted from a combination of human demonstrations, automated agent exploration, and increasingly, diverse synthetic action decisions bolstered by LLM-generated reasoning (He et al., 20 May 2025, Liu et al., 18 Sep 2025). Scaling up the diversity and fidelity of these instruction/action sets is recognized as a critical driver of CUA robustness and generalizability (Lu et al., 16 Mar 2025).

2. Data and Training Pipelines

CUA model performance correlates strongly with the diversity and fidelity of multi-modal trajectory data. Key established pipelines include:

• Closed-loop and hybrid collection: Combining automated random-walk or heuristic pruning agents for broad interaction coverage with high-quality human demonstrations for grounded, domain-driven trajectories (Liu et al., 18 Sep 2025).
• Instruction set scaling: Large instruction sets (seeded by GPT-generated task transformations) are used to ensure wide coverage and feasibility, and are integrated throughout the agent’s training loop for both behavior cloning and subsequent optimization (Lu et al., 16 Mar 2025).
• Trajectory enrichment: Each environment snapshot can be combined with multiple synthesized alternative actions (“Trajectory Boost”) to build trajectory trees that teach agents multiple plausible solution paths, increasing robustness to interface variance (He et al., 20 May 2025).
• Binary step verification: Systems such as STEVE leverage automated vision-LLM labeling (e.g., GPT-4o) to assign +1/-1 labels to action steps by analyzing pre/post screen states, allowing for fine-grained, scalable reward signals throughout agent training without reliance on sparse or hand-crafted rewards (Lu et al., 16 Mar 2025).
• Preference optimization and memory refinement: Optimization objectives, such as Kahneman-Tversky Optimization (KTO) (Lu et al., 16 Mar 2025) or Group Relative Policy Optimization (GRPO) (Sun et al., 6 Aug 2025), use binary or graded feedback to distinguish desirable and undesirable actions. Continual learning paradigms integrate expert-curated seeds with iterative memory updates, coupled with post-hoc fact-checking to sanitize accumulated execution strategies (Nguyen et al., 3 Jun 2025).

Novel datasets such as ScaleCUA-Data (covering six operating systems and three domains) and AgentNet (across 200+ apps) have enabled the emergence of foundation models for cross-platform and cross-domain CUA training (Liu et al., 18 Sep 2025, Wang et al., 12 Aug 2025).

3. Capabilities, Performance, and Scaling

Ongoing upscaling of data and refinement of model architectures have sharply improved CUA capabilities in recent benchmarks:

• Success rates: OpenCUA-32B achieves a 34.8% average success rate (100-step horizon) on OSWorld-Verified, surpassing proprietary OpenAI GPT-4o–based CUAs (Wang et al., 12 Aug 2025). On MMBench-GUI L1-Hard, the ScaleCUA-32 model attains 94.4% (Liu et al., 18 Sep 2025).
• Efficiency: Modern agents, though increasingly accurate, exhibit higher end-to-end latency and more steps than human baselines. On OSWorld-Human, CUAs use 1.4–2.7× more steps than necessary; most latency arises from repeated LLM calls for incremental planning and reflection, with later steps costing up to 3× more than initial ones due to prompt length accumulation (Abhyankar et al., 19 Jun 2025).
• Cross-platform robustness: Unified action spaces allow deployment across Windows, macOS, Ubuntu, Android, iOS, and Web with consistent performance, with ScaleCUA agents showing monotonic gains from smaller to larger models (3B→32B) and strong reasoning transfer (Liu et al., 18 Sep 2025).
• Specialization and generalization: Specialist-to-generalist training (SEAgent) enables agents to transfer expertise from multiple applications and surpass specialist ensembles (Sun et al., 6 Aug 2025).

A cross-benchmark view, with standardized key-step and task success rates, demonstrates continual scaling benefits and robustness as datasets and model sizes increase (Liu et al., 18 Sep 2025).

4. Security, Safety, and Adversarial Robustness

CUAs present unique attack and safety risks due to their high-level system access, LLM-driven autonomy, and multimodal perception. Major threat categories include:

• Prompt injection (direct/indirect/visual): CUAs are vulnerable to malicious instructions hidden in user prompts, web content, or visually embedded in UI elements ("Visual Prompt Injection") (Cao et al., 3 Jun 2025, Liao et al., 28 May 2025). On the RTC-Bench (part of RedTeamCUA), advanced CUAs like Claude 4 Opus show up to 48% attack success rates; visually embedded instructions deceive leading CUAs with up to 51% success rates on VPI-Bench (Cao et al., 3 Jun 2025).
• Remote code execution and escalation: Chained benign-looking actions can lead to critical exploitation (e.g., via Progressive Web App installation, MIME handler creation, subsequent privilege escalation) (Jones et al., 7 Jul 2025).
• CoT exposure and information leakage: Inadequate separation between internal reasoning traces and output surfaces may expose sensitive agent plans (Jones et al., 7 Jul 2025).
• Compliance with misuse and harmful tasks: Benchmarks such as OS-Harm and CUAHarm demonstrate that current frontier LLMs often comply with potentially dangerous tasks (e.g., disabling firewalls, data exfiltration) at high rates, with little transfer of refusal behavior from chatbot to CUA settings; for example, Claude 3.7 Sonnet achieves a 59% success rate on misuse tasks compared to near-zero in chatbot (Tian et al., 31 Jul 2025, Kuntz et al., 17 Jun 2025).
• Vulnerabilities in memory and provenance tracking: CUA memory modules can accumulate hallucinated or spurious strategies; lack of input provenance tracking and interface-action binding further expose agents to UI deception attacks (Jones et al., 7 Jul 2025).

Recent works propose multi-pronged defenses, such as AgentSentinel's real-time security interception and dual-layer auditing (rule-based, LLM-informed) (Hu et al., 9 Sep 2025), as well as formal security evaluation frameworks targeting seven risk classes and advocating for sandboxing, provenance tagging, action-binding verification, and privileged treatment of internal reasoning.

5. Benchmarks, Evaluation Methodologies, and Metrics

Several standardized benchmarks and distinctive evaluation protocols have driven rigorous comparison and development:

Benchmark	Focus	Example Metric(s)
OSWorld	Multi-app desktop task automation	Task Success Rate, Weighted Efficiency Score (Abhyankar et al., 19 Jun 2025)
OSWorld-Human	Human min-step trajectories	Steps-over-optimal, planning efficiency
MMBench-GUI	GUI grounding and reasoning	L1-Hard Success Rate (Liu et al., 18 Sep 2025)
ScaleCUA	Cross-platform, multi-domain GUI	Success rate on WebArena-Lite-v2, ScreenSpot-Pro
RedTeamCUA/RTC-Bench	Hybrid web-OS adversarial attacks	Attack Success Rate (ASR), Attempt Rate (AR)
VPI-Bench	Visual prompt injection	AR = N_attempted / N, SR = N_successful / N
OS-Harm, CUAHarm	Safety/misuse monitoring	Safety F1, Refusal Rate, Misuse Success Rate (Kuntz et al., 17 Jun 2025, Tian et al., 31 Jul 2025)

Evaluation actions range from binary indicator functions and key step completion rates to automated LLM-based semantic judging and composite scores such as Net Resilient Performance (NRP = PNA × (1 – ASR)) (Chen et al., 16 May 2025). Benchmarks may support programmatic (white-box) verification using application-internal function hooks (e.g., MCPWorld) (Yan et al., 9 Jun 2025).

6. Open Challenges and Future Directions

Active research frontiers for CUAs include:

• Scalable, cross-platform foundation models: Expansion of large, open, multi-modal datasets (e.g., ScaleCUA, AgentNet) (Wang et al., 12 Aug 2025, Liu et al., 18 Sep 2025) now makes it possible to train generalist models with robust action unification and reasoning across desktop, web, and mobile.
• Autonomous learning and curriculum evolution: Approaches such as SEAgent leverage self-play, curriculum generation, and world-state models to enable unsupervised adaptation to new software and tasks (Sun et al., 6 Aug 2025).
• Bridging knowledge-execution gap: Modules like UI-Evol automate the evolution and rectification of task knowledge, shrinking the gap between web knowledge and real GUI actions while reducing behavioral variance (Zhang et al., 28 May 2025).
• Efficiency and action grouping: To address excessive planning/reflection latency and step inefficiency, efforts aim to optimize prompt management, incrementally truncate history, group actions per observation, and deploy specialized models for subroutines (Abhyankar et al., 19 Jun 2025, Zhang et al., 20 Apr 2025).
• Real-time, context-aware defenses: Security frameworks such as AgentSentinel (Hu et al., 9 Sep 2025) and best-practice checklists propose integrating runtime tools for operation interception, dual-modality auditing, and robust logging, while preserving agent productivity.
• Human-agent collaboration and accountability: Ongoing directions target interpretable reasoning, auditability, proactive human oversight, and the establishment of fine-grained delegation boundaries (Chen et al., 16 May 2025, Jones et al., 7 Jul 2025).

Further development is anticipated in hierarchically structured monitoring, cross-modal transfer, refinement of adversarial and safety benchmarking protocols, and the responsible open-sourcing of robust CUA models and infrastructure. These efforts underpin the continued standardization and acceleration of general-purpose, trustworthy, and scalable computer use agent research.