Agentic Automation with Computer Use (AACU)
- AACU is a paradigm where LLM-driven agents interpret high-level goals, decompose them into multi-step workflows, and execute tasks via interfaces rather than static scripts.
- The architecture incorporates hierarchical planning, diverse action modalities (e.g., GUI actions and code execution), and dynamic observation interfaces to ensure effective task management.
- Empirical evaluations indicate AACU systems enhance efficiency and adaptability in various domains through optimized execution, reduced error surfaces, and flexible prototyping.
Agentic Automation with Computer Use (AACU) denotes a class of agentic systems in which LLM- or MLLM-based agents interpret high-level goals, decompose them into multi-step workflows, access knowledge and tools, and operate computer environments through software interfaces rather than merely producing textual advice. In the recent literature, AACU is defined in contrast to traditional Robotic Process Automation (RPA), which relies on predefined workflows, selectors, and scripting, and is also continuous with earlier notions of agentic process automation in which agents participate in both workflow construction and execution (Průcha et al., 4 Sep 2025, Ye et al., 2023). Across the surveyed work, AACU encompasses GUI-centric agents, hybrid GUI-plus-code systems, tool-augmented desktop agents, web agents, and domain-specific systems whose central property is execution on behalf of a user within bounded or partially bounded environments.
1. Definition and conceptual boundaries
AACU is characterized in the literature as automation through computer use rather than through static workflow replay alone. One line of work defines it as LLM agents that perform tasks through natural-language instructions and autonomous interaction with user interfaces, emphasizing flexibility, rapid prototyping, and adaptation to changing interfaces (Průcha et al., 4 Sep 2025). A related but broader enterprise framing defines agentic AI as systems that can interpret a goal, decompose it into intermediate steps, access relevant knowledge, call tools or systems, monitor intermediate outcomes, and return or execute a result within defined boundaries; this framing places AACU within a larger category of workflow-executing systems with varying degrees of autonomy (Koch et al., 15 Jun 2026).
The boundary between AACU and earlier automation paradigms is therefore not simply “uses a GUI” versus “uses an API.” Earlier work on Agentic Process Automation already proposed that LLM-based agents could offload the intelligent parts of process automation by participating in workflow construction and workflow execution, with deterministic code handling routine steps and agents handling semantic data-flow and control-flow decisions (Ye et al., 2023). AACU extends that trajectory toward direct operation of live computer environments, including desktops, browsers, and application GUIs, and, in later systems, toward mixed action spaces that include code, tools, or scripts alongside clicks and keystrokes (Song et al., 5 Aug 2025).
A recurrent misconception in this literature is that AACU is reducible to GUI imitation. Several recent systems explicitly reject that reduction. CoAct-1 argues that many computer-use tasks are not inherently “GUI tasks” but “system tasks,” and that forcing every backend operation into a sequence of GUI manipulations creates brittleness and inefficiency (Song et al., 5 Aug 2025). UltraCUA makes a similar point by contrasting primitive actions such as click, type, and scroll with higher-level programmatic tool calls, shortcuts, and scripts (Yang et al., 20 Oct 2025). This suggests that AACU is best understood as computer-mediated task execution with heterogeneous action modalities, not merely vision-guided mouse control.
2. Action spaces and architectural patterns
A defining design axis in AACU is the action repertoire available to the agent. Early open GUI frameworks such as Agent S constrain the agent to bounded discrete actions including click, type, scroll, hotkey, hold_and_press, drag_and_drop, save_to_buffer, switch_applications, wait, done, and fail, with one action emitted per timestep through an Agent-Computer Interface (ACI) (Agashe et al., 2024). The rationale is to bound control, force immediate feedback after each step, and separate high-level planning from low-level execution.
Later work generalizes the action space in two directions. CoAct-1 introduces coding as a first-class action modality in a three-agent hierarchy consisting of an Orchestrator, a GUI Operator, and a Programmer. The Orchestrator cannot act directly on the operating system; instead, it decomposes the task, delegates subtasks either to a GUI worker or to a coding worker, and then updates memory from returned summaries and screenshots. The Programmer writes and executes Python or Bash through a code interpreter, making it suitable for file management, data processing, configuration editing, and other operations that are cleaner as scripts than as GUI sequences (Song et al., 5 Aug 2025). UltraCUA follows a related logic at foundation-model scale, training a single model to alternate between GUI primitives and high-level tool calls, shortcuts, and scripts when those offer a safer or shorter route to completion (Yang et al., 20 Oct 2025).
A third architectural pattern inserts an explicit procedural layer between high-level intent and primitive actions. CUA-Skill defines a skill as
where is the suitable application, the natural-language intent, the argument schema, and a parameterized execution graph. Skills are retrieved dynamically, instantiated with arguments, composed through a skill composition graph, and executed through GUI primitives or script/API calls (Chen et al., 28 Jan 2026). This shifts AACU from flat trajectory prediction toward reusable procedural abstractions.
Across these systems, a common architecture emerges: a planner or manager reasons over the global task; execution modules operate at a narrower scope; memory is summarized rather than naively shared; and specialized workers are invoked only when their modality is appropriate. The main differences concern how much structure is hard-coded into the action interface, whether skills are retrieved or learned implicitly, and whether code or tool use is treated as an external augmentation or as a native component of the action space.
3. Observation, grounding, memory, and control loops
AACU performance depends not only on the action interface but also on the observation interface and the control loop that links perception to execution. Agent S illustrates a now-standard design in which the agent consumes both a screenshot and an annotated accessibility tree, augmented with OCR blocks when those are not already represented, and uses hierarchical planning with narrative memory at the manager level and episodic memory at the worker level (Agashe et al., 2024). Its manager retrieves external web knowledge and internal narrative memory to form subtasks, while workers retrieve subtask-level experience and use trajectory reflection during execution.
The observation problem becomes more acute for dynamic environments. The AOI work argues that current computer-use agents are “blind between screenshots and deaf to audio” because observation is tied to action, typically as one screenshot every 3–5 seconds with no audio channel. It introduces the Agent-Computer Observation Interface (AOI), which decouples continuous observation from discrete action through inter-step keyframe capture, volume-gated audio transcription, and CU-model-generated visual narration that persists as text (Li et al., 28 Jun 2026). A notable empirical conclusion is that keyframe selection itself matters less than converting visual changes into persistent textual narration, indicating that temporal memory and compression are central observation problems in AACU rather than mere screenshot frequency.
Long-horizon control is formalized most explicitly in CarePilot, a healthcare-specific multi-agent framework organized around an actor–critic loop. The Actor predicts the next semantic action from the current screen, task goal, grounded tool outputs, and dual memories; the Critic evaluates the proposed action, updates memory, and either accepts it or emits corrective feedback. The paper writes the action policy as
and the Critic score as
If , the action is accepted; otherwise, structured feedback is generated and memory is revised (Ghosh et al., 25 Mar 2026).
These designs converge on a general control principle: AACU agents require a persistent belief state over the task, not just the current image. Observation must encode recent change, latent workflow state, and sometimes non-visual signals such as speech or tool outputs. Planning must therefore be hierarchical or memory-aware, and execution must include some form of self-evaluation, reflection, or runtime critique.
4. Benchmarks, evaluation protocols, and empirical performance
AACU evaluation is unusually heterogeneous because different papers target different environments and notions of success. OSWorld has become the main desktop benchmark: it contains 369 real-computer Linux tasks spanning productivity suites, IDEs, browsers, file managers, and multi-application workflows, and uses a rule-based evaluator built from 134 atomic execution-based checks combined with Boolean logic (Song et al., 5 Aug 2025). On this benchmark, Agent S established an earlier open-framework milestone with 20.58% overall success rate against a best baseline of 11.21%, while CoAct-1 later reported 60.76% success in the 100+ step setting and 59.93% at the 100-step budget, together with an average of 10.15 steps on successful tasks, compared with 15.22 for GTA-1 and 14.90 for UI-TARS (Agashe et al., 2024, Song et al., 5 Aug 2025).
WindowsAgentArena measures cross-platform desktop generalization. Agent S reported 18.2% on the 154-task benchmark without explicit adaptation, exceeding NAVI at 13.3% (Agashe et al., 2024). UltraCUA later reported 21.7% success rate in out-of-domain Windows evaluation despite training on Ubuntu-based tasks, and CUA-Skill Agent reported 57.5% successful rate in best-of-three evaluation, which the paper presents as state of the art (Yang et al., 20 Oct 2025, Chen et al., 28 Jan 2026). These results indicate that performance differences are increasingly driven by action abstraction, skill reuse, and hybrid execution rather than by raw multimodal capability alone.
Web-centric AACU uses a different benchmark ecology. Fara-7B, a native screenshot-based 7B web agent trained on synthetic trajectories from FaraGen, reports 73.5% on WebVoyager, 34.1% on Online-Mind2Web, 26.2% on DeepShop, and 38.4% on WebTailBench, while also achieving 86.7 on ScreenSpot-V1 and 89.3 on ScreenSpot-V2 grounding benchmarks (Awadallah et al., 24 Nov 2025). Its significance lies as much in efficiency as in accuracy: the reported cost per task is $0.025 on WebVoyager and$0.069 on WebTailBench, much lower than several frontier-backed agents (Awadallah et al., 24 Nov 2025).
Dynamic and domain-specific benchmarks expose additional failure modes. DynaCU-Bench contains 100 dynamic browser tasks plus a 50-task static control set, and AOI improves evaluated models by roughly +17 to +48 percentage points over screenshot baselines with zero retraining; on Static-50, the AOI full configuration reaches 50/50 while the standard loop reaches 43/50 (Li et al., 28 Jun 2026). CareFlow, a healthcare benchmark of about 1,100 long-horizon tasks, shows that step-wise accuracy and task completion diverge materially: CarePilot reaches 90.18 step-wise accuracy and 51.45 task accuracy with Qwen 3 VL-8B, while strong baselines can exhibit reasonable step-wise accuracy but much lower task accuracy (Ghosh et al., 25 Mar 2026). BIMgent’s Mini Building Benchmark, comprising 25 building-modeling tasks and over 2000 action steps, reports 32% end-to-end success while all baseline models fail to complete any full task (Deng et al., 8 Jun 2025).
A consistent empirical pattern follows from these results. AACU systems are evaluated not merely by next-action quality but by execution-grounded completion, and end-to-end success is highly sensitive to action count, trajectory length, and the agent’s ability to select the right action modality. This is especially explicit in CoAct-1 and UltraCUA, both of which argue that shortening trajectories by replacing long GUI sequences with code or tool calls reduces the error surface (Song et al., 5 Aug 2025, Yang et al., 20 Oct 2025).
5. Relation to RPA, enterprise integration, and operational control
AACU is frequently compared with classical RPA because both aim to automate digital workflows. The comparative study “Are LLM Agents the New RPA?” evaluates RPA via UiPath and AACU via Anthropic’s Computer Use Agent on three rpachallenge.com workflows—data entry, monitoring, and document extraction—and measures speed, reliability, and development effort (Průcha et al., 4 Sep 2025). The reported conclusion is not that AACU already replaces RPA. RPA is significantly faster on the comparable processes, AACU fails to complete the data-entry process, and descriptive reliability favors RPA even though small-sample statistical tests do not establish significance at (Průcha et al., 4 Sep 2025). At the same time, AACU requires less development time, adapts more flexibly to dynamic interfaces, and appears attractive for rapid prototyping and lightweight automation.
This pattern aligns with later enterprise position papers. The “Integrator Advantage” paper argues that near-term value lies not in full autonomy or workforce reduction, but in controlled partial autonomy for simple and medium-complexity business processes, especially where work is frequent, semi-structured, digital, verifiable, and bounded in downside risk (Koch et al., 15 Jun 2026). It recommends API integration as the preferred path, with RPA/UI automation used as a brittle adapter when APIs do not exist. It also proposes an autonomy ladder in which small and medium-sized companies should typically target Levels 2 to 4 rather than fully autonomous Level 6 systems (Koch et al., 15 Jun 2026).
Operationally, AACU introduces uncertainty sources that conventional observability stacks do not address well. AgentOps proposes a six-stage “AI AgentOps Automation Pipeline”: Observe Behavior, Collect Metrics, Detect Issues, Identify Root Cause, Optimize Recommendations, and Automate Operations (Moshkovich et al., 15 Jul 2025). The framework treats agentic systems as stateful, adaptive entities whose probabilistic reasoning, evolving memory, dynamic tool use, and multi-agent coordination require runtime control, self-healing, and telemetry rather than one-shot debugging. For AACU, this amounts to a control plane around execution: tracing tool calls, monitoring loops and task decomposition, diagnosing failures, and applying runtime changes to prompts, tools, thresholds, workflows, or guardrails without redeployment (Moshkovich et al., 15 Jul 2025).
A plausible implication is that enterprise AACU will not converge on a single automation substrate. Instead, the literature points toward layered deployments in which deterministic APIs, RPA-style components, and agentic computer use coexist, with AACU handling ambiguity and interface variability while more structured mechanisms handle stable standard cases.
6. Governance, explanation, and human oversight
Because AACU systems act rather than merely answer, the literature places unusual emphasis on visibility, control, and accountability. The communication-oriented interface paper argues that greater agentic capability should mean less routine back-and-forth but more communication for oversight and explanation, and it proposes three explanation types: action-process explanations, uncertainty explanations, and coordination explanations (Jang et al., 2 May 2026). Action-process explanations expose what the agent did and in what order; uncertainty explanations reveal assumptions, ambiguity, and likely failure points; coordination explanations clarify how multiple agents or tools divide work and how conflicts are resolved.
Enterprise governance proposals make these requirements more concrete. The “Integrator Advantage” paper specifies a minimal governance baseline comprising an agent register, business owner, technical owner, purpose statement, risk classification, data classification, permission model, audit trail, human-in-the-loop rules, escalation logic, quality monitoring, incident process, change management, and a kill switch (Koch et al., 15 Jun 2026). It also recommends distinct agent identity, explicit delegation, least-privilege permissions, auditable actions, and revocable credentials, while warning against shared service accounts or implicit operation under a human identity (Koch et al., 15 Jun 2026).
AgentOps extends this governance view into runtime observability. Its issue taxonomy includes task failures, component issues, feedback-handling failures, metric-based failures, and security/compliance issues such as access control breaches, sensitive data exposure, resource misuse, SLA violations, and unintended side effects (Moshkovich et al., 15 Jul 2025). The framework’s “taming, not eliminating, uncertainty” formulation is especially relevant to AACU, where nondeterminism arises not only in outputs but also in intermediate routing decisions, memory retrieval, tool choice, and runtime-generated code (Moshkovich et al., 15 Jul 2025).
Human oversight in AACU is thus not merely a final approval checkbox. The literature treats it as a structured arrangement involving explanation, override rights, escalation paths, visible evidence, and calibrated trust. This suggests that the mature AACU interface is neither a silent executor nor a conventional chatbot, but an execution system with selective transparency, controllable autonomy, and explicit governance hooks.
7. Domain-specific expansion and collaborative directions
Although much AACU work centers on general desktop or web tasks, several papers show that the paradigm is expanding into specialized professional domains. In healthcare, CarePilot targets long-horizon software workflows across DICOM viewers, medical annotation tools, EMR/EHR systems, and laboratory information systems. CareFlow contains about 1,100 tasks with average trajectory lengths around 12–13 steps and a core semantic action space of CLICK, SCROLL, ZOOM, TEXT, SEGMENT, and COMPLETE (Ghosh et al., 25 Mar 2026). The paper argues that heterogeneity, partial observability, and compliance sensitivity make healthcare a stringent AACU setting, and its results support the claim that long-term memory, tool grounding, and critique are more important than isolated one-step prediction (Ghosh et al., 25 Mar 2026).
In the Architecture, Engineering, and Construction sector, BIMgent applies AACU to BIM authoring in Vectorworks. Its pipeline separates a design layer, an action-planning layer, and an execution layer, uses DeepFloorPlan and multimodal floorplan interpretation, and relies on hierarchical planning plus dynamic GUI grounding of pop-up dialogs (Deng et al., 8 Jun 2025). The system reports 32% end-to-end success on 25 real-world building-modeling tasks, with especially strong subtask results for walls and openings, while generic baselines achieve 0% end-to-end success (Deng et al., 8 Jun 2025). The case demonstrates that AACU can be extended from routine office automation to open-ended, high-parameter professional software.
A different expansion concerns software engineering and scientific tooling. “Autonomous Computer Vision Development with Agentic AI” shows an agentic system built with OpenManus and SimpleMind that interprets a natural-language prompt, generates a YAML knowledge-graph configuration, verifies it, runs SM-Learn for training and SM-Think for inference, and reports mean Dice scores of 0.963 for lungs, 0.824 for heart, and 0.830 for ribs on chest X-ray segmentation tasks (Kim et al., 11 Jun 2025). This work is not GUI-centric in the strict sense, but it fits AACU’s broader pattern of planning, configuring, verifying, and executing software workflows through live computer tools.
Finally, AACU is beginning to encompass not only delegation but also instruction. DigitalCoach introduces a multimodal dataset of 72 expert–novice coaching sessions, 22,752 dialogue turns, and 28.1 hours of recordings across Excel, Figma, Blender, FL Studio, and Onshape, and evaluates whether models can teach humans to use software (Chen et al., 30 Jun 2026). Automated and interactive evaluation show that current models give more direct instructions but fewer explanations, diagnoses, and knowledge-check questions than humans, and that human coaching yields larger post-task gains than model coaching (Chen et al., 30 Jun 2026). This shifts the scope of AACU from “agent uses the computer for the user” to the broader possibility that the agent observes, explains, and coaches computer use while preserving human agency.
Taken together, these domain-specific systems indicate that AACU is not a single benchmark-defined technique but a growing systems area. Its mature forms combine heterogeneous actions, structured memory, grounded perception, execution-based verification, runtime governance, and, increasingly, collaborative or pedagogical roles alongside autonomous workflow execution.