ComputerAgent: Autonomous Software Agents

• ComputerAgent is defined as a computer-based software agent that perceives its environment, reasons with goals and constraints, and executes actions through various interfaces.
• They employ modular architectures with a perception–reasoning–action cycle, integrating components like security, memory management, and planning to enhance functionality.
• Applications range from endpoint security and mobile agents to multi-agent orchestration, emphasizing reliability, human-intent alignment, and risk management.

Searching arXiv for papers on computer-use agents and related agent architectures/security to ground the article. ComputerAgent denotes a computer-based software agent that perceives computational environments, reasons over goals and constraints, and acts through operating-system, application, tool, or network interfaces. Across recent literature, the term spans several closely related forms: autonomous endpoint agents embedded inside an operating system, mobile agents that migrate across hosts while preserving code and state, GUI and OS agents that operate software interfaces on behalf of users, LLM-powered multi-agent systems that plan and execute long-horizon tasks, and computer-use agents whose actions carry real authority in deployed environments (R et al., 11 Nov 2025, Amro, 2014, Li et al., 22 Oct 2025, Chen et al., 8 May 2026). A unifying characterization is the continual perception–decision–action cycle, often implemented through modular architectures, explicit memory, tool invocation, and policy-driven execution over partially observable software state (R et al., 11 Nov 2025, Chen et al., 8 May 2026).

1. Definitions and conceptual scope

The concept of ComputerAgent covers several established traditions in computer science. In one formulation, it is “an autonomous, event-driven software entity embedded in a Windows operating system” that continuously senses system activity, interprets it using learned models and domain knowledge, and acts to mitigate risk (R et al., 11 Nov 2025). In another, it is a mobile agent: “a program that can exercise an individual's or organization's authority, work autonomously toward a goal, and meet and interact with other agents,” with code, state, and attributes that can migrate across agent platforms (Amro, 2014). In recent OS-agent work, it is an intelligent, persistent system that sits “inside” the operating system, observes screenshots or UI hierarchies, interprets user instructions, and issues low-level actions such as clicks, swipes, typing, or system-button events (Li et al., 22 Oct 2025).

A broader systems view treats computer-use agents as policies operating over real software environments rather than prompt-only models. In that framing, the agent acts in browsers, desktops, mobile apps, file systems, terminals, and tool backends, where reliability depends not only on task success but also on perception correctness, planning fidelity, memory use, tool mediation, permission scope, and runtime oversight (Chen et al., 8 May 2026). A plausible implication is that “ComputerAgent” is best understood not as a single architecture class but as a family of autonomous software systems coupled to digital environments through authority-bearing interfaces.

This family includes specialized endpoint security agents, desktop GUI agents, mobile GUI agents, multi-agent orchestrator-worker systems, agent protocol participants, and benchmarked computer-control policies (R et al., 11 Nov 2025, Liu et al., 20 Feb 2025, Li et al., 22 Oct 2025, Jiang et al., 6 Feb 2026, Zheng et al., 25 Mar 2026, Dong et al., 22 Sep 2025). The common denominator is operational autonomy under software mediation.

2. Architectural patterns and the perception–reasoning–action loop

A recurrent architectural pattern is modular decomposition into sensing, reasoning, execution, and coordination. The endpoint security agent in “Endpoint Security Agent: A Comprehensive Approach to Real-time System Monitoring and Threat Detection” is explicitly described as implementing the classic loop “perception → reasoning → action” through a System Monitoring Module, Data Collection Module, Security Analysis Module, Response System, Communication Module, and Event Logging and Reporting (R et al., 11 Nov 2025). The monitoring module captures process events, network connections, file system operations, registry modifications, and services and user account activities; the analysis module combines signature-based detection, behavioral analysis, machine learning–driven anomaly detection, and ATT&CK-informed logic; the response system executes policy-driven actions such as process termination, network isolation, file quarantine, and user account restriction (R et al., 11 Nov 2025).

OS agents adopt analogous decompositions but orient them toward user tasks rather than threat response. ColorAgent places an Execution module at the center and surrounds it with a Knowledge Retrieval module for app- or task-specific information, a Task Orchestration module for decomposition and memory transfer across subtasks, and a Hierarchical Reflection module for step-level, trajectory-level, and global task verification (Li et al., 22 Oct 2025). PC-Agent similarly decomposes decision-making into Instruction–Subtask–Action levels, assigning separate Manager, Progress, Decision, and Reflection agents to instruction decomposition, progress tracking, stepwise action generation, and bottom-up error feedback (Liu et al., 20 Feb 2025).

Lemon Agent formalizes a related Planner–Executor–Memory paradigm through AgentCortex. Its orchestrator routes tasks to workers, each worker executes subtasks via ReAct-style loops and dynamic parallel tool calling, and a self-evolving memory system stores execution traces, skills, decision rules, and tool-usage patterns (Jiang et al., 6 Feb 2026). This suggests that contemporary ComputerAgent design has shifted from monolithic controllers toward layered and distributed controllers in which local execution and global coordination are separated for efficiency and robustness.

A different, but compatible, architecture appears in AgentSys, which organizes agents hierarchically for security. A main agent maintains trusted long-horizon context, while worker agents process tool outputs in isolated contexts and return only schema-validated JSON results to the main agent (Wen et al., 7 Feb 2026). This design treats memory boundaries as first-class architectural elements rather than incidental prompt structure.

3. Environment interfaces and modalities of control

ComputerAgents differ significantly in the interfaces through which they perceive and act. Endpoint security agents attach directly to operating-system telemetry. The Windows endpoint agent uses native tools like WMI and ETW to subscribe to process creation and termination, service status changes, hardware and OS configuration changes, detailed process and thread events, file I/O traces, registry I/O, and network stack events (R et al., 11 Nov 2025). Such agents operate at low system levels and act through local remediation and centralized server coordination.

GUI and OS agents operate one layer above, through rendered interfaces. ColorAgent is environment-facing in the sense that it sees screenshots and UI hierarchies and emits actions such as click, long_press, swipe, system_button(Back/Home/Menu/Enter), open, type, clear_text, wait, answer, and terminate (Li et al., 22 Oct 2025). PC-Agent similarly acts through desktop GUI primitives including OpenApp, Click, DoubleClick, Select(text), Type, Drag, Scroll, Shortcut, and Stop, while its Active Perception Module augments screenshots with accessibility-tree information and OCR-based text localization (Liu et al., 20 Feb 2025). AppAgent-Pro follows the same GUI-agent paradigm in a mobile setting, operating applications such as YouTube and Amazon through real-time screenshots and app navigation, but distinguishes between shallow execution and deep execution for proactive, multi-app information acquisition (Zhao et al., 26 Aug 2025).

Another class acts through terminals and tool APIs. The computer-use agents evaluated in AdvCUA are OS-domain agents whose main actuator is shell-command execution. ReAct, AutoGPT, Gemini CLI, Cursor CLI, and Cursor IDE perceive command outputs and act by issuing terminal commands inside a multi-host sandbox, allowing them to perform privilege escalation, persistence, credential access, lateral movement, and exfiltration tasks (Luo et al., 8 Oct 2025). MadAgents exemplifies a domain-specific instance of this pattern: an orchestrator and specialized worker agents manipulate MadGraph, ROOT, FastJet, shell tools, web search, and PDF reading to install software, generate events, analyze outputs, and document simulation campaigns (Plehn et al., 28 Jan 2026).

The mobile-agent tradition adds host mobility as an interface modality. In the mobile-agent model, the entire computational component—code, state, and attributes—moves from one platform to another, suspending execution on one host and resuming on another (Amro, 2014). This differs from both GUI operation and terminal control because the agent itself is the unit of motion.

4. Learning, planning, memory, and decision structures

A central line of variation among ComputerAgents concerns how they make decisions. Several systems rely heavily on machine learning. The Windows endpoint agent extracts 45 security-related features spanning process, registry, file system, network, temporal, and contextual dimensions, and combines Isolation Forest, a fine-tuned Llama model with LoRA, and a risk scoring system (R et al., 11 Nov 2025). Its risk score is explicitly defined as

$$\text{Risk Score} = 0.3 \times \text{Anomaly Score} + 0.2 \times \text{Event Frequency Score} + 0.25 \times \text{Severity Score} + 0.15 \times \text{Asset Criticality} + 0.1 \times \text{User Risk Score}.$$

The same system reports standard classification formulations and evaluation metrics for precision, recall, and $F_1$ (R et al., 11 Nov 2025).

ColorAgent approaches long-horizon control with step-wise reinforcement learning and self-evolving training. It constructs step-wise samples $\{I, h_{t-1}, s_t, a_t\}$, applies multi-path augmentation and difficulty-based filtering, and uses Group Relative Policy Optimization with rule-based format and accuracy rewards (Li et al., 22 Oct 2025). The reward design explicitly enforces parsable and executable outputs, reflecting the operational demands of GUI control. ComputerAgent, in the strict sense of the desktop-control RL framework “Towards General Computer Control with Hierarchical Agents and Multi-Level Action Spaces,” formulates OS control as a two-level option process with a manager and subpolicy: $$\pi(a_t \mid s_t) = \pi_H(a_t^H \mid s_t) \cdot \pi_L(a_t^L \mid s_t, a_t^H).$$ It defines a hierarchical action space over single keys, hotkeys, meta-actions, and mouse control, and trains DQN, PPO, and A2C variants under a shaped reward that decomposes manager and subpolicy contributions (Dong et al., 22 Sep 2025).

Memory has emerged as a major design axis. ColorAgent uses a task extractor and task rewriter to transfer information from one subtask trajectory into the prompt for the next (Li et al., 22 Oct 2025). Lemon Agent introduces Self-Evolving Semantic Memory, which stores execution traces, skills, decision rules, and tool affinities, and retrieves top-$k$ memories subject to similarity thresholding (Jiang et al., 6 Feb 2026). AgentSys, by contrast, treats unrestricted memory accumulation as a security liability and isolates raw tool outputs and subtask traces in worker-local contexts, exposing only structured return values to the main agent (Wen et al., 7 Feb 2026). This suggests two competing but compatible memory doctrines: memory as accumulated reusable skill, and memory as a selectively bounded channel.

Planning structures range from explicit task decomposition to emergent tool sequencing. PC-Agent’s Manager Agent produces parameterized subtasks and maintains a communication hub for inter-subtask dependencies (Liu et al., 20 Feb 2025). Lemon Agent’s orchestrator decides whether to spawn one or more workers based on task complexity and decomposability (Jiang et al., 6 Feb 2026). AgentStore introduces a MetaAgent that either routes a task to a single specialist agent or enters a manager mode that predicts a top-$K$ subset of agents and decomposes the task across them (Jia et al., 2024). These systems collectively indicate that long-horizon ComputerAgent design increasingly relies on hierarchical control and explicit decomposition.

5. Security, reliability, and authority

Security and reliability are now inseparable from the definition of a ComputerAgent because many such agents operate with direct authority over files, accounts, network access, or endpoint controls. The deployment-grounded framework for computer-use agents formalizes this by treating a CUA as a policy

$\pi(a_t \mid h_t, o_t, g, c),$

where $o_t$ is partial observation, $h_t$ is internal state or history, $g$ is the user goal, and $F_1$0 is the constraint set (Chen et al., 8 May 2026). Its architectural analysis divides failure sources into Perception, Decision, and Execution, and its lifecycle analysis tracks Creation, Deployment, Operation, and Maintenance as distinct points where authority exposure and control placement matter (Chen et al., 8 May 2026).

Two recent lines of work focus specifically on indirect prompt injection and execution drift. AgentSys argues that conventional agents indiscriminately append all tool outputs and traces into the main context, allowing injected instructions to persist across many decision points (Wen et al., 7 Feb 2026). Its hierarchical isolation alone reduces attack success to 2.19%, and the full system achieves 0.78% ASR on AgentDojo and 4.25% on ASB while slightly improving benign utility over undefended baselines (Wen et al., 7 Feb 2026). Agent-Sentry instead attempts to bound agentic systems via execution provenance. It constructs benign, attack, and ambiguous functionality graphs from traces and blocks action tool calls that deviate from learned flows or fail a trusted-input intent-alignment check. On Agent-Sentry Bench under full functionality graph coverage, overall utility is 94.61% and ASR is 9.46%; on AgentDojo Dynamic Benchmark it achieves 76.30% utility with 3.70% ASR, compared with 47.69% ASR for the undefended baseline (Sequeira et al., 24 Mar 2026).

A different security perspective appears in the survey of mobile agents. There, threats are classified as agent-to-platform, platform-to-agent, agent-to-agent, and platform-to-platform, with concrete attack classes including masquerading, unauthorized access, denial of service, repudiation, alteration, and eavesdropping (Amro, 2014). The security objectives are authentication and authorization, confidentiality and privacy, accountability and non-repudiation, and availability (Amro, 2014). Although formulated for migrating agents, these categories remain applicable to modern tool-using and computer-use agents.

The need for protocol-level assurance has also become explicit. “AgentRFC: Security Design Principles and Conformance Testing for Agent Protocols” introduces the Agent Protocol Stack, a six-layer model covering transport security, message transport and wire format, session and resilience, identity/capability/trust, semantic operations and consent, and audit/accountability (Zheng et al., 25 Mar 2026). Its Agent-Agnostic Security Model defines 11 principles, including identity verifiability, capability attestation, delegation monotonicity, prompt integrity, consent explicitness, audit completeness, fail-secure defaults, credential and registry integrity, wire-format integrity, session lifecycle completeness, and composition safety (Zheng et al., 25 Mar 2026). This suggests that the secure ComputerAgent is no longer only an application artifact; it is also a protocol participant requiring formal conformance reasoning.

A major controversy concerns whether such agents can be repurposed for offensive operations. AdvCUA provides evidence that they can. Across 140 ATT&CK-aligned tasks in a realistic multi-host enterprise sandbox, mainstream terminal-based computer-use agents show substantial BSR and ASR on direct malicious tasks, TTP-based malicious tasks, and end-to-end kill chains (Luo et al., 8 Oct 2025). For example, ReAct with GPT-4o attains ASR@5 of 83.78% and BSR@5 of 98.65% on TTP-based malicious tasks, while Cursor CLI with Claude Opus 4.1 attains ASR@5 of 77.03% and BSR@5 of 91.89% on the same task class (Luo et al., 8 Oct 2025). This does not imply that ComputerAgents are intrinsically malicious; rather, it shows that capability and misuse are technically proximate when high-authority execution is exposed.

6. Coordination, multi-agent composition, and protocol ecosystems

As ComputerAgents become more capable, many systems distribute competence across multiple agents rather than rely on a single controller. ColorAgent is explicitly described as a multi-agent framework in which execution, knowledge retrieval, orchestration, and reflection are separated (Li et al., 22 Oct 2025). PC-Agent coordinates Manager, Progress, Decision, and Reflection agents for desktop tasks (Liu et al., 20 Feb 2025). Lemon Agent adopts a two-tier orchestrator–worker structure in which the orchestrator allocates workers for parallel subtask execution and workers themselves may invoke tools concurrently (Jiang et al., 6 Feb 2026). MadAgents uses an Orchestrator, Planner, Plan-Updater, Reviewer, and several domain workers including MG-Operator, Script-Operator, CLI-Operator, Plotter, PDF-Reader, and Researcher (Plehn et al., 28 Jan 2026).

AgentStore generalizes this idea into a platform architecture. It defines an AgentPool $F_1$1, where each $F_1$2 is an agent implementation and $F_1$3 its metadata document, and introduces a MetaAgent with AgentToken representations for routing and management (Jia et al., 2024). The LM head is extended so that agent tokens and word tokens share the prediction space: $F_1$4 For single-agent routing, decoding stops when an agent token is emitted; for multi-agent tasks, a top-$F_1$5 set of agent tokens serves as a hash over the relevant specialist subset (Jia et al., 2024). On OSWorld, AgentStore(AT) improves average success from 11.21% to 23.85% (Jia et al., 2024). This suggests that scalable ComputerAgent systems may increasingly resemble software ecosystems or app stores rather than individual executables.

Communication among agents is itself becoming protocolized. The Agent Network Protocol proposes a three-layer system consisting of an identity and secure communication layer, a meta-protocol negotiation layer, and an application protocol layer (Chang et al., 18 Jul 2025). It uses DIDs, DID documents, and ECDHE-based end-to-end encrypted channels, and introduces an Agent Description Protocol in JSON-LD plus an Agent Discovery Protocol for active or passive discovery (Chang et al., 18 Jul 2025). A computer agent in this setting is a first-class network participant rather than a local automation tool.

This protocol turn intersects directly with security assurance. AgentRFC’s notion of composition safety states that security properties holding for individual protocols may fail when protocols are composed through shared infrastructure (Zheng et al., 25 Mar 2026). A plausible implication is that future ComputerAgent systems will require both architectural modularity and protocol modularity, with corresponding verification mechanisms at each layer.

7. Evaluation regimes, empirical performance, and open problems

Evaluation of ComputerAgents has diversified according to environment and risk model. For mobile and desktop GUI agents, AndroidWorld, AndroidLab, OSWorld, and PC-Eval emphasize task completion in realistic interfaces (Li et al., 22 Oct 2025, Liu et al., 20 Feb 2025, Jia et al., 2024). ColorAgent reports success rates of 77.2% on AndroidWorld and 50.7% on AndroidLab, while PC-Agent reports 76.0% subtask success rate and 56.0% instruction success rate on PC-Eval, representing a 32% absolute improvement over prior state-of-the-art methods (Li et al., 22 Oct 2025, Liu et al., 20 Feb 2025). The hierarchical RL framework ComputerAgent reports 92.1% success on simple tasks with fewer than 8 steps and 58.8% on hard tasks with 8 or more steps, while reducing model size to 15M parameters and halving inference time relative to large MLLM baselines (Dong et al., 22 Sep 2025).

Security-oriented evaluations have shifted from prompt-level jailbreak success toward environment-grounded action verification. AgentSys reports both benign utility and ASR on AgentDojo and ASB (Wen et al., 7 Feb 2026). Agent-Sentry Bench measures utility, ambiguity rate, misclassification rate, and ASR under different functionality graph coverage assumptions (Sequeira et al., 24 Mar 2026). AdvCUA defines Bypass Success Rate and Attack Success Rate: $F_1$6

$F_1$7

and evaluates them through hard-coded trigger, probe, verify, and match mechanisms rather than LLM-as-a-Judge (Luo et al., 8 Oct 2025). This reflects a broader methodological shift toward executable validation.

Several open problems recur across the literature. One is long-horizon constraint preservation: CUAs must retain user constraints and safety conditions across extended trajectories without planning drift (Chen et al., 8 May 2026). Another is memory discipline: both over-retention and under-retention create problems, whether through privacy leakage, stale state, injection persistence, or loss of task continuity (Wen et al., 7 Feb 2026, Chen et al., 8 May 2026). A third is controllable grounding: reliable mapping from visual or semi-structured observations to actions remains difficult in desktop and mobile settings, motivating accessibility augmentation, OCR-assisted localization, uncertainty-aware zooming, and parser-augmented perception (Liu et al., 20 Feb 2025, Chen et al., 8 May 2026).

A further challenge concerns authority binding. Many papers implicitly assume a privileged execution substrate, but recent work argues that permission manifests, sandboxing, provenance-aware mediation, and runtime verification must be integral rather than auxiliary (Chen et al., 8 May 2026, Wen et al., 7 Feb 2026, Sequeira et al., 24 Mar 2026). Finally, evaluation remains incomplete. ColorAgent explicitly notes that current benchmarks are insufficient for comprehensive evaluation of OS agents and points to future work on evaluation paradigms, agent collaboration, and security (Li et al., 22 Oct 2025). The same conclusion appears, in different language, across protocol assurance, endpoint defense, and adversarial misuse studies.

Taken together, these developments indicate that ComputerAgent has become a cross-cutting systems concept linking autonomous control, tool use, software security, protocol interoperability, and human-intent alignment. The field no longer treats the agent merely as a convenience abstraction. It treats it as an executable, authority-bearing computational entity whose architectural decomposition, learning dynamics, memory structure, interfaces, and governance mechanisms jointly determine both capability and risk (R et al., 11 Nov 2025, Chen et al., 8 May 2026).