Agent–Computer Interface (ACI)

• Agent–Computer Interface (ACI) is defined as the formal protocol mediating perception, action, and control loops between autonomous agents and digital environments.
• ACIs employ approaches from direct GUI control and structured skill invocation to API-first pipelines and secure, context-aware policy enforcement.
• Empirical evaluations show ACIs improve task performance while addressing challenges like memory management, hierarchical planning, and state coordination.

An Agent–Computer Interface (ACI) is the formal boundary, protocol, or API that mediates the perception, action, and control loop between an autonomous agent—typically based on a large language or vision–LLM—and a digital computing environment. Unlike traditional human–computer interfaces, which are optimized for direct human interaction (ergonomic input devices, visual metaphors, cognitive affordances), an ACI exposes machine-readable, machine-executable abstractions enabling autonomous software agents to observe the environment and issue actions as first-class participants. Contemporary implementations of ACIs span direct GUI control, structured skill invocation, API-first pipelines, hardware-mediated input/output emulation, and secure context-aware policy enforcement, coalescing into a foundational substrate for robust, reliable LLM-driven computer use.

1. Conceptual Foundations and Formal Definitions

Agent–Computer Interfaces arose from the need to enable LLM, VLM, or classical agents to operate within (and across) digital environments using modalities ranging from pixels to structured APIs. The ACI is defined as the logical and physical boundary through which agents both observe the environment (screen pixels, structured UI trees, file system state) and execute atomic or higher-order actions (mouse events, keystrokes, function invocations) (Sager et al., 27 Jan 2025).

A recurring formalism models the ACI as a tuple $(S, A, T)$, where:

• $S$: state space (environment configurations, e.g., GUI layouts, files on disk)
• $A$: action space (curated set of atomic or skill-level controls)
• $T: S \times A \rightarrow S \times O$: the transition and observation function mapping prior state and action to next state plus structured observation (Yang et al., 2024).

Further abstraction in agent-centric software design treats the ACI (or "agent interface") as a mapping $$\mathcal{A} : \mathit{Agent} \times C \times \Sigma_{\text{in}} \rightarrow \Sigma_{\text{out}} \cup \mathcal{E}$$, where $C$ denotes a registry of invocable capabilities, $\Sigma_{\text{in}}$ and $\Sigma_{\text{out}}$ are type-checked schemas, and $\mathcal{E}$ is a set of machine-actionable errors (Wang et al., 19 Mar 2026). This specification emphasizes strict typing, idempotency, and machine-interpretability over human-oriented flexibility or ambiguity.

2. Taxonomies and Architectural Patterns

Systematic surveys delineate ACIs in three orthogonal dimensions (Sager et al., 27 Jan 2025):

• Domain Perspective: Web, mobile (Android/iOS), desktop (Windows/macOS); each supports ACIs at varying depth via browser APIs, accessibility trees, or direct GUI control.
• Interaction Perspective: Observation modalities—screenshots, structured DOM/UI trees, multi-modal; Action modalities—low-level events ($\text{click}(x, y)$, $\text{type}(\text{text})$), direct UI access (element IDs), or capability invocation (send_email(args)).
• Agent Perspective: Varies from memoryless policies $a_t \sim \pi_\theta(o_t)$ to full history-based or Markov-state policies for hierarchical or multi-turn planning.

Architectural realizations include:

• Pixel-to-action agents: Vision–LLMs receive raw screenshots and emit low-level commands (click, drag, type) (He et al., 20 May 2025).
• Structured skill-based frameworks: Each atomic operation is a "skill" with parameterized execution graphs and composition/topology-aware invocation logic, as in CUA-Skill (Chen et al., 28 Jan 2026).
• API-first ACIs: Agents invoke semantically meaningful, versioned capabilities via machine-readable schemas (OpenAPI, JSON-Schema), with strongly typed input/output contracts (Wang et al., 19 Mar 2026, Lu et al., 2024).
• Hardware ACIs: Emulation of HID (Human Interface Device) events using external hardware for platform-agnostic control (Bigham, 31 Jan 2026).
• Secure context-aware envelopes: Agent actions are gated by OS-level enforcement services with intent/context-aware policy engines (Gong et al., 26 Sep 2025).

3. Benchmarking, Performance Metrics, and Empirical Insights

Benchmarks and task definition rigor are critical for assessing ACI robustness. Enterprise-grade evaluation, as exemplified by UI-CUBE, organizes 226 tasks into:

• Simple UI interactions: Core control primitives (buttons, textboxes), systematically varied (activation modes, layout, validation).
• Complex workflows: Copy-paste, business-process operations; enterprise application scenarios (ERP/CRM/HR system mocks) involving conditional logic and hierarchical navigation (Cristescu et al., 21 Nov 2025).

Key evaluation metrics:

• Task success rate

$S = \frac{\sum_t s_t}{|T|}$

• Tier-specific reliability

$$R_i = \frac{\sum_{t \in \text{tier } i} s_t}{|\text{tier } i|}$$

• Capability cliff

$\Delta C = R_{\text{simple}} - R_{\text{complex}}$

• Human-relative performance

$\rho = \frac{R^a}{H}$

Empirical studies demonstrate a sharp capability cliff: while current agents approach 68–87% of human performance on simple tasks, they achieve only 15–32% on enterprise-grade workflows (Cristescu et al., 21 Nov 2025). Multi-resolution testing reveals environmental brittleness: success drops by up to 50 percentage points when scaling input resolution.

Ablation studies confirm that interface design choices—action granularity, context slicing, and error-checking—each yield quantifiable impacts (e.g., a 7.7 pp drop without an edit command in SWE-agent) (Yang et al., 2024).

4. Architectural Limitations and Design Challenges

Current ACI-empowered agents encounter persistent obstacles in operational reliability:

• Memory Management Failures: Stateless LLMs lose track across multi-step routines, leading to omissions, duplicates, or stale variable bindings.
• Hierarchical Planning Deficits: Flat, monolithic plans without subgoal abstraction or macro-action libraries; failures in adaptive replanning or partial recovery.
• State Coordination and Grounding Brittleness: Visual misalignments and perceptual errors cascade across task steps; semantic misgrounding leads to unrecoverable logical faults.
• Validation Limitations: Reliance on brittle string-matching or LLM-as-judge mechanisms; insufficient deterministic, state-based oracles exacerbate silent failures (Cristescu et al., 21 Nov 2025).

These limitations are structural, persisting across agent architectures and foundation models, resistant to mere scaling or prompt engineering.

5. Paradigms, Emerging Methodologies, and Representative Systems

Diverse implementations of ACIs reveal convergent design philosophies and distinctive trade-offs:

• Structured Skill Graphs: CUA-Skill formalizes desktop agent actions as a skill base with parameterized execution graphs and composition logic; memory-aware retrieval/failure recovery and LLM-driven reranking are central for robustness (Chen et al., 28 Jan 2026).
• API-First Planning: AXIS enforces an "API-first, UI-fallback" doctrine, exposing a minimal, agent-discoverable interface over application operations, yielding 65–70% task completion time reduction and ~50% cognitive load reduction while preserving human-level accuracy (Lu et al., 2024).
• Pixel-Based Generalization: PC Agent-E validates that screenshot-only ACIs—without privileged access to application structures—generalize effectively across OSes; ReAct-style ("Thought" + "Action") scaffolds support verifiable reasoning (He et al., 20 May 2025).
• Security and Compliance: CSAgent integrates intent/context-aware static policy enforcement with LLM-driven toolchains for policy extraction; runtime overhead is minimal (<7%), blocking >99% of adversarial agent attacks (Gong et al., 26 Sep 2025).
• Hardware Decoupling: HIDAgent enables universal device control via emulated keyboard/mouse events, facilitating platform-agnostic agent operation—even on locked-down or non-instrumented targets (Bigham, 31 Jan 2026).
• Terminal and Code-Based ACIs: SWE-agent and EnIGMA leverage structured, low-ambiguity command protocols ("DISCUSSION" + "COMMAND" blocks, REPL/interactive debugger integration) to maximize representational compatibility and transparency in code- and shell-centric environments (Yang et al., 2024, Abramovich et al., 2024, Masi, 11 Mar 2026).
• Hybrid Model Context Protocol Servers: LiteCUA (AIOS) reveals that decoupling interface complexity (GUI, application trees) from reasoning via a Model Context Protocol server (rich JSON schemas, atomic action primitives) enables competitive performance with extremely minimalist agent logic (Mei et al., 24 May 2025).

6. Design Principles, Best Practices, and Future Directions

Synthesis across empirical and theoretical work yields the following ACI design imperatives:

• Structured State and Action Spaces: Expose environment state and permissible actions in explicit, well-typed schemas (JSON, OpenAPI), decoupled from UI idiosyncrasies (Wang et al., 19 Mar 2026, Lu et al., 2024).
• Hierarchical and Modular Planning: Layer multi-level planners with libraries of parametric skills or sub-policies; support compositional workflows rather than monolithic step sequences (Cristescu et al., 21 Nov 2025, Chen et al., 28 Jan 2026).
• Explicit Memory and Checkpointing: Integrate robust state tracking (key–value or graph-based memories), enabling iterative aggregation, error recovery, and intermediate result checkpointing.
• Adaptive Perceptual Grounding: Fuse closed-loop visual modules (pixel and DOM) with semantic selectors and feedback oracles to verify, correct, or rollback perceptual errors (Cristescu et al., 21 Nov 2025).
• Security and Policy Enforcement: Mediate all agent actions with runtime policy validation tailored to user intent, environmental context, and privilege boundaries (Gong et al., 26 Sep 2025).
• Transparent Feedback and Self-Monitoring: Insert deterministic, state-based validation checks (inline oracles) and require stepwise reporting for early drift detection and rollback.
• Mixed-Initiative and Human Oversight: Provide mechanisms for human inspection, approval gates, and undo/redo at every layer (especially in security-sensitive or enterprise contexts) (Masi, 11 Mar 2026).

Future research directions emphasize the continued evolution of ACIs towards:

• Ubiquitous, capability-based platforms exposing all application functionality as programmatic skills or APIs (Wang et al., 19 Mar 2026).
• Integrated testing and monitoring frameworks that leverage synthetic agents for continuous evaluation and regression detection.
• Standardized, high-complexity benchmarks (UI-CUBE, OSWorld, WindowsAgentArena) and online human assessment pipelines (Cristescu et al., 21 Nov 2025, Sager et al., 27 Jan 2025).
• Secure, sandboxed OS-level deployment models with self-adaptive policy management and attack-surface minimization (Gong et al., 26 Sep 2025).
• Cross-platform, hardware-agnostic operation modalities for agents in both online and offline settings (Bigham, 31 Jan 2026).

7. Comparative Evaluation and Outlook

Modern ACIs are distinguished not by a single modality but by their conformance to strict machine-interpretability, modularity, safety, and feedback-integrity criteria. Empirical results demonstrate that performance ceilings in current LLM-based CUAs are architectural, not incremental—successful future ACIs must depart from monolithic, stateless prompt engineering toward structured state tracking, hierarchical planning, adaptive perceptual grounding, and robust error handling (Cristescu et al., 21 Nov 2025, Chen et al., 28 Jan 2026). As the field progresses, agent–computer interfaces will transition from fragile bridges to foundational substrates, enabling agents to reliably and securely orchestrate complex, cross-application workflows autonomously in real-world deployment scenarios.