---
title: Hybrid GUI+API Action Space
url: https://www.emergentmind.com/topics/hybrid-gui-api-action-space
type: topic
---

# Hybrid GUI+API Action Space

A hybrid GUI+API action space refers to an agent action paradigm in which both low-level, visually-interacting GUI primitives (such as clicks, keypresses, and drags over screen elements) and high-level, programmatic API (or tool) calls are simultaneously available. This design allows intelligent agents—typically those orchestrated by LLMs or RL agents—to dynamically determine, at each step of interacting with a software environment, whether to issue a primitive direct GUI action or a native API command, leveraging the strengths of both modalities. Hybrid action spaces are increasingly central in computer-using agent frameworks and are recognized for combining the reliability/efficiency of APIs with the generality and universality of GUI-level control.

## 1. Formal Action Space Definition

The hybrid GUI+API space is universally characterized by the union of two disjoint action sets:

$$
A = A_\text{GUI} \,\cup\, A_\text{API}
$$

- $A_\text{GUI}$ consists of low-level interface actions such as
  $$
  \{ \text{click}(c),~\text{type}(t),~\text{hotkey}(k),~\text{scroll}(\Delta),~\text{drag}(c_1,c_2) \mid c,c_1,c_2\in C,~t\in\Sigma^*,~k\in K,~\Delta\in\mathbb{Z} \}
  $$
  where $C$ is the set of detected UI controls, $\Sigma^*$ is the set of strings, and $K$ is the set of keyboard shortcuts.

- $A_\text{API}$ is the set of registered, high-level application-native API operations:
  $$
  A_\text{API} = \{ f_i(\theta_i)~|~f_i \text{ is a named API endpoint},~\theta_i \text{ its parameter set} \}
  $$

At each time $t$, the agent, observing system state $s_t$, selects $a_t \in A$ to induce a transition $T(s_t, a_t) = s_{t+1}$ (or $\perp$ on failure), as in $T: S \times A \to S \cup \{\perp\}$ [2504.14603].

This structure generalizes across environments: GUI primitives enable general access (regardless of program coverage), while APIs—when present—serve as robust, efficient alternatives. For example, CoAct-1 extends the hybrid set with $A_\text{code}$, denoting scripted Python/Bash execution [2508.03923]. Mobile agent work (MAS-Bench) incorporates “shortcuts” (API, deep-link, RPA) along with GUI “Tap,” “Swipe,” and navigation primitives [2509.06477]. 

Quantitatively, in benchmarks such as GUI-360, the primitive pool is explicitly cataloged (e.g., 4 GUI primitives and 14 app-specific APIs, totaling 18 distinct function names with parameter schemas) [2511.04307].

## 2. System Architectures and Unified Execution Layer

Modern hybrid agents employ modular architectures to support dynamic interleaving of GUI and API actions. UFO2 exemplifies this pattern with its HostAgent–AppAgent–Puppeteer model [2504.14603]:

- **HostAgent** orchestrates multi-step task decomposition, agent spawning per application, and maintains a global blackboard.
- **AppAgents** are application-specific workers that:
  - Fuse state perception via both Windows UI Automation (UIA) and vision-based parsing.
  - Engage in iterative ReAct loops (LLM plan, execute by Puppeteer, update local FSM).
  - Invoke **Puppeteer**, which exposes a single `Execute(a)` primitive:
    - If the requested semantic action matches a registered API, it issues the API call.
    - Otherwise, it falls back to a GUI action sequence.

Each action is represented by a JSON record specifying application, control identifier (if applicable), action type (API or GUI), and arguments.

Both levels maintain minimal finite-state machines (CONTINUE, PENDING, FINISH, FAIL) for safe preemption and recovery. These principles are mirrored in CoAct-1, where the Orchestrator agent delegates subtasks to GUI Operators or Programmer agents (for code execution), and in MCPWorld with a "unified tool-based space" where the LLM planner calls GUI or API actions at each step [2506.07672, 2508.03923].

### Architectural Pattern Comparison

| Paradigm                  | Description                                               | Example Papers           |
|---------------------------|----------------------------------------------------------|-------------------------|
| HostAgent–AppAgent split  | System-level orchestrator, app-specialized executors     | [2504.14603]            |
| Orchestrator and Toolset  | Registry of both GUI/API “tools” available at planning   | [2503.11069]            |
| ReAct or LLM loop         | LLM alternates between GUI and API invocation by prompt  | [2410.16464, 2510.17790]|

A central property is that the agent can, for each subtask, select between an API if present (favoring atomicity and efficiency) and a GUI fallback if not, enabling robust cross-application coverage.

## 3. Control and Perception: Hybrid Detection and Grounding

Hybrid agents require cross-modal perception to ground GUI primitives and APIs. UFO2 integrates two detection streams for GUI controls [2504.14603]:

- **UIA Layer**: Enumerates standard Windows controls (`C_\text{UIA}`) via accessibility API. 
- **Vision Layer**: Detects visually-drawn (custom) controls from screenshots using object detectors (YOLOv8+Florence2).

Fusion is achieved by combining $C_\text{UIA}$ with non-overlapping vision detections, using IoU-based deduplication (threshold $\tau = 0.1$). The resulting control set $C$ is annotated and passed into the LLM for grounding, ensuring generalization across both standard and bespoke UI elements.

Similarly, MAS-Bench and GUI-360 jointly use screenshot, accessibility, and tree information for grounding [2509.06477, 2511.04307]. In MCPWorld, hybrid state observations comprise both pixel-level screenshots/UI trees and API return values; the same applies to UltraCUA, which accepts screenshots plus tool signatures per step [2510.17790].

## 4. Planning, Scheduling, and Decision Criteria

Efficient utilization of the hybrid space requires task-level planning and robust policies for when to select API over GUI (or vice versa). Several strategies are empirically instantiated:

- **Speculative Multi-Action Batching (UFO2)**: AppAgents predict a batch of $k$ actions per LLM call, executing them sequentially as long as GUI control preconditions are satisfied, exiting early if any control is stale. This reduces LLM calls by up to 50% at no significant loss in success rate [2504.14603].
- **Heuristic-driven Decision (API vs. GUI) [2503.11069]**: An agent compares scores for candidate actions based on modeled efficiency, latency, reliability, security risk, and brittleness, favoring APIs below a latency (e.g., $<500$ms) or where API reliability $>90\%$, falling back to GUI where coverage is incomplete.
- **Orchestrator Delegation (CoAct-1)**: An explicit delegation policy $\pi_\text{delegate}(d|s_t)$ is learned, deciding whether a subgoal is best attempted by GUI or code.
- **MAS-Bench Shortcut Generation**: Agents may generate macros (new shortcut APIs) from prior successful action trajectories by mining frequent subsequences or semantically annotating them with selectors [2509.06477].

Decision logic is commonly implemented both in LLM prompts (structured tool schemas, explicit tagging of action types) and in the policy architecture (action-type softmax).

## 5. Empirical Effects and Quantitative Outcomes

Hybrid action spaces are consistently associated with substantial improvements in agent performance, as demonstrated across several domains:

- **Desktop Automation (UFO2, UltraCUA, CoAct-1, ComputerRL)**
  - UFO2: GUI+API integration increases OSWorld-W office SR from 16.3% (GUI-only) to 24.5% (GUI+API), with average step count dropping from 13.8 to 6.6 [2504.14603].
  - UltraCUA (OSWorld): Relative gains over GUI-only base of +23% for 7B (27.0% vs 23.4%) and +22.9% for 32B (41.0% vs 33.3%); efficiency gains of 11% fewer steps [2510.17790].
  - CoAct-1: SR improves from 53.1% (GTA-1 baseline) to 60.76% (hybrid), with 33% fewer steps per solved task [2508.03923].
  - ComputerRL: GUI-only agent achieves 11.2% SR, hybrid (API+GUI) achieves 26.2%; office domain increases from 6.2% to 27.9% [2508.14040].

- **Mobile Agents (MAS-Bench):**
  - Single-app tasks: Hybrid agent increases SR from 0.511 (T3A) to 0.641 (MAS-MobileAgent), a ΔSR of +0.195, with mean step ratio dropping from 1.056 to 0.613, and token cost dropping from 346K to 99K [2509.06477].
  - Cross-app: Hybrid agent improves SR from 0.340 to 0.617.

- **Web Automation (Beyond Browsing):**
  - Browsing-only agent: 14.8% SR; API-only: 29.2%; hybrid: 35.8% (+6.6pp vs API) [2410.16464].

- **Ablation results (GUI-360):**
  - With 19% API use rate, many long-horizon tasks are completed in fewer steps and with higher semantic precision [2511.04307].

Performance metrics used include Success Rate (SR), Average Steps (AS or MS), action and function accuracy, mean execution time (MET), and tool/call diversity.

## 6. Implementation, Training Objectives, and Policy Representation

Hybrid action spaces are typically backed by unified policy representations and model heads, which handle both discrete (action type) and continuous (arguments/coordinates/text) facets:

- **Hierarchical/Unified Action Policy Heads**: Most systems employ a softmax head over action types (GUI primitives and APIs/tools), with per-action argument heads—regression for coordinates, classifiers for element IDs, and text generation for API strings [2511.04307, 2508.14040].
- **Loss Functions**: Training combines cross-entropy (action type and argument), status prediction (CONTINUE/FINISH), and, if RL is used, step-level or trajectory-level rewards with normalization and, in some cases, auxiliary rewards for tool invocation [2510.17790].
- **Representation Learning for Hybrid Actions**: HyAR introduces an embedding table for discrete actions (GUI events), along with a conditional VAE for continuous parameters (API args), enabling DRL over hybrid spaces with minor architectural adaptation [2109.05490].
- **Observational Inputs**: Inputs to the policy network generally include screenshots, accessibility data/parsed control trees, prior action/observation tuples, and tool schema lists.
- **Efficiency Enhancements**: Speculative plan batching [2504.14603], shortcut mining [2509.06477], and reward shaping for tool use [2510.17790] are employed for improved efficiency and balanced exploration/exploitation of both action types.

This dual-head or hybrid-policy modeling has been empirically shown to scale to branching factors of ~800 (GUI*targets + API*args) per step, requiring careful sampling and curriculum design.

## 7. Limitations, Open Problems, and Benchmarks

Despite the robust empirical gains, several limitations and challenges persist:

- **Incomplete API Coverage**: Many GUI tasks still lack robust programmatic APIs; brittle GUI actions remain unavoidable [2510.17790].
- **Ambiguity and Context Drift**: Hybrid agents can struggle with ambiguous spec, drifting GUI layouts, or outdated control maps, especially for vision-based GUI grounding [2504.14603].
- **Tool Kriging and Expansion**: Automatic extraction of APIs/tools from documentation or codebases, plus dynamic mining of new macro-actions, remains an open research front [2510.17790, 2509.06477].
- **Benchmarks and Evaluation**: Standardized testbeds such as GUI-360, OSWorld, MAS-Bench, and MCPWorld explicitly support hybrid modalities and provide metrics for both per-action and end-to-end agent assessment [2511.04307, 2509.06477, 2506.07672].
- **Sample Efficiency and RL Training**: As seen in UltraCUA and ComputerRL, RL over hybrid spaces remains challenging, with sample efficiency, entropy collapse, and reward signal design demanding further work [2510.17790, 2508.14040].
- **Security and Safety**: Programmatic actions (especially code) may have safety implications; isolation and replay are common safeguards [2508.03923].

Current benchmarks consistently demonstrate that providing both GUI and API primitives yields measurable improvements in SR and execution efficiency—affirming the hybrid paradigm as the prevailing foundation for general computer-use agents.

Source: https://www.emergentmind.com/topics/hybrid-gui-api-action-space