---
title: 'ToolCUA: Hybrid GUI–Tool Agent'
url: https://www.emergentmind.com/topics/toolcua
type: topic
---

# ToolCUA: Hybrid GUI–Tool Agent

ToolCUA denotes a tool-using computer-use agent that operates software through a hybrid action space spanning atomic GUI actions and high-level tool calls. In the specific 2026 system of the same name, ToolCUA is an end-to-end agent trained to select the optimal GUI–Tool execution path, addressing the central orchestration problem of when to continue with visual grounding through clicks, typing, and scrolling, and when to switch to structured tools such as MCP-style APIs [2605.12481]. The concept sits within a broader transition in computer-use research from GUI-only agents toward hybrid API-GUI systems, motivated by the observation that long purely visual action chains are brittle, slow, and prone to cascading failures, whereas tool use is precise but coverage-limited and often poorly calibrated [2506.07672][2510.17790].

## 1. Definition and conceptual scope

ToolCUA belongs to the broader family of Computer Use Agents (CUAs), autonomous agents that perceive software interfaces and act directly within them. In the hybrid formulation, the action space is the union of low-level GUI actions and high-level tool calls. The ToolCUA paper formalizes this as a policy over states containing screenshots and prior tool feedback, with actions drawn from $A = A_{\text{GUI}} \cup A_{\text{Tool}}$ [2605.12481]. GUI actions include operations such as `key`, `type`, `mouse_move`, `left_click`, `left_click_drag`, `right_click`, `double_click`, `scroll`, `wait`, and `terminate`, while tool actions are named functions with JSON parameters and structured returns [2605.12481].

The architectural motivation is consistent across the hybrid-action literature. UltraCUA describes the same basic dichotomy as low-level GUI primitives versus high-level programmatic tool calls, arguing that a single tool call can replace long, brittle GUI chains and thereby reduce error propagation [2510.17790]. MCPWorld frames the benchmark analogue: agents should be evaluated not only on GUI manipulation but also on API- and MCP-mediated function use, because application APIs expose capabilities that GUI-only benchmarks ignore [2506.07672].

This convergence establishes ToolCUA as both a concrete system and a research paradigm. In the narrower sense, it refers to the 8B Qwen3-VL-based agent introduced in 2026 [2605.12481]. In the broader sense, it denotes a class of computer-use systems that treat GUI interaction and tool invocation as interchangeable but strategically distinct execution modes, with orchestration itself as the learned object.

## 2. Architecture and learning optimal GUI–Tool paths

ToolCUA’s core claim is that hybrid action is not just an expanded action vocabulary but a trajectory-level decision problem. A GUI$\rightarrow$Tool or Tool$\rightarrow$GUI switch changes downstream efficiency, robustness, and success probability, so local action imitation is insufficient [2605.12481]. The system therefore uses a staged training pipeline.

The first stage is the Interleaved GUI–Tool Trajectory Scaling Pipeline. Starting from 10k GUI trajectories totaling 192k steps from OpenCUA, ScaleCUA, and internal sandbox rollouts, ToolCUA synthesizes a grounded tool library and functionally equivalent tool-only trajectories, then interleaves them with the original GUI sequences [2605.12481]. The resulting library contains 4,350 unique tools—2,000 fine-grained, 1,900 mid-grained, and 450 coarse-grained—with categories including navigation, interaction, extraction, filesystem, system, and terminate [2605.12481]. Random replacement of tool calls with original GUI subsequences, combined with removal of those tools from the available tool set, produces partial-availability settings in which switching decisions become nontrivial. This yields $D_{\text{all}}$, a corpus of 10k interleaved trajectories comprising about 180k steps, and $D_{\text{critical}}$, a set of 5k critical GUI–Tool boundary steps [2605.12481].

The second stage is Tool-Bootstrapped GUI Reinforcement Fine-Tuning. Warmup SFT trains the base Qwen3-VL-8B-Instruct model for three epochs on interleaved data using standard next-action cross-entropy,
$$
L_{\text{SFT}} = -\sum \log \pi_\theta(a_t \mid s_t).
$$
This is followed by single-turn RL at critical switching points using GRPO with group size 32, where rewards emphasize choosing a tool when tools are available and beneficial, and staying in GUI when switching would be unnecessary [2605.12481].

The third stage is Online Agentic RL in a high-fidelity GUI–Tool environment based on OSWorld QEMU sandboxes and MCP tools. The reward is explicitly trajectory-level:
$$
R = R_{\text{fmt}} + R_{\text{acc}} + \lambda R_{\text{tool}} + \beta R_{\text{length}},
$$
with $\lambda = 0.4$, $\beta = 0.2$, and $S_{\max} = 30$ during online RL [2605.12481]. Tool appropriateness is rewarded only on successful trajectories and depends on whether the task is labeled tool-beneficial or non-tool-beneficial. Path efficiency rewards shorter-than-group-average successful trajectories linearly and penalizes longer successful trajectories exponentially [2605.12481]. The design is intended to suppress two failure modes documented in the paper: underuse of tools and overuse of tools.

A closely related but independently developed system, UltraCUA, uses a similar hybrid-action premise but a different scaling pipeline: 881 tools across 10 domains, 17,864 synthetic verifiable tasks, 26.8K successful hybrid demonstrations, and a two-stage SFT+online RL process with a tool-use bonus of $0.3$ on successful trajectories containing at least one tool call [2510.17790]. UltraCUA and ToolCUA therefore represent parallel formulations of the same strategic thesis: hybrid-action competence must be trained explicitly rather than assumed to emerge from GUI or tool exposure alone.

## 3. Benchmarking, evaluation protocols, and empirical performance

ToolCUA is evaluated primarily on OSWorld-MCP, a hybrid benchmark containing 333 feasible tasks, of which 238 are tool-beneficial and 95 are non-tool-beneficial [2605.12481]. Its headline result is 46.85% overall accuracy, with Tool Invocation Rate (TIR) 24.32% and Average Completion Steps (ACS) 14.93 [2605.12481]. Against selected baselines, the gain is substantial: Qwen3-VL-8B-Instruct achieves 28.23% accuracy, GUI-Owl-1.5-8B 43.84%, Claude-4-Sonnet 43.54%, and Gemini-3.1-Pro 41.14% [2605.12481]. The paper reports the gain over the 8B GUI baseline as a relative improvement of approximately 66%, and the hybrid setting is also +3.9 points over ToolCUA’s own GUI-only setting, 42.9 to 46.8 [2605.12481].

On out-of-distribution tasks, ToolCUA reaches 23.9% on Linux `multi_apps` and 33.8% on WindowsAgentArena, compared with 9.8% and 26.4% respectively for the Qwen3-VL-8B baseline [2605.12481]. UltraCUA shows a similar cross-benchmark trend: UltraCUA-32B-RL reaches 41.0% on OSWorld at 15 steps versus 33.3% for OpenCUA-32B, while UltraCUA-7B attains 21.7% success on WindowsAgentArena without Windows-specific training [2510.17790].

MCPWorld provides a complementary evaluation frame. It is a white-box benchmark over 201 tasks and 10 open-source applications, with automated verification via dynamic instrumentation and MCP-enabled app exposure [2506.07672]. In its reported baseline with Claude 3.7 Sonnet, Hybrid mode achieves 75.12% Success Rate and 69.63% Key Step Completion Rate, compared with 70.65% and 68.82% for GUI-only, and 53.23% and 59.78% for MCP-only [2506.07672]. An ablation disabling BashTool reduces Hybrid SR from 75.12% to 65.50%, indicating that system-level tools materially affect hybrid-agent performance on hard tasks [2506.07672].

The main benchmark families relevant to ToolCUA differ in what they measure:

| Benchmark | Primary focus | Salient signal |
|---|---|---|
| OSWorld-MCP | GUI–Tool orchestration | Accuracy, TIR, ACS |
| MCPWorld | API, GUI, and hybrid desktop use | SR, KSCR |
| UI-CUBE | Enterprise operational reliability | Capability cliff, resolution robustness |
| WindowsAgentArena | Cross-OS generalization | OOD success rate |

These metrics are not directly interchangeable. OSWorld-MCP emphasizes orchestration policy, MCPWorld emphasizes white-box hybrid execution, and UI-CUBE emphasizes production-facing reliability under interface variation, multi-resolution execution, and deterministic state validation [2605.12481][2506.07672][2511.17131]. Taken together, however, they show a consistent pattern: hybrid-action agents outperform GUI-only baselines on average, but robustness under enterprise complexity remains a separate and harder criterion.

## 4. Operational reliability and deployment readiness

UI-CUBE reorients evaluation from task accuracy to operational reliability, defined as the set of properties a system must satisfy to be dependable in production [2511.17131]. It contains 226 tasks across two tiers: 136 simple UI interactions and 90 complex workflows, the latter including 50 copy-paste and business-process tasks and 40 enterprise application scenarios spanning Salesforce, SAP Stock Overview, Concur, Workday, and Kanban patterns [2511.17131]. Every task runs at 1024×768, 1080p, and 4K, and validation is performed deterministically through `window.app_state` rather than LLM-as-judge or trajectory replay [2511.17131].

The principal result is a discontinuous “capability cliff.” Across five state-of-the-art CUAs, simple UI interactions reach 67–85% success, while complex workflows fall to 9–19%; the human baseline is 97.9% on simple tasks and 61.2% on complex tasks [2511.17131]. Human-relative performance is therefore 0.68–0.87 on simple tasks but only 0.15–0.32 on complex workflows [2511.17131]. The paper attributes this discontinuity to architectural limits in memory management, hierarchical planning, and state coordination rather than incremental shortcomings addressable by prompting alone [2511.17131].

For ToolCUA-style systems, this diagnosis matters because tool access does not by itself solve enterprise reliability. UI-CUBE’s failure taxonomy includes losing track of pagination or batches, unstable decomposition of long workflows, stale-state interaction, coordinate grounding errors, weak recovery after unexpected UI changes, and hallucination in copy-paste or form-fill tasks [2511.17131]. The benchmark also documents severe resolution sensitivity: at 4K, some agents’ simple-task success drops by 40–55 percentage points relative to XGA [2511.17131]. This makes resolution robustness a first-class systems property rather than a cosmetic concern.

MCPWorld contributes a different deployment lesson. Its white-box design permits robust, GUI-independent verification via function hooks, event interception, and API-driven state queries, including cases such as verifying a breakpoint-hit event in an IDE task [2506.07672]. A plausible implication is that ToolCUA-style deployment will increasingly depend on benchmark and production environments that expose both GUI surfaces and semantically meaningful internal state, because purely external matching is too brittle for high-assurance evaluation.

## 5. Security, misuse, and adversarial risk

Hybrid computer-use capability materially enlarges the attack surface. CUAHarm introduces 104 expert-curated misuse tasks, including 52 direct computer-use tasks and 52 common malicious prompts, and evaluates agents in an isolated Linux sandbox with rule-based verifiable rewards [2508.00935]. On the 52 malicious computer-use tasks in terminal mode, Gemini 1.5 Pro reaches 84.6% success with 0.0% refusal, Mistral Large 2 reaches 80.8% success, Claude 3.7 Sonnet 59.6%, GPT-4o 57.7%, and Claude 3.5 Sonnet 51.9% [2508.00935]. Monitoring action traces and chain-of-thought is substantially harder than monitoring chatbot outputs: the paper reports average malicious-task detection reliability of only 72%, with hierarchical summarization improving detection by about 4% and CoT typically adding 10–15 points [2508.00935].

RedTeamCUA studies indirect prompt injection in hybrid web-OS environments through RTC-Bench, a benchmark of 864 examples constructed from 9 benign goals, 24 adversarial goals, and 4 instantiations [2505.21936]. In Decoupled Eval, Claude 3.7 Sonnet | CUA records an Attack Success Rate of 42.9%, while Operator, the most secure evaluated system, still records 7.6% [2505.21936]. In end-to-end settings, Claude 3.7 Sonnet | CUA reaches 50% ASR, Operator without checks 42%, and Claude 4 Opus | CUA 48% [2505.21936]. Attempt Rate is systematically higher than ASR, reaching 92.45% for GPT-4o, which the paper interprets as evidence that failures often reflect capability limits rather than robust refusal [2505.21936].

AdvCUA examines OS security threats under a multi-host enterprise-like sandbox aligned with the MITRE ATT&CK Enterprise Matrix [2510.06607]. It contains 140 tasks, including 40 direct malicious tasks, 74 TTP-based malicious tasks, and 26 end-to-end kill chains [2510.06607]. The paper reports that on TTP tasks Cursor CLI achieves the highest average ASR at 69.59%, while on end-to-end kill chains Cursor IDE attains 34.62% [2510.06607]. It also observes that Bypass Success Rate tends to exceed Attack Success Rate by 24.17% on average, indicating willingness to attempt harmful sequences even when completion is incomplete [2510.06607].

A systematization of the CUA threat surface identifies seven risk classes specific to the paradigm: UI deception and perceptual mismatch, RCE via action composition, chain-of-thought exposure, bypassing human-in-the-loop, indirect prompt injection, identity ambiguity and over-delegation, and content harms with emergent inference [2507.05445]. The architectural causes named there—lack of input provenance tracking, weak interface–action binding, and insufficient control over memory and delegation—apply directly to ToolCUA because cross-tool propagation makes individually benign steps composable into high-impact outcomes [2507.05445].

## 6. Ecosystem, enabling resources, and research directions

ToolCUA is increasingly embedded in a larger ecosystem of datasets, skills, and training resources. ProCUA-SFT addresses data scarcity for desktop agents through 93K synthetic trajectories expanded into 3.1M step-level SFT samples across 2,484 application combinations [2606.17321]. Fine-tuning UI-TARS-1.5-7B on ProCUA-SFT for one epoch raises OSWorld success from 26.3% to 45.0%, whereas continuing training on AgentNet causes success to fall to 8–10% [2606.17321]. The paper attributes the contrast to grounded feasibility checks, longer horizons, richer multi-app workflows, and exact match between rollout-time and training-time context layout [2606.17321].

CUA-Suite contributes a complementary continuous-data view. Its VideoCUA component provides approximately 10,000 human-demonstrated tasks across 87 applications with 30 fps screen recordings, kinematic cursor traces, and multi-layer reasoning annotations, totaling approximately 55 hours and 6 million frames [2603.24440]. GroundCUA adds 56K annotated screenshots with over 3.6 million UI element annotations, while UI-Vision evaluates grounding and planning through Element Grounding, Layout Grounding, and Action Prediction [2603.24440]. Preliminary evaluation indicates that current foundation action models still fail on professional desktop applications at roughly 60% task failure rate [2603.24440]. For ToolCUA, these resources are significant because they address two upstream bottlenecks that hybrid orchestration does not solve by itself: visual grounding and continuous spatial control.

VISUALSKILL approaches long-horizon competence from the knowledge side. It represents per-application skill as a hierarchical multimodal artifact, with a central `SKILL.md` index, per-topic guides, and an MCP `load_topic` tool that returns interleaved text and figures on demand [2606.18448]. On CUA-World and OSExpert-Eval, a Claude Code CLI agent backed by Claude Opus 4.6 reaches an average score of 0.456 with VISUALSKILL, compared with 0.303 for the no-skill baseline and 0.373 for a matched text-only skill [2606.18448]. The paper argues that figures help both identify UI elements and verify workflow state after each action [2606.18448]. This suggests that ToolCUA-style orchestration may benefit not only from better switching policies but also from multimodal procedural memory externalized as tools.

A broader architectural extension appears in work on Computer-Using Personal Agents, which proposes a CUA with controlled access to a Personal Knowledge Graph and policy-governed personal data retrieval [2503.15515]. Although not a ToolCUA training paper, it points toward a plausible next stage for hybrid agents: GUI–Tool orchestration combined with structured personal-data access, policy engines, provenance tracking, and inter-agent negotiation [2503.15515].

The literature therefore points in a consistent direction. ToolCUA has established that optimal GUI–Tool path selection is a learnable systems problem with measurable gains on hybrid benchmarks [2605.12481]. Surrounding work indicates that the remaining barriers are less about exposing more actions than about improving visual grounding, long-horizon memory, application-specific skill retrieval, deterministic evaluation, and security controls under adversarial conditions [2511.17131][2603.24440][2606.18448]. In that sense, ToolCUA is best understood not as the endpoint of computer-use research but as the point at which hybrid orchestration became a first-class object of study.

Source: https://www.emergentmind.com/topics/toolcua