ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents

Abstract: Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations, but this hybrid action space often leaves them uncertain about when to continue with GUI actions or switch to tools, leading to suboptimal execution paths. This difficulty stems from the scarcity of high-quality interleaved GUI-Tool trajectories, the cost and brittleness of collecting real tool trajectories, and the lack of trajectory-level supervision for GUI-Tool path selection. In this paper, we propose ToolCUA, an end-to-end agent designed to learn optimal GUI-Tool path selection through a staged training paradigm. We first introduce an Interleaved GUI-Tool Trajectory Scaling Pipeline that repurposes abundant static GUI trajectories and synthesizes a grounded tool library, enabling diverse GUI-Tool trajectories without manual engineering or real tool-trajectory collection. We then perform Tool-Bootstrapped GUI RFT, combining warmup SFT with single-turn RL to improve decisions at critical GUI-Tool switching points. Finally, we optimize ToolCUA with Online Agentic RL in a high-fidelity GUI-Tool environment, guided by a Tool-Efficient Path Reward that encourages appropriate tool use and shorter execution paths. Experiments on OSWorld-MCP show that ToolCUA achieves 46.85% accuracy, a relative improvement of approximately 66% over the baseline, establishing a new state of the art among models of comparable scale. It also improves by 3.9% over GUI-only settings, demonstrating effective GUI-Tool orchestration. The results further suggest that training in a hybrid action space is a promising paradigm for real-world digital agents. Open-sourced here: https://x-plug.github.io/ToolCUA/

• The paper introduces a novel synthesis pipeline that converts static GUI data into dynamic hybrid action trajectories through interleaved tool and GUI interactions.
• It employs a two-stage training paradigm combining tool-bootstrapped GUI supervised fine-tuning and multi-turn reinforcement learning to optimize path selection.
• Empirical results show a 66% relative improvement in accuracy on OSWorld-MCP and robust performance across diverse digital tasks, confirming the framework's efficacy.

ToolCUA: Optimal GUI-Tool Path Orchestration for Computer Use Agents

Motivation and Problem Statement

Computer Use Agents (CUAs), driven by Multimodal LLMs (MLLMs), are increasingly expected to autonomously operate desktop environments using both atomic GUI actions and high-level structured tool calls. However, empirical evaluation reveals that mere access to hybrid action spaces induces severe policy confusion. Agents either overuse tools, undermining task reliability, or remain GUI-centric, failing to exploit available APIs for efficiency. The central trajectory-level challenge is optimal GUI-Tool path selection: learning a policy capable of context-sensitive switching to achieve maximal reliability and efficiency across long-horizon digital tasks.

Figure 1: (a) Tool-augmented actions outperform pure GUI actions in trajectory efficiency. (b) ToolCUA delivers superior performance relative to baselines in hybrid action space.

Figure 2: Current CUAs frequently miscoordinate GUI and Tool actions, leading to suboptimal and confused execution paths in hybrid action spaces.

Synthesis Pipeline for Interleaved GUI-Tool Trajectories

The sheer scarcity of real interleaved trajectories is a fundamental bottleneck. ToolCUA introduces a trajectory scaling pipeline that synthesizes a grounded tool library from recurrent GUI procedures by leveraging MLLMs. The system transforms static GUI-only data into rich hybrid datasets by generating functionally-equivalent tool trajectories and anchoring them to next-state observations. This synthesis is conducted at varied granularity, maximizing coverage across domains, abstraction levels, and switching contexts, yielding crucial data for hybrid-action learning.

Figure 3: Data collection and training: Interleaved GUI-Tool trajectory scaling, Tool-Bootstrapped GUI RFT, and Online Agentic RL with Tool-Efficient Path Reward.

Figure 4: Synthetic GUI-Tool interleaved trajectory, demonstrating strategic tool selection and appropriate switching between GUI and Tool actions.

Staged Training Paradigm

ToolCUA adopts a two-stage paradigm:

1. Tool-Bootstrapped GUI RFT: Warmup Supervised Fine-Tuning (SFT) on synthesized interleaved data grounds the agent in both multimodal tool-calling and GUI operations. Single-turn RL (GRPO) on critical switching steps calibrates the model's local policy for GUI-Tool decisions at path bifurcation points.
2. Online Agentic RL with Tool-Efficient Path Reward: Multi-turn RL further refines trajectory-level policy. The reward is decomposed into $R_{tool}$ (tool appropriateness: incentivize tool use only when beneficial according to task-level labels) and $R_{length}$ (path efficiency: reward relative reduction in trajectory length), supplementing conventional accuracy and format rewards. This regimen enforces global optimality in path selection, balancing execution efficiency and tool usage.

   Figure 5: RL training dynamics and ablation: Interleaved data bootstrapping and Tool-Efficient Path Reward are critical for effective learning of hybrid orchestration.

Empirical Results and Ablation

ToolCUA achieves a new SOTA on OSWorld-MCP, attaining 46.85% accuracy—a 66% relative improvement over the Qwen3-VL-8B-Instruct baseline—with minimal tool calls and the lowest Average Completion Steps among competitors. This demonstrates genuine orchestration: selective and context-appropriate tool use, not indiscriminate invocation. Cross-domain and cross-platform generalization is robust, with substantial gains on both unseen Linux multi_apps and Windows desktop tasks, outperforming even much larger models.

Figure 6: Task-level results across OSWorld-MCP: ToolCUA outperforms Gemini, Qwen, and coldstart RFT baselines in accuracy and efficiency.

Ablation studies confirm the indispensability of the synthesis-driven scaling pipeline and trajectory-level rewards. Models trained with agentic RL alone, without interleaved supervision, fail to acquire reliable tool-calling knowledge and remain stuck at low TIR. Removing tool-efficient rewards during RL disables robust path length reduction and impairs tool-appropriateness.

Case Studies

Case analyses on LibreOffice Calc and VS Code tasks illustrate ToolCUA's ability to exploit high-level tools for efficient, semantically-aligned trajectory completion while seamlessly reverting to GUI actions for tasks outside tool coverage (e.g., handling pop-up dialogs). The hybrid action space enables both deterministic editing and nuanced UI interaction within complex application workflows.

Practical and Theoretical Implications

ToolCUA demonstrates that optimal orchestration of hybrid actions in digital environments is attainable only by combining scalable trajectory synthesis, staged hybrid-action training, and trajectory-level reinforcement signals. This dismantles the assumption that action-space expansion is enough for robust agentic performance. The formal reward decomposition affords principled policy shaping, decoupling tool use from naive task completion. The approach generalizes across diverse domains and platforms, indicative of its utility for universal digital automation frameworks.

Future Directions

Key open problems include synthesis scalability and fidelity coupling to source trajectory diversity, real-world tool interface maturity, and benchmarking breadth. Reducing infrastructure dependence via lighter environments and supporting asynchronous RL for long-horizon agentic optimization are promising directions. Extension to web and mobile domains—and further granularity of tool abstraction—will afford new levels of agent applicability and control.

Figure 7: Synthesized tools projected in action space; diverse taxonomy and granularity enable flexible orchestration.

Conclusion

ToolCUA sets a systematic paradigm for learning optimal GUI-Tool path selection in CUAs. The combination of scalable interleaved trajectory synthesis, tool-bootstrapped hybrid-action training, and trajectory-level reward shaping is essential for resolving hybrid action confusion, achieving efficient and reliable automation. This research establishes hybrid action learning as the foundation for generalizable agentic systems in real-world digital environments, with implications for both practical deployments and theoretical policy learning.