Covering Human Action Space for Computer Use: Data Synthesis and Benchmark

Abstract: Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. We hypothesize that this issue largely stems from the scarcity of data for complex interactions. To address this problem, we propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions across five modalities: GUI, text, table, canvas, and natural image, as well as a variety of actions (click, drag, draw, etc.), covering a broader range of interaction types than prior click-centric benchmarks that focus mainly on GUI widgets. We also design a renderer-based data-synthesis pipeline: scenes are automatically generated for each modality, screenshots and element coordinates are recorded, and an LLM produces matching instructions and action traces. After training on this corpus, our Phi-Ground-Any-4B outperforms open-source models with fewer than 32B parameters. We will release our benchmark, data, code, and models at https://github.com/microsoft/Phi-Ground.git

• The paper presents a CUActSpot benchmark that expands GUI evaluations beyond simple clicks to cover diverse, complex interactions.
• It introduces a renderer-driven synthetic data pipeline that generates 50M annotated samples across various computer-use modalities.
• Empirical results show improved action grounding in tasks such as table manipulation and freehand drawing, underscoring the value of data diversity.

Covering Human Action Space for Computer Use: Data Synthesis and Benchmark

Motivation and Background

The paper "Covering Human Action Space for Computer Use: Data Synthesis and Benchmark" (2605.12501) systematically addresses the prevailing deficiencies in GUI-based agent research, emphasizing the narrow focus of existing GUI grounding benchmarks and datasets. Most prior work has concentrated on click-only actions over GUI widgets, underrepresenting the extensive and heterogeneous action space encountered in authentic computer-use scenarios, such as multi-point interactions, drag-and-drop, freehand drawing, and direct manipulation of tables, canvas, and natural images. The reliability of advanced Computer-Use Agents (CUAs), including recent iterations like GPT-5.4 and Claude, is consequently impaired, especially in long-tail complex interactions that dominate real-world failures.

Figure 1: Prior research has overwhelmingly focused on click actions; genuine computer-use agents must master a richer action and modality space, including editing, dragging, and drawing.

The authors conduct an in-depth error analysis of GPT-5.4 on practical tasks, revealing a striking prevalence of failures in nuanced interactions beyond simple clicks. This finding motivates a comprehensive reevaluation of how action grounding is benchmarked and learned within multimodal agents.

The CUActSpot Benchmark and Its Design

A central contribution is the CUActSpot benchmark, a manually-constructed suite targeting a broad spectrum of complex mouse-based actions crucial to computer workflows. The benchmark covers five fundamental modalities: standard GUI widgets, direct text interaction, table manipulations typical in spreadsheets, graphical canvas operations, and natural image manipulation as encountered in applications like Photoshop.

Evaluation within CUActSpot moves beyond mere point prediction. It formalizes ordered/unordered regions for multi-point or trajectory-based actions and introduces "banned regions" to penalize degenerate solutions. This protocol ensures robust assessment of capabilities such as text span selection, table cell manipulation, freehand drawing, and region-based image annotation—challenging agent abilities orthogonal to domain-specific knowledge.

Figure 3: The benchmark implements structured rules for evaluating the spatial and sequential correctness of actions, covering both order-sensitive and order-insensitive operations.

Empirically, CUActSpot exhibits minimal confounds from domain expertise or overfitting to software idiosyncrasies, in contrast to ScreenSpot-Pro and UI-Vision, where upper-tail performance is often determined by memorization of software layouts.

Synthetic Data Pipeline for General Action Grounding

Recognizing the crippling scarcity of richly-annotated data for complex computer-use actions, the paper introduces a renderer-driven synthetic data pipeline. This approach programmatically generates scenes and action annotations across modalities by leveraging domain-specific rendering tools (e.g., Playwright and Selenium for web and GUI, PyQt5 for text, custom Python/plt for canvas, and SAM-based routines for images). Action instructions and traces are synthesized by prompting high-capacity LLMs (OpenAI o3), using structured element metadata.

The pipeline records precise spatial metadata for GUI elements, table cells, canvas shapes, and image regions, significantly reducing annotation entropy and supporting automated construction of instruction-response pairs suitable for agent training.

Figure 2: The synthetic pipeline renders diverse visual modalities and curates paired action traces and spatial metadata for LLM-driven task generation.

This scalable synthetic process yields 50M samples, enabling mid-training or pre-training of VLMs for action grounding well beyond click-only regimes.

Experimental Results and Empirical Analysis

The authors instantiate their approach by training Phi-Ground-Any-4B, a 4B-parameter VLM based on Phi-3.5-VL, exclusively utilizing the synthesized corpus. Performance is rigorously compared against contemporary models on CUActSpot, ScreenSpot-Pro, UI-Vision, and agentic task suites like OSWorld.

Key results include:

• Phi-Ground-Any-4B attains the highest CUActSpot scores among all public models under 32B parameters, particularly excelling in complex interaction types (e.g., tabular and canvas actions).
• Fine-tuning with app-specific click data improves legacy benchmarks but degrades CUActSpot scores, demonstrating that raw click-centric performance is not a reliable proxy for genuine action grounding.
• On OSWorld, grounding (as opposed to planning) emerges as the limiting factor in several complex workflows, corroborating the focus on richer action space coverage.

Data ablation studies further reveal that scaling the diversity of modalities and task types (variety scaling) offers greater gains than brute-force scaling within a single modality.

Figure 4: Increasing data diversity—rather than simply scaling sample count per modality—demonstrates pronounced improvement in both in-domain and cross-domain generalization.

The authors quantify compositional generalization, documenting that their model reliably solves tasks absent from its training set by composing elemental skills across modalities (e.g., manipulating text inside images).

Analysis of Failure Modes and Complex Interaction Challenges

In-depth failure studies spotlight the persistent bottlenecks in state-of-the-art CUA agents. A substantial proportion of errors in GPT-5.4 and other benchmarks originate from deficient action grounding in non-widget contexts such as dragging or region selection in images, as opposed to classical single-click tasks.

Figure 5: Top—GPT-5.4 failure statistics reveal system errors are dwarfed by action grounding errors in real computer-use scenarios; Bottom—a qualitative end-to-end failure case in complex document creation highlights these challenges.

Such findings highlight the need for granular, diagnostic benchmarks alongside agent-level task evaluations, particularly as complex, low-frequency interactions dominate the long tail of automated computer use.

Implications and Future Directions

This work establishes an influential precedent for both the evaluation and learning of GUI-based computer-use agents. The formalization of action grounding over an expanded modality and action type spectrum, together with a scalable synthetic generation pipeline, unlocks a new regime for systematic study and model advancement. The findings directly challenge the efficacy of click-only or widget-centric evaluation methodologies, pressing the community to move toward richer, more generalizable agent capabilities.

Strategically, variety scaling and compositional generalization will become increasingly critical as CUAs move towards open-world, long-horizon, and stateful workflows. Future work should focus on increasing the realism of synthetic data, integrating longer interaction horizons, and investigating alignment to organic human-computer interaction distributions. The release of the benchmark, pipeline, and models promises to accelerate progress on these open challenges.

Conclusion

The study delivers a rigorous analysis of the limitations in current CUA grounding methodologies, culminating in the CUActSpot benchmark and a robust synthetic data pipeline that comprehensively covers complex human-computer action space. The authors provide empirical evidence advocating for diversity-driven data scaling and isolate cross-modal generalization as a crucial vector for progress. These contributions are poised to reshape how the field conceptualizes and trains truly generalist computer-use agents.