Grounding Computer Use Agents on Human Demonstrations (2511.07332v1)

Abstract: Building reliable computer-use agents requires grounding: accurately connecting natural language instructions to the correct on-screen elements. While large datasets exist for web and mobile interactions, high-quality resources for desktop environments are limited. To address this gap, we introduce GroundCUA, a large-scale desktop grounding dataset built from expert human demonstrations. It covers 87 applications across 12 categories and includes 56K screenshots, with every on-screen element carefully annotated for a total of over 3.56M human-verified annotations. From these demonstrations, we generate diverse instructions that capture a wide range of real-world tasks, providing high-quality data for model training. Using GroundCUA, we develop the GroundNext family of models that map instructions to their target UI elements. At both 3B and 7B scales, GroundNext achieves state-of-the-art results across five benchmarks using supervised fine-tuning, while requiring less than one-tenth the training data of prior work. Reinforcement learning post-training further improves performance, and when evaluated in an agentic setting on the OSWorld benchmark using o3 as planner, GroundNext attains comparable or superior results to models trained with substantially more data,. These results demonstrate the critical role of high-quality, expert-driven datasets in advancing general-purpose computer-use agents.

• The paper presents ClickNet, a dense, expert-annotated desktop GUI dataset spanning 87 applications with 56K high-res screenshots and over 3.56M annotations.
• The paper details the ClickX models that employ 700K supervised fine-tuning pairs and 10K reinforcement learning samples to achieve state-of-the-art grounding accuracy.
• The paper demonstrates that high-quality human demonstration data significantly reduces data volume needs while enabling efficient and generalizable CUAs across desktop environments.

Grounding Computer-Use Agents on Human Demonstrations

Introduction and Motivation

Computer-use agents (CUAs) capable of automating and interacting with graphical interfaces are contingent upon precise grounding of natural language instructions to on-screen UI elements. This paper addresses the substantial gap in high-quality desktop GUI grounding datasets, introducing ClickNet, a large-scale, expert-annotated desktop dataset, and ClickX models designed for efficient and accurate grounding. Unlike web and mobile-centric resources, ClickNet spans 87 open-source desktop applications, uniquely capturing the granularity, density, and resolution challenges intrinsic to desktop GUIs.

ClickNet: Dataset Construction and Properties

ClickNet’s construction is predicated on human expert demonstration, ensuring that annotated screenshots reflect authentic UI interaction trajectories. The dataset comprises 56K high-resolution screenshots and over 3.56M human-verified element annotations—threefold denser than earlier resources. Each screenshot is annotated for almost every visible element with bounding boxes and textual labels. Fine-grained category assignment covers menus, toolbars, buttons, input fields, and more, leveraging OCR methods where appropriate.

Figure 1: Overview of ClickNet and ClickX. Human demonstrations yield annotated screenshots and UI metadata, which are processed into instruction tasks. ClickX is trained in two stages: SFT on 700K samples, then RL on 10K samples, resulting in state-of-the-art performance.

The selection of open-source applications—spanning office, scientific, graphics, system utilities, and development categories—ensures broad coverage and legal accessibility. The dataset’s resolution ranges from 0.5M to 7M pixels per screenshot, and the average element occupies just 0.13% of a screenshot, emphasizing the necessity for models to resolve fine positional ambiguities. The annotation process features strict QA by multi-tiered specialists, maximizing label fidelity.

Figure 2: Left: ClickNet achieves broader pixel distributions than competing datasets. Right: Much smaller median bounding box area demonstrates fine-grained annotation unmatched in legacy resources.

Instruction Tuning Data: Diversification and Contextualization

Instruction generation leverages dense GUI annotations, producing real-world instruction tasks through three primary categories:

• Direct: Element attribute descriptions and spatial context
• Functional: Action-oriented prompts tied to the UI element’s function
• Spatial: Relational commands, referencing proximity to anchors

Instructions are synthesized via multimodal prompting (using Qwen2.5-VL-72B) with automated diversity filtering, resulting in a non-redundant, high-secondary-context corpus suitable for complex grounding.

Figure 3: Examples of instruction-tuning datapoints showing the transformation from annotated UI elements and context into diverse, semantically relevant commands.

Architecture and Training of ClickX Models

ClickX models are built atop Qwen2.5-VL-Instruct at 3B and 7B scales, updating both vision and language parameters for optimal GUI grounding. Training proceeds in two stages:

• Supervised Fine-Tuning (SFT): 700K curated image-instruction pairs drawn from ClickNet; batch size 128 on 8 × H100 GPUs; two epochs leverage the full model for better multimodal synergy.
• Reinforcement Learning (RL) Post-Training: 10K out-of-distribution samples are used for reward-based policy refinement using the RLOO objective. The discrete reward function encodes normalized click distance to ground-truth boxes, with shaped bins for penalizing error magnitude and incentivizing central localization.

The RL stage avoids the introduction of reward-model hallucination, with empirical ablation favoring discrete over continuous or binary distance-based rewards for GUI targeting stability.

Quantitative Evaluation and Benchmarking

ClickX is evaluated on five established benchmarks (ScreenSpotPro, OSWorld-G, UI-Vision, MMBench-GUI, ScreenSpot-v2), covering both desktop-centric and cross-platform GUI grounding settings.

• SFT models outperform baselines, achieving superior accuracy versus counterparts trained on multi-million sample corpora (e.g., JEDI) using only 700K instruction samples.
• RL tuning yields incremental gains, particularly for models not trained with ClickNet for SFT, indicating the initial data quality’s primacy in model convergence.
• ClickX-3B achieves comparable or better scores than larger models (JEDI-7B, OpenCUA-72B) on agentic multi-step OSWorld tasks, highlighting its efficiency.

  Figure 4: Representative grounding mistakes by ClickX-7B on ScreenSpot-V2, illustrating domain adaptation challenges and error modes in element selection.

Analysis: Generalization, Error Modes, and Application Scope

ClickX displays pronounced gains on desktop benchmarks, especially in icon recognition and workflow-intensive domains. Cross-domain transfer is competitive, with strong mobile and web scores, though some regression is observed outside the desktop OOD distribution—likely requiring future data augmentation from those platforms. Error analysis indicates a model bias toward smaller GUI components—a byproduct of SFT data selection—and highlights occasional ambiguities in context-sensitive spatial instructions.

Application-level analysis reveals that the use of open-source software for training enables effective generalization to closed-source analogs (e.g., LibreOffice vis-à-vis Microsoft Office) due to shared UI layouts and conventions.

Figure 5: Example annotation for GIMP, highlighting the density and category variation of GUI element labels in complex creative applications.

Figure 6: 7-Zip screenshot demonstrating fine element localization, typical of system utility software covered in ClickNet.

Practical and Theoretical Implications

The core claims demonstrated by ClickNet and ClickX are:

• Dense, high-quality annotation is a critical factor in desktop GUI grounding performance, outweighing the benefits of sheer data volume.
• Small language-vision models (3B–7B) can achieve or surpass large-scale counterparts when trained on data curated via expert human demonstration and robust instruction design.
• RL post-training offers only marginal gains when SFT uses highly informative supervision, suggesting diminishing returns for further tuning without significant reward engineering advances.
• Open-source desktop application selection supports broad generalization and facilitates open research, but future work should systematically analyze domain transfer bottlenecks to web and mobile.

The theoretical implication is that GUI grounding can be effectively decoupled from action-trajectory scale when annotation quality and context are optimized. Practically, resource-efficient agents with strong grounding are now tractable for deployment in task automation and accessibility tools across workflows.

Conclusion

This work bridges desktop GUI grounding’s annotation gap via ClickNet and demonstrates that robust agentic abilities can be attained with orders-of-magnitude less supervised data compared with prior paradigms. The release of both the dataset and trained ClickX models establishes a foundation for research in generalist CUA systems, continual learning, and reward signal optimization. Prospective work should explore scaling data across platforms, incorporating dynamic UI states, and leveraging programmatic demonstration for embodied interaction tasks in software environments.