GroundCUA: Dense Desktop Grounding in CUA-Suite

• GroundCUA is a large-scale, human-annotated dataset that provides dense, pixel-accurate labeling of UI elements in professional desktop applications.
• It addresses the visual grounding bottleneck by enabling precise localization and semantic interpretation of interface elements using 56K screenshots and over 3.6M annotations.
• Integrated in the CUA-Suite, GroundCUA supports training models like ClickX for advanced screen parsing and robust task execution in high-resolution desktop environments.

Searching arXiv for GroundCUA and closely related CUA-Suite papers. Searching arXiv for “GroundCUA” and “CUA-Suite”. GroundCUA is a large-scale, human-annotated desktop grounding dataset for computer-use agents (CUAs), designed to train and evaluate the precise localization and semantic interpretation of interface elements in real professional software. In the CUA-Suite framework, it is the grounding component that directly addresses the “visual grounding bottleneck” exposed by UI-Vision, providing 56K annotated screenshots and over 3.6 million UI element annotations across 87 applications; a later dataset-focused account reports 55,568 screenshots and over 3.56 million human-verified annotations across 87 applications and 12 application categories (Jian et al., 25 Mar 2026, Feizi et al., 10 Nov 2025).

1. Conceptual role and problem setting

GroundCUA is framed as a response to a specific failure mode in CUAs: agents often fail not on high-level intent, but on identifying the correct on-screen target among many visually similar controls. The dataset is therefore designed for dense, pixel-accurate supervision in professional desktop applications, rather than for sparse screenshot-action pairing or synthetic accessibility-tree reconstruction. Its core premise is that professional desktop software requires “almost every visible element” to be labeled, including small icons and controls, because desktop environments are high-resolution, dense, and populated by application-specific visual conventions (Jian et al., 25 Mar 2026).

This grounding emphasis distinguishes GroundCUA from trajectory datasets whose primary unit is an action sequence. GroundCUA supports grounding tasks rather than action execution directly: it is intended to teach models to localize target UI elements from language queries, interpret layout structure, distinguish functionality among controls, and support “generalist screen parsing,” where a system converts raw screenshots into structured UI elements, captions, and semantics-aware representations. The later dataset-focused paper states the same agenda in task form: given a screenshot $I$ and an instruction $x$, the model predicts a 2D point $\hat{p} = (\hat{u}, \hat{v})$, which is correct iff $\hat{p} \in B$, where $B$ is the target element’s axis-aligned bounding box; the evaluation metric is accuracy (Feizi et al., 10 Nov 2025).

2. Position within the CUA-Suite ecosystem

GroundCUA is one component of CUA-Suite’s three-part architecture, which combines temporal demonstrations, dense grounding supervision, and diagnostic evaluation. The paper’s central design claim is that these resources jointly provide dense, causal supervision in which “every element on screen is labeled and every action is logged” (Jian et al., 25 Mar 2026).

Component	Role	Scale
VideoCUA	Continuous expert trajectories	about 10,000 tasks; 87 apps; roughly 55 hours; 6 million frames
GroundCUA	Dense grounding supervision	56K screenshots; 3.6M element annotations
UI-Vision	Diagnostic benchmark	450 task demonstrations

Within this division of labor, VideoCUA supplies the temporal and motor side of desktop interaction, GroundCUA supplies the spatial and perceptual side, and UI-Vision tests three grounding/planning abilities: element grounding, layout grounding, and action prediction. GroundCUA is thus the training-side counterpart to UI-Vision. A plausible implication is that the dataset is intended to close the gap between screenshot-based agents and agents that can operate robustly in professional software by giving them a far denser perceptual substrate than prior open resources (Jian et al., 25 Mar 2026).

3. Data construction and annotation methodology

GroundCUA is built from expert demonstrations rather than synthetic traversal. The collection pipeline begins with continuous 30 fps expert screen recordings. From these recordings, the dataset extracts keyframes corresponding to the screenshots immediately preceding state-changing user actions. Annotators then draw bounding boxes around every visible UI element in those keyframes. Each element receives a textual label: the element’s name when available, the displayed text for shorter strings, or a concise summary for long text regions such as source code or long descriptions. OCR text is also extracted using PaddleOCR (Jian et al., 25 Mar 2026).

The human annotation process is emphasized repeatedly. The CUA-Suite account describes the corpus as human-curated rather than synthetic, with around 70 people involved in annotation and quality assurance, and with cross-checking by the authors for expert-verified accuracy. The dataset-focused paper gives a more granular workforce description: around 70 individuals were organized into annotators, QA specialists, and project managers; most were in India and Latin America; annotators were 20–35 years old, had at least a bachelor’s degree in technical fields, had prior experience in data labeling and UI research, and were trained on the platforms and annotation guidelines; each task took on average 60–90 minutes, including quality checks (Jian et al., 25 Mar 2026, Feizi et al., 10 Nov 2025).

A further structural property is the use of keyframes immediately before user actions rather than arbitrary screenshots. This preserves action-relevant screen states while maintaining dense labeling. The same data source also enables direct coupling with VideoCUA: the grounding annotations inherit their context from continuous expert trajectories rather than from isolated static images. This suggests a training regime in which perceptual grounding and temporal decision-making can be aligned without changing the underlying desktop state distribution (Jian et al., 25 Mar 2026).

4. Dataset scale, schema, and representational properties

The scale of GroundCUA is one of its defining characteristics. The CUA-Suite paper reports 56K annotated screenshots and over 3.6 million UI element annotations across 87 diverse desktop applications. The dataset-focused paper provides more exact figures: 55,568 screenshots, over 3.56 million human-verified UI element annotations, an average of 64 annotations per screenshot, and some screenshots with as many as 542 annotated elements. It further reports a mean screenshot resolution of 2.03 megapixels, a resolution range of 0.39 to 7.0 megapixels, and an average bounding-box area of 0.13% of image area (Jian et al., 25 Mar 2026, Feizi et al., 10 Nov 2025).

The application distribution is also broad. The later paper states that the dataset covers 87 applications across 12 application categories: Education, Browsers, Development, Productivity, Graphics and Design, Video and Audio Production, Communication, Entertainment, System Utilities, Security, Finance and Business Analytics, and Scientific. This breadth matters because desktop grounding differs materially from web and mobile grounding: desktop applications often lack reliable DOM or accessibility trees and frequently use canvas-based or custom-drawn widgets (Feizi et al., 10 Nov 2025).

GroundCUA’s annotation schema combines spatial and semantic information. At minimum, each visible element receives a bounding box and a text label. The CUA-Suite description adds that roughly 50% of elements are assigned to one of eight high-level functional categories. The dataset-focused account confirms the presence of fine-grained category metadata but notes a versioning discrepancy: one summary reports about 50% of elements assigned to a high-level category, the appendix states every element was assigned to one of six high-level categories, and the main text mentions eight high-level categories. The consistent factual core is that GroundCUA includes element typing/category supervision in addition to localization and text labeling (Jian et al., 25 Mar 2026, Feizi et al., 10 Nov 2025).

5. Instruction generation, model training, and GroundNext/ClickX

GroundCUA is not only a box-level corpus; it is also the source of an instruction-grounding training pipeline. The CUA-Suite paper explicitly links it to downstream training of GroundNext models and notes that the resulting 700K instruction-tuning dataset helps teach application-specific grounding strategies. The later paper states that densely labeled screenshots are deduplicated using label text matching and perceptual hashing on element crops, yielding about 900K unique elements, from which a 700K instruction-tuning dataset for supervised fine-tuning and a separate 10K set for reinforcement learning post-training are constructed (Jian et al., 25 Mar 2026, Feizi et al., 10 Nov 2025).

Three instruction families are generated. Direct instructions explicitly refer to the target element; functional instructions describe the target by purpose; spatial instructions specify the target by relation to nearby anchors. The detailed account states that Qwen2.5-VL-72B is used for many of these prompts and that generation conditions include the full screenshot, element crop, bounding box, platform name, annotated label, and nearby context. The 700K SFT split contains 50% direct instructions, 35% functional instructions, and 15% spatial instructions (Feizi et al., 10 Nov 2025).

The same paper introduces a model family whose naming is not entirely uniform in the summarized materials. The abstract introduces GroundNext, while the detailed account identifies the concrete model family as ClickX-3B and ClickX-7B, both initialized from Qwen2.5-VL-Instruct and trained by fine-tuning both the vision encoder and the LLM. The reported SFT setup uses a single node with 8 H100 GPUs, global batch size 128, per-device batch size 1, gradient accumulation 16, learning rate $3\times 10^{-6}$, cosine decay, warmup ratio 0.05, and 2 epochs. RL post-training uses Relative Leave-One-Out (RLOO), group size $n = 8$, batch size 64, one epoch, and a customized discrete reward based on normalized distance to the ground-truth box (Feizi et al., 10 Nov 2025).

6. Empirical performance, interpretation, and limits

GroundCUA-derived models are evaluated on five benchmarks: ScreenSpot-Pro, OSWorld-G, MMBench-GUI (L2), ScreenSpot-v2, and UI-Vision. The reported SFT-only results are 66.4 average for ClickX-3B and 69.2 for ClickX-7B; RL post-training raises these to 68.4 and 70.5 respectively. The paper emphasizes that these results are achieved with only 700K instructions, compared with much larger corpora such as JEDI at approximately 9M datapoints, supporting the claim that high-quality expert supervision can outperform larger but noisier datasets (Feizi et al., 10 Nov 2025).

The agentic evaluation is also notable. On OSWorld-Verified, using o3 as planner and ClickX for grounding/execution, ClickX-3B w/ o3 reaches 50.6 overall across 361 evaluated tasks, with Google Drive-related tasks excluded. The paper characterizes this as strongly outperforming other 3B-class models such as OpenCUA-A3B and Kimi-VL-A3B, surpassing OpenCUA-72B, and remaining comparable to JEDI-7B w/ o3 at 51.0. This supports the narrower claim that compact grounding specialists can materially improve end-to-end agent systems when paired with a strong planner (Feizi et al., 10 Nov 2025).

Several misconceptions are explicitly corrected by the source materials. First, GroundCUA is not primarily an action-execution dataset; it is the perceptual companion to VideoCUA and the training-side resource for UI-Vision-style grounding tasks (Jian et al., 25 Mar 2026). Second, dense grounding does not remove all CUA failure modes. The CUA-Suite paper reports that current foundation action models still exhibit approximately 60% task failure rate on professional desktop applications, indicating that grounding is necessary but not sufficient for robust general-purpose computer use (Jian et al., 25 Mar 2026). Third, grounding is not equivalent to security. Security analysis of CUAs argues that safe deployment also requires input provenance tracking, strong interface-action binding, and control over memory and delegation; a plausible implication is that GroundCUA strengthens perceptual grounding without, by itself, solving adversarial trust-boundary problems (Jones et al., 7 Jul 2025).

GroundCUA’s broader significance lies in the kind of supervision it makes available. The dataset is intended for precise visual grounding, screen parsing, and semantics-aware UI understanding at a near-panoptic level, and—when combined with VideoCUA—for world models and reward modeling. The article-length evidence across CUA-Suite and the later dataset/model paper converges on a common conclusion: dense, expert-verified desktop grounding data is a foundational ingredient for reliable CUAs, particularly in professional applications where small, visually similar, and custom-drawn controls dominate the action space (Jian et al., 25 Mar 2026, Feizi et al., 10 Nov 2025).