CUA-Suite: Desktop Computer-Use Agent Ecosystem

Updated 4 July 2026

CUA-Suite is a full-stack desktop computer-use agent ecosystem integrating continuous human demonstration videos, dense pixel-level UI annotations, and specialized benchmarking tools.
It leverages continuous 30 fps video recordings and comprehensive UI labeling to overcome data shortages in screenshot-only datasets, thus improving grounding and planning performance.
Its three core components—VideoCUA, GroundCUA, and UI-Vision—enable detailed evaluation protocols that reveal both the strengths and persistent challenges in interface localization.

CUA-Suite is a desktop computer-use-agent data ecosystem centered on continuous human demonstration video, dense pixel-level UI annotation, and desktop-specific evaluation. In its primary and most specific sense, the term denotes the resource introduced in “CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents,” whose core components are VideoCUA, GroundCUA, and UI-Vision. The suite was designed to address two bottlenecks in computer-use agents: the shortage of continuous, high-fidelity human trajectories and the weak grounding and planning performance of current models on professional desktop applications. Its released artifacts include approximately 10,000 human-demonstrated tasks across 87 applications, 55 hours and 6 million frames of 30 fps screen video, 56,000 annotated screenshots, and more than 3.6 million UI element annotations (Jian et al., 25 Mar 2026).

1. Definition and scope

CUA-Suite is presented as a full-stack desktop CUA resource rather than a single dataset. Its organizing claim is that continuous video is the missing supervision substrate for scaling computer-use agents, because sparse screenshot-action corpora discard cursor motion, transient interface states, hover feedback, tooltips, and other temporal cues that are directly relevant to GUI control. VideoCUA’s continuous streams therefore form a strict superset of information relative to screenshot-only datasets and can be losslessly transformed into the formats required by existing agent frameworks (Jian et al., 25 Mar 2026).

This positioning is explicitly comparative. The suite is framed against screenshot-based resources such as ScaleCUA and OpenCUA. ScaleCUA is described as containing about 2 million screenshots, corresponding to less than 20 hours of video, whereas VideoCUA alone totals about 55 hours and 6 million frames at 30 fps. The resulting temporal coverage is described as more than 2.5 times that of the largest existing open dataset used for comparison. The suite is also desktop-specific: it targets professional software such as IDEs, graphics packages, office applications, CAD, GIS, and scientific tools, where current foundation action models exhibit substantial failure rates (Jian et al., 25 Mar 2026).

A second defining feature is representational density. CUA-Suite does not only record actions; it aligns visual state, low-level executable actions, cursor kinematics, UI element geometry, textual element labels, and multi-layered reasoning annotations. This combination places it simultaneously in the roles of imitation-learning corpus, grounding dataset, and benchmark substrate (Jian et al., 25 Mar 2026).

2. Constituent resources

The suite consists of three tightly coupled components.

Component	Scale	Primary role
VideoCUA	~10,000 tasks; 55 hours; 6 million frames; 87 apps	Continuous human demonstrations with synchronized actions and reasoning
GroundCUA	56,000 screenshots; 3.6 million UI annotations	Dense grounding and screen-parsing supervision
UI-Vision	450 task demonstrations from the same app pool	Benchmarking of element grounding, layout grounding, and action prediction

VideoCUA is the trajectory core. It provides continuous 30 fps screen recordings, kinematic cursor traces, time-stamped input logs, and OpenCUA-style multi-layered annotations. Its 87 applications span 12 categories, including development environments such as VS Code and IntelliJ IDEA, graphics and design tools such as Blender, GIMP, Krita, and Inkscape, scientific and engineering packages such as RStudio, GNU Octave, FreeCAD, and QGIS, productivity tools such as LibreOffice and OnlyOffice, and media tools such as OBS Studio, OpenShot, Shotcut, and Audacity (Jian et al., 25 Mar 2026).

GroundCUA is the suite’s dense UI annotation layer. It covers the same professional desktop domain with human-verified bounding boxes for almost every visible element, textual labels, and semantic type annotations for about half of the elements. The paper describes this density as extending to small icons and custom controls, which is important because professional desktop interfaces frequently rely on canvas-based or non-standard widgets that are not recoverable from accessibility trees alone. GroundCUA also underlies a 700,000-example instruction-tuning dataset used to train grounding models such as GroundNext (Jian et al., 25 Mar 2026).

UI-Vision is the evaluation layer. It is a desktop-centric benchmark derived from 450 task demonstrations and is designed to isolate grounding and planning capabilities. Its tasks are organized around element grounding, layout grounding, and action prediction, making it complementary to VideoCUA’s richer but less benchmark-focused trajectory structure (Jian et al., 25 Mar 2026).

3. Data representation and annotation pipeline

CUA-Suite’s representation is step-structured but video-grounded. For each state-changing step, the suite stores a screenshot, a natural-language observation, a reasoning segment, an action description, executable pyautogui code, the subsequent screenshot, and a reflection. The paper summarizes this as

$\tau_t = (s_t, o_t, r_t, d_t, a_t, s_{t+1}, \text{ref}_t).$

Here $s_t$ is the current keyframe, $o_t$ is a detailed screen description, $r_t$ is the task-linked reasoning trace, $d_t$ is a semantic action description, $a_t$ is executable pyautogui code, and $\text{ref}_t$ is a post-hoc reflection on the action outcome (Jian et al., 25 Mar 2026).

The reasoning annotations are unusually verbose. Average per-step annotation length is approximately 496.7 words, decomposed into 157.4 words for observation, 194.3 for reasoning, 17.7 for action description, and 127.4 for reflection. This makes the resource atypically suitable for models that jointly learn perception, planning, execution, and self-critique rather than action decoding alone (Jian et al., 25 Mar 2026).

Keyframes are defined with care to avoid trivial target leakage. For evaluation, the keyframe corresponding to action time $\tau_t$ is captured at the temporal midpoint between the previous and current actions,

$\text{keyframe}_t = f_{\frac{\tau_{t-1} + \tau_t}{2}},$

so that the cursor is in transit rather than already sitting on the target. Coordinates are normalized to $[0,1]^2$ , enabling resolution-agnostic action storage and later rescaling to pixel coordinates at inference time (Jian et al., 25 Mar 2026).

The human annotation pipeline is likewise substantial. The work used a professional labeling vendor with about 70 annotators, QA specialists, and project managers. Annotators designed tasks for the 87 applications, executed those tasks while recording screen video and logging inputs, and then labeled extracted keyframes with bounding boxes, labels, and semantic categories. The annotator pool is described as being based in India and Latin America, largely in the 20–35 age range, with technical bachelor’s degrees and prior UI-related annotation experience. Quality control combined a pilot phase, vendor-side review, author cross-checks, and custom consistency validation scripts (Jian et al., 25 Mar 2026).

4. Benchmark tasks and evaluation protocols

CUA-Suite supports several research tasks, but its benchmark protocols concentrate on grounding and planning. UI-Vision defines three principal evaluation modes. Element grounding requires a model to localize the UI element referred to by a textual query. Layout grounding extends this to structural interface understanding, such as identifying related interface regions. Action prediction requires a model to infer the next GUI action from a screenshot and task context (Jian et al., 25 Mar 2026).

VideoCUA introduces a task-level action prediction protocol that is more explicitly trajectory-conditioned. The model receives the global task instruction, the previous $s_t$ 0 ground-truth actions with both code and natural-language descriptions, and the current screenshot. It must then output the next action as pyautogui code. Automatic evaluation is restricted to coordinate-based actions—click, doubleClick, rightClick, and dragTo—while keyboard and text-entry actions are reserved for human evaluation because semantic correctness cannot be reduced to raw string or coordinate matching alone (Jian et al., 25 Mar 2026).

The principal action-prediction metrics are coordinate error and thresholded success. If $s_t$ 1 is the predicted coordinate and $s_t$ 2 the ground-truth coordinate, pixel error is measured as

$s_t$ 3

Aggregate performance is then reported via mean and median pixel distance, along with $s_t$ 4 for thresholds such as 20 and 50 pixels: $s_t$ 5 This protocol reflects the suite’s emphasis on fine-grained interface grounding rather than only task-level completion (Jian et al., 25 Mar 2026).

GroundCUA additionally introduces a typed UI ontology for dense annotation. The eight reported semantic categories are Input Element, Sidebar, Information Display, Button, Navigation, Visual Elements, Menu, and Others. This ontology supports detection, classification, layout parsing, and referring-expression grounding on professional desktop interfaces (Jian et al., 25 Mar 2026).

5. Empirical findings

The suite’s baseline evaluations expose a persistent grounding bottleneck on professional desktop software. On UI-Vision’s element-grounding benchmark, the strongest reported model in the provided comparison, MAI-UI-32B, reaches 47.7 average accuracy, decomposed into 59.1 on Basic, 57.1 on Functional, and 26.9 on Spatial. The weakest listed values are markedly lower, with UI-TARS-7B at 17.6 average and Qwen3-VL-8B at 18.0 average. Across models, the spatial split remains the hardest, staying below 30 even for the top entry. The paper’s interpretation is that relational and structural grounding remain substantially less mature than simpler semantic identification (Jian et al., 25 Mar 2026).

On VideoCUA action prediction, OpenCUA-7B yields 387.5 mean pixel error, 236.0 median pixel error, 7.9% success at 20 pixels, and 16.5% success at 50 pixels. OpenCUA-32B improves these numbers to 274.2 mean pixel error, 97.0 median pixel error, 22.0% success at 20 pixels, and 37.7% success at 50 pixels. Even the stronger model therefore misses the target by more than 50 pixels in most cases, and a large fraction of errors exceed 500 pixels, indicating cross-panel or region-level confusion rather than small localization noise (Jian et al., 25 Mar 2026).

The human evaluation is more revealing because it separates action semantics from spatial grounding. Over 49 tasks and 576 steps, OpenCUA-32B attains 57.6% overall stepwise accuracy. Action correctness is much higher at 85.9%, but grounding correctness on coordinate-based steps is only 52.4%. Non-coordinate actions reach 67.6%. This suggests that current models often identify the correct operation class yet fail to bind that operation to the correct screen region. The paper’s broader summary is that preliminary evaluation reveals approximately 60% task failure for foundation action models on professional desktop applications (Jian et al., 25 Mar 2026).

GroundCUA-derived training also shows that the suite is not only diagnostic. The paper reports that GroundNext-3B, trained on the derived grounding corpus, reaches 50.6 on OS-World Verified when paired with the o3 planner. This suggests that the dense annotation regime transfers beyond the suite’s own benchmark surface (Jian et al., 25 Mar 2026).

6. Relation to the broader CUA ecosystem

CUA-Suite occupies the demonstration-and-grounding layer of a wider CUA research landscape. Adjacent work has emphasized different infrastructure bottlenecks. CUA-World, built with Gym-Anything, focuses on scalable environment construction and long-horizon interactive tasks, reporting 12,103 tasks across 200+ software applications and train/test splits with contamination filtering; its long-horizon subset, CUA-World-Long, contains 200 tasks, one per application, many requiring hundreds of steps (Aggarwal et al., 7 Apr 2026). UI-CUBE targets enterprise reliability rather than demonstration scale, with 226 tasks across two difficulty tiers, three resolutions, and deterministic application-state validation, and it reports a capability cliff from 67–85% on simple interactions to 9–19% on complex workflows for current CUAs (Cristescu et al., 21 Nov 2025). CUA-Gym addresses RL with verifiable rewards, contributing 32,112 programmatically verified training tuples grounded in 110 environments and showing performance gains on OSWorld-Verified after GSPO training (Wang et al., 25 May 2026).

Other work fills further ecosystem roles. CUARewardBench evaluates outcome and process reward models for desktop trajectories (Lin et al., 21 Oct 2025). CUA-Skill introduces a Windows skill base of 452 atomic skills spanning 17 applications and reports 57.5% best-of-three success on WindowsAgentArena for its skill-based agent (Chen et al., 28 Jan 2026). A11y-CUA adds an accessibility lens by comparing blind and low-vision users, sighted users, and CUAs across 60 Windows tasks, showing sharp performance drops under keyboard-only and magnifier conditions (Mohanbabu et al., 10 Feb 2026). Security-focused work systematizes seven vulnerability classes for computer-use agents, including UI deception, indirect prompt injection, and chain-of-thought exposure (Jones et al., 7 Jul 2025).

Within this broader landscape, CUA-Suite’s distinctive role is to provide continuous expert desktop video together with dense element grounding and long reasoning traces. Its limitations, as stated in the source material, follow from that same design. The suite is centered on open-source desktop software, not proprietary enterprise systems; its demonstrations reflect curated expert workflows rather than passive real-user telemetry; and its current coverage is desktop-focused rather than cross-platform. A plausible implication is that CUA-Suite is best understood not as a universal benchmark, but as the high-fidelity perceptual and behavioral substrate that complements environment suites, RLVR corpora, accessibility datasets, reward-model benchmarks, skill libraries, and security testbeds already emerging in the CUA literature (Jian et al., 25 Mar 2026).