VideoCUA: Expert Desktop Video Corpus

• VideoCUA is a comprehensive, continuous video corpus capturing full temporal desktop interactions, including cursor kinematics, intermediate visual feedback, and detailed reasoning annotations.
• It comprises about 10,000 expert-demonstrated tasks across 87 professional applications with 55 hours of 30 fps recordings and multi-layered annotations, enabling transformation into various training formats.
• The dataset supports grounded perception, continuous spatial control, planning, and reward modeling, while exposing current limitations in fine-grained visual grounding for action models.

Searching arXiv for VideoCUA and closely related CUA-Suite papers to ground the article. VideoCUA is the core training corpus in CUA-Suite, designed specifically for computer-use agents (CUAs) operating in professional desktop applications. It is presented as the largest open expert video dataset of its kind for desktop interaction, and its central premise is that continuous video is fundamentally more informative than sparse screenshot-action pairs because it preserves the full temporal dynamics of human interaction: cursor motion, intermediate visual feedback, hover states, drag paths, and action timing (Jian et al., 25 Mar 2026). In this formulation, VideoCUA is not merely a repository of GUI states; it is a multimodal demonstration corpus intended to support grounded perception, planning, continuous spatial control, reward modeling, and visual world modeling for general-purpose desktop agents.

1. Definition and problem setting

VideoCUA addresses what the paper identifies as the main bottleneck for generalist CUAs: the lack of continuous, high-quality human demonstrations. The dataset is positioned against prior GUI corpora that record isolated states or final click coordinates. In contrast, VideoCUA retains synchronized full-screen video and action-relevant trajectories, so that models can observe the intermediate states between actions rather than only the endpoints (Jian et al., 25 Mar 2026).

The motivation is explicitly temporal and control-theoretic. Desktop tasks unfold over time, and screenshot datasets miss the “in-between” evidence needed for planning and grounding. Cursor behavior is also treated as intrinsically continuous rather than reducible to a single click coordinate: approach, deceleration, drag, fine positioning, and correction are part of the signal. This suggests that VideoCUA is meant to support policy learning regimes in which trajectory geometry and temporal evolution are first-class supervision, rather than incidental metadata.

The paper further states that the continuous video streams in VideoCUA form “a superset of information that can be losslessly transformed into the formats required by existing agent frameworks” (Jian et al., 25 Mar 2026). That characterization is central to the dataset’s role within the broader CUA ecosystem. It implies backward compatibility with screenshot-action pipelines while preserving information needed for future frameworks that model action-conditioned transitions or continuous control.

2. Scale, composition, and annotation schema

VideoCUA provides approximately 10,000 human-demonstrated tasks across 87 diverse applications with continuous 30 fps screen recordings, kinematic cursor traces, and multi-layered reasoning annotations, totaling approximately 55 hours and 6 million frames of expert video (Jian et al., 25 Mar 2026). The comparison in the paper also lists 10 actions as the action vocabulary/statistic for the dataset.

The dataset’s main quantitative profile can be summarized as follows.

Aspect	VideoCUA
Tasks	~10,000
Applications	87
Video scale	55 hours, ~6 million frames, continuous 30 fps
Interaction signal	kinematic cursor traces
Annotation density	~497 words per step
Action statistic	10 actions

The annotation density is unusually high for a desktop interaction corpus. The paper specifies that the synthesized trajectories average 496.7 words per step, decomposed into 157.4 observation, 194.3 reasoning, 17.7 action description, and 127.4 reflection (Jian et al., 25 Mar 2026). This makes the supervision explicitly multi-layered rather than a single natural-language comment attached to each action.

VideoCUA includes both grounding-related annotations and reasoning annotations. For grounding, keyframes are extracted immediately before state-changing actions, and every visible UI element in those keyframes is manually labeled with bounding boxes. Each element also receives a text label: element name when available, displayed text for short strings, or a concise summary for long text. OCR is additionally extracted with PaddleOCR for long text regions. About 50% of elements are assigned to one of eight high-level functional categories: Input Element, Sidebar, Information Display, Button, Navigation, Visual Elements, Menu, and Others (Jian et al., 25 Mar 2026).

For reasoning, the paper adopts an OpenCUA-style synthesis pipeline with four annotation layers: Observation, Thought chain, Action description, and Reflection. The formal trajectory representation is

$$\tau_t = (s_t, o_t, r_t, d_t, a_t, s_{t+1}, \text{ref}_t),$$

where $s_t$ is the state or keyframe, $o_t$ the observation, $r_t$ the reasoning chain, $d_t$ the action description, $a_t$ the executable pyautogui code, $s_{t+1}$ the next state, and $\text{ref}_t$ the reflection (Jian et al., 25 Mar 2026). For an arXiv-reading audience, the important point is that VideoCUA pairs perceptual grounding with executable action structure and explicit post-action analysis.

3. Continuous demonstrations as a modeling substrate

A defining property of VideoCUA is that it preserves the full temporal structure of desktop behavior. The paper contrasts this explicitly with ScaleCUA, which contains 2 million screenshots, equivalent to less than 20 hours of video at 30 fps. VideoCUA instead totals approximately 55 hours and 6 million frames, described as more than $2.5\times$ the approximately 2 million screenshots ($\sim 18.5$ hours at 30 fps) in ScaleCUA (Jian et al., 25 Mar 2026).

The distinction is not only one of scale but of representational form. Screenshot datasets can encode state snapshots and sometimes final action coordinates, but they do not preserve hover states, drag paths, action timing, or intermediate interface feedback. VideoCUA therefore supports several transformations without information loss: screenshot-action pairs, state-action-next-state triplets, and continuous kinematic traces (Jian et al., 25 Mar 2026). The paper gives concrete examples of these conversions: sampling keyframes and pairing them with the next action for screenshot-action models, extracting $s_t$0 transitions for world-model training, and directly using logged mouse trajectories for cursor-control policies.

This transformability matters methodologically. It means VideoCUA is not tied to a single training paradigm. A model can treat it as a conventional imitation-learning corpus, as a grounding benchmark source, as a control dataset, or as a transition dataset for action-conditioned prediction. A plausible implication is that the dataset is intended to reduce fragmentation between discrete-action CUA pipelines and emerging video-native approaches.

4. Role inside CUA-Suite

CUA-Suite is presented as a large-scale ecosystem of expert video demonstrations and dense annotations for professional desktop computer-use agents, and VideoCUA is its core training corpus (Jian et al., 25 Mar 2026). The suite also includes UI-Vision, described as a rigorous benchmark for evaluating grounding and planning capabilities in CUAs, and GroundCUA, a large-scale grounding dataset with 56K annotated screenshots and over 3.6 million UI element annotations (Jian et al., 25 Mar 2026).

Within that ecosystem, VideoCUA occupies the role of continuous demonstration substrate rather than isolated evaluation benchmark. UI-Vision emphasizes evaluation of grounding and planning; GroundCUA emphasizes large-scale screenshot-level grounding; VideoCUA provides expert trajectories with temporal continuity, cursor kinematics, and reasoning layers. This division of labor suggests a modular research workflow in which perception, planning, execution, and evaluation can be studied jointly or separately, while sharing a common data ecosystem.

The paper’s emphasis on professional desktop applications is also significant. Difficult interfaces are characterized not merely as more cluttered images but as applications with dense toolbars, multiple panels, unusual widgets, application-specific visual vocabularies, and canvas-based or non-standard layouts. Examples given include Krita, FreeCAD, Inkscape, OBS Studio, and Darktable (Jian et al., 25 Mar 2026). In that sense, VideoCUA is designed to target failure modes that are underrepresented in lighter-weight web-centric or mobile-centric datasets.

5. Research affordances and enabled directions

The paper highlights several research directions enabled by VideoCUA’s continuous and richly annotated structure (Jian et al., 25 Mar 2026). First, its dense element annotations support grounding and generalist screen parsing, including canvas-based or custom-drawn widgets that DOM or accessibility-tree methods miss. Second, its reasoning layers and task-level trajectories support planning and action prediction, since models can learn not only where to click but what to do next.

Third, VideoCUA supports continuous spatial control. The paper argues that discrete click prediction is insufficient for high-precision GUI work, and that cursor traces capture move-to motions, deceleration near targets, drag behaviors, and fine-grained cursor adjustment. This makes the corpus directly relevant to imitation learning and offline RL for human-like cursor control (Jian et al., 25 Mar 2026).

Fourth, the paper connects VideoCUA to reward modeling. Because the dataset contains successful expert demonstrations plus step-level annotations, it can be used to train models that judge whether a task was completed successfully and perhaps provide finer reward signals than binary success/failure. This linkage is reinforced by the separate work on video-based reward modeling for CUAs, which introduces ExeVR-53k and ExeVRM and argues that execution video can serve as a scalable, model-agnostic evaluator across desktop, web, and mobile platforms (Song et al., 10 Mar 2026).

Fifth, VideoCUA supports visual world models. Since the recordings are continuous 30 fps video, the dataset supports action-conditioned future prediction: simulating the visual consequence of a click, predicting UI changes after a drag or menu action, and supporting visual lookahead planning (Jian et al., 25 Mar 2026). This suggests a convergence between CUA training corpora and video-prediction paradigms more commonly associated with embodied agents.

6. Empirical findings and current limitations

The paper uses VideoCUA to probe the performance of current foundation action models on professional desktop software and reports substantial weakness (Jian et al., 25 Mar 2026). On automated action prediction using 256 sampled tasks from VideoCUA, OpenCUA-7B achieves 16.5% @50px and OpenCUA-32B achieves 37.7% @50px. Mean pixel distance improves from 387.5 px for 7B to 274.2 px for 32B, but absolute grounding performance remains weak.

Human evaluation on 49 tasks and 576 annotated steps shows that the 32B model achieves 57.6% combined stepwise accuracy, 85.9% action correctness, and only 52.4% grounding accuracy on coordinate-based steps (Jian et al., 25 Mar 2026). The interpretation given in the paper is precise: the model often knows the right action type but fails to localize the target precisely. This is an important corrective to a common misconception that professional desktop failure is mainly a planning problem; the findings indicate that fine-grained visual grounding remains a major bottleneck.

The paper summarizes the broader implication bluntly: current foundation action models struggle substantially with professional desktop applications (Jian et al., 25 Mar 2026). This limitation is not incidental but constitutive of VideoCUA’s purpose. The dataset is explicitly meant to expose and address precisely those failure modes.

Several constraints are also acknowledged or implied. VideoCUA’s annotations are expert-curated and dense, which is valuable but expensive. The dataset is centered on professional desktop applications rather than unrestricted web-scale environments. A plausible implication is that transfer beyond the represented software ecosystems will depend on how well future models can generalize from the observed distributions of UI structure, cursor kinematics, and reasoning traces. Nonetheless, within its intended domain, VideoCUA serves as a high-fidelity corpus for studying grounded, temporally aware, planning-capable, and control-sensitive computer-use agents.