CUA-World Benchmark Suite

• CUA-World is a benchmark suite of digital twins modeling human computing contexts, featuring over 10,000 long-horizon tasks.
• It uses a multi-agent, automated creation-audit pipeline to generate and verify realistic software environments and tasks.
• Evaluation metrics such as task success rates and partial-credit scoring offer granular insights into agent performance.

CUA-World defines a comprehensive benchmarking suite and associated research infrastructure for evaluating computer-use agents (CUAs)—autonomous, model-driven systems that interact with software across diverse, economically meaningful domains. CUA-World is characterized by its breadth of supported platforms, rigorous task curation, automated environment generation at scale, and methodology for fine-grained, grounded evaluation. Its architecture models the digital workspaces of human users, enabling empirical assessment of agentic capabilities for real-world, long-horizon digital workflows spanning medical science, engineering, enterprise systems, and more (Aggarwal et al., 7 Apr 2026).

1. Definition, Scope, and Motivations

CUA-World is a benchmark suite of fully interactive environments, modeled as a "digital twin" of human computing contexts (Awadallah et al., 24 Nov 2025). It encompasses over 10,000 long-horizon tasks distributed among 200 diverse software applications, targeting all 22 U.S. SOC major occupation groups. Supported platforms include Linux, Windows, Android, and Web, with applications and tasks mapped to major sectors such as medical science (e.g., radiology), astronomy, engineering (CAD, modeling), enterprise (ERP, CRM), finance, GIS, and forensic analysis (Aggarwal et al., 7 Apr 2026). Each software environment is configured with realistic, publicly available data, with strict separation of "Train" and "Test" splits (∼9,720 training, ∼2,500 testing tasks), alongside a dedicated CUA-World-Long benchmark of tasks regularly exceeding 500 discrete steps (Aggarwal et al., 7 Apr 2026).

The principal motivation is to enable empirical, reproducible evaluation of agentic AI in settings that reflect both the complexity and the economic relevance of actual human computer use, addressing the limitations of previous benchmarks restricted to toy problems, GUI-only workflows, or tasks with limited occupational coverage.

2. Multi-Agent Automated Environment Creation

A key innovation of CUA-World is the automation of environment and task generation via a multi-agent pipeline ("creation-audit loop"):

• Creation Agent ($\mathrm{Agent}_C$): An LLM-based coding agent (Claude Opus 4.5/4.6) authors three setup scripts—install.sh (software and dependencies), configure.sh (domain data), and task_setup.sh (task-specific state)—along with a JSON config per environment (Aggarwal et al., 7 Apr 2026). It performs research on installing and configuring each software, downloads and formats relevant datasets, and produces visual and log-based "evidence documents" to substantiate correct setup.
• Audit Agent ($\mathrm{Agent}_{\mathrm{audit}}$): A separately prompted adversarial Claude agent verifies setup evidence against a comprehensive checklist that includes screenshot correspondence, structured data presence, and absence of stale outputs, ensuring full sandbox integrity (Aggarwal et al., 7 Apr 2026). Feedback triggers corrective iteration until the environment passes all checks.

This pipeline learns from a shared memory of successful and failed setups, enabling sublinear scaling as the number of environments grows.

Task generation follows propose-and-amplify strategies: a modest set of high-quality agent-generated seeds (∼5 per software) is expanded through LLM-based permutation and VLM-based validation, yielding both breadth and depth across the task taxonomy.

3. Task Taxonomies, Platforms, and CUA-World-Long

The selection of software is grounded in U.S. GDP at the occupational level: O*NET occupation data is combined with LLM and web-based research to quantify the economic significance of each software application via the formula

$$\text{GDP}_{\text{software}} = \sum_{\text{occ}} \left(\text{GDP}_{\text{occ}} \times p_{\text{computer}} \times s_{\text{category}} \times s_{\text{software}}\right)$$

where $p_{\text{computer}}$ is the fraction of work on computers, $s_{\text{category}}$ is share of software category use, and $s_{\text{software}}$ is share of the product within the category (Aggarwal et al., 7 Apr 2026).

Software is selected in a five-tier process: top GDP products, strategic/STEM coverage, round-robin by major group, niche products, and additional category coverage.

Tasks are distributed across platforms, ensuring coverage of both domain-specialized and general-purpose software. CUA-World-Long comprises exceptionally challenging, long-horizon tasks (one per software), each constructed from analysis of typical agent failures and designed such that completion requires authentic, multi-hundred-step workflows (Aggarwal et al., 7 Apr 2026). Train-test contamination is minimized using LLM-based similarity scoring to cluster similar tasks within the same split.

4. Evaluation Methodologies and Metrics

CUA-World environments support automated, checklist-based VLM verification of agent trajectories, extracting privileged ground truth signals for robust, partial-credit, and stepwise evaluation (Aggarwal et al., 7 Apr 2026). This avoids reliance on brittle or external signals such as UI screenshots or file diffs which are prone to failure with UI drift.

Models are primarily evaluated on:

• Task Success Rate (SR): Proportion of test tasks in which agent actions yield all required deliverables or states within a step/cost/time budget.
• Partial-Credit Scoring: For tasks with multiple sub-goals, models may be awarded granular credit for partial task completion.
• Scaling and Generalization: Learning curves for both number of tasks and number of supported software, separating in-domain (IID) from out-of-domain (OOD) generalization (Aggarwal et al., 7 Apr 2026).

Test-time auditing is explicitly supported: a VLM reviews completed agent trajectories, offering corrective feedback—this mechanism increased Gemini-3-Flash's pass rate on CUA-World-Long from 11.5% to 14.0% (Aggarwal et al., 7 Apr 2026).

5. Key Experimental Results

Benchmarking on CUA-World demonstrates the utility of multi-agent pipeline scaling, test-time auditing, and knowledge distillation.

Model	Avg. Score (Test)	Pass Rate (%) (Test)	Long-Horizon Pass Rate (%)
Qwen3-VL-2B (baseline)	12.7	1.6	-
Qwen3-VL-2B (distilled)	22.5	4.4	-
Qwen3-VL-4B (baseline)	19.3	3.9	-
Kimi-K 2.5	37.1	12.8	-
Gemini 3 Flash (Test)	50.1	22.6	11.5 (Long), 14.0 (+TTA)

Distilling task trajectories from larger, agentic teacher models into compact vision-language students (Qwen3-VL-2B) yields student agents outperforming models twice their size (Aggarwal et al., 7 Apr 2026). Pass rates sharply increase once the average step budget exceeds ∼100 steps for test tasks; on CUA-World-Long, step budgets and test-time auditing directly raise pass rates.

Scaling experiments show that doubling number of software or tasks results in roughly 3.5 average score points of gain, with generalization lagging for out-of-domain (OOD) software—indicating limited cross-software transferability and motivating further research (Aggarwal et al., 7 Apr 2026).

6. Security Framework and Design Principles

CUA-World surfaces both capability and security challenges. A systematization of CUA-specific risks (Jones et al., 7 Jul 2025) identifies seven threat classes relevant for large-scale agent environments:

1. UI Deception and Perceptual Mismatch
2. Remote Code Execution via Multi-Action Composition
3. Chain-of-Thought (CoT) Exposure and Leakage
4. Human-in-the-Loop Safeguard Bypass
5. Indirect Prompt Injection
6. Identity Ambiguity and Over-Delegation
7. Content Harms from Emergent Inference

A comprehensive security framework is recommended:

• Input Provenance Tracking: Source and trust tagging of every perceptual artifact.
• Interface–Action Binding Checks: Retrospective verification of plan-to-execution mapping.
• Memory and CoT Control: Treat all reasoning traces as privileged internal metadata; enforce encryption or redaction.
• Context-Aware Gating: Require explicit and cryptographically signed intent for all side-effecting actions.
• Runtime Planning Audits: Systematic auditing of planned action sequences.
• Ephemeral/Scoped Execution: Enforce container reset and session isolation for every task (Jones et al., 7 Jul 2025).

Implementation best practices include the use of hardened orchestration layers, strict policy guards on browser APIs, auditable ledgers of agent actions, and adversarial testing across all identified risk classes.

7. Research Insights and Future Directions

CUA-World demonstrates that environment and task creation bottlenecks can be addressed through agentic pipelines leveraging shared memory and adversarial audit, supporting sublinear scaling across domains (Aggarwal et al., 7 Apr 2026). Scaling, test-time auditing, and model distillation are empirically validated as key levers for performance gains.

Challenges persist: long-horizon planning remains unsolved (even frontier models pass <28% of CUA-World-Long tasks), OOD generalization is weak, and visually complex or domain-specialized software presents persistent difficulty for compact models. CUA-World’s structure is flexible and extensible to accommodate new models, specialized planners, and cross-application workflows.

A plausible implication is that open, modular environments such as CUA-World will serve as a standard research substrate for the development, evaluation, and iterative improvement of safe, robust, and general-purpose computer-use agents (Aggarwal et al., 7 Apr 2026, Jones et al., 7 Jul 2025). Continued investigation into RL refinement, self-debate, hierarchical planning, and human–agent collaboration is anticipated to drive progress on this benchmark suite.