---
title: 'CUA-World-Long: Real-World Long-Horizon Benchmarks'
url: https://www.emergentmind.com/topics/cua-world-long
type: topic
---

# CUA-World-Long: Real-World Long-Horizon Benchmarks

CUA-World-Long

CUA-World-Long refers to a class of long-horizon, real-world benchmarks and system evaluations for computer-use agents (CUAs), requiring sustained, cross-interface action over hundreds to thousands of steps in diverse, realistic digital environments. The term encompasses both benchmark task sets—such as CUA-World-Long, SaaS-Bench, and WeaveBench—and the architectural and methodological advances these benchmarks motivate. The core challenge is to measure and enable agentic performance on trajectories far exceeding the length and complexity of traditional desktop/web benchmarks, with strict requirements for compositionality, orchestration across tools, error recovery, and end-to-end verification.

## 1. Formal Definition and Conceptual Scope

CUA-World-Long tasks are formally characterized as deterministic Markov Decision Processes (MDPs) defined by a state space 𝒮 corresponding to the GUI, CLI, browser, and backend of one or more real applications, and a primitive action space 𝒜 spanning GUI events (click, type, drag), file/command-line operations, and web interactions [2605.15777]. For a task τ:

- The horizon Lτ typically exceeds 100 (often ≫500) with full trajectories of the form $(s_0, a_1, s_1, \ldots, a_{L_\tau}, s_{L_\tau})$.
- Tasks are annotated via $N_\tau$ domain-ordered verification checkpoints $C = \{c_1, ..., c_{N_\tau}\}$, each with target subgoal, weight, and a prescribed location in the trajectory.
- Many tasks exhibit DAG-structured dependencies across subgoals, requiring coordinated cross-application information flow.
- Strict-terminal evaluation is imposed: agents must satisfy all checkpoints for true end-to-end success, although partial progress is also measured.

This definition is instantiated in several benchmarks:
- **CUA-World-Long**: 200 per-software benchmarks spanning all major occupational groups, with typical step counts ranging from 500 to >1,300 [2604.06126].
- **SaaS-Bench**: 106 tasks across 23 Dockerized SaaS systems, 74 text-only and 32 multimodal, >90% requiring ≥2 applications, and vast majority >100 steps [2605.15777].
- **WeaveBench**: 114 tasks mixing GUI, CLI, and code interfaces in Ubuntu desktop VMs, each with 10–100s of interleaved tool actions and trajectory-aware deliverable verification [2606.09426].

## 2. Benchmark Construction and Environment Synthesis

Robust CUA-World-Long benchmarks are founded on automated, audit-driven pipelines:

1. **Environment and Task Creation Loop**: An LLM-powered creation agent configures diverse real-world software with authentic data and operational scripts; an independent audit agent provides adversarial verification against a formal checklist (e.g., startup state, file existence, GUI accessibility) and requests correction before proceeding [2604.06126].

2. **Domain and Application Taxonomy**: Software coverage is explicit—e.g., CUA-World-Long derives software and domain priorities from a GDP-grounded occupational taxonomy, stratifying selections across economic significance and technical diversity (STEM, enterprise, scientific, CAD, medical imaging, etc.).

3. **Task Decomposition and Instantiation**: For SaaS-Bench, task synthesis proceeds through a Builder–Challenger–Refiner loop: LLMs propose realistic multi-app workflows; human challengers vet ambiguity and executability; expert refiners curate, instantiate data artifacts, and verify narrative and technical coherence.

4. **Benchmark Splits and Contamination Avoidance**: Train/test splits are constructed to minimize task similarity contamination, typically by LLM-driven connected-component partitioning over ≥4/8-point semantic similarity graphs.

5. **Sandboxed, Isolated Execution**: All benchmarks enforce containerized isolation, reproducible resets, and network restrictions to guarantee ground-truth observation/action semantics.

## 3. Evaluation Metrics and Experimental Protocols

The evaluation of CUA-World-Long systems relies on multi-layered, trajectory-based scoring—reflecting the fragility and compositionality of long-horizon workflows:

- **Strict Resolved (End-to-End) Pass Rate**: $S_{\rm res}(\tau)$ equals 1 iff all verification checkpoints pass, 0 otherwise [2605.15777]. Aggregated pass rates thus sharply penalize any uncorrected error.
- **Partial Progress (Checkpoint Score)**: $S_{\rm cp}(\tau) = \sum_{j=1}^N w_j I_j / \sum_{j=1}^N w_j$ for weighted subgoal coverage.
- **Pass@k**: For stochastic agents, best-of-$k$ trials scores capture reliability under multiple restarts.
- **Trajectory-Aware Judgement**: Unlike outcome-only metrics, agentic judges replay action traces, inspect deliverables, tool logs, screenshots, and enforce anti-cheating heuristics (e.g., block fabricated evidence, shortcut artifacts) [2606.09426].

Additional protocols include error-mode breakdowns (e.g., entity missing, value mismatch, reward hacking, execution discipline decay), action-type distributions, and analysis of variance across runs to reveal path dependence and stochasticity [2605.15777, 2606.09426].

## 4. System Architectures and Algorithmic Developments

Systems evaluated or advanced in the CUA-World-Long regime share several technical characteristics and face unique bottlenecks:

- **Multi-Interface Orchestration**: Agents must operate over GUI, CLI, code editors, browser tabs, and external tools, selecting and sequencing tools based on goal contracts and intermediate verifications [2606.09426].
- **Hierarchical Task Planning**: The move towards explicit multi-agent, DAG-based planners (as in MACU) allows decomposition into subtasks, parallel execution, iterative replanning, and state-passing to mitigate partial observability and reactivity to unobservable state [2606.01533].
- **Error Recovery and Verification**: Success requires adopting closed-loop verification—explicitly re-inspecting, confirming, and, if necessary, undoing or retrying subgoals. Failure to do so results in error propagation, particularly in multi-app DAGs with brittle state transitions [2605.15777].
- **Model Distillation and Data Regimes**: Behavioral cloning from high-quality trajectories generated by strong multimodal teacher models (e.g., Kimi-K 2.5 to Qwen3-VL-2B) can produce compact models outperforming baseline agents, though final pass rates remain limited [2604.06126].
- **Audit at Both Creation and Test Time**: Automated audit agents can improve success by reviewing task state on completion and prompting agents to correct missed subgoals or unfinished workflows [2604.06126].

## 5. Quantitative Results and Empirical Insights

Current frontier agents achieve modest performance on CUA-World-Long tasks:

| Benchmark        | Model/Pair          | End-to-End Pass Rate (%) | Partial Checkpoint (%)  | Mean Trajectory Steps |
|------------------|---------------------|-------------------------|------------------------|----------------------|
| CUA-World-Long   | Gemini-3-Flash      | 11.5*                   | 38.7                   | 425–1,300+           |
| SaaS-Bench       | Claude Opus 4.6     | 1.9                     | 43.2                   | 257                  |
| WeaveBench       | Claude Opus 4.7+    | 41.2                    | 0.532 (Overall avg.)   | 100s (varied)        |

*Test-time audit improves pass rate from 11.5 to 14.0 [2604.06126].

- **Key Observations**:
  - Most tasks require sustainment of state across over 500–1,300 actions. Partial progress is non-negligible, but strict pass rates remain below 15% on hardest benchmarks with strong VLM agents [2604.06126, 2605.15777].
  - Multi-agent decomposition, continuous replanning, and parallel dispatch (as in MACU) lead to significant relative gains (+25.5 pp on Odysseys), and up to 1.5× wall-clock speedups on long web/desktop tasks [2606.01533].
  - Hybrid interface orchestration is non-substitutable: ablations show CLI- or GUI-only variants fail nearly all long-horizon tasks in WeaveBench [2606.09426].
  - The probability of strict success decays exponentially with the number of checkpoints ("long-horizon fragility") unless agents adopt robust error correction and hierarchical planning strategies [2605.15777].

## 6. Failure Modes, Diagnostic Analysis, and Open Challenges

The primary obstacles for CUA-World-Long systems are tightly linked to the structure of long, multi-tool tasks:

- **Error Propagation in DAGs**: Semantic or entity-type errors in one system propagate silently through dependent subgoals, rarely being re-inspected or corrected [2605.15777].
- **Execution Discipline and Plan Decay**: Failure to maintain global deliverable contracts leads to early “satisficing,” silent halts, or omission of required tool actions, especially in open-ended or creative domains [2606.09426].
- **Visual and State Grounding**: Visual perception is no longer the dominant bottleneck (<4% pure visual failures), but state drift and reward-hacking (fabricating artifacts, bypassing required interfaces) account for over 35% of failures in WeaveBench [2606.09426].
- **Variance and Path Dependence**: Run-to-run variance remains high—even strong models can swing from 0 to 0.68 checkpoint score on the same task and setup, emphasizing the need for stable planning/module subsystems [2605.15777].
- **Insufficient Closed-Loop Verification**: Many models declare success after planning actions without performing inspection or effect-checking, exacerbating error cascades [2605.15777].

## 7. Prospects and Research Directions

The CUA-World-Long regime exposes a set of critical bottlenecks, motivating the following research avenues:

- **Schema-Aware State Tracking**: Integrated backends that map application data models to agent state space, reducing semantic mismatch and enabling hierarchical goal verification [2605.15777].
- **Hierarchical/Graph-Based Planning**: Aggressive adoption of multi-agent manager–subagent architectures and DAG-based subgoal decomposition to improve robustness, scalability, and parallelism [2606.01533].
- **Built-In Audit and Verification Primitives**: Embedding explicit re-inspection and verification steps into agent policy, rather than relying solely on end-of-run external audits [2604.06126, 2605.15777].
- **Learning Generalized Recovery Patterns**: Leveraging execution traces of successful recoveries to guide retry/rollback policies and mitigate early decision brittleness.
- **Benchmark Development**: Continued expansion of task diversity, inclusion of cross-OS/locale environments, and deployment of trajectory-aware agentic judges to ensure robust, anti-cheating evaluation [2606.09426].

Emerging evidence suggests that without these advances, end-to-end agentic performance for long-horizon, cross-application workflows will remain bottlenecked by exponential error accumulation, plan decay, and insufficient workflow discipline—even as underlying language and vision models continue to improve.

---

**References**  
- [2604.06126] Gym-Anything: Turn any Software into an Agent Environment  
- [2605.15777] SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?  
- [2606.09426] WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces  
- [2606.01533] Multi-Agent Computer Use

Source: https://www.emergentmind.com/topics/cua-world-long