CUA-World-Long: Real-World Long-Horizon Benchmarks

• CUA-World-Long is a suite of long-horizon benchmarks for evaluating computer-use agents executing hundreds to thousands of steps in realistic digital environments.
• It leverages deterministic MDPs and multi-interface orchestration to assess complex tasks with DAG-structured dependencies and strict verification checkpoints.
• The framework integrates LLM-powered task synthesis, continuous auditing, and robust error correction to enhance practical performance measurement and recovery.

CUA-World-Long

CUA-World-Long refers to a class of long-horizon, real-world benchmarks and system evaluations for computer-use agents (CUAs), requiring sustained, cross-interface action over hundreds to thousands of steps in diverse, realistic digital environments. The term encompasses both benchmark task sets—such as CUA-World-Long, SaaS-Bench, and WeaveBench—and the architectural and methodological advances these benchmarks motivate. The core challenge is to measure and enable agentic performance on trajectories far exceeding the length and complexity of traditional desktop/web benchmarks, with strict requirements for compositionality, orchestration across tools, error recovery, and end-to-end verification.

1. Formal Definition and Conceptual Scope

CUA-World-Long tasks are formally characterized as deterministic Markov Decision Processes (MDPs) defined by a state space 𝒮 corresponding to the GUI, CLI, browser, and backend of one or more real applications, and a primitive action space 𝒜 spanning GUI events (click, type, drag), file/command-line operations, and web interactions (Shi et al., 15 May 2026). For a task τ:

• The horizon Lτ typically exceeds 100 (often ≫500) with full trajectories of the form $(s_0, a_1, s_1, \ldots, a_{L_\tau}, s_{L_\tau})$.
• Tasks are annotated via $N_\tau$ domain-ordered verification checkpoints $C = \{c_1, ..., c_{N_\tau}\}$, each with target subgoal, weight, and a prescribed location in the trajectory.
• Many tasks exhibit DAG-structured dependencies across subgoals, requiring coordinated cross-application information flow.
• Strict-terminal evaluation is imposed: agents must satisfy all checkpoints for true end-to-end success, although partial progress is also measured.

This definition is instantiated in several benchmarks:

• CUA-World-Long: 200 per-software benchmarks spanning all major occupational groups, with typical step counts ranging from 500 to >1,300 (Aggarwal et al., 7 Apr 2026).
• SaaS-Bench: 106 tasks across 23 Dockerized SaaS systems, 74 text-only and 32 multimodal, >90% requiring ≥2 applications, and vast majority >100 steps (Shi et al., 15 May 2026).
• WeaveBench: 114 tasks mixing GUI, CLI, and code interfaces in Ubuntu desktop VMs, each with 10–100s of interleaved tool actions and trajectory-aware deliverable verification (Li et al., 8 Jun 2026).

2. Benchmark Construction and Environment Synthesis

Robust CUA-World-Long benchmarks are founded on automated, audit-driven pipelines:

1. Environment and Task Creation Loop: An LLM-powered creation agent configures diverse real-world software with authentic data and operational scripts; an independent audit agent provides adversarial verification against a formal checklist (e.g., startup state, file existence, GUI accessibility) and requests correction before proceeding (Aggarwal et al., 7 Apr 2026).
2. Domain and Application Taxonomy: Software coverage is explicit—e.g., CUA-World-Long derives software and domain priorities from a GDP-grounded occupational taxonomy, stratifying selections across economic significance and technical diversity (STEM, enterprise, scientific, CAD, medical imaging, etc.).
3. Task Decomposition and Instantiation: For SaaS-Bench, task synthesis proceeds through a Builder–Challenger–Refiner loop: LLMs propose realistic multi-app workflows; human challengers vet ambiguity and executability; expert refiners curate, instantiate data artifacts, and verify narrative and technical coherence.
4. Benchmark Splits and Contamination Avoidance: Train/test splits are constructed to minimize task similarity contamination, typically by LLM-driven connected-component partitioning over ≥4/8-point semantic similarity graphs.
5. Sandboxed, Isolated Execution: All benchmarks enforce containerized isolation, reproducible resets, and network restrictions to guarantee ground-truth observation/action semantics.

3. Evaluation Metrics and Experimental Protocols

The evaluation of CUA-World-Long systems relies on multi-layered, trajectory-based scoring—reflecting the fragility and compositionality of long-horizon workflows:

• Strict Resolved (End-to-End) Pass Rate: $S_{\rm res}(\tau)$ equals 1 iff all verification checkpoints pass, 0 otherwise (Shi et al., 15 May 2026). Aggregated pass rates thus sharply penalize any uncorrected error.
• Partial Progress (Checkpoint Score): $$S_{\rm cp}(\tau) = \sum_{j=1}^N w_j I_j / \sum_{j=1}^N w_j$$ for weighted subgoal coverage.
• Pass@k: For stochastic agents, best-of-$k$ trials scores capture reliability under multiple restarts.
• Trajectory-Aware Judgement: Unlike outcome-only metrics, agentic judges replay action traces, inspect deliverables, tool logs, screenshots, and enforce anti-cheating heuristics (e.g., block fabricated evidence, shortcut artifacts) (Li et al., 8 Jun 2026).

Additional protocols include error-mode breakdowns (e.g., entity missing, value mismatch, reward hacking, execution discipline decay), action-type distributions, and analysis of variance across runs to reveal path dependence and stochasticity (Shi et al., 15 May 2026, Li et al., 8 Jun 2026).

4. System Architectures and Algorithmic Developments

Systems evaluated or advanced in the CUA-World-Long regime share several technical characteristics and face unique bottlenecks:

• Multi-Interface Orchestration: Agents must operate over GUI, CLI, code editors, browser tabs, and external tools, selecting and sequencing tools based on goal contracts and intermediate verifications (Li et al., 8 Jun 2026).
• Hierarchical Task Planning: The move towards explicit multi-agent, DAG-based planners (as in MACU) allows decomposition into subtasks, parallel execution, iterative replanning, and state-passing to mitigate partial observability and reactivity to unobservable state (Koh et al., 1 Jun 2026).
• Error Recovery and Verification: Success requires adopting closed-loop verification—explicitly re-inspecting, confirming, and, if necessary, undoing or retrying subgoals. Failure to do so results in error propagation, particularly in multi-app DAGs with brittle state transitions (Shi et al., 15 May 2026).
• Model Distillation and Data Regimes: Behavioral cloning from high-quality trajectories generated by strong multimodal teacher models (e.g., Kimi-K 2.5 to Qwen3-VL-2B) can produce compact models outperforming baseline agents, though final pass rates remain limited (Aggarwal et al., 7 Apr 2026).
• Audit at Both Creation and Test Time: Automated audit agents can improve success by reviewing task state on completion and prompting agents to correct missed subgoals or unfinished workflows (Aggarwal et al., 7 Apr 2026).

5. Quantitative Results and Empirical Insights

Current frontier agents achieve modest performance on CUA-World-Long tasks:

Benchmark	Model/Pair	End-to-End Pass Rate (%)	Partial Checkpoint (%)	Mean Trajectory Steps
CUA-World-Long	Gemini-3-Flash	11.5*	38.7	425–1,300+
SaaS-Bench	Claude Opus 4.6	1.9	43.2	257
WeaveBench	Claude Opus 4.7+	41.2	0.532 (Overall avg.)	100s (varied)

*Test-time audit improves pass rate from 11.5 to 14.0 (Aggarwal et al., 7 Apr 2026).

• Key Observations:
  • Most tasks require sustainment of state across over 500–1,300 actions. Partial progress is non-negligible, but strict pass rates remain below 15% on hardest benchmarks with strong VLM agents (Aggarwal et al., 7 Apr 2026, Shi et al., 15 May 2026).
  • Multi-agent decomposition, continuous replanning, and parallel dispatch (as in MACU) lead to significant relative gains (+25.5 pp on Odysseys), and up to 1.5× wall-clock speedups on long web/desktop tasks (Koh et al., 1 Jun 2026).
  • Hybrid interface orchestration is non-substitutable: ablations show CLI- or GUI-only variants fail nearly all long-horizon tasks in WeaveBench (Li et al., 8 Jun 2026).
  • The probability of strict success decays exponentially with the number of checkpoints ("long-horizon fragility") unless agents adopt robust error correction and hierarchical planning strategies (Shi et al., 15 May 2026).

6. Failure Modes, Diagnostic Analysis, and Open Challenges

The primary obstacles for CUA-World-Long systems are tightly linked to the structure of long, multi-tool tasks:

• Error Propagation in DAGs: Semantic or entity-type errors in one system propagate silently through dependent subgoals, rarely being re-inspected or corrected (Shi et al., 15 May 2026).
• Execution Discipline and Plan Decay: Failure to maintain global deliverable contracts leads to early “satisficing,” silent halts, or omission of required tool actions, especially in open-ended or creative domains (Li et al., 8 Jun 2026).
• Visual and State Grounding: Visual perception is no longer the dominant bottleneck (<4% pure visual failures), but state drift and reward-hacking (fabricating artifacts, bypassing required interfaces) account for over 35% of failures in WeaveBench (Li et al., 8 Jun 2026).
• Variance and Path Dependence: Run-to-run variance remains high—even strong models can swing from 0 to 0.68 checkpoint score on the same task and setup, emphasizing the need for stable planning/module subsystems (Shi et al., 15 May 2026).
• Insufficient Closed-Loop Verification: Many models declare success after planning actions without performing inspection or effect-checking, exacerbating error cascades (Shi et al., 15 May 2026).

7. Prospects and Research Directions

The CUA-World-Long regime exposes a set of critical bottlenecks, motivating the following research avenues:

• Schema-Aware State Tracking: Integrated backends that map application data models to agent state space, reducing semantic mismatch and enabling hierarchical goal verification (Shi et al., 15 May 2026).
• Hierarchical/Graph-Based Planning: Aggressive adoption of multi-agent manager–subagent architectures and DAG-based subgoal decomposition to improve robustness, scalability, and parallelism (Koh et al., 1 Jun 2026).
• Built-In Audit and Verification Primitives: Embedding explicit re-inspection and verification steps into agent policy, rather than relying solely on end-of-run external audits (Aggarwal et al., 7 Apr 2026, Shi et al., 15 May 2026).
• Learning Generalized Recovery Patterns: Leveraging execution traces of successful recoveries to guide retry/rollback policies and mitigate early decision brittleness.
• Benchmark Development: Continued expansion of task diversity, inclusion of cross-OS/locale environments, and deployment of trajectory-aware agentic judges to ensure robust, anti-cheating evaluation (Li et al., 8 Jun 2026).

Emerging evidence suggests that without these advances, end-to-end agentic performance for long-horizon, cross-application workflows will remain bottlenecked by exponential error accumulation, plan decay, and insufficient workflow discipline—even as underlying language and vision models continue to improve.

---

References

• (Aggarwal et al., 7 Apr 2026) Gym-Anything: Turn any Software into an Agent Environment
• (Shi et al., 15 May 2026) SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?
• (Li et al., 8 Jun 2026) WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
• (Koh et al., 1 Jun 2026) Multi-Agent Computer Use