---
title: OSWorld and AndroidWorld Benchmarks
url: https://www.emergentmind.com/topics/osworld-verified-and-androidworld-benchmarks
type: topic
---

# OSWorld and AndroidWorld Benchmarks

OSWorld-Verified and AndroidWorld are high-fidelity, large-scale benchmarks designed for evaluating autonomous GUI agents across desktop and mobile environments, respectively. These benchmarks provide end-to-end platforms with hundreds of real-world, programmatically verifiable tasks, enabling rigorous, reproducible assessment of agents’ abilities to ground, plan, and act within actual operating systems and apps. Each presents distinct challenges: OSWorld-Verified focuses on task diversity and system heterogeneity in real desktop environments, while AndroidWorld emphasizes dynamic, parameterized mobile workflows in real Android apps. Together, they form the contemporary foundation for empirical advances in generalist computer-use agents.

## 1. Benchmark Design and Scope

**OSWorld-Verified** is the programmatically-verifiable subset of OSWorld, originally introduced by Xie et al. [2404.07972]. OSWorld-Verified contains 369 tasks on Ubuntu (and optionally Windows/macOS) that span file management, system configuration, document editing, application control, and multi-app workflows. Each task is defined by an initial VM snapshot, setup scripts, and a deterministic evaluation function that inspects artifacts such as files, accessibility trees, or application states [2404.07972, 2602.00575]. The action space consists of discrete GUI operations (e.g., click, type, drag, hotkey), optionally extended with shell or code execution calls in some agents [2510.02250, 2602.00575].

**AndroidWorld** comprises 116 tasks covering 20 stock Android apps, including productivity, communication, navigation, and multimedia tasks [2405.14573]. Each task is a dynamic, parameterized template instantiated with specific inputs (e.g., contact names, dates) at runtime. Environment interaction uses ADB for event injection and system state queries. State observation is provided as high-resolution screenshots (2400×1080×3 RGB) and, in some variants, accessibility trees [2405.14573]. The action space aligns with native user gestures (tap, swipe, scroll, type) and device-level navigation.

Tasks in both benchmarks vary widely in complexity: from atomic actions (single dialog navigation) to long-horizon, multi-application workflows requiring persistent memory and context-aware planning. Parameter randomization and initial state shuffling ensure that agents cannot overfit to static templates [2405.14573]. 

## 2. Evaluation Protocols and Metrics

Task completion in OSWorld-Verified and AndroidWorld is measured by end-to-end success as determined by scripted verifiers that inspect post-episode system state rather than brittle pixel or semantic matching [2404.07972, 2405.14573]. For each task $i$ with success indicator $s_i \in \{0,1\}$, the primary metric is:

$$
\textrm{Success Rate} = \frac{1}{N}\sum_{i=1}^N s_i \times 100\%
$$

In AndroidWorld, a given task may also have gradated reward for multi-step goals, formally:

$$
r = \frac{\mathbb{1}[\textrm{goal}_1] + \mathbb{1}[\textrm{goal}_2]}{2}
$$

Pass@$k$ metrics are reported when agents can attempt tasks multiple times per seed:

$$
\mathrm{pass}@k = 1 - (1 - \mathrm{SR})^k
$$

Verifier accuracy is measured via precision, recall, F1, and overall agreement with human or ground-truth script labeling:

$$
\mathrm{Precision} = \frac{\sum_i [\hat{R}_i=1 \wedge y_i=1]}{\sum_i [\hat{R}_i=1]}
$$

$$
\mathrm{Recall} = \frac{\sum_i [\hat{R}_i=1 \wedge y_i=1]}{\sum_i [y_i=1]}
$$

$$
\mathrm{F1} = 2\frac{\mathrm{Precision} \times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}
$$

Test time scaling is a central protocol: increasing the allowed number of steps or rollouts generally yields monotonic accuracy improvements, particularly for agents leveraging best-of-$N$ or read-only reproduction [2510.02250, 2602.00575]. These protocols mitigate effects of non-determinism, environment crashes, and task stochasticity.

## 3. Baseline Results and State-of-the-Art Performance

Recent studies provide comprehensive performance baselines. On OSWorld-Verified, best-performing systems with full best-of-$N$ scaling approach human-level accuracy. On AndroidWorld, elite agents achieve success rates near or above 80%. Below is a synthesized performance summary (Pass@1 unless otherwise noted):

| Model/System       | OSWorld-Verified (%) | AndroidWorld (%) |
|--------------------|---------------------|------------------|
| Human             | 72.4                | 80.0             |
| Surfer 2          | 60.1                | 87.1             |
| UI-TARS-2         | 47.5                | 73.3             |
| Step-GUI-8B       | 48.5 (Pass@3)       | 80.2 (Pass@3)    |
| Mobile-Agent-v3   | 37.7                | 73.3             |
| Ferret-UI Lite    | 19.8 (50 steps)     | 28.0             |

Notable scaling methods include Behavior Best-of-N (bBoN) [2510.02250], achieving 69.9% on OSWorld-Verified with GPT-5 (N=10), and VAGEN, an agentic verifier that raises verification accuracy by 5–10 points on both benchmarks [2602.00575]. UI-TARS-2 and MobileRL-9B set strong open-source records [2509.02544, 2508.15144].

Performance on AndroidWorld tends to be higher than OSWorld-Verified for most agents, due to more regular mobile UI structure and shorter task horizons. Narrower agent architectures and smaller models significantly lag, especially on desktop, as seen with Ferret-UI Lite [2509.26539].

## 4. Methodological Innovations and Design Principles

Both benchmarks are uniquely distinguished by their formalization of real-system, open-ended tasks and programmatic outcome validation [2404.07972, 2405.14573, 2602.00575]. OSWorld-Verified embraces desktop task heterogeneity and partial observability, necessitating agents with robust GUI grounding, action composition, and operational knowledge. AndroidWorld is characterized by its dynamic, parameterized task generation system, yielding effectively infinite unique task instances, and hermetic initialization and verification logic guaranteeing reproducibility over random draws [2405.14573].

Proactive agentic verification approaches—such as VAGEN—demonstrate superior verification through tool-based probing (e.g., shell, python, GUI tool use), overcoming visual evidence deficiencies in partially observable settings [2602.00575]. Read-only majority voting and best-of-$N$ actor selection further stabilize evaluation and maximize sample efficiency.

In contrast, earlier baseline or LLM-as-a-Judge paradigms rely on passive, screenshot-only analysis, which fails to robustly discriminate between alternate but correct execution paths or latent state achievement [2602.00575]. The importance of execution-based, not representation-based, validation is repeatedly evidenced across ablation studies.

## 5. Analysis of Success and Failure Modes

Success in both benchmarks is highly sensitive to the quality of spatial grounding, stepwise contextual memory, and the agent’s ability to recover from intermediate errors. OSWorld-Verified agents most often fail due to inaccurate click localization, misinterpretation of task objectives, or inability to recover from modality or window state shifts [2404.07972, 2509.26539]. In AndroidWorld, although structural regularity aids agent performance, stochasticity in app data, emulator state, or parameterization can induce variance of up to 20+ percentage points for certain seeds [2405.14573].

Progressive verification and reflective reasoning chains are effective: VAGEN’s multi-stage static, visual, and proactive validation schema allows rapid dismissal of trivial errors and direct investigation of hidden states [2602.00575]. Behavior narrative construction in bBoN similarly guides model selection among divergent rollouts, allowing state-of-the-art sampling efficiency [2510.02250]. 

Test-time scaling further reveals that most agents continue to gain accuracy, with diminishing returns, as step or rollout budget increases. Desktop tasks with longer horizons and greater state branching present greater challenges than mobile-native tasks [2510.19949].

## 6. Benchmark Impact and Implications

OSWorld-Verified and AndroidWorld have catalyzed a shift toward realistic, multi-operating-system generalist agents and highlighted the necessity of robust reward modeling and verifiable evaluation. OSWorld-Verified’s suite of system-level, execution-checked tasks has become a canonical benchmark for desktop agent architecture and RL method development [2404.07972, 2510.02250]. AndroidWorld fills the prior gap in mobile-agent evaluation by enforcing dynamic, system-level validation across real apps, supporting research in mobile-specific planning, UI grounding, and parameter robustness [2405.14573].

The observation that “GUI tasks are easy to verify but hard to solve” remains central. Verifier sophistication—particularly agentic, interactive approaches—translates directly to downstream policy improvements via RL with verifiable rewards (RLVR) and test-time selection [2602.00575, 2512.15431].

A plausible implication is that further unification of desktop, mobile, and web benchmarks under a dynamic, RL-compatible reward and scaling framework will become the standard for evaluating next-generation foundation agents.

## 7. Open Challenges and Future Directions

Despite rapid progress, desktop environments in OSWorld-Verified remain more brittle for agents due to high GUI heterogeneity, multi-window complexity, and long-horizon tasks [2404.07972, 2510.19949]. Programmatic evaluation itself can yield “false negatives” when agents complete tasks via alternative valid strategies not covered by script logic, motivating ongoing work in more flexible, semantic verification [2602.00575].

AndroidWorld’s infinite parameterization exposes agents to rare and adversarial task configurations; mean-over-seeds and pass@k are now viewed as more robust reporting practices than single-seed results [2405.14573].

A plausible implication is increased research attention to self-evolving, trajectory-level reward pipelines, agentic interactive verifiers, and unified sandboxed environments capable of spanning desktop, mobile, and web domains at scale [2512.15431, 2508.15144]. Expansion to handle non-verifiable, truly open-ended tasks remains an open research challenge.

---

**References:**  
[2404.07972], [2405.14573], [2510.02250], [2510.19949], [2508.15144], [2509.26539], [2512.15431], [2509.02544], [2602.00575]

Source: https://www.emergentmind.com/topics/osworld-verified-and-androidworld-benchmarks