---
title: WindowsAgentArena Benchmark
url: https://www.emergentmind.com/topics/windowsagentarena-waa
type: topic
---

# WindowsAgentArena Benchmark

WindowsAgentArena (WAA) is a large-scale, open benchmark designed to rigorously evaluate autonomous computer-use agents operating within the Microsoft Windows operating system. Established as a counterpart to prior Linux-centric benchmarks such as OSWorld, WAA emphasizes multimodal, long-horizon GUI automation across a spectrum of realistic desktop tasks, leveraging raw desktop screenshots as its primary observation modality. WAA is now a principal yardstick for gauging the agentic capabilities of systems that seek to generalize human-like interaction patterns across diverse Windows applications and workflows [2409.08264][2510.02250][2604.21375].

## 1. Design Goals, Scope, and Motivation

WindowsAgentArena was created to address deficiencies in earlier computer-use benchmarks, which either focused on web navigation, text-based interfaces, or non-Windows environments. Its chief motivation is to simulate the breadth and heterogeneity of real-world Windows desktop usage at scale, including the interplay of multiple applications, nested dialogs, and visually intricate workflows such as document editing, code authoring, web browsing, system configuration, and multimedia manipulation [2409.08264][2510.02250].

Key properties:
- **Breadth**: 154 tasks spanning Office suites (LibreOffice Writer/Calc), Web browsers (Chrome, Edge), File Explorer, Settings, VS Code, VLC, and utilities like Notepad, Calculator, Clock, Paint.
- **Multimodality**: Tasks require agents to operate exclusively via visual feedback (RGB screenshots) and GUI interaction primitives. No DOM or accessibility tree is exposed, except in specific agent implementations tested.
- **Realism**: Uses full-featured cloud-hosted Windows VMs with genuine OS and application installations, not emulated or HTML-reconstructed UIs.
- **Reproducibility and Scalability**: Evaluations are parallelizable on public cloud (e.g., Azure, AWS), using containerized VMs with per-task snapshotting for consistent environments [2409.08264].

## 2. Task Structure, State and Action Spaces

WAA formalizes each interaction episode as a partially observable Markov decision process (POMDP) [2504.00906][2510.02250]:
- **State ($s_t$)**: The comprehensive desktop configuration at time $t$, including app windows, filesystem state, and open dialogs.
- **Observation ($o_t$)**: A pixel-level screenshot of the entire desktop.
- **History ($h_t$)**: Sequence $(o_0, a_0, o_1, ..., o_t)$, accessible to the agent.
- **Action Space**: Composed of atomic GUI actions:
  - Mouse primitives: `click(x,y)`, `double_click(x,y)`, `drag_to`, `scroll`.
  - Keyboard input: `type(text)`, `press_key`, `hotkey`.
  - Composite/utility actions: `call_code_agent()` for direct code execution (in some agent variants), `call_search_agent`.
  - Task flow: `agent.done()`, `agent.fail()`.
- **Constraints**: Step budgets of 15 (original), 30 (V2), 50, or 100 actions per task, with stricter limits for more challenging settings [2504.00906][2505.13909][2510.02250][2604.21375].

Tasks are specified by natural language instructions. The diversity and complexity of tasks range from simple file renames to multi-application workflows and manipulations involving dynamic visual states (e.g., delayed UI feedback, transient dialogs) [2409.08264][2510.02250].

## 3. Evaluation Criteria and Protocol

The primary metric is **Success Rate (SR)**, a normalized binary measure:
\[
\mathrm{SR} = \frac{1}{N}\sum_{i=1}^{N} \mathbb{1}[\text{task}_i \text{ succeeded}]
\]
where a task is marked as succeeded only if the post-episode WM state matches an executable, hand-written evaluator script for that task. The typical protocol involves:
- Each task evaluated from a clean snapshot, ensuring independence and reproducibility [2505.13909][2510.02250].
- Agents must complete tasks within the step budget; exceeding the budget or issuing `agent.fail()` results in failure.
- No agent receives demonstration or task-specific training for WAA; zero-shot transfer is emphasized [2410.08164].

Recent enhancements (WindowsAgentArena-V2) enforce strict task feasibility, snapshot resets per task, and human-in-the-loop verification for nuanced completion criteria, eliminating infeasible or ambiguous objectives [2505.13909].

## 4. Agent Architectures and State-of-the-Art Performance

WindowsAgentArena has catalyzed several lines of agent research. The following table summarizes success rates by leading approaches (max 50/100 steps) [2604.21375][2510.02250][2509.02544][2504.00906][2410.08164][2409.08264]:

| Method              | 15–30 Steps | 50 Steps | 100 Steps |
|---------------------|------------:|---------:|----------:|
| NAVI (baseline)     |     19.5%   |     –    |    –      |
| Agent S             |     18.2%   |     –    |    –      |
| Agent S2            |     29.8%   |     –    |    –      |
| UI-TARS-1.5         |     –       |   42.1%  |    –      |
| UI-TARS-2           |     –       |   50.6%  |    –      |
| Agent S3 (no scal.) |     –       |   49.0%  |   50.2%   |
| bBoN (Agent S3/GPT5)|     –       |   54.1%  |   56.6%   |
| VLAA-GUI            |     –       |   60.4%  |   61.0%   |
| Human               |     –       |   74.5%  |    –      |

Over time, SR has improved from ∼19% (NAVI) to over 60% (VLAA-GUI), with recent systems leveraging:
- Hierarchical modular planning and compositional architectures (Agent S2/S3) [2504.00906][2510.02250].
- Multi-turn RL with value pretraining and enhanced PPO objectives (UI-TARS-2) [2509.02544].
- Rollout scaling and trajectory selection via Behavior Best-of-N (bBoN) using behavior narratives and VLM-based comparative judges [2510.02250].
- Modular “STOP–RECOVER–SEARCH” frameworks, incorporating mandatory visual verifiers, loop-breaking strategies, and on-demand search/coding modules (VLAA-GUI) [2604.21375].

**VLAA-GUI** (Gemini 3 Flash backbone) currently sets the highest published SR at 60.4% (50-step) and 61.0% (100-step), substantially outperforming both previously dominant methods and all earlier baselines [2604.21375].

## 5. Key Methodological Innovations and Failure Modes

- **Best-of-N Rollout with Vision-Language Judging (bBoN)**: Generates $N$ stochastic trajectories per task and, for each, summarizes the action-effect sequence (“behavior narrative”). A VLM judge compares these in MCQ format, directly selecting the most successful trajectory, yielding significant robustness in the face of stochasticity and compounding errors [2510.02250].
- **Hierarchical and Specialist Agents**: Architectures such as Agent S2 decompose high-level tasks into dynamically replanned subgoals, using a mixture of grounding strategies to localize GUI elements precisely, which is essential in visually complex, multi-app or multi-dialog workflows [2504.00906].
- **Reinforcement Learning and Value Shaping**: UI-TARS-2 employs decoupled-GAE, length-adaptive GAE, and PPO with value pretraining, which empirically stabilizes policy learning across sparse rewards and long-horizon tasks, resolving challenges in stale window focus or over-scrolling [2509.02544].
- **Modular Action Control (VLAA-GUI)**: Integrates mandatory Completeness Verifier (for visual evidence at each agent.done), Loop Breaker (switches interaction modes or resets on loops), and on-demand Search Agent to mitigate premature stopping and repetitive failures. Ablations indicate each module contributes 5–11pp SR improvement, with greatest gains in Web/System categories [2604.21375].

Dominant failure modes across agents include: menu mis-grounding, failure to reorient in multi-dialog settings, excessive click loops, and incomplete verification of visually subtle completion cues.

## 6. Comparative Analysis with Other Benchmarks

WAA is distinct from other benchmarks in several respects:
- **OSWorld** is primarily Ubuntu-based, though many of its tasks are re-expressed in WAA with Windows-specific adaptations (PowerShell for shell, Edge/Chrome integration, UIA trees). Unlike OSWorld, WAA covers additional Windows-only scenarios (ribbon menus, proprietary Office workflows) [2409.08264][2504.00906].
- **AndroidWorld** and **Mind2Web** address mobile or browser-level automation, not the desktop-wide agentic setting of WAA [2504.00906].
- WAA uniquely supports direct binary script-based evaluators, enabling reinforcement learning paradigms with executable reward functions, not just dataset-style logs [2409.08264].
- Its cloud-native architecture allows execution and evaluation at a scale and speed (O(10–20 minutes) for ∼150 tasks) impractical with traditional, serial benchmarks [2409.08264].

## 7. Significance, Current Limitations, and Future Directions

WAA is the current standard for measuring generalist Windows computer-use agent robustness and adaptability. It exposes key challenges—cross-OS generalization, precise GUI grounding, and robust multi-application planning—that drive architectural innovation in the field. The leading systems (VLAA-GUI, bBoN, Agent S3) have narrowed the human-agent performance gap to within 13–14pp (61% agent vs 74–75% human at 100 steps) [2510.02250][2604.21375]. A plausible implication is that modular control and trajectory-level selection will be core to further advances.

Current limitations include:
- Persistently low SR in certain applications (notably Office/LibreOffice), due to intricate UI and interaction workflows.
- Remaining ungrounded failures in visually ambiguous or non-standard interface elements.
- The need for richer feedback beyond binary success/failure and improved benchmarking in settings where tools (e.g., direct search, coding agents) are essential.

Continued evolution (e.g., WAA-V2) is directed toward tighter task curation, fairer evaluation (snapshot resets, infeasible task removal), and greater alignment with robust autonomous desktop automation paradigms [2505.13909].

---
**References**

- [2409.08264] Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
- [2410.08164] Agent S: An Open Agentic Framework that Uses Computers Like a Human
- [2504.00906] Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
- [2505.13909] Efficient Agent Training for Computer Use
- [2509.02544] UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning
- [2510.02250] The Unreasonable Effectiveness of Scaling Agents for Computer Use
- [2604.21375] VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation

Source: https://www.emergentmind.com/topics/windowsagentarena-waa