WindowsWorld Benchmark for Autonomous GUI Agents

• WindowsWorld is a process-centric benchmark for assessing autonomous GUI agents executing complex, multi-application workflows with occupational grounding across Windows desktops.
• It employs a human-in-the-loop multi-agent framework with generation, refinement, human review, and environment synthesis to create realistic, executable tasks.
• Empirical results highlight significant performance gaps in cross-application coordination, conditional reasoning, and failure recognition in current autonomous agents.

Searching arXiv for papers on WindowsWorld and closely related Windows GUI agent benchmarks to ground the article. WindowsWorld is a process-centric benchmark for autonomous GUI agents in realistic Windows desktop environments, designed to evaluate whether such agents can execute complex, profession-specific workflows that span multiple applications rather than merely solve isolated single-screen tasks (Li et al., 30 Apr 2026). Its defining premise is that real computer use is structured as a process: information must be moved across applications, intermediate results must be inspected, conditional decisions must be made during execution, and some tasks must be recognized as infeasible. The benchmark therefore combines occupational grounding, cross-application workflow composition, and intermediate checkpointing, yielding 181 tasks across 16 personas, 17 desktop applications, four difficulty levels, and a task distribution in which 77.9% of tasks are multi-application (Li et al., 30 Apr 2026).

1. Concept and motivation

WindowsWorld was introduced to address three shortcomings in prior GUI-agent evaluation. First, existing benchmarks were dominated by isolated or single-application tasks. Second, they typically reduced long workflows to all-or-nothing final success. Third, benchmark construction pipelines often did not scale well while preserving realistic files, dependencies, and environment artifacts (Li et al., 30 Apr 2026). WindowsWorld responds by making professional workflow execution, rather than single-app manipulation, the primary object of study.

The benchmark’s key abstraction is that desktop autonomy should be assessed as progress through semantically necessary sub-goals. In this framing, a workflow such as extracting information from one application, transforming it in another, and communicating it in a third is qualitatively different from a short atomic task, even when the raw horizon length is similar. WindowsWorld therefore treats process completion, cross-application coordination, and failure recognition as first-class evaluation targets (Li et al., 30 Apr 2026).

This design is explicitly occupational. Tasks are generated from 16 personas across 5 categories, including Administrative/Support, Business/Management, Creative/Content, Technical/IT, and General User, with examples such as Accountant and Software Engineer (Li et al., 30 Apr 2026). That occupational conditioning is not decorative: it determines which applications co-occur, what kinds of artifacts are created, and which intermediate states count as meaningful progress.

2. Benchmark construction and generation pipeline

WindowsWorld is built with a human-in-the-loop multi-agent framework organized into four stages: Generator, Refiner, Human Reviewer, and Environment Generator (Li et al., 30 Apr 2026). The Generator uses an LLM, implemented with DeepSeek-V3.2, to produce tasks conditioned on persona and difficulty level. The generation prompt constrains app usage, workflow structure, and evaluation specification, and requires a structured output containing the instruction, persona, involved applications, preconditions, environment setup, success criterion, and intermediate checks (Li et al., 30 Apr 2026).

The Refiner stage consists of four nodes. A Semantic Deduplicator computes pairwise cosine similarity between instruction embeddings and prunes near-duplicates when the similarity exceeds the threshold $\tau = 0.85$. A Validity Auditor verifies URL accessibility with asynchronous HTTP requests and checks file mentions against environment setup. A Dependency Reasoner rewrites procedural preconditions into declarative state conditions. A Metric Refiner standardizes evaluation criteria and intermediate checkpoints into path-agnostic assertions and expected final states (Li et al., 30 Apr 2026).

Human review remains central. Four annotators reject tasks that are ambiguous or under-specified, judged by subjective rather than objective criteria, or dependent on unavailable proprietary software or inaccessible external services (Li et al., 30 Apr 2026). This stage is what turns the benchmark from purely synthetic generation into curated executable evaluation.

The Environment Generator then synthesizes the task artifacts needed to instantiate workflows, including .xlsx, .docx, .py, and .json files (Li et al., 30 Apr 2026). It also uses a Smart File Merging strategy so that tasks within the same persona can share coherent resources instead of relying on unrelated one-off files. This suggests that WindowsWorld’s realism comes not only from natural-language instructions but also from the consistency of the supporting desktop state.

3. Task structure, difficulty levels, and application coverage

WindowsWorld contains 181 tasks, with an average of 5.0 sub-goals in the abstract and 4.97 intermediate checkpoints per task in the detailed description (Li et al., 30 Apr 2026). It uses four difficulty levels. L1 consists of single-app atomic tasks: complex operations inside one application. L2 consists of multi-app linear workflows spanning multiple applications. L3 consists of dynamic-reasoning tasks that require conditional logic, reasoning, or decisions during execution. L4 consists of infeasible tasks made impossible by dead URLs, missing files, authentication barriers, or similar constraints, and is intended to test abstention and failure recognition rather than completion (Li et al., 30 Apr 2026).

The benchmark spans 17 common desktop applications. The reported counts are Word (40), Excel (73), PowerPoint (18), Acrobat (11), Thunderbird (72), Chrome (70), File Explorer (59), Calculator (7), Task Manager (2), Snipping Tool (1), GIMP (14), Paint (11), Photos (7), VLC (1), VS Code (30), PowerShell (18), and Windows Terminal (9) (Li et al., 30 Apr 2026). This mix gives the benchmark a strongly office- and workflow-centered profile, with recurring pairings such as Excel–Thunderbird and Word–Thunderbird that model routine analysis-to-communication pipelines.

The intermediate checks are semantic rather than action-specific. The benchmark removes constraints such as “click button X” and instead evaluates states such as “the target file is open” or “the correct attachment has been added to the email” (Li et al., 30 Apr 2026). This makes the benchmark path-robust: keyboard shortcuts, menu navigation, and alternative valid procedures can all satisfy the same sub-goal. Infeasible tasks are treated differently. For L4, success means correctly identifying infeasibility rather than hallucinating completion (Li et al., 30 Apr 2026).

4. Environment model, observations, actions, and scoring

WindowsWorld runs in controlled Windows virtual machines and models GUI interaction as a partially observable Markov decision process, $$(\mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R})$$, with trajectory $\tau = (o_1, a_1, \dots, o_T, a_T)$ (Li et al., 30 Apr 2026). Agents interact only through standard GUI actions rather than privileged application hooks, which aligns the benchmark with the operational constraints of real desktop use.

Three observation modalities are evaluated: Raw Screenshot, Set-of-Marks (SoM), and Screenshot + Accessibility Tree, also called the Hybrid setting (Li et al., 30 Apr 2026). Two action spaces are supported: free-form PyAutoGUI and computer_13 from OSWorld. Most experiments use PyAutoGUI, while computer_13 includes actions such as MOVE_TO(), CLICK(), RIGHT_CLICK(), DOUBLE_CLICK(), DRAG_TO(), SCROLL(), TYPE(), PRESS(), HOTKEY(), WAIT(), [FAIL](https://www.emergentmind.com/topics/flow-matching-adversarial-imitation-learning-fail)(), and DONE() (Li et al., 30 Apr 2026).

Evaluation is split into an Intermediate Check Score and a Final Check Score. The first measures the mean fraction of intermediate checkpoints satisfied; the second is a binary final success judgment (Li et al., 30 Apr 2026). The automated judge is implemented with Qwen3-VL-Plus. On 100 stratified tasks covering 518 checkpoints, agreement with two human annotators yielded Pearson correlation of 0.9108 for the intermediate score and 0.8316 for the final score, with Cohen’s $\kappa$ of 0.8668 at checkpoint level and 0.8271 for final judgments (Li et al., 30 Apr 2026). The scoring design therefore serves two purposes: it preserves end-state evaluation while making long-horizon failure modes visible.

Step budgets are level-specific: 15 for L1, 25 for L2, 40 for L3, and 20 for L4 (Li et al., 30 Apr 2026). These limits matter because one of the benchmark’s central findings is that contemporary agents often consume many more steps than a human would need while still failing to complete the workflow.

5. Empirical results and characteristic failure modes

The main empirical result is that contemporary computer-use agents perform poorly on realistic multi-application workflows. The best reported result is 20.44% final success, achieved by Gemini-3-flash-preview under the Screenshot + Accessibility Tree setting (Li et al., 30 Apr 2026). Under that same setting, the model attains 50.32% on the Intermediate Check Score, producing a large progress–completion gap. This gap is one of WindowsWorld’s most important measurements: agents often satisfy some early sub-goals but fail to integrate them into a correct terminal state (Li et al., 30 Apr 2026).

Performance degrades sharply with task difficulty. For Gemini-3-flash-preview in the Hybrid setting, L1 final success is 35.90%, L2 is 17.50%, L3 is 14.00%, and L4 is 16.67% (Li et al., 30 Apr 2026). The drop from L1 to L3 indicates that cross-application state maintenance and conditional reasoning are major bottlenecks. L4 further shows that failure recognition is weak: many systems hallucinate success instead of correctly reporting infeasibility (Li et al., 30 Apr 2026).

WindowsWorld also isolates multi-application difficulty from mere horizon length. On a step-matched comparison, L1 tasks have an average minimum expert action count of 10.92 and L2 tasks 11.26, yet the Intermediate Check Score drops from 65.74% to 35.14% and the Final Check Score from 46.15% to 14.29% (Li et al., 30 Apr 2026). This indicates that cross-application coordination is not reducible to task length alone.

Checkpoint-wise analysis shows that many failures occur early. For example, under screenshot-only input, Gemini-3-flash first fails at the first checkpoint in 23.1% of feasible multi-app tasks, at the second in 26.2%, and at the third in 21.5% (Li et al., 30 Apr 2026). This suggests that agents often stall near the beginning of a workflow rather than failing only on the final integration step.

The benchmark also documents characteristic desktop failure modes, including inability to open Chinese-named files reliably, input method editor conflicts when typing Chinese through PyAutoGUI, unstable application switching, incorrect clipboard transfer across applications, and pasting back into the source application instead of the intended target (Li et al., 30 Apr 2026). These are not incidental artifacts. They expose the fact that realistic GUI autonomy depends on focus verification, state tracking, and error recovery, not only on perception.

6. Relation to adjacent benchmarks, significance, and limitations

WindowsWorld is closely related to OSWorld but differs in both scope and evaluative emphasis. OSWorld established a scalable real-computer environment across Ubuntu, Windows, and macOS and reported a benchmark of 369 tasks, with 27.4% multi-app workflow tasks and a best model result of 12.24% overall success (Xie et al., 2024). WindowsWorld narrows the platform focus to controlled Windows VMs and reorients evaluation around process-centric professional workflows, raising the multi-app share to 77.9% and adding explicit intermediate inspection (Li et al., 30 Apr 2026). This suggests that WindowsWorld should be understood not as a replacement for OSWorld, but as a specialization toward the workflow regime that OSWorld under-measured.

WorldGUI addresses a different axis of realism. It is a Windows-centric benchmark with 107 meta tasks and 208 augmented tasks, built to test robustness to diverse initial states across “Win. + Web” applications (Zhao et al., 12 Feb 2025). Where WorldGUI asks whether plans adapt when the starting desktop state is non-canonical, WindowsWorld asks whether agents can maintain a semantically coherent process across multiple applications, occupations, and infeasible branches (Li et al., 30 Apr 2026). The two benchmarks are therefore complementary rather than redundant.

WinClick isolates the grounding subproblem on Windows screenshots and reports 56.2% click accuracy on WinSpot for its best full fine-tuned model (Hui et al., 27 Jan 2025). This suggests that grounding remains a necessary but insufficient component for WindowsWorld-style agents. WindowsWorld’s best runs fail not only because of pixel-level localization but also because of conditional reasoning, cross-application memory, and execution efficiency (Li et al., 30 Apr 2026).

The benchmark’s significance lies in making these deficits measurable. It introduces state-based intermediate checks, professional persona grounding, infeasible-task evaluation, and a task mix dominated by cross-application workflows. At the same time, the paper is explicit about limitations. Intermediate scoring depends on full-trajectory execution and manually reviewed checkpoints, which constrains scalability. The environment is primarily optimized for a single-language OS interface rather than true multilingual localization. The benchmark does not yet include MCP tool evaluation (Li et al., 30 Apr 2026). A plausible implication is that WindowsWorld functions less as a final benchmark than as a stress test for the next generation of desktop agents: systems that can ground reliably, maintain state across windows, reason conditionally across $\geq 3$ applications, and terminate correctly when a task is impossible.