OSWorld V2: Next-Generation Benchmark

• OSWorld V2 is a prospective successor to OSWorld, envisioned to include broader environments, richer action modalities, and longer and dependent workflows.A key characteristic includes process-level diagnostics, human-normalized efficiency, and explicit robustness and safety evaluations.Available as an evolving research direction rather than a formally specified benchmark, it addresses limitations and extensions identified across the OSWorld research line.
• key_takeaways1
• OSWorld V2 is a still evolving research infrastructure/adaptation, not a formally specified benchmark

OSWorld V2 is an editorial designation for a prospective or evolving successor to OSWorld rather than a formally specified benchmark release documented under that name in the supplied research. The original OSWorld is an executable benchmark and virtualized real-computer environment for multimodal agents performing open-ended tasks across desktop, web, command-line, and cross-application workflows (Xie et al., 2024). Subsequent work has proposed extensions addressing GUI grounding, efficiency, MCP tool invocation, long-horizon composition, and process-based evaluation. Collectively, these studies define a technically coherent agenda for an OSWorld V2: broader environments, richer action modalities, longer and dependent workflows, human-normalized efficiency, process-level diagnostics, and explicit robustness and safety evaluation.

1. Origins and conceptual scope

OSWorld was introduced to address two deficiencies in earlier computer-use benchmarks. Non-executable datasets such as Mind2Web, Android-in-the-Wild, and OmniACT provide demonstrations or action annotations but do not allow agents to execute tasks or learn from environmental feedback. Conversely, executable benchmarks such as MiniWoB++, WebShop, WebArena, InterCode, mobile GUI environments, and AgentBench generally constrain interaction to a particular application domain or interface. OSWorld instead exposes a complete virtual computer supporting arbitrary desktop, web, and command-line applications (Xie et al., 2024).

The original benchmark models computer use as a partially observable interaction problem involving visual perception, GUI grounding, operational knowledge, long-horizon planning, file I/O, application-specific behavior, and recovery from environmental noise. Its formal environment is represented as a POMDP,

$$(\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{T},\mathcal{R}),$$

where $\mathcal{S}$ is the state space, $\mathcal{O}$ the observation space, $\mathcal{A}$ the action space, $\mathcal{T}$ the transition function, and $\mathcal{R}$ the reward function.

The public benchmark contains 369 Ubuntu tasks, 43 Windows tasks for a separate analytical set, and infrastructure-level support for Ubuntu/Linux, Windows, and macOS. Its Ubuntu image is based on Ubuntu 22.04 and includes Chrome, VLC, Thunderbird, VS Code, LibreOffice Calc, Writer and Impress, GIMP, a terminal, file manager, image viewer, PDF viewer, and other operating-system applications. The benchmark includes 268 single-application tasks and 101 multi-application workflows, with 302 distinct initial states and 134 unique evaluation functions.

OSWorld tasks begin from reproducible intermediate states rather than uniformly clean launches. A task configuration can specify the VM snapshot, files to transfer, applications and documents to open, window sizes and layouts, preprocessing commands, post-completion processing, data retrieval, and a task-specific evaluator. Virtual machines provide isolation, resettable state, heterogeneous operating systems and software, headless execution, and parallel evaluation.

The principal observations are full-desktop screenshots, filtered accessibility trees, and terminal output. The main action space consists of executable Python code using pyautogui, supplemented by WAIT, FAIL, and DONE. A structured computer_13 action space is also defined for reinforcement learning. Evaluation is execution-based: the agent changes the environment, and a task-specific evaluator inspects resulting files, application state, browser cookies, accessibility trees, configuration files, exported artifacts, or dynamically retrieved web information.

The original benchmark exposed a substantial capability gap. Human evaluators achieved 72.36% overall accuracy, whereas the best reported model, GPT-4 using a filtered accessibility tree, achieved 12.24%. The dominant failures involved GUI grounding, operational knowledge, environmental noise, state tracking, and long-horizon coordination rather than language interpretation alone (Xie et al., 2024).

2. Benchmark architecture and task semantics

OSWorld’s central architectural contribution is the combination of real operating systems, arbitrary applications, reproducible virtual-machine initialization, intermediate human-like states, cross-application workflows, universal mouse-and-keyboard interaction, and example-specific execution-based evaluation.

A task proceeds through initialization, agent interaction, post-processing, data retrieval, and evaluation. The environment can initialize a VM from a snapshot, transfer or generate files, open applications and documents, resize windows, navigate to a specified location, and establish a partially completed workflow. The task then provides an instruction and observation; the agent emits executable action code; the simulator executes it; and the environment returns a new observation. The loop terminates with DONE, FAIL, or exhaustion of the step limit.

The action space supports ordinary GUI operations including movement, clicking, right-, middle-, double- and triple-clicking, dragging, scrolling, typing, key presses, hotkeys, modifier-key control, and timing operations. Unlike browser-specific action spaces, it permits arbitrary combinations of GUI and CLI interaction. Tasks may require creating, renaming, copying, moving, exporting, or uploading files; editing configuration files; manipulating cookies; preparing unsent email drafts; comparing generated files with reference files; and producing output in a specified location and format.

Thirty tasks in the original benchmark are intentionally infeasible because they involve deprecated features, hallucinated application capabilities, or requirements unsatisfiable in the specified environment. Correctly emitting FAIL can receive positive or full reward depending on the evaluator, although models sometimes overuse FAIL and produce false positives.

The reward function is bounded:

$$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow[0,1].$$

A successful task receives $1$, partial achievement may receive a positive value below $1$, correct prediction of an infeasible task receives a positive value, and other outcomes receive $0$. However, benchmark tables primarily report binary success rates rather than a standardized partial-credit aggregation. The original benchmark does not specify a universal weighting scheme, confidence intervals, or statistical significance tests.

A plausible V2 interpretation is therefore not a replacement of the original execution paradigm but an extension of its task semantics. The core requirement would remain evaluation of resulting computer state while making process, efficiency, robustness, and action modality explicit.

3. Grounding, observation, and action diversity

GUI grounding—the mapping from an instruction and visual observation to a concrete action coordinate or bounding box—is a central bottleneck in OSWorld-style agents. “Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis” introduced OSWorld-G, a grounding benchmark derived from OSWorld rollouts, and the 4-million-example Jedi dataset (Xie et al., 19 May 2025). Neither resource is described as OSWorld V2, but both address limitations directly relevant to a successor benchmark.

OSWorld-G contains 564 annotated desktop screenshot–instruction examples. Its categories overlap and include text matching, element recognition, layout understanding, fine-grained manipulation, and refusal. The benchmark covers 32 UI types and annotates target bounding boxes, fine-grained UI-type tags, refined instructions, and infeasibility labels. Evaluation uses coordinate containment: a prediction is correct when the predicted point lies within the annotated target region.

The benchmark distinguishes capabilities that are often conflated in end-to-end evaluation. Text matching concerns lexical localization. Element recognition requires mapping icons or visual forms to functions. Layout understanding requires reasoning over containers, panels, tabs, notification bars, and interface hierarchy. Fine-grained manipulation concerns sliders, table cells, text positions, dividers, steppers, and other spatially precise controls. Refusal requires recognizing that a target is absent or impossible.

Jedi decomposes grounding data into icons, components, layouts, office interactions, descriptions, executable actions, and refusal examples. Its data-generation pipeline combines source UI code, design files, real screenshots, metadata, LLM-generated descriptions, action details, sampled grounding actions, and quality filtering. The component data includes buttons, tabs, menus, dialogs, drawers, tables, forms, sliders, steppers, calendars, date pickers, color pickers, accordions, lists, grids, and related controls. Office augmentation covers document scrolling, text selection, spreadsheet cells, row and column headers, slide thumbnails, scrollbars, autofill, and cell-edge selection.

Jedi improved OSWorld end-to-end performance in a two-model system consisting of a GPT-4o planner and a Jedi grounder. GPT-4o alone achieved 5.0% OSWorld success, while the combined system achieved 22.4–22.7% at 15 steps and 27.0% at 100 steps. This result supports the interpretation that low-level grounding is an independent systems bottleneck, although it does not show that improved grounding resolves planning, state tracking, or application reasoning.

OSWorld V2 could consequently treat observations as a controlled experimental dimension rather than a fixed interface. Relevant conditions include screenshot-only input, accessibility-tree-only input, multimodal input, Set-of-Mark representations, corrupted or incomplete accessibility trees, stale accessibility data, changed screen resolution, altered window position and size, background clutter, randomized themes, and different desktop interfaces. The original benchmark found that more than 75% of sampled failures involved mouse-click inaccuracies, while layout perturbations reduced GPT-4V Set-of-Mark performance from 50.79% on an easy subset to 15.04% when windows were minimized or made small (Xie et al., 2024).

4. Efficiency and human-normalized evaluation

The original benchmark emphasizes task success, but OSWorld-Human demonstrates that success alone does not capture practical usability. “OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents” constructs human-determined trajectories for all 369 original OSWorld tasks (Abhyankar et al., 19 Jun 2025). The paper reports that planning and reflection account for 75%–94% of task latency in the studied system, and that later interaction steps can take approximately three times longer than early steps because prompt histories accumulate.

OSWorld-Human provides two trajectory granularities. A single-action trajectory lists primitive actions separately. A grouped-action trajectory combines consecutive actions that can be performed from the same observation without requiring a new prediction. Grouped trajectories therefore contain no more steps than single-action trajectories and approximate an interaction protocol in which an agent emits safe action sequences rather than alternating a model call with every primitive operation.

The paper reports that strong agents use approximately 1.4–2.7 times more steps than necessary, depending on the agent and comparison convention. This is an excess-trajectory result rather than a direct wall-clock measurement. Human reference trajectories are not necessarily mathematically minimal: they encode manually validated, practical procedures and permit ordinary application knowledge such as keyboard shortcuts while generally avoiding programmatic shortcuts unavailable to typical users.

The principal proposed metric is Weighted Efficiency Score (WES). For successful tasks, it weights success by the ratio of expected human steps to actual agent steps. For unsuccessful tasks, it applies a penalty proportional to the fraction of the available step budget consumed. In intended form,

$\mathcal{S}$0

and

$\mathcal{S}$1

where $\mathcal{S}$2 indicates task success, $\mathcal{S}$3 is the human expected trajectory length, $\mathcal{S}$4 the agent trajectory length, and $\mathcal{S}$5 the maximum step budget.

Agent S2 with Gemini 2.5 achieved 41.4% original OSWorld success, but 28.2% single-action WES and 17.4% grouped-action WES. These values demonstrate that accurate completion and efficient completion are distinct properties. Accessibility trees also incur computational costs: their generation required approximately 3–26 seconds depending on the application and number of UI elements, while their inclusion added thousands of tokens to prompts.

A V2 benchmark should therefore report task success together with action count, grouped and atomic step count, wall-clock latency, model-call latency, prompt and completion tokens, retrieval and reflection overhead, time-to-failure, and human-normalized trajectory length. Step count should be interpreted cautiously because a single step may contain multiple executable commands and because different actions have different computational costs.

5. Tools, MCP, and hybrid interaction

OSWorld-MCP extends the original OSWorld environment with Model Context Protocol tools and evaluates GUI operation, tool invocation, and decision-making jointly (Jia et al., 28 Oct 2025). The paper does not identify OSWorld-MCP as OSWorld V2, but it provides a model for a successor that permits agents to choose among GUI actions, command-line operations, and structured external tools.

The benchmark analyzes 361 OSWorld tasks after excluding eight Google Drive tasks. MCP tools are applicable to 250 tasks, while 111 are classified as non-tool-beneficial. The final collection contains 158 manually validated tools covering VS Code, Google Chrome, LibreOffice Calc, Writer and Impress, OS utilities, VLC, filesystem operations, and Git. GIMP and Thunderbird do not receive MCP servers because of software-version constraints.

The tool-construction pipeline generates code-based solutions, filters them by executing them against the corresponding task, wraps verified solutions as MCP tools, incorporates tools from existing MCP servers, and subjects the resulting collection to review by at least two experienced reviewers. Retrieval-augmented generation selects application-relevant tools rather than presenting all 158 tools simultaneously.

At each step, an agent can select one of the basic GUI operations or invoke an MCP tool with structured arguments. Hybrid trajectories can interleave tool calls, visual observations, mouse and keyboard actions, and termination. The benchmark uses 15- and 50-step episode limits and reports Task Accuracy, Tool Invocation Rate, and Average Completion Steps.

MCP generally improves performance. At 15 steps, OpenAI o3 improves from 8.3% GUI-only accuracy to 20.4% with MCP, while Claude 4 Sonnet improves from 30.2% to 35.3%. At 50 steps, Claude 4 Sonnet improves from 40.1% to 43.3%, with a 36.3% Tool Invocation Rate. Agent-S2.5 achieves 49.5% MCP accuracy at 50 steps. Tool invocation remains relatively infrequent, and long-horizon tool composition remains difficult.

Tool Invocation Rate is not merely the fraction of tasks containing a tool call. It counts successful tool use on tool-beneficial tasks and successful non-use on non-tool-beneficial tasks. Removing tool retrieval reduces Gemini-2.5-Pro overall accuracy from 20.5% to 15.5% at 15 steps and reduces TIR from 16.8% to 11.6%. Shuffling tool descriptions changes performance, revealing prompt-order sensitivity.

An OSWorld V2 incorporating tools would need to distinguish GUI competence, tool selection, argument construction, tool–GUI arbitration, multi-tool planning, recovery from tool errors, and compliance with process constraints. It should also report whether a tool invocation was appropriate, whether it produced a valid state transition, and whether the resulting shortcut complied with the task instruction. The original OSWorld analysis already noted cases in which agents used ffmpeg to complete a task that explicitly required GIMP, producing the desired artifact while violating the application constraint (Xie et al., 2024).

6. Long-horizon, process-based, and cross-environment evaluation

Two later directions extend OSWorld beyond atomic, outcome-only tasks. ChainWorld composes atomic OSWorld tasks into long-horizon desktop workloads using directional compatibility search while preserving source evaluators (Siu et al., 19 Jun 2026). Its workload contains 347 chains of length two to four and compares single-turn evaluation, in which all tasks are presented together, with multi-turn evaluation, in which tasks are revealed sequentially. Across four current computer-use agents, maximum chain completion is 31%. Single-turn failures concentrate on artifact precision, whereas multi-turn failures more often involve session management, fragmented progress, and later-turn disengagement.

OSWorld-Pro instead decomposes tasks into sequentially dependent subgoals and evaluates process progression (Wang et al., 21 Sep 2026). It contains 305 tasks, 2,814 progressive subgoals, 67,264 human annotation labels, 31 applications, 19 Linux distributions, and 13 desktop interfaces. The task categories are Diversity, Coordination, and Robustness. Trajectories average 55.1 steps, with a maximum of 149, while the evaluation harness permits up to 250 turns.

For each step, OSWorld-Pro annotates the targeted subgoal, feasibility, progression, and completion. This makes it possible to distinguish a wrong keyboard input from an inaccurate click, an irrelevant action, a failure to detect infeasibility, and an error that propagates from an earlier subgoal. The benchmark reports all-feasible-subgoal success, progression likelihood by action type, Successful Effort, Failure Persistence, Infeasible Persistence, steps, output tokens, and estimated token cost.

The benchmark exposes process failures hidden by final-state scoring. One Claude Opus 5 trajectory spent 59 steps editing presentation master-slide fonts and colors while the active subgoal was to insert and position a logo. The actions were valid UI operations but irrelevant to the current milestone. Click progression likelihood was 39.0% for Minimax M3 and 41.9% for Kimi K3, compared with approximately 72.4–92.5% for stronger models. Completed subgoals required an average of 6.2 steps, while attempted but infeasible subgoals consumed 18.8 steps on average.

OSWorld-Pro reports 75.7% for Claude Opus 5, compared with 83.4% on OSWorld, and 28.9% for Minimax M3, compared with 75.2% on OSWorld. These scores should not be treated as a direct capability ranking because the benchmarks differ in task composition, environments, turn limits, evaluators, and scoring. They indicate that dependent subgoals, rare applications, cross-application coordination, unfamiliar Linux distributions, desktop-interface transfer, and recovery impose additional burdens.

A V2 architecture could combine the original final-state evaluator with process annotations and long-horizon workload composition. Its reporting should separate final outcome correctness, subgoal completion, progression likelihood, action efficiency, infeasibility recognition, state recovery, cross-application continuity, and robustness across operating systems and desktop interfaces. Human trajectories could provide atomic and grouped references, while source evaluators could preserve compatibility with existing OSWorld results.

7. Prospective OSWorld V2 design and limitations

No supplied paper defines an official OSWorld V2 task inventory, protocol, environment, scoring system, or release. OSWorld-G, Jedi, OSWorld-Human, OSWorld-MCP, ChainWorld, and OSWorld-Pro are separate contributions built on or related to the original OSWorld environment. They should not be represented as components of an official V2 without independent documentation from the OSWorld maintainers.

Nevertheless, the research supports several design principles for a prospective successor:

1. Retain executable real-computer environments. Tasks should run in isolated, resettable virtual machines with reproducible initial states, arbitrary applications, GUI and CLI interaction, and execution-based state verification.
2. Expand cross-operating-system and cross-interface coverage. The original benchmark is Ubuntu-centric, contains a smaller Windows set, and has no macOS task set. OSWorld-Pro demonstrates the value of Linux-distribution and desktop-interface variation.
3. Separate grounding from end-to-end competence. A V2 release should include diagnostic grounding tasks covering text, icons, layout, precise manipulation, refusal, and accessibility-tree robustness, while retaining long-horizon end-to-end tasks.
4. Support hybrid action modalities. GUI actions, terminal commands, application APIs, MCP tools, and safe combinations should be available where task instructions permit them. Tool-beneficial and non-tool-beneficial tasks should be distinguished, and process compliance should be evaluated.
5. Introduce human-normalized efficiency. Success should be accompanied by atomic and grouped step counts, wall-clock time, model-call latency, token use, tool-call latency, and excess trajectory relative to validated human references.
6. Evaluate dependent workloads. Atomic tasks should be supplemented by multi-application chains and sequentially dependent subgoals. Single-turn and multi-turn protocols should be reported separately because they expose different failure profiles.
7. Expose process-level failure modes. Step-level labels for target subgoal, feasibility, progression, and completion can distinguish grounding errors, keyboard errors, irrelevant actions, infeasibility persistence, and error propagation.
8. Stress-test generalization. Resolution, window position, window size, clutter, theme, accessibility completeness, application versions, desktop interfaces, operating systems, and task wording should be systematically perturbed.
9. Strengthen evaluator validation. Task-specific evaluators should be tested against multiple valid solution paths, harmless implementation differences, alternative action modalities, and unintended side effects. Binary success, partial credit, per-subgoal progress, safety, and process compliance should be reported separately.
10. Measure safety and side effects. Current OSWorld evaluation primarily checks whether the requested result was achieved. A successor should additionally measure unintended deletion, modification of unrelated files, credential exposure, unauthorized network access, excessive permissions, privacy violations, and recovery after unsafe actions.
11. Address contamination and source leakage. The original benchmark draws from documentation, tutorials, forums, public blogs, public benchmarks, and other online sources. A V2 evaluation should include private or periodically refreshed test sets, temporal splits, held-out applications and layouts, paraphrased instructions, and contamination audits.
12. Report uncertainty and reproducibility. Repeated runs, confidence intervals, per-task results, model and environment versions, evaluator reliability, hardware configuration, inference latency, and random seeds are necessary for interpreting small performance differences.

The resulting benchmark would preserve OSWorld’s defining contribution—execution in realistic, stateful computer environments—while broadening the unit of evaluation from final artifact correctness to a multidimensional account of grounding, planning, action selection, tool use, process compliance, efficiency, robustness, safety, and recovery. In this sense, OSWorld V2 is best understood not as a documented release represented by one paper, but as a research direction assembled from the limitations and extensions identified across the OSWorld research line.