OS-MAP: Computer Automation Benchmark

• OS-MAP is a benchmark for computer-use automation that defines task complexity through a two-axis matrix of autonomy levels and generalization scopes.
• It organizes 416 realistic tasks across 15 Ubuntu applications, enabling granular evaluation of execution, planning, adaptation, and orchestration in agents.
• The benchmark highlights significant limitations in current models, revealing failures in instruction following, multi-app coordination, and recovery from unexpected changes.

OS-MAP is a benchmark for computer-using agents designed to measure progress along two orthogonal dimensions of practical computer automation: the depth of autonomy required to complete a task and the breadth of generalization across real user demands. It organizes 416 realistic daily computer-use tasks across 15 applications in a dynamic, executable virtual-machine environment, and evaluates agents through a structured performance-generalization matrix rather than a single undifferentiated success score. Its central premise is that prior benchmarks flatten heterogeneous task requirements and therefore obscure whether an agent is improving in execution, planning, adaptation, or orchestration, and whether those gains transfer beyond narrow scenarios. In this sense, OS-MAP reframes benchmark design around autonomy structure and demand coverage rather than app lists alone (Chen et al., 25 Jul 2025).

1. Motivation and benchmark rationale

OS-MAP was introduced to address two limitations attributed to earlier computer-use benchmarks. First, existing evaluations often ignore internal task heterogeneity. A computer-use task may be dominated by grounding and execution, by deterministic planning, by adaptation to dynamic conditions, or by long-horizon coordination across applications. When these are aggregated into a flat task pool, benchmark scores become difficult to interpret diagnostically. Second, prior benchmarks are described as insufficiently aligned with actual user demand, because they are often organized around applications or templates rather than a hierarchy of real-world needs (Chen et al., 25 Jul 2025).

The benchmark’s stated goal is to bridge research evaluation and practical deployment. It therefore emphasizes realistic daily-use scenarios, explicit task structure, and analysis of where present-day agents fail. A recurring implication in the paper is that benchmark design should support targeted capability development rather than merely ranking models. This suggests a shift from aggregate end-to-end success toward decomposition of capability regimes such as execution, planning, adaptability, and orchestration.

OS-MAP is positioned against benchmarks including OSWorld, WindowsAgentArena, AndroidWorld, WebArena, WorkArena, GAIA, and Mind2Web. Its distinguishing feature is not only that it uses an executable environment, but that it combines such an environment with a two-dimensional evaluation structure grounded in user-demand coverage (Chen et al., 25 Jul 2025).

2. Dataset composition and execution environment

OS-MAP is built on a dynamic, executable virtual-machine environment inherited from and extended beyond OSWorld. The benchmark contains 416 tasks over 15 Ubuntu applications and uses state-based success verification. Observations are raw screenshots only, and the action interface is based on atomic mouse and keyboard operations plus a dedicated CALL_USER action (Chen et al., 25 Jul 2025).

The task inventory is reported as including 283 single-app tasks and 154 multi-app tasks. The paper also notes that approximately 37.7% of tasks are multi-app, while elsewhere it reports 35.7%, which is explicitly identified as a reporting inconsistency in the supplied description. Task sources are mixed: 161 tasks were authored directly, while 255 were adapted and labeled from OSWorld. The appendix is described as stating that 255 OSWorld tasks were retained after filtering and re-annotation. Benchmark statistics report an average instruction length of 34.3 words and an average of 11.4 steps per task (Chen et al., 25 Jul 2025).

The environment is configured for reproducibility through snapshots and host-side control. Task specification is JSON-based and includes initialization, instruction, and evaluation protocol. Initialization uses reusable scripts for simple setup and manual snapshot creation for complex setup. Evaluation combines state comparison, file comparison, system-state inspection, and execution-based verification when needed. The observation space excludes accessibility trees and Set-of-Marks annotations; the rationale given is that screenshots are closer to human perception, more general, and avoid the verbosity or inaccuracy of structured UI representations. The default screen resolution is 1920 × 1080 (Chen et al., 25 Jul 2025).

The action space extends OSWorld’s Computer_13 interface with CALL_USER, yielding 17 actions in total: MOVE_TO, CLICK, MOUSE_DOWN, MOUSE_UP, RIGHT_CLICK, DOUBLE_CLICK, DRAG_TO, SCROLL, TYPING, PRESS, KEY_DOWN, KEY_UP, HOTKEY, WAIT, [FAIL](https://www.emergentmind.com/topics/flow-matching-adversarial-imitation-learning-fail), DONE, and CALL_USER. The CALL_USER action is intended for sensitive steps such as login credentials, making human-agent collaboration a first-class part of the benchmark design rather than an out-of-band exception (Chen et al., 25 Jul 2025).

3. Two-dimensional evaluation structure

The defining methodological contribution of OS-MAP is its organization of tasks along two axes: automation level and generalization scope. The first axis measures how much autonomy the agent must exercise; the second measures how widely that capability transfers across the demand hierarchy (Chen et al., 25 Jul 2025).

The automation axis is a five-level taxonomy, L1 through L5, inspired by SAE driving automation and defined in terms of the human-agent division of labor:

• L1: Reactive executor: the agent performs user-specified atomic actions, with the user responsible for planning.
• L2: Deterministic planner: the user provides the goal, and the agent plans and executes under ideal conditions.
• L3: Adaptive agent: the user sets the goal, and the agent independently handles disturbances in dynamic, noisy, partially observable environments.
• L4: Global conductor: the agent decomposes and orchestrates complex end-to-end workflows, including cross-application context switching, dependency tracking, tool use, and long-horizon coordination.
• L5: Proactive companion: the agent proactively monitors context, anticipates needs, and initiates actions.

The paper explicitly states that OS-MAP does not yet include L5 tasks. Thus the taxonomy is partly evaluative and partly roadmap-like: it defines a capability frontier beyond the current benchmark coverage.

The second axis is derived from a real-world user demand hierarchy comprising domains, scenarios, and representative tasks with applications. This hierarchy was constructed using the State of Mobile 2025 report and the OECD ICT Access and Usage Database, then adapted for desktop use by removing mobile-specific categories such as payments and adding desktop-relevant categories such as office work. The hierarchy contains 6 top-level domains, 18 sub-needs, and 45 concrete scenarios. The six domains are Work, Study, Life services, Entertainment, Creative production, and System management (Chen et al., 25 Jul 2025).

From this hierarchy OS-MAP defines three generalization scopes:

• S1: Narrow: competence within a single scenario.
• S2: Domain-level: competence across multiple scenarios within one domain.
• S3: General: competence across most or all six domains.

The resulting evaluation matrix is intended to expose whether an agent is merely narrow but competent, broadly capable only on simple tasks, or moving toward the “upper-right” region of high autonomy and high generality. The paper identifies that upper-right region as high-value but currently unachieved (Chen et al., 25 Jul 2025).

4. Formalization, task lifecycle, and curation

OS-MAP formalizes computer-use automation as a partially observable Markov decision process,

$$(\mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}),$$

with state $s_t \in \mathcal{S}$, observation $o_t \in \mathcal{O}$, action $a_t \in \mathcal{A}$, transition

$s_{t+1} = \mathcal{T}(s_t, a_t),$

and binary final reward

$r \in \mathcal{R} = \{0,1\}.$

The benchmark uses final-outcome reward only, with success mapped to 1 and failure to 0; it does not provide intermediate process rewards (Chen et al., 25 Jul 2025).

Each task proceeds through five stages: initialization, task execution, optional post-execution configuration, state extraction, and evaluation. Episodes terminate on DONE, FAIL, or a maximum step limit. The reported step caps are 15 for L1, 15 for L2, 30 for L3, and 50 for L4. In the baseline experiments, hyperparameters are reported as temperature = 1.0, top_p = 0.9, and max generation tokens = 1500 (Chen et al., 25 Jul 2025).

Task creation is described as a six-step manual pipeline: task selection, exploration and specification, instruction and configuration, reference state preparation, evaluation setup, and cross-validation. Nine computer science students contributed approximately 600 hours of work. OSWorld-derived tasks were filtered to remove redundant tasks within the same scenario, tasks with low general relevance, and tasks judged infeasible or ambiguous. This emphasis on filtering is important to the benchmark’s stated purpose: OS-MAP prefers representative and transferable tasks over sheer task count (Chen et al., 25 Jul 2025).

Representative tasks are used to illustrate the taxonomy. Examples include an L4 “Rotate Wallpapers” task involving downloading wallpapers, naming files, adding them as candidates, setting today’s wallpaper, and configuring a cron task for daily switching; an L4 “Zotero Citation” task involving article search, citation formatting, paper download, and integration with existing citations; and an L3 “Meet Schedule” task involving reading a Rocket.Chat message and adding meeting details to Calendar. These examples are meant to demonstrate that OS-MAP spans single-action execution, explicit planning, adaptation, and multi-app orchestration (Chen et al., 25 Jul 2025).

5. Baselines and empirical findings

OS-MAP evaluates three baseline categories. The first consists of general-purpose VLMs used directly for end-to-end control: GPT-4o, Claude-3.7-Sonnet, Gemini-2.5-Pro, Qwen2.5-VL-72B, and InternVL3-8B. The second is a GUI-specific baseline, UI-TARS-72B. The third consists of planning-grounding systems using GPT-4o for planning and GUI grounding models for execution, including Aguvis-7B, OS-ATLAS-Base-7B, UGround-7B, and GUI-Actor-7B (Chen et al., 25 Jul 2025).

The headline empirical result is that overall performance remains low. The strongest reported overall results are approximately 11.4–11.5% task success: 11.4% for UI-TARS-72B and 11.5% for GUI-Actor-7B. Human performance is reported as 71.9% overall. The gap between human and agent performance is therefore large, even under structured evaluation (Chen et al., 25 Jul 2025).

Performance degrades sharply with increasing automation level. For UI-TARS-72B, the paper reports 48.0% at L1, 14.0% at L2, 1.0% at L3, and 0.0% at L4. The same qualitative pattern is said to hold broadly across models: some grounding and basic planning are achievable, but present systems collapse on adaptation and orchestration. The benchmark therefore makes visible a specific failure profile rather than only low aggregate success (Chen et al., 25 Jul 2025).

The paper further states that UI-TARS-72B provides the best balance of visual grounding, robust planning, and generalization among the evaluated systems. It also notes that open-source GUI-centric models can outperform proprietary general-purpose models in this setting, and that planning-grounding decomposition can improve over pure end-to-end general VLM use. A plausible implication is that GUI-specialized data and modular control pipelines remain competitive, or even advantageous, for desktop computer-use tasks that require precise interaction with visual interfaces.

6. Failure modes, interpretation, and relation to adjacent work

OS-MAP includes a structured failure analysis. Two general failure classes are emphasized: poor instruction following and severe hallucination. Poor instruction following includes wrong output format and invalid actions, and is said to account for around 15% of failures. Severe hallucination includes assuming that actions succeeded when they did not, misidentifying UI elements, and confusing windows or pages (Chen et al., 25 Jul 2025).

The benchmark also reports level-specific bottlenecks. At L1, failures are dominated by weak grounding and a tendency to choose “magical” shortcuts such as command-line invocations or direct URLs instead of interacting with the presented interface. At L2, models can often produce a reasonable plan but are distracted by similar options or ignore task-specific constraints. At L3, they fail to recover from unexpected UI changes and show weak awareness of implicit in-window context. At L4, they exhibit poor task decomposition, dependency misordering, context-switching errors, weak tool use, and aimless clicking. These failure modes reinforce the benchmark’s central claim that execution, planning, adaptation, and orchestration are separable capability layers rather than a single continuum (Chen et al., 25 Jul 2025).

The benchmark’s broader significance lies in making this layered failure structure measurable. For research, it supports capability decomposition and targeted development. For deployment, it supports the paper’s conclusion that current state-of-the-art agents are not ready for general daily computer use, and that practical systems will need stronger robustness, better long-horizon planning, explicit recovery strategies, better multi-app orchestration, and safe human-in-the-loop mechanisms such as CALL_USER (Chen et al., 25 Jul 2025).

Within the broader GUI-agent ecosystem, OS-MAP occupies an evaluative role rather than a training-data-generation role. This distinguishes it from OS-Genesis, which is described as a data synthesis pipeline for training GUI agents on Android and web environments through reverse task synthesis and a trajectory reward model, rather than an agent planning architecture or a desktop benchmark (Sun et al., 2024). Taken together, these works suggest a division of labor in the field: OS-MAP structures evaluation of breadth and depth in desktop computer use, while trajectory-synthesis systems such as OS-Genesis target the production of higher-quality training data for GUI agents in related environments.