OSWorld-Human: A Human Efficiency Benchmark

• OSWorld-Human is a benchmark extension that provides manually curated, minimal human step trajectories for each of 369 computer tasks.
• It evaluates computer-use agents by comparing task execution steps to a human baseline using the Weighted Efficiency Score.
• Empirical findings show that even top-performing agents use 1.4–2.7 times more steps than humans, underscoring a gap in practical efficiency.

OSWorld-Human is a manually curated extension of the OSWorld benchmark that provides, for each of the original 369 tasks, a human-determined “gold” trajectory of the minimal steps required to complete the task. It was introduced to measure the temporal efficiency of computer-use agents, not merely their task success, by comparing agent trajectories against a human baseline for step count and, by implication, latency. In this formulation, OSWorld-Human reframes evaluation from a purely outcome-centric benchmark into one that also quantifies procedural economy on real desktop tasks (Abhyankar et al., 19 Jun 2025).

1. Origin and relationship to OSWorld

OSWorld-Human is defined relative to OSWorld, the underlying benchmark of 369 computer tasks involving real web and desktop applications in open domains, OS file I/O, and workflows spanning multiple applications. OSWorld executes agents in full virtualized computer environments and evaluates final states with custom execution-based scripts rather than next-action matching. In the original OSWorld study, human performance was measured through completion time and success rates, but the paper did not package a separate “OSWorld-Human Dataset”; instead, it reported human logs under controlled conditions in the same environment (Xie et al., 2024).

The distinction is important. OSWorld-Human does not replace OSWorld’s execution-based evaluation; it adds a human-derived efficiency reference for each of the original tasks. The result is a benchmark layer that preserves OSWorld’s task inventory while introducing a normative notion of how many steps a task should require when performed minimally by a human. This makes OSWorld-Human specifically a benchmark for efficiency of computer-use agents rather than a new task suite.

2. Motivation: latency and temporal performance

The principal motivation for OSWorld-Human is the discrepancy between benchmark accuracy and practical usability. The associated study argues that state-of-the-art systems have focused on improving accuracy on leading benchmarks while remaining “practically unusable” because end-to-end latency can reach tens of minutes for tasks that humans typically complete in a few minutes. To examine this gap, the authors conducted a latency study on 37 representative OSWorld tasks using Agent S2 with GPT-4.1 for planning and reflection, UI-TARS-7B-DPO for grounding, and a standard Ubuntu VM (Abhyankar et al., 19 Jun 2025).

That study decomposed task latency into screenshot capture, action execution, planning, reflection, retrieval, and grounding. Planning and reflection together accounted for 75%–94% of total latency across applications, while retrieval contributed 1%–9%, grounding 3%–6%, and screenshot plus action execution together less than 3%. The study also found that per-step LLM latency and prompt length grow roughly linearly with step index: steps 40–50 incur approximately 3 times higher latency than steps 1–10 because each prompt contains the full history of prior steps. These findings motivate two optimization targets: reducing the number of LLM calls and reducing the number of steps per task.

A plausible implication is that a benchmark based only on success rate obscures a major systems bottleneck. Under this view, temporal inefficiency is not a secondary implementation issue but a first-order property of current computer-use agents.

3. Construction and annotation protocol

OSWorld-Human maps every OSWorld task to a minimal human trajectory. For most tasks, the original OSWorld “source,” such as forum posts or provided scripts, already described a sequence of interactions; these descriptions were translated into the OSWorld action space but without fixed screen coordinates. For tasks lacking explicit sources, two graduate-student annotators independently performed and logged the minimal steps required, and disagreements were adjudicated by replaying the trajectory in the OSWorld VM (Abhyankar et al., 19 Jun 2025).

The final dataset contains 369 examples, matching OSWorld exactly, and provides two trajectories for each example.

Trajectory type	Definition	Role
Single-action trajectory	Each atomic interaction, such as a mouse click or keypress, is one step	Fine-grained human baseline
Grouped-action trajectory	Consecutive interactions that do not require an intermediate observation or decision are bundled as one step	Coarser efficiency baseline

The grouped-action representation is intended to collapse mechanically contiguous operations into a single decision unit. The example given in the paper is “click text field → type text → press Enter” as one grouped step. This suggests that OSWorld-Human distinguishes between motor-level effort and decision-level effort, allowing evaluation either at the granularity of raw UI actions or at the granularity of semantically coherent action bundles.

The dataset also includes application-level step statistics. Table 3 in the paper reports, for example, that LibreOffice Calc tasks require 13.6 single steps on average but only 5.9 grouped steps.

4. Evaluation formalization and metrics

OSWorld-Human introduces the Weighted Efficiency Score (WES) as a combined accuracy-and-efficiency metric over the benchmark tasks. The metric is defined using four task-level quantities: the binary success indicator $r_t \in \{0,1\}$, the expected human number of steps $t_{\mathrm{exp}}$, the actual agent number of steps $t_{\mathrm{actual}}$, and the agent’s maximum allowed step budget $S$. The paper decomposes WES into a positive component, $\text{WES}^+$, rewarding efficient successes, and a negative component, $\text{WES}^-$, penalizing slow failures; it further states that $\text{WES}^+ \in [0,1]$ and $\text{WES}^- \in [-1,0]$ (Abhyankar et al., 19 Jun 2025).

The evaluation protocol re-runs or reuses trajectories for 16 agents on all 369 OSWorld tasks under the same VM setup. Each agent retains its own maximum step budget, such as 50 or 100 steps. Agents are grouped by observation modality: raw screenshots (SS), SS plus accessibility tree (A11y), SS plus A11y plus Set-of-Marks (SoM), or SS plus SoM. For each trajectory, the OSWorld-Human grouping rules are applied so that both single-action and grouped-action efficiency can be measured.

Four reported metrics structure the benchmark:

• Original (%): the success rate on OSWorld.
• Single WES$^+$ (%): $\text{WES}^+$ computed against single-action trajectories.
• Grouped WES$t_{\mathrm{exp}}$0 (%): $t_{\mathrm{exp}}$1 computed against grouped-action trajectories.
• WES$t_{\mathrm{exp}}$2: the shared failure-side penalty metric.

This metric design makes a direct conceptual separation between solving tasks and solving them with near-human procedural efficiency. An agent can therefore score highly on OSWorld success while remaining substantially inefficient relative to the human baseline.

5. Empirical findings

The central empirical result is that strong OSWorld accuracy does not translate into human-level efficiency. The top agent by original success rate, UI-TARS-1.5 at 100 steps with 42.5% success, achieves only 23.7% single-WES$t_{\mathrm{exp}}$3 and 14.3% grouped-WES$t_{\mathrm{exp}}$4. Agent S2 with Gemini 2.5, evaluated at 50 steps and 41.4% success, attains 28.2% single-WES$t_{\mathrm{exp}}$5 and 17.4% grouped-WES$t_{\mathrm{exp}}$6. The study interprets this as implying that the latter uses, on average, 1.5 times more steps than a human under the single-action reference and 2.4 times more steps under the grouped-action reference. Across all 16 agents, even the best systems require 1.4–2.7 times more steps than necessary (Abhyankar et al., 19 Jun 2025).

These results operationalize a gap that had been visible but weakly quantified in earlier OSWorld measurements. In the original OSWorld benchmark, humans completed 72.36% of tasks overall, while the best model achieved only 12.24%, revealing a broad capability deficit in GUI grounding and operational knowledge. OSWorld-Human adds a different diagnosis: even when agents are comparatively successful, they remain temporally inefficient in ways that materially affect usability (Xie et al., 2024).

A plausible implication is that efficiency should be treated as an orthogonal optimization axis rather than as a by-product of accuracy gains. The latency study reinforces this interpretation by showing that long trajectories amplify prompt length and LLM latency step by step.

6. Scope, release, and distinctions from adjacent artifacts

The planned release for OSWorld-Human includes the 369 tasks with single- and grouped-action gold trajectories in JSON format aligned to the original OSWorld schema, together with a Python evaluation script for computing $t_{\mathrm{exp}}$7 and $t_{\mathrm{exp}}$8. The dataset is slated for release under CC BY 4.0, with the evaluation code under MIT, at the repository https://github.com/OSWorld/OSWorld-Human (Abhyankar et al., 19 Jun 2025).

OSWorld-Human should be distinguished from several nearby but different forms of “human” involvement in the OSWorld ecosystem. The original OSWorld paper reports human completion times and success rates, along with quality-control procedures, but explicitly does not package a separate human demonstration corpus under the name OSWorld-Human (Xie et al., 2024). OSWorld 2.0 likewise uses human-authored workflows, human timing, and human QA, but its authors state that there is no separately named “OSWorld-Human” artifact in that benchmark release (Yuan et al., 28 Jun 2026). A further source of confusion is ChainWorld: the provided document associated with that title contains no substantive ChainWorld content and does not mention any “OSWorld-Human Dataset,” beyond citing OSWorld in the bibliography (Siu et al., 19 Jun 2026).

Within this landscape, OSWorld-Human is best understood as a narrowly defined benchmark extension: it is not a replacement for OSWorld, not a long-horizon workload suite, and not a general human demonstration corpus for desktop grounding. Its specific contribution is to provide minimal human step trajectories for the original 369 OSWorld tasks and a metric framework that makes efficiency measurable, comparable, and auditable at benchmark scale.