OSWorld-Verified Tool Assurance

• OSWorld-Verified is a rigorous methodology defined by automated tool generation, dual-reviewer curation, and empirical validation for robust agent benchmarking.
• It employs a multi-phase pipeline that includes code generation, sandbox testing, and manual curation to ensure tools perform accurately in real-world environments.
• By integrating OSWorld-Verified tools, researchers achieve enhanced agent performance metrics such as improved task success, lower completion steps, and more reliable tool invocation rates.

The term “OSWorld-Verified” refers to a rigorous methodology and concrete label within the OSWorld-MCP ecosystem for assuring the functional, practical, and evaluative soundness of Model Context Protocol (MCP) tools used for benchmarking computer-use agents. A tool is deemed “OSWorld-Verified” only if it passes a three-phase pipeline—automated generation, dual-reviewer human curation, and empirical validation through agent benchmarking—thereby providing a reproducible, standards-driven basis for measuring tool-augmented agent capabilities in real computer environments (Jia et al., 28 Oct 2025).

1. Definition and Rationale

An “OSWorld-Verified” tool is one that has satisfied stringent criteria:

• It can be automatically generated from a natural-language specification.
• It passes execution-based validation against realistic tasks in a sandboxed OSWorld VM.
• Dual expert reviewers confirm its correctness, generality, and utility through hands-on inspection and stress-testing in diverse OSWorld target environments (Linux, Windows, macOS).
• It demonstrably improves state-of-the-art multimodal agent performance, both in raw task success and operational efficiency, when incorporated into OSWorld-MCP benchmarking.

This standard addresses prior deficiencies in agent evaluation, where comparing GUI-only and tool-augmented agents led to unfair or misleading conclusions. Explicit measurement of tool invocation skills is essential for quantifying the decision-making and operational capabilities of advanced computer-use agents.

2. Automated Tool Generation Pipeline

The OSWorld-Verified toolset is constructed via a code-generation pipeline, which systematically processes each of the 369 original OSWorld benchmark tasks:

• Code Generation Module: OpenAI’s o3 model, prompted with CoAct-inspired strategies and equipped with task specifications and I/O examples, emits standalone (Python) scripts intended to complete each target task using only public libraries and fully self-contained logic.
• Code Filter Module: Generated scripts are executed in an OSWorld sandbox; only those passing all functional success criteria are retained. Out of 369 initial scripts, 72 survived this filter.
• Tool Wrap Module: Successful scripts are programmatically translated into JSON-RPC–compliant MCP tool specifications, describing invocation signatures, parameters, expected outputs, and error schemas suitable for integration in tool-enabled agent architectures.
• Tool Harvesting: This pipeline is complemented by careful selection of 192 existing tools (from MCP servers, CLI utilities, VS Code plugins, filesystem helpers), for broad functional coverage.

The union of these processes yields a candidate set for further validation (Jia et al., 28 Oct 2025).

3. Manual Dual Reviewer Curation and Validation

Automated generation produces diverse and sometimes brittle outputs, necessitating rigorous human filtering:

• Dual-Reviewer Inspection: Two GUI-agent experts independently audit each tool for task-robustness, generality, thorough documentation, and resilience to edge cases. Criteria include utility in everyday computing, interface clarity, and proper handling of error conditions or corner-case input.
• Practical Applicability: Reviewers verify tool behavior across distinct, previously unseen files and UI states, and in multiple OS variants, to expose dependencies and prevent overfitting to the original benchmark case.
• Versatility Check: Any tool hard-coded to a single file or parameterizes weakly fails this stage. Only tools with robust argument ranges, support for wildcards, and comprehensive exception handling are retained.

Any tool that fails even one reviewer’s criteria is immediately discarded. This process reduced the set from 264 to 158 final "OSWorld-Verified" tools. Application domains covered include LibreOffice Calc/Writer/Impress, VS Code, Chrome, VLC, generic OS utilities, and filesystem operations.

4. Benchmarking Methodology and Metrics

To empirically validate the value of MCP tool invocation, OSWorld-MCP introduces metrics beyond traditional success rates, focusing on nuanced agent behaviors:

• Task Accuracy (Acc):

$$\mathrm{Acc} = \frac{\#\,\text{successful tasks}}{\text{total}\ \#\,\text{tasks}}$$

• Tool Invocation Rate (TIR):

With $N_t$ tool-beneficial and $N_g$ non-tool-beneficial tasks, and $n_t$, $n_g$ the number of successes with and without tool invocation, respectively:

$\mathrm{TIR} = \frac{n_t + n_g}{N_t + N_g}$

TIR directly quantifies agent judgment regarding when to invoke (or refrain from) MCP tools.

• Average Completion Steps (ACS):

For $N$ tasks and step counts $S_i$,

$\mathrm{ACS} = \frac{1}{N}\sum_{i=1}^N S_i$

Lower ACS reflects greater decision and tool-use efficiency.

Performance benchmarking is conducted with leading large multimodal models (LMMs) and agentic frameworks. Notably, the incorporation of “OSWorld-Verified” tools results in statistically significant increases in overall Acc (4–19 percentage points), consistent reductions in ACS, and improved TIR, with Claude-4-Sonnet at 50 steps setting the highest TIR at 36.3%. All models, however, underperform in absolute TIR, highlighting headroom for better tool-invocation learning (Jia et al., 28 Oct 2025).

5. Criteria for the “OSWorld-Verified” Label

A tool is “OSWorld-Verified” only if it satisfies all of the following:

1. Generation from natural-language task prompts using the automated pipeline.
2. Passing functional/sandboxed execution in the OSWorld environment.
3. Human validation: two independent domain experts confirm correctness, generality, and utility.
4. Demonstrated empirical efficacy: measurable improvement in agent success and efficiency on real tasks when the tool is included in the evaluation.

This enforces both top-down (human-driven) and bottom-up (empirical/automatic) assurance, setting a reproducible and transfer-ready benchmark methodology.

6. Implications and Future Directions

“OSWorld-Verified” establishes an empirical and methodological baseline for further research into tool-augmented agent architectures, curriculum design, and system integration. By complementing GUI-action skills with explicit, verifiable tool-use, this approach enables fair, granular comparisons of agent models in holistic digital environments. It further motivates the design of agents and RL pipelines that can learn to judge relevance, utility, and efficiency of tools in complex, multi-application workflows.

The pipeline model also suggests applicability for future benchmark suites, as modular MCP tools spanning other application domains can be generated, curated, and validated at scale. Potential extensions include broader task diversity, more dynamic tool environments, and integration with methods for formal verification, further raising the assurance level for both tool and agent evaluation (Jia et al., 28 Oct 2025).

7. Summary Table: OSWorld-Verified Tool Creation Stages

Stage	Main Technique(s)	Selection Criteria / Output
Automated Generation	LLM code gen + sandbox exec	Passes task's functional success criteria
Manual Validation	Dual-reviewer hands-on test	Usefulness, robustness, parameterization, OS portability
Empirical Evaluation	Benchmark metrics (Acc, TIR, ACS)	Boosts agent performance vs. GUI-only baseline

By providing this rigorous, integrated standard, “OSWorld-Verified” fundamentally raises expectations for what constitutes a trustworthy and generalizable tool-evaluation benchmark in multimodal agent research.