- The paper establishes a verifier-grounded framework that constructs auditable benchmarks for computer-use agents in native desktop applications.
- It employs a self-evolving verification process and a diverse task generation pipeline that ensure realistic task evaluation and robust reward computation.
- The approach demonstrates superior performance of frontier models and clear advantages over LLM judges, achieving high verifier-human agreement rates.
OpenComputer: Verifiable Software Worlds for Computer-Use Agents
Introduction and Motivation
OpenComputer establishes a verifier-grounded paradigm for constructing scalable, auditable benchmarks tailored for the evaluation and training of computer-use agents interacting with native desktop applications. Unlike prior work that heavily depends on static, manual task construction or visual proxy benchmarks, OpenComputer addresses a fundamental bottleneck: constructing realistic software environments and delivering programmatic, fine-grained state verification. This enables reproducibility, partial-credit evaluation, and direct reward assignment important for robust RLHF, supervised learning, and large-scale systematic benchmarking of agents.
Framework Design
OpenComputer integrates four tightly-coupled components:
- Verifier Generation: Each supported application receives an app-specific Python-based verifier module. This module programmatically inspects all relevant application state via reliable interfaces (e.g., browser CDP, UNO, D-Bus, SQLite, direct file parsing) and exposes stable CLI endpoints producing structured JSON outputs. Verifiers are treated as production software: they undergo systematic unit and integration testing, with documented inspection surfaces and failure case analysis.
- Self-Evolving Verification: Recognizing the brittleness of hard-coded verification logic, OpenComputer incorporates an execution-driven feedback mechanism. Calibration tasks probe verifiers with real agent trajectories, and discrepancies between programmatic judgments and LLM-assigned references are used to automatically iterate on checker code, endpoint coverage, and documentation. This bootstraps both coverage and reliability, yielding high alignment with human labelers.
- Task Generation Pipeline: The task generator emphasizes diversity, realism, and checkability. Tasks are proposed with user goals independent of verifier endpoint limitations, then filtered and grounded through the verification stack. Tasks lacking adequate programmatic inspectability are either discarded or force extension of the verifier, ensuring every finalized benchmark task is robustly machine-checkable.
- Evaluation Harness: The evaluation framework executes agents within isolated desktop sandboxes, capturing full action-screenshot trajectories. Rewards are computed via precise execution of verifier commands on terminal software state, supporting both exact success and partial credit. This process provides auditable, stepwise agent evaluation unconstrained by the ambiguities of screenshot-only or LLM-judged metrics.
Empirical Evaluation and Results
OpenComputer encompasses 33 desktop applications and 1,000 highly varied tasks. The framework is evaluated against a spectrum of proprietary and open-source computer-use agents.
Key empirical observations are as follows:
- Frontier models (e.g., GPT-5.4, Claude-Sonnet-4.6) display superior robustness and efficiency. GPT-5.4 achieves a mean checklist-based reward of 88.4% and a strict success rate of 68.3%—it is notably efficient, requiring fewer steps (avg. 19.0) and lower interaction latency (16.5s/step) than competitors. However, even best-in-class models routinely fail one-third of tasks, demonstrating the difficulty and realism of the benchmark.
- Open-source models exhibit significant generalization failures. Several models with moderate to strong scores on previous benchmarks (e.g., OSWorld) experience sharp drops: for instance, GUI-OWL-1.5-8B declines from 52.3% OSWorld-Verified to 5.7% on OpenComputer, exposing limited transfer across diverse software domains and increased task complexity.
- Verifier-grounded scoring achieves clear empirical superiority over LLM-as-judge. On a human-labeled comparison set, hard-coded verifiers match human verdicts on 113/120 tasks (97.3% per-item agreement), outperforming LLM-based judges (95/120, 92.2%). Crucially, dense interfaces and terminal workflows introduce ambiguous, latent errors that are reliably detected only by structured state inspection.
Furthermore, the self-evolving verification layer repairs 89.4% of checker-side errors in calibration tasks, increasing human-verifier agreement substantially (from 85.2% to 94.1%).
Theoretical and Practical Implications
OpenComputer demonstrates that robust, scalable evaluation necessitates verifier-centric benchmarks. Reliance on LLM judgments, even with advanced models and detailed prompts, is insufficient for settings requiring precise, multi-surface verification—fine-grained spreadsheet changes, persistent metadata, log-dependent workflows, and non-trivial document manipulations illustrate this limitation.
By shifting the locus of evaluation to programmatic, auditable reward computation, OpenComputer lays a stable foundation for:
- Agent Training: Enabling machine-checkable, reward-driven self-improvement pipelines (e.g., RLHF, rejection sampling, SFT) that depend on reproducible feedback.
- Failure Analysis: Allowing systematic study of partial task completion, subtask-specific failures, and software-specific bottlenecks, which are invaluable for the development of modular or specialized agents.
- Hybrid Evaluation: Open tasks unfit for machine-only checks (e.g., requiring visual or geometric judgments) to future work on hybrid verification protocols, combining executable and visual criteria.
Limitations and Prospective Directions
Despite its robustness, certain classes of tasks—primarily those requiring semantic interpretation of rendered geometry, GUI layout, or complex visual alignment—transcend the capabilities of current programmatic verifiers. OpenComputer excludes such tasks from its core benchmark but releases them for diagnostic and hybrid-verification research.
Future development could focus on:
- Expanding inspection interfaces to support richer geometric or visual state extraction.
- Integrating hybrid LLM-verifier adjudication for a broader set of workflows.
- Scaling to cross-platform and non-desktop environments (e.g., mobile, web, embedded systems).
- Leveraging the evaluation harness for trajectory-level data collection and advanced agent training strategies.
Conclusion
OpenComputer provides an extensible, verifier-grounded infrastructure for systematically benchmarking and advancing computer-use agents in realistic software environments. Its architecture and dataset are pivotal for advancing research on autonomous computer-use, enabling reproducible benchmarks, interpretable error analysis, and effective reward-driven learning. The empirical demonstration of a persistent validity gap between LLM-based and hard-coded evaluation, combined with observed difficulties in agent end-to-end reliability, underscore the continuing necessity for well-aligned, robust verifier-focused benchmarks in the field (2605.19769).