OpenComputer Verification Framework

• OpenComputer is a verifier-grounded framework that integrates task synthesis, self-evolving verification, and an evaluation harness to benchmark desktop agents.
• It utilizes app-specific state verifiers and robust calibration protocols to ensure high fidelity and reproducibility in agent assessments.
• The framework supports 33 desktop applications with 1,000 realistic tasks, automating evaluation with structured, machine-checkable rewards.

OpenComputer is a verifier-grounded framework for constructing machine-checkable software worlds designed to benchmark and advance the capabilities of computer-use agents. Addressing a long-standing gap between synthetic environment generation and verifiable evaluation, OpenComputer provides a pipeline centered on hard-coded, structured state verifiers and self-correcting mechanisms. It encompasses support for 33 widely-used desktop applications and 1,000 finalized, realistic tasks across domains including browsers, office suites, creative software, development environments, file managers, and communication tools. Agents are evaluated in sandboxed environments, with success quantified via exact, auditable partial-credit rewards. Unlike LLM-as-judge paradigms, OpenComputer’s systematic observability architecture achieves higher fidelity with human adjudication when fine-grained application state is decisive, thereby establishing rigorous foundations for measuring agent progress and exposing persistent challenges in computer automation (Wei et al., 19 May 2026).

1. Framework Architecture

OpenComputer is structured as an integrated pipeline consisting of four principal components:

1. App-Specific State Verifiers: Each application is instrumented with a dedicated verifier module that exposes a stable interface for state inspection and criterion checking.
2. Self-Evolving Verification Layer: A feedback-driven calibration protocol identifies and repairs residual verifier bugs under realistic execution, systematically improving verifier accuracy.
3. Task-Generation Pipeline: Large, diverse sets of desktop tasks are synthesized to maximize realism and verifiability, grounded in verifiers and supported by tightly-scoped environment initialization.
4. Evaluation Harness: Agents interact with instrumented sandboxes, with every interaction logged and all success criteria evaluated automatically to yield granular, auditable rewards.

A task instance $\tau = (x, e, c)$ consists of a natural language instruction $x$, an initialization script $e$ (for seeding app state), and a checklist $c$ of machine-checkable criteria. Agents begin in initial state $s_0 \sim e$ and proceed through a sequence of actions to reach $s_T$, at which point $c$ is evaluated for exact reward $$R = \frac{|\{\text{checks passed}\}|}{|\{\text{total checks}\}|}$$, supporting partial credit.

2. App-Specific State Verifiers

Verifiers are Python modules providing CLI subcommands for both coarse- and fine-grained state inspection:

• query-* endpoints return structured (JSON) dumps of relevant application state (e.g., open-tabs for browsers, cell-values for spreadsheets),
• check-* subcommands evaluate individual success criteria (pass/fail).

Internal inspection mechanisms are tailored per application family: browsers utilize Chrome DevTools Protocol or WebDriver; office suites use UNO APIs or file parsing; SQLite-backed applications (e.g., bookmarks) are checked via direct database inspection; Linux apps leverage D-Bus or native accessibility APIs; and general artifacts rely on filesystem or metadata analysis.

Verifier modules are developed iteratively using a debug-fix-retry protocol:

1. Enumeration of all inspectable application surfaces.
2. Mapping to concrete inspection channels.
3. Implementation of query and checker endpoints.
4. Validation via unit/integration tests on positive and negative fixtures.
5. Iteration until all tests pass with high reliability.

On average, each supported application is equipped with approximately 17.7 verifier endpoints.

3. Self-Evolving Verification and Calibration

Programmatic verifiers may exhibit residual errors or “blind spots” revealed only under live execution. OpenComputer’s self-evolving verification layer employs an iterative calibration routine:

• Inputs: Initial verifier $V_a$ for application $a$; calibration task set $x$0 (∼15 representative tasks per app).
• Procedure:

1. For each calibration task, run an agent in a sandbox to produce execution trace and final state $x$1.
2. Obtain reference verdicts from an LLM evaluator (per-criterion, $x$2).
3. Run the programmatic verifier ($x$3).
4. Compare $x$4 and $x$5; if the checker fails where human/LLM agrees the agent succeeded, flag as a verifier bug.
5. Repair only the verifier code/endpoints/documents—static data (task and state) remain unchanged.
6. Re-test and iterate per item up to $x$6 attempts or until agreement is achieved.

A simplified pseudocode:

$c$2

Empirical results over 450 calibration tasks indicate that 159 exhibited LLM–verifier disagreement, of which 76 were attributed to verifier bugs. 89.4% of these were corrected within three refinement iterations. Held-out human–checker agreement improved from 85.2% to 94.1%.

4. Task Synthesis Pipeline

The task-generation process targets realism, diversity, and machine-checkability:

1. Proposal: LLMs propose candidate $x$7 pairs, with “user goals” $x$8 sampled to maximize scenario variety.
2. Filtering: Tasks are filtered according to difficulty, excluding trivial or linear tasks and retaining multi-step workflows of moderate-to-high complexity.
3. Verifier Grounding: For each candidate, if all success criteria $x$9 are checkable by the current verifier, the task is retained; otherwise, the verifier is actively extended following the established protocol.
4. Environment Synthesis: Construct seed files, folders, profiles, or database snapshots such that $e$0 yields a coherent, replayable initial state $e$1 in a fresh VM.
5. Finalization: Each new task $e$2 is added to the benchmark suite.

Periodic coverage analysis discovers missing workflows by application or feature area, prompting further task generation and verifier extension as necessary.

5. Evaluation Harness and Reward Computation

The evaluation harness automates large-scale, reproducible agent assessment:

1. Load $e$3 and $e$4 into a Dockerized desktop VM.
2. Sample $e$5 and instantiate the application.
3. For up to $e$6 steps or until agent STOP:
   • Provide screenshot $e$7.
   • Receive $e$8 (GUI/CLI action).
   • Apply and record action, screenshot, and reasoning.
4. Upon episode termination:
   • Execute all checkers $e$9 via $c$0 in situ.
   • Compute reward: $c$1.

The protocol fully supports partial credit, with reward granularity matching per-criterion pass/fail status. All logs and scores are exact and machine-checkable.

6. Empirical Results and Benchmark Statistics

OpenComputer’s first release covers 33 desktop applications with 1,000 finalized tasks. Summary statistics:

Apps	Tasks	Endpoints/App	Checks/Task	SeedFiles/Task
33	1000	17.7	6.9	1.3

Agent performance metrics on the benchmark are as follows:

Model	OSWorld	SuccessRate	AvgSteps	Time/Step	AvgReward
GPT-5.4	75.0%	68.3%	19.0	16.5 s	88.4%
Claude-Sonnet-4.6	72.5%	64.4%	31.5	20.8 s	76.6%
Kimi-K2.6	73.1%	58.8%	35.7	33.0 s	70.7%
Qwen-3.5-27B	56.2%	32.3%	33.1	57.3 s	59.4%
Gemini-3-Flash	–	16.4%	25.4	9.0 s	37.0%
EvoCUA-8B	46.1%	10.9%	67.0	9.7 s	38.1%
Qwen-3.5-9B	41.8%	7.8%	39.3	17.8 s	31.7%
GUI-OWL-1.5-8B	52.3%	5.7%	73.6	9.4 s	27.8%

Verifier-based adjudication yields higher concordance with human judgment compared to LLM-only evaluation: on a 120-task sample, task-level agreement was 94.1% (verifier) vs. 79.2% (LLM), while per-item agreement was 97.3% (verifier) vs. 92.2% (LLM). In direct GUI vs. CLI assessment across 343 tasks, GPT-5.4 achieved 75.2% GUI-mode success, while Claude Code reached 67.2% in CLI mode. The vast majority of self-evolving verifier bugs were corrected within 1–2 rounds.

7. Limitations and Future Directions

Several challenges and prospective research avenues remain:

• Visual/Geometric Criteria: Approximately 17 benchmark tasks involve requirements dependent on layout or rendered appearance; such criteria are currently excluded pending development of hybrid verification via combined state inspection and vision-based LLM adjudication.
• Coverage Gaps: Although 33 applications are supported, coverage does not extend to highly specialized suites or mobile platforms. Expansion to broader OS and UI paradigms is an anticipated extension.
• Training Integration: The framework is fully amenable to RL with machine-checkable rewards, but end-to-end agent fine-tuning on the OpenComputer benchmark has yet to be realized.
• Dynamic UIs and Version Drift: Applications change over time, which can invalidate verification protocols. The self-evolving layer partially mitigates breakage, but ongoing monitoring and community-driven verifier maintenance are necessary.
• LLM Judge as Fallback: For inherently visual criteria, robust, reproducible “visual verifiers” (potentially LLM- or vision-based) are an open research topic.

A plausible implication is that OpenComputer establishes a reproducible baseline for high-fidelity agent evaluation and creates a compelling direction for the joint development of scalable task synthesis, robust verifiers, and grounded reward mechanisms (Wei et al., 19 May 2026).