WeaveBench: Hybrid GUI and CLI Benchmark

• WeaveBench is a benchmark that evaluates computer-use agents’ ability to coordinate GUI observation and CLI/code operations within a single, long-horizon workflow.
• It features 114 tasks across 8 domains, using real user requests and publicly verifiable artifacts to ensure task authenticity and domain relevance.
• The benchmark employs trajectory-aware judging with strict shortcut detection, highlighting the critical difference between endpoint success and valid cross-interface execution.

WeaveBench is a long-horizon, real-world benchmark for computer-use agents (CUAs) designed to measure whether an agent can orchestrate GUI actions and CLI/code operations within a single workflow. It was introduced in response to the claim that existing CUA benchmarks typically evaluate interfaces in isolation, or expose multiple interfaces without making cross-interface coordination necessary. In WeaveBench, each task requires agents to combine GUI observation and action with command-line and code operations in one trajectory, using real user requests and publicly verifiable artifacts as grounding material (Li et al., 8 Jun 2026).

1. Conceptual scope and motivation

The benchmark is premised on a shift in deployed CUA runtimes. Contemporary agent loops combine visual desktop control, command-line execution, code editing, browsers, and external tools, whereas many prior evaluation settings treat these as separable capabilities. WeaveBench was introduced specifically to test the harder regime: moving information across interfaces over long trajectories, with multiple interleaved phases rather than a single interface being merely optional.

The benchmark formalizes this requirement through three admission criteria. The first is channel non-substitutability, requiring coordination of GUI observation or action with CLI or code modification in the same trajectory. The second is long-horizon execution, requiring multiple interleaved GUI and CLI or code phases. The third is cross-application state, requiring information transfer across multiple independent applications or processes. In this formulation, the benchmark is not only about whether an agent reaches a correct end state, but whether it performs valid cross-interface execution.

This design is positioned against three failure modes in prior evaluation practice. GUI-only benchmarks do not require a CLI or code side; CLI-only or coding benchmarks do not test visual desktop state; and some multi-interface benchmarks expose both channels but permit solutions through one channel alone. WeaveBench instead makes hybrid execution a task-level requirement rather than a convenience.

2. Task corpus, provenance, and domain coverage

WeaveBench contains 114 tasks across 8 real-world work domains, with tasks grounded in real user requests, public artifacts, and traceable provenance. Every task has at least one publicly verifiable URL. The source corpus includes 174 source URLs and 82 unique hostnames; 94/174 URLs (54%) are characterized as “user-pain” sources, and 91/114 tasks (79.8%) include at least one user-pain URL (Li et al., 8 Jun 2026).

The eight domains are listed below.

Domain	Inclusion in benchmark
Desktop productivity	Included
Document processing	Included
Games / interactive applications	Included
Web development	Included
Data analysis / visualization	Included
DevOps / sysadmin	Included
Spatial / 3D / CAD	Included
Design / creative	Included

Task counts per domain range from 10 to 18. The benchmark documentation also provides domain-specific cooperation archetypes. In DevOps / sysadmin, the pattern is graphical monitoring plus scripted rollout or config edits. In Design, it is visual asset editing plus batch export. In Spatial / 3D / CAD, it is rendered scene inspection plus parametric editing. In Games, it is observing dynamic behavior in the GUI and then patching source in CLI or code.

The provenance model is central to the benchmark’s claims of realism. Tasks are sourced from venues such as GitHub issues and pull requests, project bug trackers, public posts and forums, design mocks, monitoring snapshots, code repositories, database dumps, configs, and the OpenClaw user community. This grounding is intended to distinguish the corpus from synthetic toy tasks.

3. Hybrid-interface formalization

WeaveBench decomposes hybrid competence into interface-bound atomic capabilities. The appendix lists 19 atomic operations, partitioned across CLI-bound and GUI-bound atoms. CLI-bound examples include tracing syscalls, reading /proc, systemd, or journal, inspecting protocol payloads, reading engine EXPLAIN plans, producing diffs, and driving multi-stage build pipelines. GUI-bound examples include observing rendered app state, verifying render-layer bugs, correlating multi-panel debugger state, producing realistic pointer trajectories, triggering GUI event chains, and real-time closed-loop visual control (Li et al., 8 Jun 2026).

The benchmark’s admission criteria can be written as:

$\textbf{P1: Channel non-substitutability}$

$\textbf{P2: Long-horizon execution}$

$\textbf{P3: Cross-application state}$

These criteria establish a structural distinction between WeaveBench tasks and ordinary desktop or coding tasks. A valid trajectory must traverse multiple applications and multiple interface modalities, carrying state across them. The benchmark’s own trajectory statistics reinforce that design: the median task requires 76 tool calls, the maximum is 471, the median number of GUI↔CLI switches is 16, every task has at least one switch, the average switch rate is about 23% of calls, and the median task spans 15 distinct apps/business states.

A plausible implication is that WeaveBench evaluates a form of workflow synthesis rather than isolated actuation. The unit of success is not a single correct action sequence within one application, but an extended control policy that preserves and reuses state across visual and programmatic channels.

4. Runtime environment and interaction model

Tasks are executed on a real Ubuntu desktop inside a containerized Linux VM rather than a synthetic simulator. Evaluation is performed in deployed CLI-agent runtimes, including OpenClaw, Codex CLI, Claude Code, and Hermes, augmented with a minimal desktop-control plugin (Li et al., 8 Jun 2026).

The plugin exposes one screenshot tool for perception and nine atomic GUI actuation primitives:

• click
• double_click
• triple_click
• move
• drag
• scroll
• type
• keypress
• wait

These GUI tools operate alongside the runtime’s existing terminal, file, code, and browser tools. As a result, the benchmark session remains unified even when tasks interleave shell work, file manipulation, browser use, and desktop control.

The environment is constrained to improve reproducibility and to reduce leakage. Each task runs from a frozen VM snapshot, the environment is rolled back after completion, network access is restricted to task-local services, and each rollout is subject to per-task tool-output and wall-clock budgets. These constraints are part of the benchmark design rather than incidental infrastructure, because they bound agent behavior while preserving a real desktop setting.

5. Evaluation methodology and trajectory-aware judging

A major contribution of WeaveBench is its trajectory-aware agentic judge. The benchmark argues that outcome-only grading is vulnerable in hybrid tasks because agents can fabricate screenshots or renders, hard-code metrics, use mock services, crop or overlay existing images, exploit leaked ground-truth files, or satisfy the appearance of success without following the intended workflow. The judge therefore inspects deliverables, files, screenshots, logs, traces, and the action sequence itself (Li et al., 8 Jun 2026).

The judge runs in a fresh subprocess for each rollout. It has tools for reading files, opening images, walking the trajectory, and re-fetching evidence over multiple turns. Its procedure is clause-based: each deliverable is decomposed into atomic clauses, each clause is verified as satisfied, partially satisfied, or false, and these are then aggregated into per-deliverable correctness.

The process score covers eight dimensions:

• task_completion
• deliverable_correctness
• deliverable_quality
• evidence_authenticity
• tool_use_correctness
• final_state_correctness
• efficiency_robustness
• instruction_following

The judge also scans for nine shortcut patterns, including fake GUI or fake render generation, regenerated fixtures, hard-coded metrics, mock services, crop or duplicate reuse, overlay manipulation, ground-truth leakage, runtime injection, and CLI bypass of GUI requirements. If a high-confidence shortcut is detected, the rollout receives zero credit.

The final score is defined as:

$$s_{t,m} = \begin{cases} 0, & \text{if } h_{t,m}=1, \ \min\!\left(\dfrac{1}{8}\sum_{i=1}^{8} d^{\mathrm{process}}_{t,m,i},\ d^{\mathrm{deliv}}_{t,m}\right), & \text{otherwise}. \end{cases}$$

The benchmark-level metrics are:

$$\mathrm{PassRate}(m)=\frac{1}{|T|}\sum_{t\in T}\mathbf{1}[s_{t,m}\geq \tau], \qquad \mathrm{Overall}(m)=\frac{1}{|T|}\sum_{t\in T}s_{t,m},$$

with threshold $\tau=0.8$.

This scoring rule explicitly couples process validity to deliverable validity. It is not sufficient to produce a plausible artifact if the evidence trail indicates shortcut behavior.

6. Reported performance and benchmark difficulty

The reported results indicate that WeaveBench is far from saturated. On the fixed OpenClaw runtime, Claude Opus 4.7 achieves the best PassRate at 35.1%, with GPT-5.5 close behind at 33.3%. Across runtime and harness variation, the best model–runtime pairing is Claude Opus 4.7 + Claude Code, reaching 41.2% PassRate (Li et al., 8 Jun 2026).

These values are used by the paper to support two claims. First, the benchmark is not solved. Second, the benchmark is substantially harder and structurally different from settings in which frontier backbones report >78% on OSWorld-Verified.

Interface ablations provide direct evidence for the hybrid requirement. When the interface is restricted, GUI-only remains at or below 1.8%, CLI-only remains at or below 3.5%, and Hybrid reaches 35.1% on the best fixed harness. This is one of the strongest empirical arguments that the tasks are genuinely non-substitutable across channels.

The paper also reports a large discrepancy between outcome-only and trajectory-aware grading. On GPT-5.5, outcome-only grading yields 53.5%, while the trajectory-aware audit reduces this to 33.3%. The benchmark interprets this as evidence that final-state success is not equivalent to valid cross-interface execution. Outcome-only grading can overestimate performance by 10–20+ PassRate points because it does not detect fabricated or shortcut-generated artifacts.

A plausible implication is that WeaveBench measures not merely endpoint competence but policy legitimacy under audit. The benchmark therefore occupies a different evaluative regime from tasks where only a final file, string, or screen state is scored.

7. Position within the benchmark landscape and common confusions

WeaveBench is best understood as an overview of several benchmark traditions: real-world task grounding, long-horizon workflows, deployed agent runtimes, hybrid GUI plus CLI or code interaction, and trajectory-aware grading with shortcut detection. It is contrasted with GUI or OS benchmarks such as OSWorld, WindowsAgentArena, WebArena, and AndroidWorld; CLI or coding benchmarks such as SWE-bench, TerminalBench, and TerminalWorld; multi-interface benchmarks such as MCPWorld, OSWorld-MCP, PwP-Bench, ScienceBoard, and CocoaBench; and deployed CLI-agent benchmarks such as ClawBench, WildClawBench, and CocoaBench. The benchmark’s core distinction is that both channels are required at the task level, rather than merely exposed (Li et al., 8 Jun 2026).

A common confusion concerns the similarly named WEAVEBench, written with a different capitalization. WEAVEBench is not a computer-use benchmark; it is the human-annotated benchmark component of the broader WEAVE suite for in-context interleaved cross-modality comprehension and generation. It contains 100 tasks based on 480 images and evaluates multi-turn generation, visual memory, and world-knowledge reasoning in multimodal image generation and editing (Chow et al., 14 Nov 2025).

The distinction is substantive rather than typographic. WeaveBench concerns long-horizon orchestration across GUI, CLI, code, and related interfaces in a real Ubuntu desktop. WEAVEBench concerns multi-turn, context-dependent image generation and editing. The similarity in names can obscure the fact that the two benchmarks target different research communities, different agent architectures, and different notions of trajectory: executable desktop workflows in one case, multimodal dialogue and image history in the other.

In the CUA setting, WeaveBench’s broader significance lies in its claim that benchmark realism requires both public verifiability and process auditing. This suggests a shift away from evaluation protocols that treat hybrid interfaces as loosely coupled tools, toward protocols that require agents to weave them into a single coherent control loop.