Claw-SWE-Bench: Agent Harness Evaluation
- Claw-SWE-Bench is a benchmark that evaluates OpenClaw-style agent harnesses by standardizing prompt templates, runtime budgets, and workspace contracts in real-world GitHub issues.
- It employs a unique adapter protocol with five lifecycle methods to isolate the effects of harness design from the underlying model’s performance under SWE-bench conditions.
- The benchmark offers both a full 350-instance workload and an 80-instance Lite subset to balance performance, cost accounting, and reproducibility in agent evaluations.
Searching arXiv for the benchmark and closely related SWE-bench/OpenClaw papers to ground the article in current literature. Claw-SWE-Bench is a benchmark and execution protocol for evaluating OpenClaw-style agent harnesses on real GitHub issue-resolution tasks under a SWE-bench-style scoring contract. It was introduced to make heterogeneous agent harnesses, or “claws,” comparable under fixed settings including a prompt template, runtime budget, workspace contract, patch extraction procedure, and evaluator. The benchmark contains 350 instances across 8 languages and 43 repositories, together with an 80-instance Lite subset for lower-cost iteration, and it treats harness design and cost accounting as first-class variables alongside model choice (Zheng et al., 10 Jun 2026).
1. Problem Setting and Relation to SWE-bench
Claw-SWE-Bench addresses a gap between general-purpose tool-using agents and the evaluation contract of SWE-bench. In SWE-bench, a model receives a codebase, an issue description, and a test suite, and succeeds only if its patch causes all fail-to-pass tests to pass without breaking previously passing tests across 2,294 real GitHub issue instances drawn from 12 Python repositories (Jimenez et al., 2023). That setup assumes a clean Docker workspace, a well-defined patch artifact, and a prediction format compatible with the evaluator.
The motivating observation behind Claw-SWE-Bench is that a general-purpose agent such as OpenClaw does not, by itself, satisfy the Docker, patch, and prediction contract required for SWE-bench scoring. A second observation is that prior SWE-bench-style reports bundle model, harness loop, prompt, test harness, and termination logic together, making it difficult to isolate whether performance differences arise from the LLM or from the surrounding agent harness. Claw-SWE-Bench therefore fixes the evaluation envelope and elevates the harness into an explicit experimental variable (Zheng et al., 10 Jun 2026).
2. Adapter Protocol and Execution Contract
The core technical contribution is a small adapter interface that wraps a heterogeneous harness into the SWE-bench execution pipeline. Every adapter implements exactly five lifecycle methods: create_agent(…), send_task(problem_statement, repo_state), backup_session(…), delete_agent(…), and get_docker_args(…) (Zheng et al., 10 Jun 2026).
The shared orchestrator executes each instance in four stages. First, it performs Docker workspace preparation by checking out the repository at the issue’s base_commit inside the official SWE-bench image, with any future commits pruned. Second, it instantiates a fixed-format prompt with the problem_statement and base_commit, instructing the agent to edit only files under /testbed and not to change test files. Third, it drives the agent loop through create_agent → send_task → wait for finish → backup_session → delete_agent under a single wall-clock timeout. Fourth, it extracts the candidate patch by running git [diff](https://www.emergentmind.com/topics/differential-transformer-diff) against the base_commit, removing non-solution artifacts, and writing model_patch into a SWE-bench-compatible prediction.jsonl (Zheng et al., 10 Jun 2026).
This contract has three explicit consequences. All harnesses operate in the same filesystem namespace; all candidate solutions are harvested from filesystem edits rather than custom output parsers; and no harness can gain by writing test files or inspecting future commits. A common misconception is that patch extraction is a minor implementation detail. The benchmark’s results instead show that adapter design materially determines whether model outputs become valid, scorable patches.
3. Benchmark Composition and Lite Selection
The full workload merges two upstream sources. The first is SWE-bench-Multilingual, contributing 300 non-Python issues spanning Java, Go, Rust, JS/TS, C/C++, Ruby, and PHP over 43 repositories. The second is SWE-bench-Verified-Mini, contributing 50 human-verified Python issues. After future-commit cleanup, the combined workload totals 350 real GitHub issue-resolution tasks across 8 languages (Zheng et al., 10 Jun 2026).
To support lower-cost validation, the benchmark also releases Claw-SWE-Bench Lite, an 80-instance subset. Lite selects exactly instances per language under a fixed within-language difficulty quota of $2/3/3/2$ across quartiles –, and it is calibrated over 17 columns consisting of 9 OpenClaw × model runs plus 8 non-OpenClaw claw × model cells. The selection minimizes three stated biases simultaneously: resolve-rate parity, pairwise ranking preservation, and cost parity. A local 1-swap search with 200 restarts finds a stable $K^\*$ in , and the released choice is for robustness (Zheng et al., 10 Jun 2026).
| Variant | Instances | Construction |
|---|---|---|
| Full benchmark | 350 | 300 SWE-bench-Multilingual + 50 SWE-bench-Verified-Mini |
| Lite subset | 80 | per language; calibrated over 17 columns |
Lite is not merely a random sample. Its reported mean Pass@1 over the 17 calibration columns is $0.643$ versus $0.639$ on the full-350 benchmark, a difference of $2/3/3/2$0 percentage points, while costing only $2/3/3/2$1 of the end-to-end API bill. This suggests that the Lite subset is intended to preserve comparative structure rather than only reduce runtime (Zheng et al., 10 Jun 2026).
4. Evaluation Methodology, Metrics, and Cost Accounting
The primary reported metric is Pass@1. More generally, the benchmark defines
$2/3/3/2$2
where $2/3/3/2$3 is the probability that the $2/3/3/2$4-th sample resolves the issue. In the benchmark’s single-run setting, $2/3/3/2$5, so Pass@1 is simply (#Resolved)/(#Instances) (Zheng et al., 10 Jun 2026).
Each instance is allotted at most $2/3/3/2$6 of wall-clock time across all turns and tool calls. Concurrency is fixed, usually at 3 workers, and each harness may retry only once with max_retries=1. The orchestrator enforces three additional constraints: no git add or commit by agents, no future-commit log inspection, and a single final diff harvest (Zheng et al., 10 Jun 2026).
A distinctive feature of Claw-SWE-Bench is explicit cost accounting. The benchmark records actual API spend per full-350 run from provider billing logs, along with token usage and cache read/write statistics when available. Cache hit rate is defined as
$2/3/3/2$7
Main result tables include total cost in USD, wall-clock duration in minutes, token counts in millions, and CacheHit. The benchmark therefore does not treat accuracy as the only outcome variable; systems with similar Pass@1 can differ substantially in total API cost (Zheng et al., 10 Jun 2026).
5. Empirical Findings
The benchmark’s adapter diagnostic isolates the effect of the execution wrapper under a fixed model. With OpenClaw and GLM 5.1 on the 350-instance benchmark, a bare adapter that simply asks the model to emit a unified diff resolves 67 of 350 instances, or 19.1% Pass@1, and 69.1% of its outputs fail to apply. The full adapter, which requires file edits in /testbed and uses git diff extraction, resolves 257 of 350 instances, or 73.4% Pass@1, with apply failure below 1.5%. The reported gain is $2/3/3/2$8 percentage points, establishing that adapter design is essential rather than incidental (Zheng et al., 10 Jun 2026).
Holding the harness fixed and varying the model also produces large swings. In the OpenClaw × 9-model sweep on the full-350 benchmark, GPT 5.5 reaches 78.0% Pass@1 at a reported cost of \$2/3/3/2Q1Q1$1133.1; for GLM 5.1, <code>generic</code> reports \$Q1$2768.8 (Zheng et al., 10 Jun 2026).
These results support three substantive conclusions. First, harness is a first-class axis of evaluation, not a neutral wrapper. Second, similar resolved rates can correspond to very different token bills and wall-clock costs. Third, low-level protocol details such as how edits are written into the workspace can dominate nominal model capability.
6. Interpretation, Reproducibility, and Naming Ambiguity
Claw-SWE-Bench provides a common CLI, a single JSONL prediction format, a shared prompt template, a fixed per-instance budget, and open-source adapters for five claws, together with code, data splits, and scripts to regenerate figures (Zheng et al., 10 Jun 2026). In methodological terms, it separates model capability, harness design, workload, and cost accounting into orthogonal dimensions. That separation is its principal contribution to coding-agent evaluation.
The benchmark also sits within a larger ecosystem of “Claw”-prefixed evaluation efforts. The original SWE-bench established the real-repository issue-resolution paradigm that Claw-SWE-Bench adapts to heterogeneous agent harnesses (Jimenez et al., 2023). Other contemporaneous benchmarks use similar naming for different targets: ATBench-Claw defines a trajectory-safety and diagnosis track over OpenClaw-specific risk taxonomies (Yang et al., 16 Apr 2026); SecureClaw evaluates a dual-boundary security architecture across AgentDojo, AgentLeak, and Agent Security Bench (Ma et al., 8 Jun 2026); ClawBench studies 153 everyday write-heavy tasks on live production websites (Zhang et al., 9 Apr 2026); ClawGym-Bench evaluates 200 workspace-grounded tasks over files, tools, and environment states (Bai et al., 29 Apr 2026); and ClawsBench evaluates 44 structured productivity-agent tasks in simulated Gmail, Slack, Calendar, Docs, and Drive services (Li et al., 6 Apr 2026).
This suggests a recurring naming pattern rather than a single unified benchmark family. In the strict sense, however, “Claw-SWE-Bench” denotes the benchmark introduced in "Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks" (Zheng et al., 10 Jun 2026). Its defining object is not generic agent capability, safety alone, or live-web task execution, but reproducible comparison of OpenClaw-style coding harnesses under a SWE-bench-compatible contract.
