---
title: 'Claw-SWE-Bench: Agent Harness Evaluation'
url: https://www.emergentmind.com/topics/claw-swe-bench
type: topic
---

# Claw-SWE-Bench: Agent Harness Evaluation

Searching arXiv for the benchmark and closely related SWE-bench/OpenClaw papers to ground the article in current literature.
Claw-SWE-Bench is a benchmark and execution protocol for evaluating OpenClaw-style agent harnesses on real GitHub issue-resolution tasks under a SWE-bench-style scoring contract. It was introduced to make heterogeneous agent harnesses, or “claws,” comparable under fixed settings including a prompt template, runtime budget, workspace contract, patch extraction procedure, and evaluator. The benchmark contains 350 instances across 8 languages and 43 repositories, together with an 80-instance Lite subset for lower-cost iteration, and it treats harness design and cost accounting as first-class variables alongside model choice [2606.12344].

## 1. Problem Setting and Relation to SWE-bench

Claw-SWE-Bench addresses a gap between general-purpose tool-using agents and the evaluation contract of SWE-bench. In SWE-bench, a model receives a codebase, an issue description, and a test suite, and succeeds only if its patch causes all fail-to-pass tests to pass without breaking previously passing tests across 2,294 real GitHub issue instances drawn from 12 Python repositories [2310.06770]. That setup assumes a clean Docker workspace, a well-defined patch artifact, and a prediction format compatible with the evaluator.

The motivating observation behind Claw-SWE-Bench is that a general-purpose agent such as OpenClaw does not, by itself, satisfy the Docker, patch, and prediction contract required for SWE-bench scoring. A second observation is that prior SWE-bench-style reports bundle model, harness loop, prompt, test harness, and termination logic together, making it difficult to isolate whether performance differences arise from the language model or from the surrounding agent harness. Claw-SWE-Bench therefore fixes the evaluation envelope and elevates the harness into an explicit experimental variable [2606.12344].

## 2. Adapter Protocol and Execution Contract

The core technical contribution is a small adapter interface that wraps a heterogeneous harness into the SWE-bench execution pipeline. Every adapter implements exactly five lifecycle methods: `create_agent(…)`, `send_task(problem_statement, repo_state)`, `backup_session(…)`, `delete_agent(…)`, and `get_docker_args(…)` [2606.12344].

The shared orchestrator executes each instance in four stages. First, it performs Docker workspace preparation by checking out the repository at the issue’s `base_commit` inside the official SWE-bench image, with any future commits pruned. Second, it instantiates a fixed-format prompt with the `problem_statement` and `base_commit`, instructing the agent to edit only files under `/testbed` and not to change test files. Third, it drives the agent loop through `create_agent → send_task → wait for finish → backup_session → delete_agent` under a single wall-clock timeout. Fourth, it extracts the candidate patch by running `git diff` against the `base_commit`, removing non-solution artifacts, and writing `model_patch` into a SWE-bench-compatible `prediction.jsonl` [2606.12344].

This contract has three explicit consequences. All harnesses operate in the same filesystem namespace; all candidate solutions are harvested from filesystem edits rather than custom output parsers; and no harness can gain by writing test files or inspecting future commits. A common misconception is that patch extraction is a minor implementation detail. The benchmark’s results instead show that adapter design materially determines whether model outputs become valid, scorable patches.

## 3. Benchmark Composition and Lite Selection

The full workload merges two upstream sources. The first is SWE-bench-Multilingual, contributing 300 non-Python issues spanning Java, Go, Rust, JS/TS, C/C++, Ruby, and PHP over 43 repositories. The second is SWE-bench-Verified-Mini, contributing 50 human-verified Python issues. After future-commit cleanup, the combined workload totals 350 real GitHub issue-resolution tasks across 8 languages [2606.12344].

To support lower-cost validation, the benchmark also releases Claw-SWE-Bench Lite, an 80-instance subset. Lite selects exactly $K=10$ instances per language under a fixed within-language difficulty quota of $2/3/3/2$ across quartiles $Q1$–$Q4$, and it is calibrated over 17 columns consisting of 9 OpenClaw × model runs plus 8 non-OpenClaw claw × model cells. The selection minimizes three stated biases simultaneously: resolve-rate parity, pairwise ranking preservation, and cost parity. A local 1-swap search with 200 restarts finds a stable $K^\*$ in $[8,10]$, and the released choice is $K=10$ for robustness [2606.12344].

| Variant | Instances | Construction |
|---|---:|---|
| Full benchmark | 350 | 300 SWE-bench-Multilingual + 50 SWE-bench-Verified-Mini |
| Lite subset | 80 | $K=10$ per language; calibrated over 17 columns |

Lite is not merely a random sample. Its reported mean Pass@1 over the 17 calibration columns is $0.643$ versus $0.639$ on the full-350 benchmark, a difference of $+0.4$ percentage points, while costing only $22.9\%$ of the end-to-end API bill. This suggests that the Lite subset is intended to preserve comparative structure rather than only reduce runtime [2606.12344].

## 4. Evaluation Methodology, Metrics, and Cost Accounting

The primary reported metric is Pass@1. More generally, the benchmark defines
$$
\mathrm{Pass@k} = 1 - \prod_{i=1}^{k}(1-p_i),
$$
where $p_i$ is the probability that the $i$-th sample resolves the issue. In the benchmark’s single-run setting, $k=1$, so Pass@1 is simply `(#Resolved)/(#Instances)` [2606.12344].

Each instance is allotted at most $T_{\text{wall}}=3600\,\text{s}$ of wall-clock time across all turns and tool calls. Concurrency is fixed, usually at 3 workers, and each harness may retry only once with `max_retries=1`. The orchestrator enforces three additional constraints: no `git add` or `commit` by agents, no future-commit log inspection, and a single final diff harvest [2606.12344].

A distinctive feature of Claw-SWE-Bench is explicit cost accounting. The benchmark records actual API spend per full-350 run from provider billing logs, along with token usage and cache read/write statistics when available. Cache hit rate is defined as
$$
\mathrm{CacheHit}=\frac{\#\mathrm{CacheReadTokens}}{\#\mathrm{InputTokens}+\#\mathrm{CacheReadTokens}}.
$$
Main result tables include total cost in USD, wall-clock duration in minutes, token counts in millions, and CacheHit. The benchmark therefore does not treat accuracy as the only outcome variable; systems with similar Pass@1 can differ substantially in total API cost [2606.12344].

## 5. Empirical Findings

The benchmark’s adapter diagnostic isolates the effect of the execution wrapper under a fixed model. With OpenClaw and GLM 5.1 on the 350-instance benchmark, a bare adapter that simply asks the model to emit a unified diff resolves 67 of 350 instances, or 19.1% Pass@1, and 69.1% of its outputs fail to apply. The full adapter, which requires file edits in `/testbed` and uses `git diff` extraction, resolves 257 of 350 instances, or 73.4% Pass@1, with apply failure below 1.5%. The reported gain is $+54.3$ percentage points, establishing that adapter design is essential rather than incidental [2606.12344].

Holding the harness fixed and varying the model also produces large swings. In the OpenClaw × 9-model sweep on the full-350 benchmark, GPT 5.5 reaches 78.0% Pass@1 at a reported cost of \$1,399.1 with 97.3% CacheHit, while Seed 2.0-mini reaches 48.6% at \$19.4 with 79.4% CacheHit. GLM 5.1 reaches 73.4% at \$277.0, and DeepSeek-V4 Flash reaches 70.3% at \$8.2. The spread in Pass@1 across models is 29.4 percentage points, indicating that model choice alone materially changes outcome quality and cost profile [2606.12344].

Holding the model fixed and varying the harness yields comparably large differences. In the five-claw × two-model sweep, GLM 5.1 ranges from 73.4% Pass@1 on `openclaw` to 60.9% on `nanobot`, a spread of 12.5 percentage points. Under Qwen 3.6-flash, performance ranges from 66.0% on `openclaw` to 38.6% on `generic`, a spread of 27.4 percentage points. Costs also vary sharply: for Qwen 3.6-flash, `generic` reports \$14.5 while `nanobot` reports \$133.1; for GLM 5.1, `generic` reports \$85.8 while `nanobot` reports \$768.8 [2606.12344].

These results support three substantive conclusions. First, harness is a first-class axis of evaluation, not a neutral wrapper. Second, similar resolved rates can correspond to very different token bills and wall-clock costs. Third, low-level protocol details such as how edits are written into the workspace can dominate nominal model capability.

## 6. Interpretation, Reproducibility, and Naming Ambiguity

Claw-SWE-Bench provides a common CLI, a single JSONL prediction format, a shared prompt template, a fixed per-instance budget, and open-source adapters for five claws, together with code, data splits, and scripts to regenerate figures [2606.12344]. In methodological terms, it separates model capability, harness design, workload, and cost accounting into orthogonal dimensions. That separation is its principal contribution to coding-agent evaluation.

The benchmark also sits within a larger ecosystem of “Claw”-prefixed evaluation efforts. The original SWE-bench established the real-repository issue-resolution paradigm that Claw-SWE-Bench adapts to heterogeneous agent harnesses [2310.06770]. Other contemporaneous benchmarks use similar naming for different targets: ATBench-Claw defines a trajectory-safety and diagnosis track over OpenClaw-specific risk taxonomies [2604.14858]; SecureClaw evaluates a dual-boundary security architecture across AgentDojo, AgentLeak, and Agent Security Bench [2606.09549]; ClawBench studies 153 everyday write-heavy tasks on live production websites [2604.08523]; ClawGym-Bench evaluates 200 workspace-grounded tasks over files, tools, and environment states [2604.26904]; and ClawsBench evaluates 44 structured productivity-agent tasks in simulated Gmail, Slack, Calendar, Docs, and Drive services [2604.05172].

This suggests a recurring naming pattern rather than a single unified benchmark family. In the strict sense, however, “Claw-SWE-Bench” denotes the benchmark introduced in "Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks" [2606.12344]. Its defining object is not generic agent capability, safety alone, or live-web task execution, but reproducible comparison of OpenClaw-style coding harnesses under a SWE-bench-compatible contract.

Source: https://www.emergentmind.com/topics/claw-swe-bench