---
title: 'Claw-SWE-Bench Lite: Low-Cost Multilingual Evaluation'
url: https://www.emergentmind.com/topics/claw-swe-bench-lite
type: topic
---

# Claw-SWE-Bench Lite: Low-Cost Multilingual Evaluation

Searching arXiv for the benchmark and closely related evaluation papers to ground the article in current literature.
Claw-SWE-Bench Lite is an 80-instance subset of the full Claw-SWE-Bench benchmark, released as the low-cost companion to a multilingual SWE-bench-style evaluation suite for “OpenClaw-style” agent harnesses on coding tasks [2606.12344]. It is not a separate benchmark family with a different task contract; rather, it inherits the same standardized adapter-and-evaluator protocol as the 350-instance full benchmark and differs only in the chosen instance IDs [2606.12344]. The benchmark is designed for faster iteration during adapter debugging, prompt tuning, backbone swapping, and regression testing, while aiming to preserve “the Pass@1 scale, per-language distribution, cross-claw relative behavior, and run-cost structure of full-350” at about one quarter of the cost [2606.12344]. In this sense, Claw-SWE-Bench Lite occupies a specific role within contemporary software-engineering-agent evaluation: it is a cost-aware, rank-aware proxy for a larger multilingual benchmark, rather than a replacement leaderboard or a new issue collection [2606.12344].

## 1. Definition and benchmark role

Claw-SWE-Bench was introduced as “a benchmark for evaluating OpenClaw-style agent harnesses on coding tasks,” motivated by the mismatch between general-purpose autonomous agents and the strict scoring contract expected by SWE-bench-style evaluators [2606.12344]. The paper states that a generic agent “does not by itself satisfy the clean Docker workspace, patch, and prediction contract required for scoring,” and therefore defines an adapter protocol that makes heterogeneous agent harnesses, or claws, comparable under fixed conditions [2606.12344].

Within that framework, Claw-SWE-Bench Lite is explicitly a subset of the full benchmark rather than an independently curated benchmark with a distinct evaluation philosophy [2606.12344]. The full benchmark contains 350 GitHub issue-resolution instances across 8 languages and 43 repositories, while Lite contains 80 instances selected from that full set [2606.12344]. The paper is explicit that the full 350-instance benchmark remains “the standard evaluation surface,” and that Lite is intended as a “practical entry point” for low-cost development workflows rather than final leaderboard claims [2606.12344].

This placement matters because benchmark interpretation depends on which object is being evaluated. A Claw-SWE-Bench Lite score is, by construction, a measurement under the same fixed prompt, runtime budget, workspace contract, patch extraction procedure, and evaluator as full-350, but over a reduced, carefully calibrated instance set [2606.12344]. This suggests that Lite is best understood as an operational proxy with claim scope deliberately narrower than the full release.

## 2. Corpus composition and multilingual coverage

The full benchmark is composed of 350 real GitHub issue-resolution instances across 8 programming languages and 43 repositories, built from 300 non-Python instances from SWE-bench-Multilingual and 50 Python instances from SWE-bench-Verified-Mini after future-commit cleanup [2606.12344]. Claw-SWE-Bench Lite mirrors that source composition at smaller scale: it contains 80 instances, with exactly 10 instances from each of the 8 languages [2606.12344]. Of these 80 instances, 70 non-Python instances come from SWE-bench-Multilingual and 10 Python instances come from SWE-bench-Verified-Mini [2606.12344].

The subset covers 34 of the 43 repositories in the full benchmark, which the paper quantifies as \(34/43 = 79\%\) repository coverage [2606.12344]. This is a salient design choice because Lite is not simply a random cost-trimmed sample; it aims to remain broad in both language and repository diversity despite its reduced size [2606.12344].

The paper’s positioning of multilinguality is also important. Claw-SWE-Bench is not a Python-only benchmark in the manner of original SWE-bench [2310.06770]; rather, it combines multilingual issue-resolution instances with a standardized claw adapter protocol [2606.12344]. A plausible implication is that Lite’s value is partly methodological: it enables low-cost validation of harness behavior across language heterogeneity, rather than only across models.

## 3. Subset construction as constrained selection

The construction of Lite is one of the defining technical contributions of the benchmark. The paper states that Lite is not a random sample, but a constrained binary subset-selection problem over the 350 full-benchmark instances [2606.12344]. The selection variable is written as
\[ x_i \in \{0,1\}, \]
where \(x_i\) indicates whether full-benchmark instance \(i\) is included in Lite [2606.12344].

Two hard constraints are imposed. First, Lite must contain exactly 10 instances per language. Second, within each language, it must follow a fixed difficulty-quartile allocation of \(2/3/3/2\) across \(Q_1/Q_2/Q_3/Q_4\) [2606.12344]. These quartiles are computed “from the mean resolved rate over the calibration pool,” rather than from one model alone, so difficulty is defined relative to multiple models and multiple harnesses [2606.12344].

The optimization objective controls three biases. The first is resolve-rate parity: over the full \(17 \times 8\) grid of calibration column by language, Lite minimizes the \(L_1\) difference between Lite-estimated and full-benchmark resolved rates [2606.12344]. The second is a pairwise ranking hinge: if two calibration columns differ by more than \(\textrm{RANK\_EPS}=0.03\) on full-350, a penalty is applied if Lite reverses the order or falls within a \(0.05\) margin, with \(\lambda = 1.0\) [2606.12344]. The third is cost parity: for each calibration column, Lite minimizes the log-cost discrepancy between Lite and full-350, with \(\textrm{cost\_alpha}=1\) [2606.12344].

The benchmark is therefore both “rank-aware” and “cost-aware” in a literal sense. “Rank-aware” refers to preservation of relative ordering among multiple harness-model systems; “cost-aware” refers to preserving the operating-cost structure of the full benchmark rather than matching only Pass@1 [2606.12344]. This sharply distinguishes Lite from naive subsampling.

## 4. Calibration columns, optimization method, and size selection

The calibration pool contains 17 concrete benchmark result columns, each corresponding to one harness-model system [2606.12344]. The first 9 columns come from an OpenClaw \(\times\) 9-model sweep:

1. OpenClaw × GPT 5.5  
2. OpenClaw × Claude Opus 4.7  
3. OpenClaw × GLM 5.1  
4. OpenClaw × DeepSeek-V4 Pro  
5. OpenClaw × DeepSeek-V4 Flash  
6. OpenClaw × Kimi 2.6  
7. OpenClaw × Qwen 3.6-flash  
8. OpenClaw × MiniMax M2.7  
9. OpenClaw × Seed 2.0-mini [2606.12344]

The remaining 8 columns come from 4 non-OpenClaw claws evaluated on 2 shared models:

10. hermes × GLM 5.1  
11. hermes × Qwen 3.6-flash  
12. nanobot × GLM 5.1  
13. nanobot × Qwen 3.6-flash  
14. zeroclaw × GLM 5.1  
15. zeroclaw × Qwen 3.6-flash  
16. generic × GLM 5.1  
17. generic × Qwen 3.6-flash [2606.12344]

This heterogeneous calibration pool is central to the meaning of Lite. The subset is chosen not merely to preserve one harness’s accuracy profile, but to preserve behavior across both model variation and claw variation [2606.12344].

The released implementation does not use an external ILP solver. Instead, the paper specifies “per-language 200-restart within-quartile 1-swap local search,” with swaps restricted to the same language and same difficulty quartile so that all hard constraints remain satisfied throughout search [2606.12344]. The appendix restates this as 200 random restarts followed by same-quartile 1-swap local search independently per language [2606.12344].

The final 80-instance size is also justified empirically rather than chosen arbitrarily. The paper performs a \(K\)-sweep, where \(K\) is the number of instances per language, under perturbation scenarios involving margins, restart counts, seed offsets, and mirror parity [2606.12344]. It concludes that the minimum acceptable size lies in
\[ K^* \in [8,10], \]
and releases the conservative stable point \(K=10\), yielding \(8 \times 10 = 80\) instances [2606.12344]. The paper summarizes this by stating that at \(K=10\), the resolve gates (R-A/R-B/R-C), cost gates (C-A/C-B/C-C), and operational composite gate all pass [2606.12344].

## 5. Execution protocol and scoring contract

Claw-SWE-Bench Lite inherits the full benchmark’s execution protocol unchanged. The benchmark is built around the observation that SWE-bench does not score chat logs or natural-language answers; it expects a prediction entry containing at least `instance_id`, `model_name_or_path`, and `model_patch`, where `model_patch` is a string-valued diff patch [2606.12344]. The evaluator checks out the target repository at `base_commit` inside a Docker environment, applies the patch under `/testbed`, and runs repository-level tests [2606.12344].

To make heterogeneous claws comparable, the benchmark fixes several aspects of execution [2606.12344]. Every harness receives the same task prompt template, which includes the issue description (`problem_statement`) and `base_commit`, tells the agent that the repository lives at `/testbed`, forbids `git add` and `git commit`, forbids modifying test files, and prescribes an 8-phase workflow: Reading, Running, Exploration, Test Creation, Fix Analysis, Fix Implementation, Verification, and Final Review [2606.12344]. Every run uses one run per instance, a per-instance wall-clock timeout of 3600 seconds, and worker concurrency fixed at 3 [2606.12344].

The workspace contract is equally explicit. Each instance runs inside its official SWE-bench evaluation Docker image with the repository reset to `base_commit` and mounted at `/testbed` [2606.12344]. For the seven non-Python multilingual languages, reachable future commits later than `base_commit` are removed, because some containers leaked future Git history via `git log` or `git show` [2606.12344]. The paper frames this future-commit cleanup as both a fairness control and a threat-mitigation step [2606.12344].

Patch extraction is also standardized. Candidate solutions are not parsed from the model’s final message; instead, the benchmark computes the repository diff against `base_commit` after the harness terminates, times out, or errors, removes known non-solution artifacts, and exports the resulting Git diff as the patch [2606.12344]. This design is critical for OpenClaw-style agents because success depends on editing files in the workspace rather than emitting perfectly formatted unified diffs [2606.12344].

The evaluator is the official upstream SWE-bench evaluator [2606.12344]. This links Claw-SWE-Bench Lite directly to the broader SWE-bench lineage, where a resolved issue is defined by test-based validation rather than textual judgment [2310.06770].

## 6. Empirical parity, cost structure, and benchmark behavior

The paper’s main validation claim is that Lite behaves as a reliable low-cost proxy for the full benchmark [2606.12344]. Across the 17 calibration columns, mean Pass@1 is 0.639 on full-350 and 0.643 on Lite-80, a difference of \(+0.004\), or about \(+0.4\) percentage points [2606.12344]. This is the core parity statistic.

Per-language parity is reported in detail. Java changes from 0.699 to 0.694 (\(-0.005\)); Go remains 0.476 (\(+0.000\)); Rust changes from 0.748 to 0.741 (\(-0.007\)); JS/TS from 0.616 to 0.612 (\(-0.004\)); C/C++ from 0.647 to 0.676 (\(+0.029\)); Ruby from 0.603 to 0.629 (\(+0.027\)); PHP from 0.647 to 0.641 (\(-0.006\)); Python from 0.666 to 0.671 (\(+0.005\)) [2606.12344]. The worst per-language deviations are therefore C/C++ at \(+2.94\) pp and Ruby at \(+2.65\) pp [2606.12344].

Cross-claw parity on the 5-claw \(\times\) 2-model grid is similarly close. The paper summarizes the mean absolute Lite-vs-full difference as \(1.88\) pp, with maximum difference \(3.68\) pp for nanobot × Qwen 3.6-flash [2606.12344]. This is the main evidence that Lite preserves system comparisons reasonably well [2606.12344].

The cost side is explicitly treated as a first-class property. Relative to full-350, Lite’s full-run resource ratios are approximately: actual API cost \(22.9\%\), input tokens \(22.2\%\), output tokens \(23.6\%\), cache-read tokens \(22.6\%\), and wall-clock duration \(23.0\%\) [2606.12344]. The benchmark therefore behaves like a scaled-down version of full-350, roughly four times cheaper [2606.12344].

The article-length interpretation should be careful here. Lite is validated as an operational proxy, not as an interchangeable substitute. The paper explicitly recommends using Lite for model screening, harness/adapter debugging, prompt adjustment, regression testing, and preliminary result checking under budget constraints, while reserving full-350 for final reporting [2606.12344].

## 7. Benchmark interpretation, limitations, and surrounding debates

Claw-SWE-Bench Lite sits within an active methodological debate about what SWE-bench-style evaluations really measure. One issue is that adapter and harness design can dominate raw model capability. The benchmark’s most striking diagnostic is the OpenClaw × GLM 5.1 comparison between a bare adapter and a full adapter: the bare adapter resolves 67/350 instances for \(19.1\%\) Pass@1 with 69.1% apply failure, whereas the full adapter resolves 257/350 for \(73.4\%\) Pass@1 with \(<1.5\%\) apply failure [2606.12344]. This is the paper’s central evidence that adapter design is essential rather than incidental.

A second issue is benchmark validity. Work on SWE-bench-style evaluation has shown that official tests can be insufficient and parsers can mis-score patches. “UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench” reports 36 task instances with insufficient test cases and 345 erroneous patches incorrectly labeled as passed across SWE-Bench Lite and Verified, yielding 18 and 11 ranking changes respectively [2506.09289]. The paper explicitly reports that 40.9% of SWE-Bench Lite leaderboard entries and 24.4% of SWE-Bench Verified entries changed rank after reevaluation [2506.09289]. This suggests that any Lite-style proxy benchmark, including Claw-SWE-Bench Lite, should be interpreted with awareness that test-based resolution rates are only as reliable as the underlying harness and tests.

A third issue is contamination and memorization. “The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason” shows that state-of-the-art models can identify buggy file paths on SWE-Bench Verified with up to 76% accuracy using only issue descriptions, but achieve only up to 53% on tasks from repositories not included in SWE-Bench [2506.12286]. The paper argues that public GitHub-issue benchmarks may partially reward memorization of issue–file associations or repository-specific familiarity rather than generalizable repository reasoning [2506.12286]. A plausible implication is that any public Lite subset may be especially vulnerable if it concentrates on small, canonical, highly exposed tasks.

The original SWE-bench paper is also relevant background. It defined the task as repository-scale issue resolution over 2,294 problems from 12 Python repositories, with success requiring the patch to apply and the evaluation tests to pass [2310.06770]. Claw-SWE-Bench Lite preserves that execution-based philosophy but shifts the object of comparison from models alone to harness-plus-adapter systems under a fixed scoring contract [2606.12344].

The benchmark’s own limitations are stated clearly. Results are mostly single-run aggregates, so small differences of a few percentage points should not be overinterpreted [2606.12344]. Lite is calibrated on 17 columns, which is substantial but not exhaustive of all future harness-model combinations [2606.12344]. Cost accounting depends on provider-side pricing and caching behavior [2606.12344]. More broadly, the paper cautions that harness × model interactions are not fully decomposed by the current sweep and that future work should study variance and wider model coverage [2606.12344].

In aggregate, Claw-SWE-Bench Lite is best understood as a benchmark subset with a narrow but well-defined purpose: fast, low-cost, multilingual validation of OpenClaw-style coding-agent harnesses under a standardized SWE-bench-compatible contract [2606.12344]. Its main contribution is not new tasks, stronger agents, or a new leaderboard logic, but a calibrated evaluation surface that preserves relative system behavior and operating-cost structure closely enough to support reproducible development cycles before returning to the full 350-instance benchmark [2606.12344].

Source: https://www.emergentmind.com/topics/claw-swe-bench-lite