Claw-SWE-Bench Lite: Low-Cost Multilingual Evaluation

• Claw-SWE-Bench Lite is an 80-instance subset designed for fast prototyping of multilingual coding-agent harnesses while preserving key evaluation metrics.
• It employs a constrained selection strategy with fixed language distribution and difficulty quartile allocation to maintain resolution parity and cost structure.
• Empirical results show near-identical Pass@1 scores and reduced resource usage, making it a practical proxy for preliminary system validation.

Searching arXiv for the benchmark and closely related evaluation papers to ground the article in current literature. Claw-SWE-Bench Lite is an 80-instance subset of the full Claw-SWE-Bench benchmark, released as the low-cost companion to a multilingual SWE-bench-style evaluation suite for “OpenClaw-style” agent harnesses on coding tasks (Zheng et al., 10 Jun 2026). It is not a separate benchmark family with a different task contract; rather, it inherits the same standardized adapter-and-evaluator protocol as the 350-instance full benchmark and differs only in the chosen instance IDs (Zheng et al., 10 Jun 2026). The benchmark is designed for faster iteration during adapter debugging, prompt tuning, backbone swapping, and regression testing, while aiming to preserve “the Pass@1 scale, per-language distribution, cross-claw relative behavior, and run-cost structure of full-350” at about one quarter of the cost (Zheng et al., 10 Jun 2026). In this sense, Claw-SWE-Bench Lite occupies a specific role within contemporary software-engineering-agent evaluation: it is a cost-aware, rank-aware proxy for a larger multilingual benchmark, rather than a replacement leaderboard or a new issue collection (Zheng et al., 10 Jun 2026).

1. Definition and benchmark role

Claw-SWE-Bench was introduced as “a benchmark for evaluating OpenClaw-style agent harnesses on coding tasks,” motivated by the mismatch between general-purpose autonomous agents and the strict scoring contract expected by SWE-bench-style evaluators (Zheng et al., 10 Jun 2026). The paper states that a generic agent “does not by itself satisfy the clean Docker workspace, patch, and prediction contract required for scoring,” and therefore defines an adapter protocol that makes heterogeneous agent harnesses, or claws, comparable under fixed conditions (Zheng et al., 10 Jun 2026).

Within that framework, Claw-SWE-Bench Lite is explicitly a subset of the full benchmark rather than an independently curated benchmark with a distinct evaluation philosophy (Zheng et al., 10 Jun 2026). The full benchmark contains 350 GitHub issue-resolution instances across 8 languages and 43 repositories, while Lite contains 80 instances selected from that full set (Zheng et al., 10 Jun 2026). The paper is explicit that the full 350-instance benchmark remains “the standard evaluation surface,” and that Lite is intended as a “practical entry point” for low-cost development workflows rather than final leaderboard claims (Zheng et al., 10 Jun 2026).

This placement matters because benchmark interpretation depends on which object is being evaluated. A Claw-SWE-Bench Lite score is, by construction, a measurement under the same fixed prompt, runtime budget, workspace contract, patch extraction procedure, and evaluator as full-350, but over a reduced, carefully calibrated instance set (Zheng et al., 10 Jun 2026). This suggests that Lite is best understood as an operational proxy with claim scope deliberately narrower than the full release.

2. Corpus composition and multilingual coverage

The full benchmark is composed of 350 real GitHub issue-resolution instances across 8 programming languages and 43 repositories, built from 300 non-Python instances from SWE-bench-Multilingual and 50 Python instances from SWE-bench-Verified-Mini after future-commit cleanup (Zheng et al., 10 Jun 2026). Claw-SWE-Bench Lite mirrors that source composition at smaller scale: it contains 80 instances, with exactly 10 instances from each of the 8 languages (Zheng et al., 10 Jun 2026). Of these 80 instances, 70 non-Python instances come from SWE-bench-Multilingual and 10 Python instances come from SWE-bench-Verified-Mini (Zheng et al., 10 Jun 2026).

The subset covers 34 of the 43 repositories in the full benchmark, which the paper quantifies as $34/43 = 79\%$ repository coverage (Zheng et al., 10 Jun 2026). This is a salient design choice because Lite is not simply a random cost-trimmed sample; it aims to remain broad in both language and repository diversity despite its reduced size (Zheng et al., 10 Jun 2026).

The paper’s positioning of multilinguality is also important. Claw-SWE-Bench is not a Python-only benchmark in the manner of original SWE-bench (Jimenez et al., 2023); rather, it combines multilingual issue-resolution instances with a standardized claw adapter protocol (Zheng et al., 10 Jun 2026). A plausible implication is that Lite’s value is partly methodological: it enables low-cost validation of harness behavior across language heterogeneity, rather than only across models.

3. Subset construction as constrained selection

The construction of Lite is one of the defining technical contributions of the benchmark. The paper states that Lite is not a random sample, but a constrained binary subset-selection problem over the 350 full-benchmark instances (Zheng et al., 10 Jun 2026). The selection variable is written as

$x_i \in \{0,1\},$

where $x_i$ indicates whether full-benchmark instance $i$ is included in Lite (Zheng et al., 10 Jun 2026).

Two hard constraints are imposed. First, Lite must contain exactly 10 instances per language. Second, within each language, it must follow a fixed difficulty-quartile allocation of $2/3/3/2$ across $Q_1/Q_2/Q_3/Q_4$ (Zheng et al., 10 Jun 2026). These quartiles are computed “from the mean resolved rate over the calibration pool,” rather than from one model alone, so difficulty is defined relative to multiple models and multiple harnesses (Zheng et al., 10 Jun 2026).

The optimization objective controls three biases. The first is resolve-rate parity: over the full $17 \times 8$ grid of calibration column by language, Lite minimizes the $L_1$ difference between Lite-estimated and full-benchmark resolved rates (Zheng et al., 10 Jun 2026). The second is a pairwise ranking hinge: if two calibration columns differ by more than $\textrm{RANK\_EPS}=0.03$ on full-350, a penalty is applied if Lite reverses the order or falls within a $0.05$ margin, with $x_i \in \{0,1\},$0 (Zheng et al., 10 Jun 2026). The third is cost parity: for each calibration column, Lite minimizes the log-cost discrepancy between Lite and full-350, with $x_i \in \{0,1\},$1 (Zheng et al., 10 Jun 2026).

The benchmark is therefore both “rank-aware” and “cost-aware” in a literal sense. “Rank-aware” refers to preservation of relative ordering among multiple harness-model systems; “cost-aware” refers to preserving the operating-cost structure of the full benchmark rather than matching only Pass@1 (Zheng et al., 10 Jun 2026). This sharply distinguishes Lite from naive subsampling.

4. Calibration columns, optimization method, and size selection

The calibration pool contains 17 concrete benchmark result columns, each corresponding to one harness-model system (Zheng et al., 10 Jun 2026). The first 9 columns come from an OpenClaw $x_i \in \{0,1\},$2 9-model sweep:

1. OpenClaw × GPT 5.5
2. OpenClaw × Claude Opus 4.7
3. OpenClaw × GLM 5.1
4. OpenClaw × DeepSeek-V4 Pro
5. OpenClaw × DeepSeek-V4 Flash
6. OpenClaw × Kimi 2.6
7. OpenClaw × Qwen 3.6-flash
8. OpenClaw × MiniMax M2.7
9. OpenClaw × Seed 2.0-mini (Zheng et al., 10 Jun 2026)

The remaining 8 columns come from 4 non-OpenClaw claws evaluated on 2 shared models:

1. hermes × GLM 5.1
2. hermes × Qwen 3.6-flash
3. nanobot × GLM 5.1
4. nanobot × Qwen 3.6-flash
5. zeroclaw × GLM 5.1
6. zeroclaw × Qwen 3.6-flash
7. generic × GLM 5.1
8. generic × Qwen 3.6-flash (Zheng et al., 10 Jun 2026)

This heterogeneous calibration pool is central to the meaning of Lite. The subset is chosen not merely to preserve one harness’s accuracy profile, but to preserve behavior across both model variation and claw variation (Zheng et al., 10 Jun 2026).

The released implementation does not use an external ILP solver. Instead, the paper specifies “per-language 200-restart within-quartile 1-swap local search,” with swaps restricted to the same language and same difficulty quartile so that all hard constraints remain satisfied throughout search (Zheng et al., 10 Jun 2026). The appendix restates this as 200 random restarts followed by same-quartile 1-swap local search independently per language (Zheng et al., 10 Jun 2026).

The final 80-instance size is also justified empirically rather than chosen arbitrarily. The paper performs a $x_i \in \{0,1\},$3-sweep, where $x_i \in \{0,1\},$4 is the number of instances per language, under perturbation scenarios involving margins, restart counts, seed offsets, and mirror parity (Zheng et al., 10 Jun 2026). It concludes that the minimum acceptable size lies in

$x_i \in \{0,1\},$5

and releases the conservative stable point $x_i \in \{0,1\},$6, yielding $x_i \in \{0,1\},$7 instances (Zheng et al., 10 Jun 2026). The paper summarizes this by stating that at $x_i \in \{0,1\},$8, the resolve gates (R-A/R-B/R-C), cost gates (C-A/C-B/C-C), and operational composite gate all pass (Zheng et al., 10 Jun 2026).

5. Execution protocol and scoring contract

Claw-SWE-Bench Lite inherits the full benchmark’s execution protocol unchanged. The benchmark is built around the observation that SWE-bench does not score chat logs or natural-language answers; it expects a prediction entry containing at least instance_id, model_name_or_path, and model_patch, where model_patch is a string-valued diff patch (Zheng et al., 10 Jun 2026). The evaluator checks out the target repository at base_commit inside a Docker environment, applies the patch under /testbed, and runs repository-level tests (Zheng et al., 10 Jun 2026).

To make heterogeneous claws comparable, the benchmark fixes several aspects of execution (Zheng et al., 10 Jun 2026). Every harness receives the same task prompt template, which includes the issue description (problem_statement) and base_commit, tells the agent that the repository lives at /testbed, forbids git add and git commit, forbids modifying test files, and prescribes an 8-phase workflow: Reading, Running, Exploration, Test Creation, Fix Analysis, Fix Implementation, Verification, and Final Review (Zheng et al., 10 Jun 2026). Every run uses one run per instance, a per-instance wall-clock timeout of 3600 seconds, and worker concurrency fixed at 3 (Zheng et al., 10 Jun 2026).

The workspace contract is equally explicit. Each instance runs inside its official SWE-bench evaluation Docker image with the repository reset to base_commit and mounted at /testbed (Zheng et al., 10 Jun 2026). For the seven non-Python multilingual languages, reachable future commits later than base_commit are removed, because some containers leaked future Git history via git log or git show (Zheng et al., 10 Jun 2026). The paper frames this future-commit cleanup as both a fairness control and a threat-mitigation step (Zheng et al., 10 Jun 2026).

Patch extraction is also standardized. Candidate solutions are not parsed from the model’s final message; instead, the benchmark computes the repository diff against base_commit after the harness terminates, times out, or errors, removes known non-solution artifacts, and exports the resulting Git diff as the patch (Zheng et al., 10 Jun 2026). This design is critical for OpenClaw-style agents because success depends on editing files in the workspace rather than emitting perfectly formatted unified diffs (Zheng et al., 10 Jun 2026).

The evaluator is the official upstream SWE-bench evaluator (Zheng et al., 10 Jun 2026). This links Claw-SWE-Bench Lite directly to the broader SWE-bench lineage, where a resolved issue is defined by test-based validation rather than textual judgment (Jimenez et al., 2023).

6. Empirical parity, cost structure, and benchmark behavior

The paper’s main validation claim is that Lite behaves as a reliable low-cost proxy for the full benchmark (Zheng et al., 10 Jun 2026). Across the 17 calibration columns, mean Pass@1 is 0.639 on full-350 and 0.643 on Lite-80, a difference of $x_i \in \{0,1\},$9, or about $x_i$0 percentage points (Zheng et al., 10 Jun 2026). This is the core parity statistic.

Per-language parity is reported in detail. Java changes from 0.699 to 0.694 ($x_i$1); Go remains 0.476 ($x_i$2); Rust changes from 0.748 to 0.741 ($x_i$3); JS/TS from 0.616 to 0.612 ($x_i$4); C/C++ from 0.647 to 0.676 ($x_i$5); Ruby from 0.603 to 0.629 ($x_i$6); PHP from 0.647 to 0.641 ($x_i$7); Python from 0.666 to 0.671 ($x_i$8) (Zheng et al., 10 Jun 2026). The worst per-language deviations are therefore C/C++ at $x_i$9 pp and Ruby at $i$0 pp (Zheng et al., 10 Jun 2026).

Cross-claw parity on the 5-claw $i$1 2-model grid is similarly close. The paper summarizes the mean absolute Lite-vs-full difference as $i$2 pp, with maximum difference $i$3 pp for nanobot × Qwen 3.6-flash (Zheng et al., 10 Jun 2026). This is the main evidence that Lite preserves system comparisons reasonably well (Zheng et al., 10 Jun 2026).

The cost side is explicitly treated as a first-class property. Relative to full-350, Lite’s full-run resource ratios are approximately: actual API cost $i$4, input tokens $i$5, output tokens $i$6, cache-read tokens $i$7, and wall-clock duration $i$8 (Zheng et al., 10 Jun 2026). The benchmark therefore behaves like a scaled-down version of full-350, roughly four times cheaper (Zheng et al., 10 Jun 2026).

The article-length interpretation should be careful here. Lite is validated as an operational proxy, not as an interchangeable substitute. The paper explicitly recommends using Lite for model screening, harness/adapter debugging, prompt adjustment, regression testing, and preliminary result checking under budget constraints, while reserving full-350 for final reporting (Zheng et al., 10 Jun 2026).

7. Benchmark interpretation, limitations, and surrounding debates

Claw-SWE-Bench Lite sits within an active methodological debate about what SWE-bench-style evaluations really measure. One issue is that adapter and harness design can dominate raw model capability. The benchmark’s most striking diagnostic is the OpenClaw × GLM 5.1 comparison between a bare adapter and a full adapter: the bare adapter resolves 67/350 instances for $i$9 Pass@1 with 69.1% apply failure, whereas the full adapter resolves 257/350 for $2/3/3/2$0 Pass@1 with $2/3/3/2$1 apply failure (Zheng et al., 10 Jun 2026). This is the paper’s central evidence that adapter design is essential rather than incidental.

A second issue is benchmark validity. Work on SWE-bench-style evaluation has shown that official tests can be insufficient and parsers can mis-score patches. “UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench” reports 36 task instances with insufficient test cases and 345 erroneous patches incorrectly labeled as passed across SWE-Bench Lite and Verified, yielding 18 and 11 ranking changes respectively (Yu et al., 10 Jun 2025). The paper explicitly reports that 40.9% of SWE-Bench Lite leaderboard entries and 24.4% of SWE-Bench Verified entries changed rank after reevaluation (Yu et al., 10 Jun 2025). This suggests that any Lite-style proxy benchmark, including Claw-SWE-Bench Lite, should be interpreted with awareness that test-based resolution rates are only as reliable as the underlying harness and tests.

A third issue is contamination and memorization. “The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason” shows that state-of-the-art models can identify buggy file paths on SWE-Bench Verified with up to 76% accuracy using only issue descriptions, but achieve only up to 53% on tasks from repositories not included in SWE-Bench (Liang et al., 14 Jun 2025). The paper argues that public GitHub-issue benchmarks may partially reward memorization of issue–file associations or repository-specific familiarity rather than generalizable repository reasoning (Liang et al., 14 Jun 2025). A plausible implication is that any public Lite subset may be especially vulnerable if it concentrates on small, canonical, highly exposed tasks.

The original SWE-bench paper is also relevant background. It defined the task as repository-scale issue resolution over 2,294 problems from 12 Python repositories, with success requiring the patch to apply and the evaluation tests to pass (Jimenez et al., 2023). Claw-SWE-Bench Lite preserves that execution-based philosophy but shifts the object of comparison from models alone to harness-plus-adapter systems under a fixed scoring contract (Zheng et al., 10 Jun 2026).

The benchmark’s own limitations are stated clearly. Results are mostly single-run aggregates, so small differences of a few percentage points should not be overinterpreted (Zheng et al., 10 Jun 2026). Lite is calibrated on 17 columns, which is substantial but not exhaustive of all future harness-model combinations (Zheng et al., 10 Jun 2026). Cost accounting depends on provider-side pricing and caching behavior (Zheng et al., 10 Jun 2026). More broadly, the paper cautions that harness × model interactions are not fully decomposed by the current sweep and that future work should study variance and wider model coverage (Zheng et al., 10 Jun 2026).

In aggregate, Claw-SWE-Bench Lite is best understood as a benchmark subset with a narrow but well-defined purpose: fast, low-cost, multilingual validation of OpenClaw-style coding-agent harnesses under a standardized SWE-bench-compatible contract (Zheng et al., 10 Jun 2026). Its main contribution is not new tasks, stronger agents, or a new leaderboard logic, but a calibrated evaluation surface that preserves relative system behavior and operating-cost structure closely enough to support reproducible development cycles before returning to the full 350-instance benchmark (Zheng et al., 10 Jun 2026).