---
title: 'SWE-MERA: Dynamic Benchmark for Bug Fixes'
url: https://www.emergentmind.com/topics/swe-mera
type: topic
---

# SWE-MERA: Dynamic Benchmark for Bug Fixes

SWE-MERA is a dynamic benchmark for evaluating large language models on software engineering tasks, specifically repository-level bug fixes drawn from real-world GitHub issues and paired with bug-fix patches and test suites. It is described as a “living,” continuously updated benchmark whose primary goals are to minimize data contamination, ensure real-world relevance by monthly refreshing, and preserve discriminative power as models improve. The benchmark is motivated in part by limitations identified in static benchmarks, especially SWE-bench: recent studies cited by the SWE-MERA paper report that in SWE-bench, 32.67% of successful patches involve direct solution leakage and 31.08% pass due to inadequate test cases. SWE-MERA addresses these concerns through automated task collection, multi-stage quality validation, and sandboxed execution, yielding approximately 10,000 potential tasks with 300 samples currently available as of June 2025 [2507.11059].

## 1. Conceptual definition and relation to earlier benchmarks

SWE-MERA targets software-engineering tasks in the form of real-world GitHub issues that require bug-fix patches. Its unit of evaluation is not an isolated code snippet but a repository-level repair problem, executed through agentic frameworks such as Aider. In this formulation, each task includes an issue description, the associated code context, a gold patch, and a test suite capable of verifying whether a proposed repair is correct.

The benchmark is explicitly positioned against the static design of SWE-bench. SWE-bench is described as a one-time collection of 2,294 Python issues and pull requests, whereas SWE-MERA is refreshed monthly. The intended effect of this design is to reduce exposure to stale tasks, lower leakage risk, and maintain benchmark difficulty as frontier models improve. This suggests a shift from fixed corpus evaluation toward continual benchmark maintenance as a first-class methodological requirement in LLM-based software-engineering evaluation [2507.11059].

The paper formalizes several benchmark-level quantities. Let $N$ denote the total number of tasks, $n_{\text{contaminated}}$ the number of tasks with known solution leakage, and $C = n_{\text{contaminated}} / N$ the contamination rate. For model evaluation, if $n_{\text{solved}}$ is the number of tasks successfully solved under a given protocol, then $P_k = n_{\text{solved}@k} / N$ defines pass@$k$. These definitions place contamination and task solvability on equal footing within the benchmark design.

## 2. Automated monthly collection pipeline

SWE-MERA uses a seven-stage, monthly pipeline that is fully automated in a Python package named `repotest`. The pipeline begins with repository selection and proceeds through pull-request mapping, metadata filtering, patch extraction, build validation, end-to-end replay, and LLM-based screening. The sequence is designed to eliminate repositories or issue–PR pairs that are too small, weakly linked, insufficiently testable, or low quality.

| Stage | Operation | Key criterion |
|---|---|---|
| 1 | Repository selection | Python, at least 10 stars, at least 10 forks, recent activity, open-source license |
| 2 | PR–issue mapping | One-to-one linked or commented mapping; PR merged after issue opening |
| 3 | Metadata filtering | Title plus body length at least 25 characters |
| 4 | Patch extraction | At least one source file and one test file modified; fewer than 15 source files touched |
| 5 | Build validation | In Docker with Python 3.11, `pytest`; keep tasks with at least 1 passing test |
| 6 | End-to-end execution | Failing tests pass only after applying the gold patch |
| 7 | LLM-based evaluation | Qwen3-32B scoring; remove bottom 25 percentile on task/test correctness or completeness |

The repository filter requires a primary language of Python, at least 10 stars, at least 10 forks, recent activity in the current year, and an open-source license. For each selected repository, the system finds closed issues updated in the previous month and retains only one-to-one issue–PR mappings, either linked or commented, where the PR is merged after the issue opens. Metadata extraction downloads the issue title, body, and comments, and discards tasks for which the title and body together are shorter than 25 characters [2507.11059].

Patch extraction requires the pull request diff to modify at least one source file and at least one test file, and excludes diffs touching 15 or more source files. Build validation is performed in Docker using Python 3.11 and `pytest`, retaining only tasks with at least one passing test. End-to-end validation replays the repository’s “before” state, applies tests from the “after” state, and verifies that the failing tests pass only after the gold patch is applied. This combination of syntactic, build-level, and execution-level checks is central to the benchmark’s quality-control logic.

## 3. Contamination control and quality validation

A defining feature of SWE-MERA is its explicit treatment of contamination. The benchmark defines contamination as a case where the issue description or PR comments contain the solution code. This is operationalized alongside a quality-filtering stage designed to reject tasks whose problem statements or tests are weak even when no direct leakage is observed.

The final screening stage uses Qwen3-32B to rate each candidate task on four axes: task correctness, test correctness, test completeness, and complexity. The prompt returns JSON containing scores in $[1,10]$ and confidences in $[0,1]$. Let $s_{\text{task}}, s_{\text{test}}, s_{\text{compl}} \in [1,10]$ denote the first three scores. SWE-MERA rejects any task for which
$$
s_{\text{task}} \le Q_1(s_{\text{task}}) \;\; \text{or} \;\; s_{\text{test}} \le Q_1(s_{\text{test}}) \;\; \text{or} \;\; s_{\text{compl}} \le Q_1(s_{\text{compl}}),
$$
where $Q_1$ is the first quartile of the corresponding score distribution. The complexity score is retained for analysis but not used for filtering [2507.11059].

The paper also introduces a test-coverage notion,
$$
TC = \frac{\#\text{ assertions in test suite}}{\#\text{ distinct code branches in patch}},
$$
and describes early human-in-the-loop checks. In early development, a human-validated subset of 300 tasks was spot-checked. In the ongoing benchmark lifecycle, regular leaderboard review is used to surface obvious contamination or trivial tasks for manual removal. A plausible implication is that SWE-MERA treats automated filtering and periodic manual curation as complementary rather than interchangeable mechanisms.

## 4. Benchmark composition and task characteristics

As of June 2025, SWE-MERA contains approximately 300 fully validated tasks, with a potential pool of approximately 10,000 candidate tasks if longer history is used. The current task set is entirely Python: 100% of the available tasks are drawn from Python repositories. The benchmark is nevertheless designed to extend to Java, JavaScript, TypeScript, Go, and C++ through the same metadata pipeline [2507.11059].

The task taxonomy includes regression fix, API misuse, off-by-one, edge-case handling, performance bug, and related categories. Complexity is scored on a 1–10 scale by Qwen3 and is reported to uniformly span easy (1–3), medium (4–7), and hard (8–10) tasks. The paper further summarizes the corpus distribution in descriptive terms: approximately one-third of tasks involve boundary-value fixes such as off-by-one errors, another third require multi-file edits, and the remainder cover specialized API or performance issues. The test suites average 5–8 assertions each, with branch coverage above 70 percent on average.

These properties distinguish SWE-MERA from benchmarks dominated by narrow bug templates or single-file edits. At the same time, the benchmark remains grounded in mined GitHub maintenance activity rather than synthetic task generation. This suggests that its difficulty profile is intended to reflect the operational heterogeneity of real repository repair, not merely the hardness of isolated code synthesis problems.

## 5. Evaluation protocol and metrics

SWE-MERA evaluates models through the Aider agent. For each issue, the framework allows up to $k = 6$ independent repair attempts, and each attempt allows up to $r = 4$ reflection steps, described as lint/test feedback loops. Final verification is performed in sandboxed Docker containers using unmodified gold tests. This makes the benchmark agentic in a specific sense: performance depends not only on patch generation but also on iterative interaction with repository feedback [2507.11059].

The benchmark reports several metrics. Pass@1 is
$$
\text{pass@1} = \frac{\# \text{ issues solved on first attempt}}{N},
$$
and pass@6 is
$$
\text{pass@6} = \frac{\# \text{ issues solved in } \le 6 \text{ attempts}}{N}.
$$
Two auxiliary measures decompose agent behavior. File localization rate is
$$
FL = \frac{\# \text{ attempts that modify the correct files}}{\# \text{ total attempts}},
$$
and patch validity is
$$
PV = \frac{\# \text{ patches that compile + pass repo tests}}{\# \text{ total attempts}}.
$$

Failure analysis is categorized by wrong file, syntax error, wrong logic with assertion failure, and token-limit exceedance. Error bars in pass@$k$ plots use binomial 95% two-sided quantiles. This evaluation design allows a distinction between solving the task, localizing the repository region correctly, and producing a syntactically and operationally valid patch.

## 6. Empirical results, discriminative power, and maintenance

The benchmark reports results for a dozen recent LLMs on tasks collected between September 2024 and June 2025. On the 2025 subset with $N \approx 300$, DeepSeek-R1-0528 attains 27.8% pass@1 and 40.2% pass@6; Devstral-Small-2505 attains 17.2% and 28.2%; Qwen3-32B attains 12.3% and 26.1%, respectively [2507.11059].

| Model | pass@1 | pass@6 |
|---|---:|---:|
| DeepSeek-R1-0528 | 27.8% | 40.2% |
| Devstral-Small-2505 | 17.2% | 28.2% |
| Qwen3-32B | 12.3% | 26.1% |

The paper also reports year-over-year degradation from 2024 to 2025 on the same benchmark framework. DeepSeek-R1 drops from 50.0% to 40.2% in pass@6, a change of $\Delta = -9.8$ percentage points. Devstral-Small drops from 34.0% to 28.2%. Other models show smaller or negligible changes. These results are presented as evidence of “strong discriminative power in state-of-the-art models,” indicating that the monthly-refresh design preserves separation even as model capabilities evolve.

The error profile is likewise diagnostic. Larger models with code-tuned objectives localize files correctly about 90% of the time and generate valid patches about 98% of the time. The most common failure modes are edge-case logic mistakes, accounting for 40% of failures; missing import or dependency errors, accounting for 20%; and exceeding a 32k token limit, accounting for about 1%. The dominant issue is therefore not basic syntactic viability but semantic correctness in repository context.

SWE-MERA is maintained through a monthly refresh schedule that yields about 50–100 new high-quality tasks per month. Community contributions are handled through a leaderboard via GitHub PRs, with submitted results triggering automated validation. The maintenance policy also specifies continued monitoring of pass rates: if pass@6 approaches 90% across all tasks, the benchmark will raise the difficulty threshold or introduce domain-specific puzzles. Periodic manual audits of randomly sampled tasks are intended to guard against drift or automation errors. A plausible implication is that SWE-MERA treats benchmark validity as an ongoing systems problem, not a one-time dataset release [2507.11059].

Source: https://www.emergentmind.com/topics/swe-mera