---
title: 'SWE-bench Pro: Contamination-Resistant LLM Benchmark'
url: https://www.emergentmind.com/topics/swe-bench-pro-66b95c70-a668-4259-9df1-67bbe0b733f6
type: topic
---

# SWE-bench Pro: Contamination-Resistant LLM Benchmark

SWE-bench Pro is a contamination-resistant benchmark suite for evaluating the long-horizon, real-world software engineering abilities of large language models (LLMs) and autonomous coding agents. Conceived as a successor to the widely-used SWE-Bench family, it was specifically designed to address the saturation, triviality, and memorization artifacts exposed in earlier benchmarks. SWE-bench Pro consists of hundreds to thousands of issues sourced from complex, diverse codebases and focuses on tasks representative of professional software engineering, including multi-file bug fixes and reasoning under realistic environment and dependency constraints. It serves as a testbed for measuring genuine generalization, cross-repository transfer, and practical autonomy in automated software development systems [2509.16941][2506.12286][2503.07701].

## 1. Motivation, Scope, and Distinctions

SWE-bench Pro was motivated by the limitations in previous benchmarks—namely, data contamination, overrepresentation of popular repositories in pretraining corpora, and predominance of short-horizon or trivial tasks. For example, SWE-Bench Verified includes only 500 tasks from 12 repositories and has been shown to admit substantial memorization bias, with top LLMs achieving >70% Pass@1 accuracy, a level not representative of genuine real-world difficulty [2506.12286]. Empirical studies using minimal context subtasks (file path identification and function reproduction) revealed up to 76% accuracy on in-benchmark tasks but a collapse to ≈53% accuracy for out-of-benchmark repositories, indicating significant contamination and instance memorization.

SWE-bench Pro explicitly targets enterprise-scale realism and contamination resistance through the following criteria:
- Use of GPL-licensed and proprietary datasets to avoid inclusion in open-crawled corpora.
- Inclusion of long-horizon tasks spanning multiple files (average 4.1 files, 107.4 LOC).
- Human verification and dockerized execution to ensure task resolvability and environmental fidelity.
- Heavy emphasis on business, B2B, and developer tool domains with diverse programming languages (Python, Go, JavaScript/TypeScript) [2509.16941].

## 2. Dataset Composition, Task Structure, and Collection Protocol

SWE-bench Pro aggregates 1,865 problems from 41 actively maintained repositories, structured into three partitions:
- Public set: 731 tasks from 11 GPL-licensed repositories.
- Held-out set: 858 tasks from 12 repositories, reserved for contamination diagnostics and overfitting controls.
- Commercial set: 276 tasks from 18 proprietary startup repositories (results released, but code not public).

Each problem is reconstructed from historical GitHub issues/PRs, preserving realistic issue texts but rewritten to remove implementation leakage. Task record construction includes:
- Bullet-list requirements authored by human annotators to ground expected test behaviors.
- Explicit function/class signatures to prevent naming mismatches in test harnesses.
- Docker images specifying all runtime dependencies, environment setup, and test commands, ensuring reproducibility regardless of ecosystem changes.

Each task involves multi-turn reasoning, code search, and patching, and is anchored with a fail2pass and pass2pass test suite designed to distinguish successful regression fixes from undesired side effects [2509.16941][2503.07701].

## 3. Evaluation Protocols, Metrics, and Contamination Controls

SWE-bench Pro employs a unified harness (SWE-Agent) constraining agent actions to a fixed interface (bash, file viewing, patch submit), ensuring model-agnostic and reproducible trajectories. Key evaluation protocols are:

- **Primary metric:** Pass@1 (resolve rate)—fraction of tasks for which a single agent output transitions the test suite from fail to pass across all required tests.
  \[
  \text{Pass@1} = \frac{N_{\text{resolved}}}{N_{\text{total}}}
  \]
  For $k$-shot evaluations, Pass@$k$ is defined as $1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$ where $n$ is the number of samples and $c$ is the correct output count.

- **Diagnostic subtasks:** As established in [2506.12286], tasks include path identification and ground-truth function reproduction, with accuracy and longest common subsequence (LCS) similarity formally specified for measurement of memorization.
  - Path accuracy:
    \[
    \mathrm{Acc}_{\mathrm{path}} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}[\hat p_i \in p_i^*]
    \]
  - Function reproduction LCS similarity and exact match.
- **Contamination analysis:** The contamination rate $C$ and memorization bias $\Delta$ are formally quantified:
  \[
  C = \frac{|\mathcal{T} \cap \mathcal{E}|}{|\mathcal{E}|}
  \qquad
  \Delta = \mathrm{Acc}(\mathcal{E}\,|\,\mathcal{T}\neq\emptyset) - \mathrm{Acc}(\mathcal{E}'\,|\,\mathcal{T}\cap\mathcal{E}'=\emptyset)
  \]
- **Additional diagnostics:** Repository-anonymization, query mutation, and McNemar’s significance test to probe robustness and overfitting.

Release datasets include versioned snapshots, contamination scripts, Dockerized evaluation harness code, and exhaustive logs for baseline model configurations [2506.12286][2605.11030].

## 4. Empirical Results and Failure Analysis

Convergent evaluation across multiple studies [2509.16941][2511.13646][2606.16038] consistently shows that SWE-bench Pro is substantially more challenging than prior benchmarks. For the public set (731 issues), the following Pass@1 (resolve rate) figures are reported:

| Model/Agent            | Pass@1 (%) |
|------------------------|------------|
| GPT-5                  | 23.3       |
| Claude Opus 4.1        | 22.7       |
| Claude Sonnet 4        | 17.6       |
| Gemini 2.5 Pro         | 13.5       |
| SWE-Smith-32B          | 6.8        |
| Qwen3-32B              | 3.4        |
| **Live-SWE-agent**     | 45.8 (on Pro v2) |
| **Open-SWE-Agent**     | 36.8       |

Live-SWE-agent, featuring runtime scaffold evolution with self-generated tools, outperforms all previously published open-source baselines (45.8%), narrowing the gap to proprietary agents [2511.13646]. Similarly, Open-SWE-Traces' distilled Qwen3-30B model achieves 36.8%, with no-think mode outperforming think mode [2606.16038].

Failure analysis indicates the primary error clusters are wrong_solution (logical errors, patch applied in the wrong place), syntax_error, tool_error, context overflow, and endless file listing. The most capable models remain heavily bottlenecked by multi-file contextual reasoning, fragile tool use, and context window management [2509.16941].

## 5. Technical Implementation and Benchmarking Suite Architecture

The benchmarking suite is released as an "executable benchmarking suite" [2605.11030] comprising:
- **Workload adapters:** Support loading of SWE-bench Pro tasks as Dockerized environments with precise snapshotting.
- **Task manifests:** Each task is uniquely identified by (family, task_id, env snapshot, etc.), ensuring cross-run traceability.
- **Runner and event schema:** All agent actions, tool invokes, and test outcomes are logged as type-tagged events with per-step timing and provenance.
- **Evidence-admission gate:** Only runs with complete manifest, driver declaration, outcome, schema metadata, and provenance bindings are "paper-facing." Non-admitted runs are quarantined for audit/debug.

The formal evidence-admission contract is:
\[
A(r) = m(r) \cdot d(r) \cdot t(r) \cdot o(r) \cdot s(r) \cdot f(r) \cdot x(r) = 1
\]
where predicates represent manifest resolved, driver declared, trace completeness, terminal outcome, schema version, replay/freeze metadata, and exclusion of fixture-only runs, respectively.

All admitted runs carry a FreezeRecord tuple encoding complete provenance, enabling fully deterministic replay. Metrics such as model latency, patch cost, token counts, and invalid-action penalties are reported per-run, supporting rigorous, reproducible results analysis [2605.11030].

## 6. Interpretations, Impact, and Future Directions

SWE-bench Pro, by enforcing repository hold-out, temporal splits, explicit contamination scoring, and multi-faceted diagnostic subtasks, establishes a new standard for measuring the genuine, contamination-resistant reasoning ability of LLM-based coding agents. The step change in difficulty—reflected by a sharp drop in agent accuracy from >70% on SWE-Bench Verified to <25% on Pro for the best proprietary models [2509.16941][2506.12286]—underscores the necessity of evaluating development at a professional level.

Current agent architectures reveal critical deficits in long-horizon, multi-file reasoning, semantic correctness, tool coordination, and context management. Promising future avenues include hierarchical task decomposition, memory-efficient context retrieval, integration with static analysis, collaborative multi-agent workflows, and expanded evaluation criteria encompassing maintainability, security, and code quality [2509.16941]. The release of open, contamination-audited, and reproducible testbeds such as SWE-bench Pro is shaping the direction of LLM-for-code research by exposing hidden failure modes and quantifying authentic generalization in software engineering automation.

Source: https://www.emergentmind.com/topics/swe-bench-pro-66b95c70-a668-4259-9df1-67bbe0b733f6