---
title: SWE-Bench-Pro Benchmark Suite
url: https://www.emergentmind.com/topics/swe-bench-pro-ecd5fbe8-1171-4842-b741-58d3df0ec409
type: topic
---

# SWE-Bench-Pro Benchmark Suite

SWE-Bench-Pro is a contemporary benchmark suite for evaluating autonomous software engineering agents on realistic, long-horizon coding tasks requiring nontrivial code modifications across diverse codebases. It augments the SWE-Bench and SWE-Bench Verified frameworks, introducing rigorous contamination resistance, comprehensive test coverage, and a suite of methodological improvements designed to drive progress toward truly autonomous professional-level software agents [2509.16941][2503.15223][2511.13646][2503.07701].

## 1. Benchmark Construction and Dataset Structure

SWE-Bench-Pro comprises 1,865 human-verified tasks sourced from 41 actively maintained repositories. The benchmark emphasizes enterprise and real-world diversity with domains covering business applications, B2B services, and developer tooling. All codebases are partitioned into:

- **Public set:** 731 instances, 11 GPL repositories; fully released for open evaluation.
- **Held-Out set:** 858 instances, 12 GPL repositories; kept private to mitigate future overfitting.
- **Commercial set:** 276 instances, 18 proprietary, startup repositories; code not released, only evaluation results published [2509.16941].

Tasks are derived from consecutive commit pairs where the later “instance” fixes a bug or adds a feature. Nontriviality is enforced: only patches spanning ≥10 lines and ≥2 files are retained (mean: 107.4 LOC, 4.1 files). Each instance undergoes automated and human verification, including:

- Conversion of raw PR/commit/issue context into a concise “Problem Statement” (GitHub issue style).
- Explicit human-authored “Requirements” for unambiguous specification.
- “Interface” specification for new or modified APIs when needed.

Public and held-out sets derive from GPL/copy-left repositories, chosen to exclude from commercial or pre-training corpora, while commercial sets come from private codebases for maximal contamination resistance.

## 2. Evaluation Protocol and Performance Metrics

Agents operate within a unified evaluation scaffold. Each agent is presented the problem statement, requirements, and interface description, alongside access to a containerized, language-specific environment (e.g., Python venv, Node.js, Go modules). The typical interaction model is turn-based with a cap (≤ 200 interactions).

The primary metric is **Pass@k**, the probability that at least one of k candidate solutions is correct for a given task. For Pass@1 (the most commonly reported):

$$
\text{Pass@k} = 1 - \prod_{j=1}^k \left(1 - \frac{n_i}{N}\right)
$$

where $N$ is the number of candidate solutions, $n_i$ the number correct, and $M$ is the number of tasks [2509.16941]. Success requires all test suite assertions (not just those modified in the patch) to pass, with three test reruns per instance to root out flakiness.

Sample performance (public set, Pass@1):

| Model                     | Resolve (%) | Notes                       |
|---------------------------|-------------|-----------------------------|
| GPT-5                     | 23.3        | Highest open, public result |
| Claude Opus 4.1           | 22.7        |                             |
| Claude Sonnet 4           | 17.6        |                             |
| Gemini 2.5 Pro Preview    | 13.5        |                             |
| SWE-Smith 32B             | 6.8         | Open-source                 |
| OpenAI GPT-4o             | 4.9         |                             |
| Qwen-3 32B                | 3.4         |                             |

State-of-the-art open-source agents (Live-SWE-agent: 45.8%; SWE-agent: 43.6%) can close much of the gap to proprietary agents (Kimi K2-thinking: ∼51.8%) [2511.13646].

## 3. Automated Generation, Scale, and Distributional Shift

Automated benchmark generation is realized via SETUPAGENT, which reconstructs historically accurate dependency setup and test execution, supporting large-scale curation (hundreds of repositories, thousands of instances). SETUPAGENT iteratively improves install and test commands using LLM-driven repair, ensuring ≥95% test pass coverage before tasks are admitted to the benchmark [2503.07701].

Compared to prior manually curated benchmarks:

- **Repository count:** 366 (SWEE-Bench, extended) vs 12 (SWE-Bench)
- **Instance count:** 884 (SWEE-Bench) vs 2,294 (SWE-Bench)
- **Fix complexity:** Mean patch size 169.9 lines (SWEE), 41.0 (SWE); mean files touched: 2.05 (SWEE), 1.52 (SWE).
- **Agent success rates:** Up to 40% lower on SWEE/SWA than the original SWE, due to lower issue description quality, higher fix complexity, and less pre-training contamination.

This expanded coverage mitigates distributional mismatch: success rates sharply decline beyond the limited scope of popular, well-documented libraries, demonstrating that benchmarks restricted to such repositories are not representative of general agent performance [2503.07701].

## 4. Robust Validation: Semantic Correctness and PatchDiff

SWE-Bench-Pro adopts best-practice validations to address the problem of plausible-but-incorrect patches. Studies on SWE-Bench Verified reveal that limited test coverage (patch-level only) can cause up to 7.8% of “plausible” patches to count as correct while failing full developer test suites. Moreover, 29.6% of plausible patches induce functional behavior divergent from human-written ground truths; roughly 11% are certainly incorrect, inflating reported resolution rates by ≈6.2 percentage points [2503.15223].

PatchDiff is introduced as an automated differential patch testing pipeline:

- **Procedure:** Instrument execution traces in both oracle and agent-patch repo versions, extract target functions, prune context, and prompt an LLM to synthesize “differentiating tests” that discriminate agent and oracle behaviors.
- **Detection:** A patch is flagged divergent if there exists a synthesizable test that passes in one repo but fails in the other.
- **Scalability:** Deployed on 500 tasks, PatchDiff differentiated 146 patches, revealing subtle semantic divergences largely missed by traditional test coverage.

SWE-Bench-Pro recommends integrating PatchDiff into continuous integration pipelines, augmenting canonical test suites with community-contributed differentiators, and hardening issue specifications via executable contracts.

## 5. Agent Architectures and Dynamic Adaptation

Live-SWE-agent exemplifies successful agent adaptation for SWE-Bench-Pro [2511.13646]. Rather than relying exclusively on a fixed toolset, Live-SWE-agent self-evolves scaffolding and tool-creation at runtime:

- **Basic Loop:** Proposes individual bash commands (one-at-a-time), observes results, decides whether to create/modify utility tools on-the-fly.
- **Reflection Prompt:** After each command, prompts deliberation on tool synthesis. This increases solve rate (from 64% to 76% on SWE-Verified; analogous gains on Pro).
- **Cost and Efficiency:** Maintains low mean LLM cost (\$0.73/issue on Pro) and outperforms all known open-source baselines.

Performance comparison (SWE-Bench-Pro, public split; Table from [2511.13646]):

| Tool (LLM)                  | Solve-Rate | Avg. Cost | Notes                             |
|-----------------------------|------------|-----------|-------------------------------------|
| SWE-agent (Claude 4.5)      | 43.6%      | –         | Open-source leaderboard             |
| Live-SWE-agent (Claude 4.5) | 45.8%      | \$0.73    | State-of-the-art open/non-proprietary|
| GPT-5.1 dynamic patch       | ∼49.5%     | –         | Proprietary                         |
| Gemini 2.5 Pro              | ∼49.4%     | –         | Proprietary (DeepMind)              |
| Kimi K2-thinking            | ∼51.8%     | –         | Commercial customized agent         |

Agents employing flexible, context-sensitive tool synthesis demonstrate an explicit advantage over those with rigid scaffolds, particularly on long-horizon, multi-file tasks.

## 6. Real-World Fidelity, Contamination Resistance, and Future Directions

SWE-Bench-Pro’s design enforces contamination resistance by utilizing GPL/copyleft repositories (legal exclusion from commercial pre-train corpora) and sourcing additional tasks from proprietary, private startup repositories—minimizing chance of direct overlap with model training sets.

Failure mode analyses on frontier LLMs (GPT-5, Claude Opus, Gemini 2.5 Pro) reveal dominant error classes:

- Semantic/algorithmic errors dominate (35–52% of failures)
- Syntax errors persist (23–33%)
- Tool mismanagement and context navigation issues (endless file reading, context overflow) afflict smaller or less capable models [2509.16941]

Despite >70% Pass@1 on simpler SWE-Bench Verified tasks, solve rates remain <25% for SWE-Bench-Pro’s full public set among frontier LLMs, reflecting real-world complexity and the enduring gap to autonomous engineering proficiency.

Recommended future directions include:

- Expanded framework/language support (Java, C#, Rust).
- Enhanced evaluation: code architecture, performance, security, and collaborative workflows.
- Advanced agent scaffolding: dynamic planning, improved context retrieval, tool integration.
- Continuous evolution: on-the-fly agent adaptation, persistent and transferable “skills,” and training objectives incorporating scaffold and tool refinement [2511.13646].

## 7. Significance and Benchmarking Ecosystem

SWE-Bench-Pro establishes a contamination-resistant, semantically rigorous testbed with far-reaching consequences for code-agent research. Its methodology—integrating exhaustively validated test suites, semantic differential testing, and collaborative suite augmentation—creates a self-reinforcing cycle: solvers expose new deficiencies, benchmarks are extended, and agents converge toward true developer intent.

The benchmark, together with supporting frameworks like SETUPAGENT and PatchDiff, marks a paradigmatic shift from manual, limited benchmarking to scalable, realistic evaluation protocols essential for the future of autonomous software engineering [2503.07701][2503.15223].

Source: https://www.emergentmind.com/topics/swe-bench-pro-ecd5fbe8-1171-4842-b741-58d3df0ec409