SWE-Bench ProMax: Multilingual Refactoring Benchmark

• SWE-Bench ProMax is an expert-curated benchmark of 170 multilingual, repository-level refactoring tasks across Python, Java, TypeScript, Go, C, C++, and Rust, emphasizing behavior preservation across large codebases.
• The benchmark is designed to test long-horizon, cross-file transformations, with tasks averaging 11.4 modified source files and 261.6 lines of code, far exceeding predominantly single-file issue-resolution evaluations.
• GPT-5.2 with OpenHands achieves the top reported resolve rate of 41.2%, while failure analysis shows that AI agents most often miss peripheral call sites, configuration, documentation, generated files, and other dependent changes.

SWE-Bench ProMax is an expert-curated, multilingual benchmark for evaluating AI coding agents on large-scale code refactoring. It contains 170 instances drawn from real commits across Python, Java, TypeScript, Go, C, C++, and Rust. Unlike conventional repository-level issue-resolution benchmarks, ProMax deliberately filters for challenging, cross-file transformations that require behavior preservation across extensive codebases. Its instances average 11.4 modified source files and 261.6 lines of code, and the best reported configuration—GPT-5.2 with OpenHands—achieves a 41.2% resolve rate under the benchmark protocol (Shi et al., 10 Aug 2026).

1. Motivation and scope

SWE-Bench ProMax was developed in response to concerns about evaluation quality, benchmark saturation, and possible training-data contamination in SWE-bench-style evaluations. Earlier benchmarks have been criticized for relying on issue descriptions that may be terse or ambiguous and test suites that may be either overly narrow or overly broad. Overly narrow tests can reject behaviorally correct implementations by enforcing a particular internal strategy, whereas overly broad tests can impose requirements absent from the task description.

The ProMax paper cites an audit of SWE-bench Verified reporting that, among unsolved instances, 35.5% had overly narrow tests and 18.8% had overly broad tests. Together, these categories affected “nearly 60%” of unsolved instances. This concern is consistent with studies showing that passing selected benchmark tests is weaker than satisfying developer intent. One empirical study found that 7.8% of plausible patches generated for SWE-bench Verified were incorrect when all developer tests were executed, while PatchDiff identified behavioral differences from the reference patch in 29.6% of plausible patches (Wang et al., 19 Mar 2025). UTBoost likewise identified insufficient tests and parser defects that caused erroneous patches to be labeled as passed (Yu et al., 10 Jun 2025).

ProMax also addresses the scale of conventional SWE-bench tasks. The paper reports that 86% of SWE-bench Verified instances modify only one file, while ProMax instances modify an average of 11.4 source files and 261.6 source lines of code. Thirty percent of ProMax instances modify more than 10 files, and 32% change more than 200 lines of code. The benchmark therefore targets long-horizon repository transformation rather than predominantly localized issue repair.

The benchmark focuses on large-scale code refactoring: restructuring an existing codebase while preserving externally observable behavior. Refactoring may involve definitions, declarations, callers, implementations, generated code, templates, configuration, build files, documentation, examples, test fixtures, adapters, routers, drivers, and serialization layers. The task is consequently characterized by coordinated propagation of changes and preservation of behavior across repository boundaries.

ProMax does not claim to eliminate contamination. Its tasks are drawn from public GitHub repositories and real commits, so source code, commit histories, and gold patches may have appeared in training data. Its mitigation strategy consists primarily of rewriting issue descriptions from scratch, reviewing test suites, filtering for more complex transformations, and using large-scale tasks that are harder to solve through direct memorization. Research on SWE-bench has nevertheless found evidence consistent with instance-specific and repository-bias memorization, including high file-path identification accuracy from issue descriptions alone (Liang et al., 14 Jun 2025).

2. Dataset construction

The benchmark contains 170 instances retained from an initial pool of 29,782 candidates. Candidates were drawn from actively maintained GitHub repositories satisfying several automated criteria:

• At least 500 GitHub stars.
• An approved open-source license.
• A primary language constituting at least 80% of the codebase.
• A primary language among Python, Java, TypeScript, Go, C, C++, and Rust.

Candidate commits were submitted after January 2025, had commit messages containing the keyword “refactor,” did not contain “bug fix,” and modified both test and non-test files. These criteria define the initial collection process rather than the final semantic definition of a ProMax task.

For each candidate, the authors constructed an isolated Docker environment containing the repository at the commit immediately before the refactoring, build dependencies, and the repository’s test infrastructure. The gold patch, including source and test changes, was applied and the full test suite was run. Candidates were discarded if the environment could not be made functional, dependencies could not be installed or configured, or the gold patch failed the test suite.

The surviving candidates were subjected to expert filtering and rewriting. Experts, assisted by LLM tools, analyzed commit diffs and affected components, removed tasks that were too simple or insufficiently cross-file, reviewed and edited tests, rewrote issue descriptions, and performed final human consistency checks. The LLM was used as an interactive writing and analysis assistant rather than as an autonomous curator.

Only 170 of the 29,782 initial candidates survived the complete pipeline. This reduction reflects the benchmark’s emphasis on curated difficulty and evaluation validity rather than raw dataset size. Tasks involving single-file changes, too few modified lines, overly simple refactoring patterns, or insufficient cross-file coordination were filtered out.

The construction methodology differs from simply pairing an issue with a linked pull request. Research on SWE-bench-like benchmarks has shown that PR–issue pairings can be misaligned through scope creep, defective or incomplete pull requests, follow-up dependencies, specification drift, and tests that assert unspecified literals. A study of SWE-bench Verified found misalignment in 13.6% of its 500 instances (Wang et al., 30 Jul 2026). ProMax’s rewritten descriptions and manual test review are intended to reduce analogous construction defects, although the paper does not report a direct ablation isolating the contribution of each curation stage.

3. Languages, repositories, and task scale

SWE-Bench ProMax covers seven programming languages:

1. Python
2. Java
3. TypeScript
4. Go
5. C
6. C++
7. Rust

The languages span dynamically typed, statically typed, gradually typed, garbage-collected, systems-oriented, and ownership-based programming models. The benchmark spans 70 repositories, although its language distribution is uneven. TypeScript contributes 28 tasks from only two repositories, with Angular contributing 25. Go has the broadest repository coverage, with 16 repositories producing 23 tasks.

Language	Repositories	Instances	Average files	Average LOC
C	9	20	17.9	424.1
C++	9	22	21.4	196.3
Go	16	23	16.0	227.4
Java	11	26	20.8	309.8
Python	18	29	10.6	299.8
Rust	5	22	14.5	284.8
TypeScript	2	28	11.9	122.6
Overall	70	170	15.9	261.6

C has the highest average lines of code per task at 424.1. C++ and Java have the largest average file counts, at 21.4 and 20.8 respectively. Python, Rust, and TypeScript have lower average file counts, roughly 10.6–14.5, but remain substantially more multi-file than single-file evaluations.

The benchmark’s overall mean and maximum scales are as follows:

Component	Mean	Maximum
Issue-description tokens	685.3	2,092
Gold source files	11.4	182
Gold source LOC	261.6	4,503
Gold source tokens	8,179.5	72,623
Test-patch files	4.5	66
Test-patch LOC	185.5	1,959
Test-patch tokens	3,980.8	52,031
Total files, source plus tests	15.9	244

The largest task shown in the paper is a C++ NASA F´Prime refactoring involving 244 files and approximately 559 modified lines of code. The detailed figure displays approximately +591 and −514 source-line changes, reflecting the distinction between additions and deletions and the summarized LOC statistic.

ProMax is compared with RefactorBench, SWE-Refactor, SWE-EVO, and SWE-bench Verified. RefactorBench contains 100 handcrafted Python tasks averaging 4.3 modified files. SWE-Refactor contains 1,099 Java instances across 18 repositories. SWE-EVO averages 21 modified files but is not specifically focused on multilingual refactoring. ProMax’s distinguishing combination is repository-level execution, multilingual coverage, explicit refactoring focus, large average patch scope, and expert review of descriptions and tests.

4. Task specification and evaluation protocol

Each ProMax instance contains four principal components:

1. A preconfigured Docker environment.
2. A rewritten issue description.
3. A test suite.
4. The original developer’s gold patch.

The agent starts from the repository state immediately before the refactoring and must autonomously modify the repository. The formal success condition is outcome-oriented: an instance is resolved if and only if the agent’s final repository state passes every test in the suite. The evaluator does not require the agent to reproduce the developer’s commands or exact patch.

The gold patch is therefore a reference outcome rather than a textual target. Because behavior-preserving refactoring admits multiple valid implementations, a generated patch can succeed without matching the original diff. Conversely, syntactic similarity to the gold patch does not establish correctness.

The benchmark’s primary metric is resolve rate, also called Pass@1:

$$\text{Resolve Rate (Pass@1)} = \frac{R}{N}\times 100\%,$$

where $N=170$ is the number of benchmark instances and $R$ is the number resolved by a model and scaffold combination. An instance contributes one success only if the final repository state passes every test in the suite. The benchmark does not report a separate partial-credit score; a mostly correct refactoring that leaves one required call site unchanged is unresolved if a test fails.

All evaluated models used two agent scaffolds:

• mini-swe-agent: a minimal SWE-agent-style system providing file viewing, search, editing, Bash execution, and iterative observe–think–act interaction.
• OpenHands: a richer software-engineering runtime providing sandboxed command execution and structured file-editing tools.

Both scaffolds used a 300-step limit and a $10 per-instance cost limit. Reported measurements include overall resolve rate, per-language resolve rates, average interaction steps, and average API cost per instance.

The benchmark’s test-review strategy is intended to align three elements: the rewritten description, the test suite, and the gold patch. The descriptions were checked as a necessary condition, meaning a correct solution should satisfy them, and as a sufficient condition, meaning they should not permit unintended solutions. Tests were reviewed to remove assertions that reject valid alternative implementations and requirements not stated in the description.

This protocol strengthens ordinary test-based evaluation but does not establish complete semantic equivalence. Studies of related benchmarks have shown that selected tests may miss regressions, behaviorally divergent implementations, and over-broad changes. PatchDiff distinguishes behavioral divergence from confirmed incorrectness, while UTBoost combines generated tests, intramorphic comparison, manual validation, and parser correction to audit apparent successes (Wang et al., 19 Mar 2025, Yu et al., 10 Jun 2025).

5. Models, scaffolds, and results

Six models were evaluated:

• Gemini-3-Pro
• Claude Sonnet 4.6
• GPT-5.2
• GLM-5
• Kimi-K2.5
• Qwen3.5

The complete resolve-rate results are:

Scaffold	Model	Overall	Average steps	Average cost
mini-swe-agent	Gemini-3-Pro	26.5%	58.0	$0.60
mini-swe-agent	Claude Sonnet 4.6	30.6%	99.5	$2.32
mini-swe-agent	GPT-5.2	21.8%	25.2	$0.19
mini-swe-agent	GLM-5	22.9%	108.9	$0.10
mini-swe-agent	Kimi-K2.5	26.5%	85.3	$0.37
mini-swe-agent	Qwen3.5	20.6%	155.4	$0.93
OpenHands	Gemini-3-Pro	19.4%	51.2	$1.49
OpenHands	Claude Sonnet 4.6	38.8%	117.9	$4.77
OpenHands	GPT-5.2	41.2%	115.1	$3.60
OpenHands	GLM-5	36.5%	114.2	$0.24
OpenHands	Kimi-K2.5	32.9%	99.6	$0.72
OpenHands	Qwen3.5	36.5%	141.2	$0.78

The best result is achieved by GPT-5.2 with OpenHands at 41.2%, with 115.1 average steps and an average cost of $3.60 per instance. This is below the cited 75% or higher frontier performance on SWE-bench Verified and supports the authors’ claim that ProMax remains unsaturated.

OpenHands generally improves performance relative to mini-swe-agent. GPT-5.2 increases from 21.8% to 41.2%, Claude Sonnet 4.6 from 30.6% to 38.8%, GLM-5 from 22.9% to 36.5%, Kimi-K2.5 from 26.5% to 32.9%, and Qwen3.5 from 20.6% to 36.5%. Gemini-3-Pro is an exception, declining from 26.5% to 19.4%.

Under OpenHands, GPT-5.2 achieves 41.2% at $3.60 per instance, Claude Sonnet 4.6 achieves 38.8% at$4.77, GLM-5 achieves 36.5% at $0.24, Qwen3.5 achieves 36.5% at$0.78, and Kimi-K2.5 achieves 32.9% at $0.72. The results indicate that greater expenditure or more interaction does not guarantee higher resolve rates.

No model dominates every language. The strongest reported per-language results are:

• TypeScript: Claude Sonnet 4.6, 53.6%.
• Rust: Claude Sonnet 4.6, 63.6%.
• Java: GLM-5, 34.6%.
• Python: GPT-5.2, 48.3%.
• C: GPT-5.2, 75.0%.
• Go: Kimi-K2.5, 43.5%.
• C++: Qwen3.5, 54.5%.

Performance variance is substantial. Gemini-3-Pro obtains 0% on TypeScript under OpenHands, Claude Sonnet 4.6 obtains 63.6% on Rust, Kimi-K2.5 obtains 18.2% on Rust, and GPT-5.2 obtains 75.0% on C but only 19.2% on Java. The authors suggest that language-specific training data and model specialization may explain some of this variance.

6. Failure modes, task composition, and limitations

The dominant failure mode is incomplete refactoring. Agents often locate and modify the central implementation but fail to propagate the transformation to peripheral call sites, documentation, configuration, generated or template files, test fixtures, and downstream dependencies. For small patches of roughly up to five files, agent-modified-file distributions track gold patches relatively closely. For larger patches, agents modify substantially fewer files: the gold-patch cumulative distribution reaches 90% only around 20 files, whereas agent distributions reach 90% by approximately 10 files.

Failed instances consume more interaction rounds than successful instances. The paper describes repeated cycles of reading, editing, testing, reverting, and retrying as “unproductive exploration.” Qwen3.5 takes the most steps under mini-swe-agent, at 155.4, and 141.2 under OpenHands, but does not lead in resolve rate. This suggests that the principal bottleneck is global planning and management of cascading repository-wide consequences rather than only local code generation.

ProMax tasks are compound maintenance tasks rather than exclusively pure refactorings. The reported task categories are:

• Refactoring Cleanup: 66.5%.
• API Interface Change: 65.3%.
• New Feature: 43.5%.
• Bug Fix: 41.2%.
• Documentation: 21.2%.

Every instance belongs to at least two categories, and 46.5% belong to three or more. API Interface Change and Refactoring Cleanup co-occur in 79 instances; Refactoring Cleanup and Bug Fix co-occur in 48; API Interface Change and Bug Fix co-occur in 33.

The reported required skills are:

• Cross-file reasoning: 99.4%.
• API-semantics understanding: 98.8%.
• Interface-contract reasoning: 97.1%.
• Pattern matching: 91.8%.
• Data-flow reasoning: 88.8%.
• Domain knowledge: 79.4%.
• Type-system reasoning: 50.6%.

Representative tasks include a 244-file C++ NASA F´Prime header-organization refactoring, a 94-file Java PlantUML change adding hour-level time resolution to a Gantt engine, a 62-file C Betaflight configuration-field renaming, a 49-file Rust Burn scalar-argument unification, a 47-file Go OpenList upload-stream refactoring, a 30-file Python Google LangExtract provider-output transformation, and a 27-file TypeScript Ant Design API unification.

The benchmark has several limitations. It contains only 170 instances, and language and repository distributions are uneven. The 28 TypeScript tasks come from two repositories, mostly Angular, so results may reflect repository-specific conventions. Candidate selection depends on commit messages containing “refactor,” excluding “bug fix,” modifying both test and non-test files, and satisfying repository-level criteria. Genuine refactorings described differently may therefore be omitted.

The success criterion remains test-based. Manual review improves alignment between descriptions and tests but does not prove semantic equivalence or guarantee detection of all regressions. Refactoring tasks may also include new features, bug fixes, documentation changes, API changes, security work, or dependency work, so the benchmark does not measure only narrow textbook refactoring.

The experimental comparison is limited to six models and two scaffolds, and no independent human completion-time or human success-rate baseline is provided. The authors also report no controlled ablation comparing original and rewritten descriptions, reviewed and unreviewed tests, different complexity thresholds, or contaminated and decontaminated subsets. Consequently, the causal contribution of individual curation steps is not separately quantified.

7. Significance and interpretation

SWE-Bench ProMax extends repository-level software-engineering evaluation along three dimensions: multilinguality, transformation scale, and curation. Its instances are deliberately selected to require coordinated modifications across many files and to preserve behavior under structural change. The benchmark therefore evaluates whether an agent can maintain a global representation of a repository while executing a long-horizon transformation.

Its 41.2% best resolve rate should be interpreted as performance on ProMax’s particular distribution of expert-curated refactoring tasks, not as a universal measure of software-engineering competence. Public repositories and commits remain potential sources of contamination, and the benchmark does not report a formal overlap analysis. Rewritten descriptions reduce direct leakage from original commit messages and issue formulations but do not establish that underlying code or gold patches were absent from model training data.

ProMax’s evaluation design also illustrates the importance of separating task difficulty from benchmark validity. PR–issue misalignment studies show that historical links, patches, and tests can encode scope creep, defective solutions, incomplete specifications, follow-up dependencies, or unspecified literals (Wang et al., 30 Jul 2026). Research on multilingual issue resolution further demonstrates that language, repository complexity, issue type, patch size, and agent framework can substantially alter measured performance (Zan et al., 3 Apr 2025). These findings reinforce the need to publish per-language results, repository distributions, environment specifications, task identifiers, and explicit aggregation rules.

The benchmark’s principal contribution is not merely its multilingual task count. It establishes a curated evaluation regime for large-scale, behavior-preserving repository transformation in which success requires all relevant changes to be propagated and all tests to pass. Its reported failure analyses indicate that current agents often perform the central edit but fail to complete the surrounding transformation. ProMax thus targets a capability gap between local code modification and globally coherent software maintenance.