SWE-Perf: Repo-Level Optimization Benchmark

• SWE-Perf is a benchmark for repository-level performance optimization that evaluates coordinated multi-file edits using curated pull requests and runtime metrics.
• It employs two settings—Oracle for file-level patches and Realistic for full-repository edits—to distinguish localized patch synthesis from end-to-end optimization.
• Empirical results reveal a significant gap between expert patches and LLM-generated solutions, underscoring challenges in achieving stable, reproducible performance gains.

Searching arXiv for papers on SWE-Perf and closely related repository-level performance-optimization benchmarks. SWE-Perf is a repository-level benchmark for code performance optimization with LLMs, introduced to evaluate whether systems can optimize real-world repositories under executable, performance-sensitive conditions rather than merely preserve functional correctness (He et al., 16 Jul 2025). In its benchmark sense, SWE-Perf comprises 140 carefully curated instances derived from performance-improving pull requests from popular GitHub repositories, and each instance includes the original repository codebase, an executable environment, target functions, performance-related unit tests, expert-authored patches, and runtime metrics before and after the human patch (He et al., 16 Jul 2025). Subsequent audit work has characterized SWE-Perf as a useful but fragile benchmark signal, because many reference patches exhibit near-zero runtime change that does not remain statistically supported under cross-machine replay (Chen et al., 1 Jul 2026). The term “SWE-Perf” is also used in an earlier, unrelated VSCode extension for interactive static software performance analysis, creating a naming ambiguity that is relevant in bibliographic and systems contexts (Beigelbeck et al., 2021).

1. Definition and scope

SWE-Perf was introduced as the first benchmark aimed specifically at repository-level code performance optimization for LLMs (He et al., 16 Jul 2025). Its motivating claim is that prior efficiency benchmarks such as Mercury, EFFIBENCH, and EvalPerf focus mainly on function-level optimization, whereas practical repository optimization often requires coordinated edits across multiple files, modules, and call sites (He et al., 16 Jul 2025). The benchmark therefore targets a different capability profile from ordinary code generation and from bug-fixing benchmarks centered on functional correctness (He et al., 16 Jul 2025).

The benchmark contains 140 carefully curated instances drawn from 9 popular GitHub repositories (He et al., 16 Jul 2025). Each instance provides the original repository codebase, an executable environment, target functions, performance-related unit tests, the expert-authored patch from the underlying pull request, and runtime metrics for relevant tests before and after the human patch (He et al., 16 Jul 2025). This packaging makes SWE-Perf an executable benchmark rather than a static patch-prediction dataset.

A central design feature is its distinction between two evaluation settings. In the Oracle (File-Level) setting, the model is given the oracle target functions and relevant file context; in the Realistic (Repo-Level) setting, the system is given the functions exercised by the performance tests and must operate end-to-end across the repository, potentially modifying any file needed to achieve the speedup (He et al., 16 Jul 2025). This separation formalizes a difference between localized patch synthesis and open-world repository optimization.

This suggests that SWE-Perf is intended not merely as a test of code-editing competence, but as a benchmark for repository-scale localization, planning, and optimization under measurement noise. A plausible implication is that performance optimization is being treated as a distinct software-engineering workload rather than a minor variant of repair.

2. Benchmark construction and instance structure

The construction pipeline begins with pull-request mining and progressively narrows the candidate set through executable and statistical validation (He et al., 16 Jul 2025). The paper reports an initial pool of 102,241 pull requests collected from the same broad set of repositories used in SWE-Bench-style work (He et al., 16 Jul 2025). In Phase 1, the authors re-crawled pull requests from high-star repositories and kept only those that resolve an issue, deliberately not requiring test additions because the objective is performance improvement rather than correctness repair (He et al., 16 Jul 2025).

In Phase 2, the benchmark builders constructed executable Docker environments and measured unit-test runtimes for the original and modified codebases (He et al., 16 Jul 2025). At this stage, 34,397 distinct codebases were gathered, and runtimes were successfully obtained for 19,499 of them (He et al., 16 Jul 2025). In Phase 3, they retained pull requests where both pre- and post-patch runtimes were available and performance improvements were observable and stable, yielding 1,696 valid instances after screening (He et al., 16 Jul 2025). In Phase 4, only instances with statistically stable gains above the threshold remained, producing the final 140-instance benchmark (He et al., 16 Jul 2025).

The data collection environment is constrained during collection to one CPU core and 16 GB memory, while later evaluation uses 5 CPU cores (He et al., 16 Jul 2025). For each codebase, pytest runtimes are measured three times before more stringent stability checks are applied (He et al., 16 Jul 2025). The benchmark then warms up the environment, runs each unit test 20 times, removes outliers with the IQR rule, and computes a conservative statistically significant gain via a one-sided Mann-Whitney U test; only unit tests with $\delta > 0.05$ are retained (He et al., 16 Jul 2025).

Target-function extraction is also bifurcated by evaluation mode. In the Oracle setting, target functions are derived directly from the human patch using AST analysis and unified-diff matching; in the Realistic setting, the authors use dynamic tracing with yappi plus AST parsing of the unit tests to identify the functions directly exercised by performance-related tests (He et al., 16 Jul 2025). The benchmark explicitly avoids using the unit test code itself as the target, to prevent leakage and avoid degenerate “functional pruning” solutions that merely remove behavior to accelerate a test (He et al., 16 Jul 2025).

The instances are substantial in repository and patch scale. Across the 140 instances, the mean repository size is about 447.3 non-test files and 170k non-test lines, with maxima of 1,972 files and 502k lines (He et al., 16 Jul 2025). Human patches edit on average 131.1 lines, 4.3 files, and 7.6 functions per instance; the average number of related tests is 8.1; and the expert-patch performance improvement ratio averages 10.9%, with a maximum of 87.8% (He et al., 16 Jul 2025). The realistic setting is materially broader than the oracle one, with mean target functions of 30.1 versus 7.6 and maxima of 256 versus 94 (He et al., 16 Jul 2025).

3. Evaluation protocol and performance metric

SWE-Perf uses a three-level evaluation framework: Apply, Correctness, and Performance (He et al., 16 Jul 2025). Apply measures whether a generated patch can be merged into the codebase without conflicts; Correctness measures whether all performance-related tests still pass after the patch is applied; Performance measures the statistically significant minimum runtime gain (He et al., 16 Jul 2025). This structure separates syntactic applicability, semantic preservation, and non-functional improvement.

The paper defines: $Apply = \tfrac{N_{apply}}{N_{total}}$ and

$$Correctness = \frac{\sum_{i=1}^{N_{total}} \left[ \bigwedge_{j=1}^{N_i} result\_post_{i,j} = pass \right]}{N_{total}}$$

(He et al., 16 Jul 2025). For performance, per-test gains are computed using the same conservative statistical procedure as in data collection and averaged at the instance and benchmark levels: $$Performance = \frac{1}{N_{total}} \sum_{i=1}^{N_{total}} P_i, \qquad P_i = \frac{1}{n_i} \sum_{j=1}^{N_i} p_{i,j}$$ (He et al., 16 Jul 2025).

The benchmark’s per-test runtime improvement ratio is defined as: $$Ratio = \frac{R_{original} - R_{modified}}{R_{original}}$$ (He et al., 16 Jul 2025). Stability filtering then relies on repeated timing, IQR outlier removal, and a significance-aware minimum-gain estimate $\delta$ derived through a one-sided Mann-Whitney U procedure, with a final gain threshold using $p < 0.1$ and retention only for unit tests satisfying $\delta > 0.05$ (He et al., 16 Jul 2025).

A later audit paper reproduces SWE-Perf’s official validity rule as follows: each selected efficiency unit test must pass on the original and modified commits, and after 20 timing repetitions and IQR outlier filtering, the recomputed Mann–Whitney minimum-gain value must remain above the dataset threshold, $\delta_i > 0.05$ (Chen et al., 1 Jul 2026). This restatement is important because it frames SWE-Perf as a significance-aware benchmark rather than a simple raw-runtime comparison.

This suggests that SWE-Perf treats runtime improvement as a statistical property of a validated test workload, not as a single-point measurement. A plausible implication is that benchmark difficulty partly resides in measurement reproducibility rather than solely in edit synthesis.

4. Empirical results and capability gap

The reported results show a large gap between language-model systems and expert-authored performance patches (He et al., 16 Jul 2025). The expert baseline achieves 100.00% Apply, 100.00% Correctness, and 10.85% Performance (He et al., 16 Jul 2025). In the Oracle (File-Level) setting, representative model results include Claude-3.7-sonnet at 66.43% Apply, 61.43% Correctness, and 1.24% Performance; Claude-4-sonnet at 73.57%, 70.00%, and 1.76%; Claude-4-opus at 85.71%, 78.57%, and 1.28%; GPT-4o at 63.57%, 56.43%, and 0.60%; OpenAI-o3 at 78.57%, 76.43%, and 1.37%; and Gemini-2.5-Pro at 95.00%, 83.57%, and 1.48% (He et al., 16 Jul 2025).

In the Realistic (Repo-Level) setting, the benchmark reports OpenHands (Claude-3.7-sonnet) at 87.86% Apply, 77.86% Correctness, and 2.26% Performance, while Agentless (Claude-3.7-sonnet) reaches 88.57% Apply, 70.71% Correctness, and 0.41% Performance (He et al., 16 Jul 2025). The paper explicitly notes that OpenHands outperforms Agentless, which is consistent with the claim that iterative agent-based reasoning is better suited to open-ended repository optimization than a fixed pipeline (He et al., 16 Jul 2025).

The paper also notes that LLM-based systems can occasionally be competitive on specific repositories: on sklearn, OpenHands slightly outperforms the expert by 0.4% (He et al., 16 Jul 2025). However, this is described as an exception rather than the dominant pattern (He et al., 16 Jul 2025). The broader result is that existing systems remain far from expert-level optimization.

The analysis section further reports that optimization becomes harder as the number of target functions increases: expert performance decreases with more functions, and the gap between expert and model widens substantially in the multi-function regime (He et al., 16 Jul 2025). It also reports that as original runtime increases, the expert’s achievable performance ceiling rises, while model performance tends to plateau (He et al., 16 Jul 2025). The keyword analysis of patches indicates that model-generated edits tend to focus on low-level infrastructure and environment-related changes, whereas expert patches more often reflect higher-level abstractions, domain-specific reasoning, and performance-critical refactoring (He et al., 16 Jul 2025).

This suggests that the main unresolved difficulty is not simply test preservation, but repository-scale performance reasoning under multi-function and cross-file dependencies. A plausible implication is that SWE-Perf exposes a deficit in optimization strategy and bottleneck localization rather than only in code generation fidelity.

5. Reliability, fragility, and benchmark auditing

A later audit paper evaluates SWE-Perf alongside GSO and SWE-fficiency and argues that repository-level performance-optimization benchmarks may conflate runtime instability, benchmark-specific scoring rules, and task saturation (Chen et al., 1 Jul 2026). For SWE-Perf specifically, the audit’s central conclusion is that the benchmark is fragile because many reference patches produce runtime changes extremely close to zero (Chen et al., 1 Jul 2026).

The audit reports that SWE-Perf contains 140 tasks drawn from 9 repositories and describes its runtime comparison tests as “Unit tests filtered from source repositories to compare original and PR-modified commits” (Chen et al., 1 Jul 2026). In cross-machine replay, 138 of the 140 official tasks are complete-by-machine on each of the four cloud machine profiles, while 2 are not evaluable due to missing dependencies in the evaluation image (Chen et al., 1 Jul 2026). When the authors ask only whether the reference patch beats the base code in every machine-round replay, 48 of 140 SWE-Perf tasks satisfy this faster-than-base condition (Chen et al., 1 Jul 2026). When they reapply SWE-Perf’s original validity rule across all 12 replay conditions, only 11 of 140 tasks remain valid (Chen et al., 1 Jul 2026).

The audit states directly that “The drop is the largest for SWE-Perf, where 129 reference patches no longer show statistically supported gain across machines” (Chen et al., 1 Jul 2026). It attributes this to extremely small effect sizes rather than unusually large noise: median runtime change for SWE-Perf reference patches is $-0.03\%$, median runtime standard deviation is 1.41 percentage points, and the std./signal ratio is 43.23$Apply = \tfrac{N_{apply}}{N_{total}}$0 (Chen et al., 1 Jul 2026). It also reports that 101 of 138 SWE-Perf tasks have median changes within 5 percentage points of zero (Chen et al., 1 Jul 2026).

The audit formalizes a cross-machine replay criterion: $Apply = \tfrac{N_{apply}}{N_{total}}$1 where $Apply = \tfrac{N_{apply}}{N_{total}}$2 is the runtime of the reference patch and $Apply = \tfrac{N_{apply}}{N_{total}}$3 is the runtime of the base code for replay round $Apply = \tfrac{N_{apply}}{N_{total}}$4 and machine $Apply = \tfrac{N_{apply}}{N_{total}}$5 (Chen et al., 1 Jul 2026). It also defines a replay-variation diagnostic: $Apply = \tfrac{N_{apply}}{N_{total}}$6 (Chen et al., 1 Jul 2026). These diagnostics make the fragility argument quantitative.

The significance of this critique is methodological. If reference patches sit near the decision boundary, then leaderboard differences may reflect machine sensitivity or scoring artifacts rather than robust differences in agent capability (Chen et al., 1 Jul 2026). The audit therefore recommends validating that workloads stress the optimized path strongly enough, ensuring accepted reference patches have clear margins over runtime noise, distinguishing stable from unstable tasks, and avoiding interpretation of a single leaderboard number as a direct capability measure (Chen et al., 1 Jul 2026).

6. Relation to adjacent benchmarks and research directions

SWE-Perf occupies one part of a broader emerging literature on software-engineering benchmarks for non-functional or long-horizon tasks. A closely related benchmark, SWE-fficiency, evaluates repository-level performance optimization on real workloads and contains 498 tasks across nine Python repositories; there the objective is to produce a patch that matches or exceeds expert speedup on a workload while passing relevant unit tests (Ma et al., 8 Nov 2025). Unlike SWE-Perf’s reliance on performance-related unit tests, SWE-fficiency uses workload scripts that are separate from correctness tests and reports that state-of-the-art agents achieve less than $Apply = \tfrac{N_{apply}}{N_{total}}$7 expert speedup on average (Ma et al., 8 Nov 2025). This suggests a neighboring but methodologically distinct benchmark family.

PerfBench addresses real performance bug fixing in .NET repositories and differs more sharply by requiring the agent to generate its own performance benchmark tests, typically using BenchmarkDotNet (Garg et al., 28 Sep 2025). It contains 81 tasks from 32 repositories and reports that a baseline OpenHands agent achieves about a 3% success rate, while OpenHands-Perf-Agent reaches about a 20% success rate after adding performance-aware instructions and benchmark-output processing (Garg et al., 28 Sep 2025). A plausible implication is that benchmark design choices around workload provisioning versus agent-authored benchmarking materially change the task distribution.

SWE-Perf also sits within a wider software-engineering-agent literature emphasizing long-horizon reasoning, interaction budgets, and execution-grounded training. Skywork-SWE argues that repository-level software engineering requires both sustained iterative problem solving over more than 50 interaction rounds and long-context dependency resolution beyond 32k tokens, and reports 38.0% pass@1 on SWE-bench Verified without verifiers or multiple rollouts (Zeng et al., 24 Jun 2025). SWE-Dev similarly argues that training and inference scaling are central bottlenecks for SWE agents and reports 36.6% on SWE-bench-Verified for SWE-Dev-32B (Wang et al., 9 Jun 2025). These works are not performance-optimization benchmarks, but they contextualize why SWE-Perf is difficult: repository-scale optimization inherits the same long-horizon constraints while adding costly, noisy runtime measurement.

At a broader software-engineering level, work on Software Performance Engineering in CI/CD environments argues that SPE has largely become a post hoc activity based on production data, requires frequent re-evaluation of priorities, and demands broader systems expertise than traditional performance engineering (Kudrjavets et al., 2022). This suggests that SWE-Perf’s focus on executable repository contexts and performance-sensitive validation is aligned with the shift from isolated micro-optimization toward operationally grounded performance engineering.

7. Naming ambiguity and interpretive cautions

The name “SWE-Perf” is not unique. An earlier 2021 paper uses SWE-Perf to denote a VSCode extension for interactive static software performance analysis that integrates Infer’s performance results into the IDE through annotations, CodeLenses, side panels, and change tracking (Beigelbeck et al., 2021). That system is unrelated to the 2025 repository-level benchmark: it is an IDE tool for static complexity feedback, not an LLM benchmark (Beigelbeck et al., 2021). In bibliographic and systems discussions, distinguishing the benchmark from the IDE extension is therefore necessary.

Within benchmark interpretation, a second caution concerns what SWE-Perf actually measures. The benchmark is explicitly repository-level and performance-oriented, but its evaluation is mediated by selected performance-related unit tests and significance-aware runtime gains rather than full end-to-end workloads (He et al., 16 Jul 2025). The later audit argues that many reference gains are too small to survive cross-machine replay (Chen et al., 1 Jul 2026). This does not imply that the benchmark is meaningless; rather, it suggests that leaderboard scores should be read as measurements conditioned on workload design, statistical thresholds, and environment stability.

A further misconception is to equate high Apply or Correctness with strong optimization ability. SWE-Perf’s reported results show that model systems can achieve relatively high Apply and Correctness while remaining far below expert Performance (He et al., 16 Jul 2025). This indicates that correctness preservation and mergeability are necessary but insufficient conditions for repository-level optimization competence.

Taken together, these observations place SWE-Perf in a transitional position within LLM evaluation. It establishes a benchmark for repository-level optimization under executable conditions (He et al., 16 Jul 2025), but subsequent analysis shows that its reference signal is fragile and that stronger workload design and clearer performance margins are likely required for more stable capability measurement (Chen et al., 1 Jul 2026).