SWE-fficiency: Repo-Level Performance Benchmark

• SWE-fficiency is a benchmark that evaluates repository-level performance optimization by testing whether agents can localize bottlenecks and generate semantics-preserving patches.
• The benchmark comprises 498 tasks across nine prominent Python libraries and utilizes normalized speedup ratios to compare agent-generated patches against expert references.
• A five-stage automated pipeline, including Docker-based verification and statistical validation, ensures reproducibility and robust evaluation of performance improvements.

Searching arXiv for the benchmark and closely related SWE-efficiency papers to ground the article in the cited literature. SWE-fficiency is a benchmark for evaluating repository-level performance optimization on real workloads. Introduced to assess whether an agent can investigate code semantics, localize bottlenecks and relevant tests, and produce a semantics-preserving patch whose runtime on a slow workload matches or exceeds the speedup achieved by an expert human, it contains 498 tasks across nine widely used Python scientific and data-engineering libraries. Unlike software-engineering benchmarks that emphasize what to fix, SWE-fficiency is explicitly designed to evaluate how to fix code under full-repository context, executable workloads, and repository-owned correctness tests (Ma et al., 8 Nov 2025).

1. Benchmark scope and task model

Each SWE-fficiency task is defined by a repository snapshot at commit $c_i$, a Python performance workload $W_i$, a set of “guard” unit tests $\mathcal{T}_i$, and an expert reference diff. The workload is a standalone script that imports the repository, may define an optional setup(), defines workload(), and measures wall-clock time using timeit.repeat. An agent outputs a unified-diff patch $\Delta_i$ that is evaluated under two simultaneous conditions: every guard test must still pass, and the post-edit runtime must be as low as possible (Ma et al., 8 Nov 2025).

The core runtime quantities are the pre-edit mean runtime $T_{\mathrm{pre}}$, the post-edit mean runtime $T_{\mathrm{post}}$, and the expert post-edit mean runtime $T_{\mathrm{post}}^\ast$. These induce an agent speedup $$S_{\mathrm{agent}} = T_{\mathrm{pre}} / T_{\mathrm{post}}$$ and an expert speedup $$S_{\mathrm{gold}} = T_{\mathrm{pre}} / T_{\mathrm{post}}^\ast$$. Benchmark performance is normalized through the speedup ratio

$$\mathrm{SR}_i = \frac{S_{\mathrm{agent}}}{S_{\mathrm{gold}}} = \frac{T_{\mathrm{post}}^\ast}{T_{\mathrm{post}}},$$

so that $W_i$0 corresponds to parity with the expert reference patch, $W_i$1 means the agent exceeds the expert, and $W_i$2 means it underperforms the expert baseline (Ma et al., 8 Nov 2025).

This formulation is notable because correctness is necessary but not sufficient. A patch that passes tests yet fails to improve runtime remains subpar, while a fast patch that breaks guard tests is invalid. A common misconception is to interpret the benchmark as a standard issue-resolution task; in fact, it is closer to automated performance engineering, where semantics preservation and workload-specific acceleration are coupled objectives.

2. Dataset construction and verification pipeline

The benchmark is built through a five-stage automated pipeline. First, the authors scrape 96,457 merged pull requests from nine repositories up to March 2025 via the GitHub API. Second, they apply performance-oriented filtering: titles or metadata must contain keywords such as “perf,” “benchmark,” or “speedup”; test files must not be modified; and the diff must make a non-trivial change to the AST of at least one source file. This reduces the pool to 9,257 candidates. Third, the surviving candidates are built in pinned Docker images and executed with line coverage; only pull requests with at least one passing test that covers changed lines are retained, yielding 1,041 candidates. Fourth, human experts write minimal workload scripts that reproduce the claimed speedup. Fifth, each expert patch is replayed under pinned CPU cores and 16 GB RAM, and only instances with statistically significant expert speedup are retained, producing the final suite of 498 tasks (Ma et al., 8 Nov 2025).

The repositories are astropy, dask, matplotlib, numpy, pandas, scikit-learn, scipy, sympy, and xarray. They were chosen for performance sensitivity in real workloads, mature test suites, and substantial histories of committed optimizations. The workloads span array broadcasting, statistical fitting, groupby operations, and HPC-style loops, with pre-edit runtimes ranging from a few milliseconds to hundreds of seconds (Ma et al., 8 Nov 2025).

The final verification criterion requires

$W_i$3

where $W_i$4 is the standard deviation of patched workload runs. This is intended to guarantee that expert speedups are statistically significant before the task enters the released benchmark (Ma et al., 8 Nov 2025).

This pipeline makes SWE-fficiency “oracle-free” in the sense described in the source material: tasks are mined from real pull requests and verified by execution rather than being authored solely as synthetic prompts. A plausible implication is that benchmark difficulty is driven not only by code-editing complexity but also by the need to infer profiling-relevant program structure from repository context.

3. Evaluation rule and score geometry

SWE-fficiency aggregates task-level performance through the harmonic mean

$W_i$5

This choice heavily penalizes failures on individual tasks. If an agent submits an empty patch or a failing patch, the benchmark assigns $W_i$6, which is strictly worse than making even a small valid improvement (Ma et al., 8 Nov 2025).

In addition to $W_i$7 and $W_i$8, the benchmark reports three auxiliary statistics: percentage of tests passing after agent patches, percentage faster than baseline, and percentage exceeding expert. The speedup ratio is explicitly described as a factor-based metric anchored at human parity, so it rewards partial progress below expert level and continues rewarding improvement above expert level (Ma et al., 8 Nov 2025).

The later audit by Chen et al. clarifies the exact leaderboard rule used in public evaluation. Per-task ratios are floored at $W_i$9,

$\mathcal{T}_i$0

and the final score is the harmonic mean over all $\mathcal{T}_i$1 tasks: $\mathcal{T}_i$2 The same audit defines the denominator contribution of a task as

$\mathcal{T}_i$3

which makes clear why very poor tasks can dominate aggregate ranking (Chen et al., 1 Jul 2026).

A common misconception is that a single benchmark score distributes influence roughly evenly across tasks. The audit shows that this is not the case for SWE-fficiency’s public leaderboard rule: because the harmonic mean amplifies near-zero ratios, a small number of badly solved tasks can dominate the overall score (Chen et al., 1 Jul 2026).

4. Empirical findings and observed failure modes

The initial empirical evaluation uses two open-source agent harnesses, OpenHands and SWE-agent, with models from OpenAI, Anthropic, Google DeepMind, Z.ai, Moonshot AI, and DeepSeek. The reported outcome is that the top agent, GPT-5, reaches only 0.15× of expert speedup, while most models achieve less than 0.05× of expert speedup. Roughly 20–40% of submitted patches introduce test failures. Among correct patches, only 30–45% run faster than the pre-edit code, and fewer than 10–15% beat the expert reference patch (Ma et al., 8 Nov 2025).

The benchmark paper attributes this underperformance to several recurrent failure modes. In bottleneck mislocalization, agents edit the wrong file or function; a flamegraph-based study reports that about 70% of expert speedup mass occurs in functions untouched by the agent. In cross-function reasoning failures, agents struggle to follow data flow through multiple modules. Under shortcut bias, agents prefer narrow fast paths such as ad hoc early exits or workload-specific caching rather than the expert’s algorithmic rewrite. Correctness drift appears when patches overfit the measured workload or exploit test-harness artifacts. Maintainability problems arise because generated diffs are often large and invasive rather than concise and composable (Ma et al., 8 Nov 2025).

These findings distinguish performance optimization from conventional repository repair. Passing tests is only a partial indicator of success, since an optimization patch must also improve measured runtime under the benchmark’s workload. The source material describes this as a gap in “long-horizon software reasoning,” especially when the required change spans bottleneck localization, multi-module execution reasoning, and non-trivial algorithmic edits (Ma et al., 8 Nov 2025).

A later benchmark, SWE-Pro, reinforces this diagnosis from a different angle. It evaluates 102 expert-written optimizations under runtime, peak memory, and Time-Weighted Memory Usage, and reports that current LLMs show negligible runtime gains and nearly non-existent memory optimizations, whereas expert implementations achieve an aggregate speedup of 15.5x and peak memory reduction of 171.3x (Sarıkayak et al., 24 Jun 2026). This suggests that SWE-fficiency’s difficulty is not an isolated artifact of one leaderboard, but part of a broader deficiency in automated performance engineering.

5. Reliability, replay stability, and leaderboard interpretation

Chen et al. audited repository-level performance-optimization benchmarks, including SWE-fficiency, by replaying official reference patches for 740 tasks across four Google Cloud CPU profiles: n2-standard-64, n2d-standard-64, n4-standard-64, and n4d-standard-64. For SWE-fficiency specifically, all 498 tasks compile and run on each machine profile. However, only 470/498 tasks satisfy the “faster-than-base” check in every machine-round, and only 411/498 tasks satisfy the benchmark’s original-rule validity in every cross-machine replay (Chen et al., 1 Jul 2026).

The audit identifies three main causes of instability. Some reference patches become slower than the base code on at least one machine. Tasks with small speedup margins lose statistical validity under cross-machine noise. A handful fail due to build problems, missing dependencies, or CPU-specific instructions (Chen et al., 1 Jul 2026).

The same study shows that ranking sensitivity is not merely a measurement problem but also a scoring-rule problem. Among eight public submissions shared with GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise comparisons. Moreover, for SWE-fficiency, the worst ten tasks account for 58.5%–82.8% of the total denominator weight. The audit also reports that raising the floor from 0.001 to 0.5 reshuffles 6 of 8 SWE ranks (Chen et al., 1 Jul 2026).

These results do not invalidate the benchmark, but they do constrain interpretation. A plausible implication is that aggregate leaderboard scores should be read together with per-task signal strength, replay validity, and task-weight concentration. The audit explicitly recommends identifying “stable” tasks with large reference-patch speedups and low cross-machine variance, surfacing per-task score weights, and designing scoring rules with bounded penalties so that a single outlier failure cannot overwhelm aggregate progress (Chen et al., 1 Jul 2026).

6. Broader uses of “SWE-fficiency” in adjacent literature

Closely related literature uses “SWE-fficiency” to denote resource-adjusted evaluation and computational efficiency in software-engineering agents, rather than only repository-level performance optimization. SWE-Effi introduces four resource-adjusted effectiveness scores—Effectiveness under Token Budget, Cost Budget, CPU-Time Budget, and Inference-Time Budget—and defines each as the normalized area under a cumulative success curve up to a resource cap. It also identifies the “token snowball” effect, where cumulative token use grows like $\mathcal{T}_i$4, and “expensive failures,” where unresolved issues consume far more resources than resolved ones (Fan et al., 11 Sep 2025).

SWEnergy studies energy efficiency when four agentic issue-resolution frameworks are constrained to small LLMs. On fixed hardware, it measures energy, duration, token usage, and memory over 150 runs per configuration, finding that framework architecture is the primary driver of energy consumption. AutoCodeRover with Gemma consumes 9.4x more energy on average than OpenHands with Gemma, yet task resolution rates are near-zero except for AutoCodeRover+Qwen at 4%, indicating that much of the energy is spent in unproductive reasoning loops (Tripathy et al., 10 Dec 2025).

Several agent-design papers address efficiency bottlenecks that are also relevant to long-horizon performance tasks. SWE-AGILE manages reasoning context through a sliding window of full traces plus compressed digests, reporting approximately 820 tokens per step versus approximately 1,075 for Current-Step Thinking and a global compression rate of approximately 55.7% after SFT (Lian et al., 13 Apr 2026). SWE-Edit decomposes editing into Viewer and Editor subagents, improving resolved rate by 2.1% while reducing inference cost by 17.9% on SWE-bench Verified (Zhang et al., 28 Apr 2026). SWE-Replay reuses archived trajectories during test-time scaling and reports cost reductions of up to 17.4% while maintaining or improving performance (Ding et al., 29 Jan 2026). SWE-TRACE combines shortest-path-style supervised traces, rubric-based process reward models, and heuristic-guided test-time scaling, reporting lower latency and fewer environment executions than parallel scaling at comparable budgets (Han et al., 16 Apr 2026).

Taken together, these adjacent uses indicate that “SWE-fficiency” has acquired a broader meaning in the literature: not only optimizing repository runtime, but also measuring and improving how economically software-engineering agents consume tokens, time, compute, memory, and energy. The benchmark named SWE-fficiency remains the most direct repository-level test of real-world optimization ability, while the neighboring works define the efficiency axes on which future agents may need to improve to perform well on such tasks (Ma et al., 8 Nov 2025).