---
title: 'SWE-fficiency: Repo-Level Performance Benchmark'
url: https://www.emergentmind.com/topics/swe-fficiency
type: topic
---

# SWE-fficiency: Repo-Level Performance Benchmark

Searching arXiv for the benchmark and closely related SWE-efficiency papers to ground the article in the cited literature.
SWE-fficiency is a benchmark for evaluating repository-level performance optimization on real workloads. Introduced to assess whether an agent can investigate code semantics, localize bottlenecks and relevant tests, and produce a semantics-preserving patch whose runtime on a slow workload matches or exceeds the speedup achieved by an expert human, it contains 498 tasks across nine widely used Python scientific and data-engineering libraries. Unlike software-engineering benchmarks that emphasize what to fix, SWE-fficiency is explicitly designed to evaluate how to fix code under full-repository context, executable workloads, and repository-owned correctness tests [2511.06090].

## 1. Benchmark scope and task model

Each SWE-fficiency task is defined by a repository snapshot at commit \(c_i\), a Python performance workload \(W_i\), a set of “guard” unit tests \(\mathcal{T}_i\), and an expert reference diff. The workload is a standalone script that imports the repository, may define an optional `setup()`, defines `workload()`, and measures wall-clock time using `timeit.repeat`. An agent outputs a unified-diff patch \(\Delta_i\) that is evaluated under two simultaneous conditions: every guard test must still pass, and the post-edit runtime must be as low as possible [2511.06090].

The core runtime quantities are the pre-edit mean runtime \(T_{\mathrm{pre}}\), the post-edit mean runtime \(T_{\mathrm{post}}\), and the expert post-edit mean runtime \(T_{\mathrm{post}}^\ast\). These induce an agent speedup \(S_{\mathrm{agent}} = T_{\mathrm{pre}} / T_{\mathrm{post}}\) and an expert speedup \(S_{\mathrm{gold}} = T_{\mathrm{pre}} / T_{\mathrm{post}}^\ast\). Benchmark performance is normalized through the speedup ratio
\[
\mathrm{SR}_i = \frac{S_{\mathrm{agent}}}{S_{\mathrm{gold}}}
= \frac{T_{\mathrm{post}}^\ast}{T_{\mathrm{post}}},
\]
so that \(\mathrm{SR}_i = 1\) corresponds to parity with the expert reference patch, \(\mathrm{SR}_i > 1\) means the agent exceeds the expert, and \(\mathrm{SR}_i < 1\) means it underperforms the expert baseline [2511.06090].

This formulation is notable because correctness is necessary but not sufficient. A patch that passes tests yet fails to improve runtime remains subpar, while a fast patch that breaks guard tests is invalid. A common misconception is to interpret the benchmark as a standard issue-resolution task; in fact, it is closer to automated performance engineering, where semantics preservation and workload-specific acceleration are coupled objectives.

## 2. Dataset construction and verification pipeline

The benchmark is built through a five-stage automated pipeline. First, the authors scrape 96,457 merged pull requests from nine repositories up to March 2025 via the GitHub API. Second, they apply performance-oriented filtering: titles or metadata must contain keywords such as “perf,” “benchmark,” or “speedup”; test files must not be modified; and the diff must make a non-trivial change to the AST of at least one source file. This reduces the pool to 9,257 candidates. Third, the surviving candidates are built in pinned Docker images and executed with line coverage; only pull requests with at least one passing test that covers changed lines are retained, yielding 1,041 candidates. Fourth, human experts write minimal workload scripts that reproduce the claimed speedup. Fifth, each expert patch is replayed under pinned CPU cores and 16 GB RAM, and only instances with statistically significant expert speedup are retained, producing the final suite of 498 tasks [2511.06090].

The repositories are astropy, dask, matplotlib, numpy, pandas, scikit-learn, scipy, sympy, and xarray. They were chosen for performance sensitivity in real workloads, mature test suites, and substantial histories of committed optimizations. The workloads span array broadcasting, statistical fitting, groupby operations, and HPC-style loops, with pre-edit runtimes ranging from a few milliseconds to hundreds of seconds [2511.06090].

The final verification criterion requires
\[
T_{\mathrm{pre}} - T_{\mathrm{post}}^\ast > 2\sigma_{\mathrm{post}^\ast},
\]
where \(\sigma_{\mathrm{post}^\ast}\) is the standard deviation of patched workload runs. This is intended to guarantee that expert speedups are statistically significant before the task enters the released benchmark [2511.06090].

This pipeline makes SWE-fficiency “oracle-free” in the sense described in the source material: tasks are mined from real pull requests and verified by execution rather than being authored solely as synthetic prompts. A plausible implication is that benchmark difficulty is driven not only by code-editing complexity but also by the need to infer profiling-relevant program structure from repository context.

## 3. Evaluation rule and score geometry

SWE-fficiency aggregates task-level performance through the harmonic mean
\[
\mathrm{SR}_{\mathrm{agg}}
=
\left(\frac{1}{N}\sum_{i=1}^{N}\frac{1}{\mathrm{SR}_i}\right)^{-1}.
\]
This choice heavily penalizes failures on individual tasks. If an agent submits an empty patch or a failing patch, the benchmark assigns \(\mathrm{SR}_i = 1/S_{\mathrm{gold}}\), which is strictly worse than making even a small valid improvement [2511.06090].

In addition to \(\mathrm{SR}_i\) and \(\mathrm{SR}_{\mathrm{agg}}\), the benchmark reports three auxiliary statistics: percentage of tests passing after agent patches, percentage faster than baseline, and percentage exceeding expert. The speedup ratio is explicitly described as a factor-based metric anchored at human parity, so it rewards partial progress below expert level and continues rewarding improvement above expert level [2511.06090].

The later audit by Chen et al. clarifies the exact leaderboard rule used in public evaluation. Per-task ratios are floored at \(f=0.001\),
\[
\widehat{\mathrm{SR}}_{i,t} = \max(\mathrm{SR}_{i,t}, 0.001),
\]
and the final score is the harmonic mean over all \(T=498\) tasks:
\[
\mathrm{Score}(i)
=
\frac{T}{\sum_{t=1}^{T} \frac{1}{\widehat{\mathrm{SR}}_{i,t}}}.
\]
The same audit defines the denominator contribution of a task as
\[
w_t
=
\frac{1/\widehat{\mathrm{SR}}_{i,t}}
{\sum_{j=1}^{T} 1/\widehat{\mathrm{SR}}_{i,j}},
\]
which makes clear why very poor tasks can dominate aggregate ranking [2607.01211].

A common misconception is that a single benchmark score distributes influence roughly evenly across tasks. The audit shows that this is not the case for SWE-fficiency’s public leaderboard rule: because the harmonic mean amplifies near-zero ratios, a small number of badly solved tasks can dominate the overall score [2607.01211].

## 4. Empirical findings and observed failure modes

The initial empirical evaluation uses two open-source agent harnesses, OpenHands and SWE-agent, with models from OpenAI, Anthropic, Google DeepMind, Z.ai, Moonshot AI, and DeepSeek. The reported outcome is that the top agent, GPT-5, reaches only 0.15× of expert speedup, while most models achieve less than 0.05× of expert speedup. Roughly 20–40% of submitted patches introduce test failures. Among correct patches, only 30–45% run faster than the pre-edit code, and fewer than 10–15% beat the expert reference patch [2511.06090].

The benchmark paper attributes this underperformance to several recurrent failure modes. In bottleneck mislocalization, agents edit the wrong file or function; a flamegraph-based study reports that about 70% of expert speedup mass occurs in functions untouched by the agent. In cross-function reasoning failures, agents struggle to follow data flow through multiple modules. Under shortcut bias, agents prefer narrow fast paths such as ad hoc early exits or workload-specific caching rather than the expert’s algorithmic rewrite. Correctness drift appears when patches overfit the measured workload or exploit test-harness artifacts. Maintainability problems arise because generated diffs are often large and invasive rather than concise and composable [2511.06090].

These findings distinguish performance optimization from conventional repository repair. Passing tests is only a partial indicator of success, since an optimization patch must also improve measured runtime under the benchmark’s workload. The source material describes this as a gap in “long-horizon software reasoning,” especially when the required change spans bottleneck localization, multi-module execution reasoning, and non-trivial algorithmic edits [2511.06090].

A later benchmark, SWE-Pro, reinforces this diagnosis from a different angle. It evaluates 102 expert-written optimizations under runtime, peak memory, and Time-Weighted Memory Usage, and reports that current LLMs show negligible runtime gains and nearly non-existent memory optimizations, whereas expert implementations achieve an aggregate speedup of 15.5x and peak memory reduction of 171.3x [2606.25530]. This suggests that SWE-fficiency’s difficulty is not an isolated artifact of one leaderboard, but part of a broader deficiency in automated performance engineering.

## 5. Reliability, replay stability, and leaderboard interpretation

Chen et al. audited repository-level performance-optimization benchmarks, including SWE-fficiency, by replaying official reference patches for 740 tasks across four Google Cloud CPU profiles: n2-standard-64, n2d-standard-64, n4-standard-64, and n4d-standard-64. For SWE-fficiency specifically, all 498 tasks compile and run on each machine profile. However, only 470/498 tasks satisfy the “faster-than-base” check in every machine-round, and only 411/498 tasks satisfy the benchmark’s original-rule validity in every cross-machine replay [2607.01211].

The audit identifies three main causes of instability. Some reference patches become slower than the base code on at least one machine. Tasks with small speedup margins lose statistical validity under cross-machine noise. A handful fail due to build problems, missing dependencies, or CPU-specific instructions [2607.01211].

The same study shows that ranking sensitivity is not merely a measurement problem but also a scoring-rule problem. Among eight public submissions shared with GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise comparisons. Moreover, for SWE-fficiency, the worst ten tasks account for 58.5%–82.8% of the total denominator weight. The audit also reports that raising the floor from 0.001 to 0.5 reshuffles 6 of 8 SWE ranks [2607.01211].

These results do not invalidate the benchmark, but they do constrain interpretation. A plausible implication is that aggregate leaderboard scores should be read together with per-task signal strength, replay validity, and task-weight concentration. The audit explicitly recommends identifying “stable” tasks with large reference-patch speedups and low cross-machine variance, surfacing per-task score weights, and designing scoring rules with bounded penalties so that a single outlier failure cannot overwhelm aggregate progress [2607.01211].

## 6. Broader uses of “SWE-fficiency” in adjacent literature

Closely related literature uses “SWE-fficiency” to denote resource-adjusted evaluation and computational efficiency in software-engineering agents, rather than only repository-level performance optimization. SWE-Effi introduces four resource-adjusted effectiveness scores—Effectiveness under Token Budget, Cost Budget, CPU-Time Budget, and Inference-Time Budget—and defines each as the normalized area under a cumulative success curve up to a resource cap. It also identifies the “token snowball” effect, where cumulative token use grows like \(O(n^2)\), and “expensive failures,” where unresolved issues consume far more resources than resolved ones [2509.09853].

SWEnergy studies energy efficiency when four agentic issue-resolution frameworks are constrained to small language models. On fixed hardware, it measures energy, duration, token usage, and memory over 150 runs per configuration, finding that framework architecture is the primary driver of energy consumption. AutoCodeRover with Gemma consumes 9.4x more energy on average than OpenHands with Gemma, yet task resolution rates are near-zero except for AutoCodeRover+Qwen at 4%, indicating that much of the energy is spent in unproductive reasoning loops [2512.09543].

Several agent-design papers address efficiency bottlenecks that are also relevant to long-horizon performance tasks. SWE-AGILE manages reasoning context through a sliding window of full traces plus compressed digests, reporting approximately 820 tokens per step versus approximately 1,075 for Current-Step Thinking and a global compression rate of approximately 55.7% after SFT [2604.11716]. SWE-Edit decomposes editing into Viewer and Editor subagents, improving resolved rate by 2.1% while reducing inference cost by 17.9% on SWE-bench Verified [2604.26102]. SWE-Replay reuses archived trajectories during test-time scaling and reports cost reductions of up to 17.4% while maintaining or improving performance [2601.22129]. SWE-TRACE combines shortest-path-style supervised traces, rubric-based process reward models, and heuristic-guided test-time scaling, reporting lower latency and fewer environment executions than parallel scaling at comparable budgets [2604.14820].

Taken together, these adjacent uses indicate that “SWE-fficiency” has acquired a broader meaning in the literature: not only optimizing repository runtime, but also measuring and improving how economically software-engineering agents consume tokens, time, compute, memory, and energy. The benchmark named SWE-fficiency remains the most direct repository-level test of real-world optimization ability, while the neighboring works define the efficiency axes on which future agents may need to improve to perform well on such tasks [2511.06090].

Source: https://www.emergentmind.com/topics/swe-fficiency