SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (2511.06090v2)

Abstract: Optimizing the performance of large-scale software repositories demands expertise in code reasoning and software engineering (SWE) to reduce runtime while preserving program correctness. However, most benchmarks emphasize what to fix rather than how to fix code. We introduce SWE-fficiency, a benchmark for evaluating repository-level performance optimization on real workloads. Our suite contains 498 tasks across nine widely used data-science, machine-learning, and HPC repositories (e.g., numpy, pandas, scipy): given a complete codebase and a slow workload, an agent must investigate code semantics, localize bottlenecks and relevant tests, and produce a patch that matches or exceeds expert speedup while passing the same unit tests. To enable this how-to-fix evaluation, our automated pipeline scrapes GitHub pull requests for performance-improving edits, combining keyword filtering, static analysis, coverage tooling, and execution validation to both confirm expert speedup baselines and identify relevant repository unit tests. Empirical evaluation of state-of-the-art agents reveals significant underperformance. On average, agents achieve less than 0.15x the expert speedup: agents struggle in localizing optimization opportunities, reasoning about execution across functions, and maintaining correctness in proposed edits. We release the benchmark and accompanying data pipeline to facilitate research on automated performance engineering and long-horizon software reasoning.

• The paper introduces a systematic benchmark to evaluate language models on performance-oriented, codebase-level optimizations using real workloads and expert reference patches.
• The paper details a multi-stage pipeline that filters performance-relevant PRs, employs coverage-based test selection, and calculates a speedup ratio to assess improvements.
• The paper reveals that current language models underperform experts, largely due to issues in function-level localization and scalability, pointing to directions for agentic AI advancements.

SWE-fficiency: Benchmarking LLM Capability in Real-World Software Performance Optimization

Benchmark Motivation and Design

SWE-fficiency introduces a systematic benchmark targeting performance-oriented codebase-level optimization in popular data-science, ML, and HPC Python repositories. Unlike prior benchmarks focused on bug-fixing or function completion, SWE-fficiency emphasizes how LMs modify software to reduce runtime on realistic workloads, while strictly maintaining functionality as measured by repository test suites.

Agents are presented a full codebase and an explicit slow workload. The requirements demand end-to-end investigative workflows: identifying bottlenecks, locating relevant correctness tests via coverage, editing the codebase, and verifying achieved speedups, with real PRs from nine major repositories serving as ground truth (Figure 1).

Figure 1: SWE-fficiency evaluates the investigative, pass-to-pass workflow of performance engineering: agents must speed up real workloads in real codebases while preserving correctness enforced by repo-specific unit tests.

Data collection leverages a multi-stage pipeline: filtering PRs for performance relevance, ensuring no introduction of new behaviors, coverage-based selection of guarding tests, and manual workload annotation grounded in domain context (Figure 2). This pipeline produces 498 well-structured optimization tasks with expert-provided reference patches.

Figure 2: SWE-fficiency's task generation pipeline prunes, annotates, and verifies candidate optimization PRs, yielding instances with reproducible workloads and expert patches.

Dataset Properties and Task Evaluation

The task instances span diverse optimization types—algorithmic improvements, memory management, parallelization, and caching (Figure 3). Workloads vary substantially in runtime and complexity, and expert patches yield speedups ranging up to orders of magnitude, with substantial edits required for difficult tasks. An LM's success is scored by the speedup ratio (SR): the speedup achieved relative to the expert's patch, aggregated via the harmonic mean across instances.

Figure 3: SWE-fficiency shows substantial variation in workload runtime, expert patch speedup, and expert optimization types, establishing the benchmark's diversity.

Performance evaluation is hardened by requiring that optimized code passes all pre-existing relevant tests—strictly measuring regression-free improvement. Tasks are open-ended: agents may submit any patch, not just mimic the expert, encouraging creative but verifiable solutions.

LLM Performance Analysis

Empirical results show frontier LMs underperform experts by a wide margin: the highest SR observed was 0.15×; most agents cluster below 0.05× for the full benchmark (Table 1 from the paper). Agents frequently introduce correctness bugs or deliver only superficial speedups. Notably, easier instances (short runtimes, smaller patch edits, lower expert speedup) are solved more successfully, but LM performance deteriorates rapidly with task complexity (Figure 4).

Figure 4: LMs achieve strong results on easy tasks but struggle with increasing workload duration and larger expert speedups.

Task trajectories confirm that when LMs do find strong optimizations, they converge early. Otherwise, agents tend to stop after achieving any noticeable win, seldom iterating toward expert-level improvements (Figure 5).

Figure 5: LMs frequently find sub-expert optimizations and terminate without pursuing parity; expert-level wins are obtained early in action trajectories.

A key finding is function-level mislocalization (Figure 6): over 68% of speedup mass in the dataset comes from functions that agents do not touch. Models often alter the right files but miss optimizing actual bottleneck routines (Jaccard file overlap ∼0.55, function-level Expert-Relative Coverage ∼0.28–0.31 as detailed in quantitative metrics).

Figure 6: Example flamegraph—LM modifies deep function but misses expert's higher-impact routine, forfeiting major speedup.

Qualitative analysis reveals shortcut bias (e.g., ad-hoc caching, fast-paths triggering only under specific input conditions) and overfitting to properties detectable in the workload, which rarely generalize. Generated edits are less maintainable, with frequent reliance on module-level state or brittle assumptions. In contrast, expert patches typically restructure code for lower per-element cost or exploit backend optimizations (Cython, vectorization).

Dataset Utility, Technical Challenges, and Scaling

SWE-fficiency's evaluation environment is robustly containerized, enforces resource isolation (CPU/memory pinning), and employs memory-isolated measurement to prevent reward hacking (as detailed in the appendices). The benchmark's design—real repo optimization, separated correctness and performance evaluation, coverage-based test selection—establishes a reproducible and hard baseline for LM-driven performance engineering.

SWE-fficiency also exposes the limits of synthetic workload generation: only 53% of LM-generated workloads show statistically significant performance deltas compared to manual annotation, underlining the need for expert-guided data collection for benchmarking.

Scaling considerations suggest bottlenecks remain: longer workloads, more complex library stacks (e.g., C++, Rust), and hardware-heterogeneous evaluation would demand further advances in agentic planning and systems-level reasoning.

Comparison with Prior Art

Relative to prior benchmarks (SWE-bench, PIE, EffiBench, Mercury, GSO, SWE-Perf, etc.), SWE-fficiency emphasizes:

• Repository-scale, pass-to-pass optimization (no new behavior).
• Agent discovery of both optimization opportunities and relevant correctness-verifying tests.
• Realistic, manually curated workloads, as opposed to purely synthetic or LM-generated tests.
• Factor-based speedup ratio metric, which supports long-term progress and expert/superhuman parity comparison.
• Containerized environments for robust evaluation.

Implications and Future Directions

The substantial gap between LMs and expert performance in real-world optimization tasks reflects deficiencies in cross-function semantic reasoning, bottleneck localization, and principled system-level modification. Closing this gap likely requires integrating explicit profiling, symbolic execution, planning over long code trajectories, and harnesses that support deeper exploration. Advancements in cross-module understanding and multi-agent collaboration may improve localization and robust optimization.

In practical terms, SWE-fficiency is poised to drive agentic AI development for codebases where efficiency is business-critical, e.g., numerical computing, ML libraries, and large-scale data infrastructure. The metric design and open-ended evaluation framework are suitable for tracking future progress toward autonomous software performance engineers and cost-saving deployment at scale.

Conclusion

SWE-fficiency establishes a rigorous dataset and methodology for evaluating LLMs in the context of real-world software performance optimization. Despite strong results on trivial instances, current LMs fall short in function-level localization and scalable optimization, leaving extensive room for progress. SWE-fficiency's benchmark, analysis techniques, and evaluation harness provide a foundation for future systems seeking to bridge the expertise gap in automated performance engineering and long-horizon codebase reasoning.