Uncertainty of Further Optimization without Optimal References

Determine whether the real-world repository codebases targeted by SWE-Perf possess remaining performance optimization potential beyond their current implementations, given the absence of human reference implementations that specify an “optimal” efficiency baseline for comparison.

Background

The paper highlights a core challenge in evaluating repository-level code performance optimization: unlike correctness-focused tasks, there is no established human reference for what constitutes optimal efficiency. This lack of an optimal baseline complicates both measurement and benchmarking of whether additional improvements are achievable.

SWE-Perf addresses evaluation by providing expert-authored patches and reproducible environments to confirm measurable improvements; however, the dataset acknowledges that these patches may not represent true optimal performance. Consequently, determining the existence and extent of further optimization remains an unresolved question central to repository-level performance evaluation.