Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

Abstract: Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.

• The paper critically assesses benchmark reliability by replaying 740 reference patches across multiple hardware platforms to identify fragilities in performance measurements.
• It reveals that only a small fraction of tasks maintain consistent performance gains across environments, with metrics like 39/102 tasks passing GSO's strict criteria, highlighting sensitivity to noise.
• The study shows that leaderboard rankings are heavily influenced by aggregation methods, urging designers to adopt robust scoring metrics to better capture genuine coding agent improvements.

Assessing Reliability in Performance-Optimization Benchmarks for Coding Agents

Introduction

The increasing deployment of coding agents for software performance optimization has motivated rapid construction and adoption of repository-level evaluation benchmarks such as GSO, SWE-Perf, and SWE. These benchmarks apply candidate patches to real-world repositories and assess runtime improvements relative to unoptimized baselines and carefully curated reference patches, with their leaderboards routinely used to substantiate claims of agent progress. This paper rigorously interrogates the reliability and interpretability of such benchmark-based evaluations, scrutinizing the validity of reference patches under cross-platform execution, the sensitivity of aggregate scoring to metric definitions, and the extent to which leaderboard coverage masks remaining optimization gaps among state-of-the-art agent submissions (2607.01211).

Cross-Machine Reference-Patch Validity

Despite benchmarks’ focus on task- and code-level validity, their underlying assumption—that reference patches yield consistent, measurable speedup across close-but-different compute environments—remains largely unevaluated. To address this, the study systematically replays 740 official reference patches from GSO, SWE-Perf, and SWE-fficiency across four modern Google Cloud CPU configurations (Intel and AMD, 2019–2025 generations) and three rounds per environment, assessing both raw "faster-than-base" status and benchmark-specific original inclusion criteria.

The vast majority of tasks are technically replayable across all platforms. However, the stricter check—requiring every reference patch to satisfy its benchmark’s original performance validity criterion in all 12 replay combinations—substantially shrinks the stable core: only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE tasks survive.

Figure 1: Task counts after each replay check. Counts require passing all 12 machine-round replays (four machines across three rounds).

The failures are primarily not due to gross infrastructural mismatches, but rather to the small signal margin characterizing many reference patches, particularly in SWE-Perf. Runtime reductions for valid patches in GSO and SWE typically exceed 50%, while in SWE-Perf, the median change is $-0.03\%$, allowing trivial timing noise or platform effects to erase statistical support for reference patch superiority.

Figure 2: Reference-patch runtime-change distributions by machine and replay round. Negative values indicate runtime reduction; SWE-Perf clusters near zero change, exhibiting fragility to minor noise.

This demonstrates that the underlying performance signal—rather than just machine-level "noise"—limits benchmark portability and interpretability. SWE-Perf's methodology, which leverages existing project unit tests and accepts marginal (>5%) efficiency gains, produces tasks highly sensitive to environmental fluctuations, whereas GSO and SWE employ dedicated workloads and more robust construction.

Benchmark Scoring Rules and Leaderboard Interpretation

The transformation from per-task outcomes to final leaderboard scores varies sharply across benchmarks. GSO employs a binary "reference-level success" rule—crediting only patches that exactly match or beat the reference speedup, with equal weight for each task. In contrast, SWE computes a harmonic mean of relative speedup ratios (subject to aggressive flooring of underperforming submissions at $0.001$), amplifying the effect of a handful of low-performing tasks.

The reported rankings are highly sensitive to these aggregation choices. For eight public agent submissions evaluated on both GSO and SWE, rank order agrees in only 19/28 pairwise comparisons; for example, the GPT-5 model moves five positions depending on the metric.

More importantly, SWE’s harmonic mean aggregation results in extreme tail dominance: the worst single task may carry up to 33.6% of the score weight, and the ten worst together can account for 58.5%–82.8%, effectively allowing near-failure on a few testcases to overshadow broad, consistent improvements.

Figure 3: Score weight for the worst 1, 5, and 10 tasks under SWE’s harmonic mean, showing extreme sensitivity to low-speedup outcomes.

Applying a bounded penalty diagnostic—where the maximum penalty for a single slow task is capped to match the maximal possible positive contribution—materially reshuffles leaderboard orderings, sometimes flipping 8/28 head-to-head comparisons among top models. Thus, conclusions about agent superiority are inseparable from implicit or explicit penalties inherent in the scoring function; broad generalization on "agent quality" is not warranted absent careful, context-matched metric design.

Coverage and Residual Gaps in Public Agent Solutions

Consideration of leaderboard rankings in isolation easily leads to an over- or underestimation of technological headroom. The study analyzes outputs of 10 public agent submissions per task on all replay-valid GSO and SWE tasks (39 and 411, respectively). Coverage is already extensive: all 450 tasks are solved by at least one public submission (correct patch passing functionality checks), 449/450 have a patch outperforming base code, and 384/450 are matched or surpassed at reference-patch speedup level.

Figure 4: Task outcomes across 10 public submissions for replay-valid tasks—test-passing and >base speedup are nearly saturated; remaining frontier is closing the reference-patch gap.

The residual set of 66 tasks not reaching the reference speed are rarely cases of outright failure. On these, the best public patch achieves a median of 85.3% (GSO) or 87.9% (SWE) of the reference speedup. Only one task has a best solution that fails to improve over base runtime. The majority of these gaps are narrow: 27 reach 90–100% of the reference.

Figure 5: Best public patch speed relative to the reference patch for the 66 tasks still trailing reference performance.

In-depth strategy analysis (using automated patch categorization) reveals that while 32/66 best patches align with the reference patch’s high-level optimization strategy, 34 do not. However, employing the benchmark-designated strategy class is neither necessary nor sufficient for closing the speed gap: substantial overlap exists between "aligned" and "unaligned" best-patch groups around the 90% threshold. This underscores that the remaining performance delta is driven by implementation detail and optimization depth rather than lack of problem comprehension or strategic orientation.

Figure 6: Distribution of best public/reference speedup as grouped by high-level category alignment between the public and reference patches.

Figure 7: Distribution of best public/reference speedup by high-level strategy alignment; same-category patches often still underperform the reference.

Implications and Future Directions

The presented findings necessitate nuanced interpretation and inform recommendations across the agent benchmarking ecosystem:

• Benchmark users must decouple "signal" (genuine, robust speedup) from "measurement design" (e.g., patch fragility, scoring-artifact dominance). Reliance on aggregate scores or ranks alone is inadequate; per-task verification, signal robustness (cross-machine stability), and scoring function sensitivity must be foregrounded.
• Agent builders should interpret benchmark improvements in terms of incremental coverage of the remaining hard core, not overall leaderboard ascension. As nearly all current replay-valid tasks are already covered by at least one public submission, further metric gains will be contingent on closing narrow performance gaps, not solving fundamentally unaddressed tasks.
• Benchmark designers are advised to engineer future repository-level benchmarks with stronger reference signals, careful workload construction, and appropriately risk-balanced aggregation metrics. There is also a need to approach the full spectrum of performance engineering, moving beyond patching known hot spots to incorporating profile- or trace-driven localization, ambiguous optimization targets, comprehensive resource metrics (runtime, memory, allocation, regression potential), and test scenarios where the agent must both discover the bottleneck and validate a nuanced improvement.

Conclusion

This audit reveals that current repository-level performance-optimization benchmarks for coding agents suffer from reference patch fragility under environmental perturbations and score aggregation rules that may mask the locus and depth of coding-agent innovation. Most replay-valid tasks are already solved at or near reference-patch performance by leading submissions. Remaining performance gaps are typically implementation-level and subtle, not fundamental or strategic. As such, future benchmarks must present more challenging, diagnostically rich tasks and adopt scoring methodologies that more faithfully reflect generalizable performance engineering skill rather than overemphasizing rare tail failures or marginal signals (2607.01211).