PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

Abstract: SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.

• The paper identifies PR–issue misalignment as a construction defect, finding that 13.6% of SWE-bench Verified tasks and 44.3% of SWE-Gym tasks are misaligned.
• PAIChecker combines specialized artifact analyzers, coordinator-based label synthesis, and code validation, achieving up to 92.12% binary accuracy and 90.67% exact match across benchmark datasets.
• The findings show that misaligned tasks distort leaderboard rankings—64.1% of evaluated agents changed rank after their removal—supporting systematic audits of benchmark and training-data quality.

Problem and motivation

SWE-bench and its derivatives (SWE-bench Verified, SWE-Gym, SWE-bench Multilingual, SWE-PolyBench, SWE-Smith, SWE-Bench-Live) share a common construction pipeline: a PR is paired with an issue via regex extraction of issue references from the PR description; the issue text becomes the problem statement, and the PR patch serves as the test oracle. The paper identifies a critical assumption underlying this pipeline — that each PR exclusively and completely addresses its linked issue — and shows it is frequently violated. The consequences are twofold: evaluation instances may be unfair or unsolvable from the stated problem alone, and training datasets built on the same pairing inherit noisy supervision that encourages models to learn incomplete or defective patches.

Empirical study: taxonomy and prevalence

The authors manually annotate all 500 SWE-bench Verified instances via open coding, deriving a taxonomy of five misalignment patterns across 11 fine-grained scenarios, and find that 13.6% (68/500) of instances are misaligned despite Verified's human curation:

Pattern	Description	Share of misaligned
Defective PR (DP)	PR introduces bugs or an incomplete fix	44.1%
PR Scope Creep (SC)	PR resolves multiple issues or adds unstated features	32.4%
Incomplete Specification (IS)	Details supplied or revised in later discussion	26.5%
Unspecified Literal (UL)	Tests assert exact literals absent from the issue	low
Follow-up PR (FP)	PR supplements/repairs an earlier PR for the same issue	low

The IS criterion is deliberately conservative: an issue is flagged only when a maintainer explicitly requests missing components or redefines expected behavior, grounding the judgment in community consensus rather than annotator subjectivity.

Impact on evaluation reliability

Correlating misalignment labels with per-instance resolution data from 131 leaderboard agents, the paper reports a monotonic trend: 41.2% of the 34 never-resolved instances are misaligned, versus 5.7% among instances resolved by more than 100 agents. The authors are careful to note this correlation is not causal evidence, but it indicates that construction defects inject extraneous difficulty beyond task complexity.

A leaderboard re-ranking after excluding the 68 misaligned instances shows 84 of 131 agents (64.1%) change rank, with shifts up to +5 (ugaiforge) and −4 (Bracket.sh, AutoCodeRover-v2.1); 9 of the top 10 positions change, and pass rates rise by 2.77 pp on average with uneven per-agent gains (−0.24 to +4.49 pp). This demonstrates that misalignment distorts standings non-uniformly, so it cannot be dismissed as uniform noise. The finding is consistent with OpenAI's concurrent analysis of SWE-bench Verified, though reached from a construction-level root-cause perspective rather than contamination or test coverage.

The PAIChecker framework

The detection approach follows a text-driven, code-validation principle: textual PR-side artifacts (descriptions, discussions, reviews) are compared against the issue at a shared level of abstraction, then validated against code. The framework addresses three failure modes of monolithic approaches (a Mini-SWE-Agent baseline with GitHub API access): lack of specialization, poor generalization beyond the taxonomy (force-fitting into the nearest predefined category, e.g., labeling an accidentally linked PR as SC rather than Others), and reasoning–label inconsistency.

PAIChecker operates in three phases with a strict separation of responsibility — only Phase I may assign predefined labels; Phases II and III hold veto power only:

• Phase I — specific identification: three parallel subagents, each with a focused artifact subset: an Issue Analyzer (IS, comparing the description against the discussion thread via the reproduce/actual/expected structure), a PR Scope Analyzer (SC and UL via scope comparison), and a PR Connection Analyzer (DP and FP, using GitHub API access to trace cross-PR chronology).
• Phase II — label synthesis: a coordinator synthesizes subagent outputs, assigns Others from collected suspicious clues (enabling beyond-taxonomy generalization), and drops labels unsupported by their cited evidence.
• Phase III — code validation: a validator with repository-level file retrieval cross-references textual claims against the implementation, filtering false positives such as claimed multi-issue fixes that are actually duplicates sharing one root cause (illustrated on mypy-11632).

Evaluation

Two annotators independently labeled SWE-Gym (2,438 Python tasks) and SWE-bench Multilingual (300 tasks, eight languages) with Cohen's $\kappa = 0.91$ (binary) and $\kappa = 0.86$ (fine-grained). Notably, misalignment prevalence is far higher in uncuration-free datasets: 44.3% in SWE-Gym and 24.3% in Multilingual, versus 13.6% in Verified. Underspecification-related patterns (UL, IS, SC) drop sharply under human curation, whereas DP (6–8%) persists across datasets, indicating that relationship-level defects escape human review.

Against three prompting baselines and four agent baselines (Mini-SWE-Agent, OpenHands, Claude Code, Codex) across four backbones (GPT-5.3-Codex, Claude-Sonnet-4.6, Gemini-3.1-Pro-Preview, Qwen-3.5-Plus), PAIChecker achieves the best binary accuracy and exact match (EM) on every backbone and both datasets, with all differences significant at $p < 0.001$:

• SWE-Gym: up to 92.12% binary accuracy and 84.66% EM (Gemini), leading the best baseline by 5.13–12.39 accuracy and 9.02–17.76 EM points.
• SWE-bench Multilingual: up to 91.67% accuracy (Claude) and 90.67% EM (Gemini), leading by 2.33–5.67 accuracy and 3.66–8.00 EM points.

Even the rare Others category (0.7%, 16 instances) is best detected by PAIChecker across backbones (e.g., 46.5 F1 with Gemini versus near-zero for most baselines).

Ablations on Gemini confirm each component: removing all Phase I subagents costs −24.29 BA and −19.88 EM; the coordinator and code validator each contribute smaller but meaningful gains (−4.71 and −3.15 BA), with EM drops exceeding BA drops, indicating the later phases refine label-level precision. The PR Scope Analyzer is the single most impactful subagent (−23.50 EM), consistent with SC+UL's 33.5% prevalence. Label-change analysis shows corrections (IC) consistently outnumber regressions (CI) across both phases and all backbones, confirming net-positive self-correction.

Real-world transfer and cost

On 200 live PR–issue pairs from nine high-star repositories, PAIChecker flagged 78; of 17 maintainer responses to posted GitHub comments, 16 confirmed the flag (94%). Cost is moderate: 42K–59K tokens per instance, roughly $0.08–$0.23 per instance, with the code validator accounting for ~40% of tokens.

Limitations and open questions

The paper concedes several constraints. Task validity depends on agent tooling — an issue specified only via an external link is deemed valid if developers raise no questions, yet is unsolvable for agents without web access; the study does not evaluate validity across tool configurations. External validity rests on a limited set of repositories, though the authors argue the patterns stem from process-level factors rather than language-specific features. Backbone mixing (assigning different models to different agents) is left unexplored, as is the combinatorial search space it introduces. The correlation between misalignment and resolution difficulty remains non-causal; a controlled experiment isolating misalignment's effect on agent performance is an open question.

Conclusion

This paper establishes PR-Issue misalignment as a measurable construction-level defect in SWE-bench-like benchmarks, quantifying it at 13.6% even in the human-validated SWE-bench Verified subset and up to 44.3% in SWE-Gym, and showing it materially distorts agent rankings. PAIChecker demonstrates that a specialized multi-agent design — pattern-specific detection, coordinator synthesis with veto power, and code-level validation — reliably detects and categorizes such misalignment at scale, with strong maintainer confirmation on live repositories. The released annotations and tooling provide a concrete basis for auditing and repairing future benchmark and training-data curation.