- The paper quantitatively assesses the integration of 8,106 fix-related AI PRs, revealing 65.0% were merged, 26.1% closed without merging, and 8.9% remained unmerged,
- Variable merge success across agents is highlighted, with OpenAI Codex performing best at 81.6% and Devin lowest at 42.9%, signifying the importance of real-world integration metrics.
- Routine process dynamics, not technical defects alone, heavily influence PR failure, emphasizing problems like further testing and submission gaps.
Motivation and scope
Autonomous coding agents such as OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code now submit pull requests (PRs) to real-world repositories at scale, yet their practical value hinges on whether maintainers accept and merge those contributions. Prior evaluations of LLM-based repair largely rely on benchmark success rates or developer productivity metrics, abstracting away the downstream review-and-integration process. This paper addresses that gap with an empirical study of fix-related PRs — contributions that directly address defects and therefore face the strictest correctness, testing, and validation requirements. The motivating example is PR #553 to Azure/CloudShell, where an agent's dependency-version fix repeatedly failed CI tests despite iterative commits responding to reviewer feedback, and was ultimately closed unmerged.
The study pursues two research questions. RQ1 quantifies integration outcomes (merged, closed without merging, open) and submission-to-merge latency for fix-related PRs authored by five agents. RQ2 constructs a structured catalog of failure reasons for closed-but-unmerged PRs through manual qualitative analysis.
Methodology
The authors analyze 8,106 fix-related PRs drawn from the AIDEV-POP dataset of 33,596 AI-authored PRs mined from GitHub. The agent distribution is heavily skewed: OpenAI Codex contributes 4,338 PRs, GitHub Copilot 1,993, Devin 1,249, Cursor 411, and Claude Code only 115. For RQ1, each PR is classified by its status at data-collection time, and time-to-merge is computed as the difference between creation and closing timestamps.
For RQ2, from the 2,113 closed-but-unmerged cases identified in RQ1, a random sample of 326 PRs was drawn — sufficient for a 95% confidence level with a 5% margin of error. Two authors performed independent open coding over roughly 100 person-hours, after a calibration phase on 20 PRs (excluded from the final sample). Annotators examined PR descriptions, diffs, review comments, commit history, and closure context to inductively derive failure categories. Cohen's kappa reached 0.82, indicating substantial inter-rater agreement; all disagreements were resolved by consensus. A replication package is publicly available.
Integration outcomes and latency (RQ1)
Overall, 65.0% of the 8,106 fix-related PRs were merged, 26.1% were closed without merging, and 8.9% remained open. The headline result is the substantial cross-agent variance:
| Agent |
Total |
Merged |
Closed w/o Merge |
Open |
| OpenAI Codex |
4,338 |
3,539 (81.6%) |
616 (14.2%) |
183 (4.2%) |
| GitHub Copilot |
1,993 |
845 (42.4%) |
713 (35.8%) |
435 (21.8%) |
| Devin |
1,249 |
536 (42.9%) |
675 (54.0%) |
38 (3.0%) |
| Cursor |
411 |
281 (68.4%) |
84 (20.4%) |
46 (11.2%) |
| Claude Code |
115 |
66 (57.4%) |
25 (21.7%) |
24 (20.9%) |
Merge rates span nearly a factor of two, from 81.6% (Codex) down to 42.9% (Devin). Notably, more than half of Devin's fix-related PRs are closed without merging, a striking outcome given Devin's positioning as an autonomous software engineer. The implication is that real-world integration success depends strongly on agent design and on how agent output aligns with project conventions and review workflows — not merely on patch plausibility.
Merge-latency analysis reveals similarly wide dispersion. OpenAI Codex exhibits consistently short merge times with a tightly concentrated interquartile range (median ≈ 0.02 hours), whereas Copilot shows much broader distributions (median ≈ 18 hours) and heavier upper tails extending beyond 1,000 hours. Latency thus reflects differences in review effort and coordination overhead across agents and projects, though the paper does not disentangle agent effects from repository- or task-complexity confounds.
Failure causes (RQ2)
Manual analysis yielded a catalog of 12 failure reasons. The most frequent cause is not technical at all: 22.1% of unmerged PRs were closed because the underlying issue had already been resolved by another PR (RAPR), followed by test case failures (18.1%) and incorrect or incomplete fixes (15.3%). Build failures (2.1%) and deployment failures (3.1%) are comparatively rare — a notable finding, since it indicates agents generally produce compilable code but frequently fail at validation readiness.
| Failure cause |
Share |
| Resolved by another PR (RAPR) |
22.1% |
| Test case failures (TCF) |
18.1% |
| Incorrect/incomplete fixes (IIF) |
15.3% |
| Closed due to inactivity (CDI) |
9.2% |
| Low priority/obsolete issue (LPO) |
8.0% |
| Incomplete review process (IRP) |
5.5% |
| Fix rejected by maintainers (FRM) |
4.9% |
| No review conducted (NRC) |
4.6% |
| Closed without rationale (CWR) |
4.0% |
| Other reasons (OTH) |
3.1% |
| Deployment failures (DF) |
3.1% |
| Build failures (BF) |
2.1% |
Agent-wise normalization exposes sharply different failure profiles. For OpenAI Codex, failures are dominated by test case failures — 50 of 91 unmerged PRs (54.9%) — pointing to validation gaps rather than review friction. Devin's unmerged PRs are instead dominated by inactivity closures (27 of 50, 54.0%), suggesting difficulty sustaining engagement during longer review cycles. GitHub Copilot's largest category is redundancy (47 of 148, 31.7%), implying timing mismatches between agent submissions and parallel human activity. Cursor and Claude Code have few unmerged PRs, skewed toward prioritization- and rationale-related causes rather than systematic technical errors.
The authors synthesize the twelve categories into three groups: technical causes (IIF, TCF, BF, DF), reflecting limits in generating fully correct, validation-ready fixes; process-related causes (FRM, IRP, NRC, CWR, CDI), reflecting breakdowns in review interaction and sustained engagement; and priority-related causes (RAPR, LPO), reflecting redundancy and shifting maintainer priorities. The synthesis makes clear that a substantial share of non-integration arises from workflow dynamics rather than intrinsic flaws in the generated patches — meaning improved patch quality alone would not eliminate all failures.
Limitations and open questions
Several constraints qualify these findings. First, the dataset is heavily imbalanced toward OpenAI Codex (over half of all PRs), so aggregate statistics weight Codex behavior disproportionately; Claude Code contributes only 115 PRs, making its percentages fragile. Second, the study classifies outcomes at a single snapshot in time — the 726 open PRs may later be merged or closed, and no longitudinal follow-up is reported. Third, the failure taxonomy relies on manual interpretation of closure context; the 4.0% of PRs closed without any explicit rationale necessarily involve inferential labeling, and single-primary-cause assignment may understate multi-factor failures. Fourth, the paper does not control for repository characteristics, issue difficulty, or project review culture, so cross-agent comparisons conflate agent capability with where and how each agent is deployed. Finally, whether high merge rates correlate with post-merge code quality (e.g., revert rates or defect introduction) remains unexamined — a merged agent PR is not demonstrably a correct one.
Conclusion
This empirical study of 8,106 AI agent–authored fix-related PRs shows that while 65.0% are merged, integration success varies dramatically across agents (81.6% for Codex versus ~42–43% for Copilot and Devin), and that non-integration stems as much from timing, redundancy, and review-workflow dynamics as from technical defects. The 12-category failure catalog — anchored by test failures, redundant fixes, and incomplete patches, with build failures notably rare — provides a concrete diagnostic basis for improving agent validation strategies and human-AI collaboration in software maintenance.
