- The paper evaluates AI coding agents (Codex, Copilot, Cursor, Devin, and Claude) across task-level acceptance, review discussion volume, and commit message quality using 33,549 personalized PRs from 2,807 open-source GitHub repositories.
- Codex shows the highest PR acceptance rate (0.83) but the lowest commit message quality (0.32), while Claude leads in message quality (0.68) but has varied acceptance across tasks. Copilot attracts the most review comments (2.56) despite a low acceptance rate (0.45).
- Online PR acceptance does not strongly depend on commit message quality, indicating other factors (such as diffs and CI status are more critical. 90.6% of the PRs have no explicit review comments which highlights the potential to improve the visibility of reviewer feedback.
Overview and motivation
This paper presents a comparative, agent-centric empirical study of five autonomous coding agents—OpenAI Codex, GitHub Copilot, Cursor, Devin, and Claude—using the AIDev-pop dataset of AI-authored pull requests (PRs) in open-source repositories (2602.02345). The study is motivated by the observation that prior evaluations of AI coding agents tend to examine individual agents or aggregate behavior, emphasizing productivity or human perception rather than relative technical performance under comparable conditions. As agents increasingly submit complete PRs directly to real repositories, the authors argue that task-aware, cross-agent comparison is needed to support tool selection and deployment decisions.
The analysis covers 33,549 PRs (after excluding the rare revert and other categories) authored by the five agents across 2,807 GitHub repositories with at least 100 stars. Each PR is labeled with one of ten Conventional Commits-style task types (feat, fix, docs, build, ci, refactor, test, perf, style, chore). The distribution is heavily skewed: Codex contributes 21,799 PRs (about 65% of the dataset), while Claude contributes only 459. Feature additions (14,450 PRs) and bug fixes (8,106 PRs) dominate the corpus.
The paper addresses three research questions spanning the PR lifecycle: RQ1 — differences in PR acceptance rates across agents and task types; RQ2 — differences in review discussion volume; and RQ3 — differences in commit message quality as measured by the C-Good classifier, a BERT-based BiLSTM model that labels a message high quality only if it states both what changed and why, with a reported precision of 81.6%.
Methodology
Three metrics are computed per agent–task pair. Acceptance rate is the ratio of merged to submitted PRs. Review discussion volume is the mean number of comments per PR, separated into bot-generated and human-authored comments. Commit message quality is the proportion of commits classified as high quality by C-Good. Statistical significance is assessed with Mann–Whitney-Wilcoxon tests comparing the two closest agents on each dimension. A public replication package accompanies the study.
A notable methodological caveat, acknowledged by the authors, is that review comment counts capture only explicit written feedback: 90.6% of all PRs in the dataset have zero recorded review comments, so the metric reflects documented discussion rather than total review activity.
PR acceptance (RQ1)
Codex achieves the highest average acceptance rate at 0.83, with the lowest variability across task types (SD = 0.06), while Copilot records the lowest at 0.45. Cursor (0.67) and Claude (0.66) fall in between but with higher variance (SD = 0.13 each). A Mann–Whitney-Wilcoxon test comparing Codex and Cursor yields p=4.27×10−63, confirming the gap is statistically significant.
| Task |
Claude |
Copilot |
Cursor |
Devin |
Codex |
| build |
0.88 |
0.50 |
0.57 |
0.58 |
0.87 |
| chore |
0.71 |
0.44 |
0.74 |
0.57 |
0.84 |
| ci |
0.80 |
0.63 |
0.94 |
0.61 |
0.86 |
| docs |
0.75 |
0.61 |
0.74 |
0.71 |
0.92 |
| feat |
0.57 |
0.38 |
0.60 |
0.54 |
0.81 |
| fix |
0.57 |
0.42 |
0.68 |
0.43 |
0.82 |
| perf |
0.67 |
0.27 |
0.46 |
0.35 |
0.68 |
| refactor |
0.50 |
0.48 |
0.65 |
0.57 |
0.80 |
| style |
– |
0.40 |
0.72 |
0.68 |
0.85 |
| test |
0.50 |
0.37 |
0.62 |
0.50 |
0.84 |
Maintenance-oriented tasks (docs, build, ci) are accepted at higher rates than functional tasks (feat, fix) for all agents. Codex is the only agent exceeding 0.80 acceptance on both feat and fix. Because Codex also contributes the largest number of PRs, its high rate cannot be attributed to small sample size. The implication is that integration success varies systematically with both the agent and the nature of the task, so aggregate acceptance figures obscure meaningful task-level differences.
Review discussion volume (RQ2)
Copilot's PRs attract by far the most review activity, averaging 1.25 bot comments and 1.31 human comments per PR (2.56 combined)—the only agent whose PRs consistently trigger sustained discussion. All other agents average below one comment per PR, with Codex receiving almost none (0.02 bot, 0.05 human). For Copilot, bot and human comment volumes are roughly equal; for every other agent, human comments dominate. The difference between Copilot and Devin (the next-closest agent) is statistically significant (p=7.09×10−50).
| Agent |
Bot avg |
Human avg |
Total avg |
| Copilot |
1.25 |
1.31 |
2.56 |
| Devin |
0.11 |
0.60 |
0.71 |
| Claude |
0.11 |
0.52 |
0.63 |
| Cursor |
0.12 |
0.29 |
0.41 |
| Codex |
0.02 |
0.05 |
0.07 |
Two findings carry particular weight here. First, Copilot's high discussion volume coincides with its lowest acceptance rate, suggesting its contributions impose greater review overhead for worse integration outcomes. Second, and more strikingly, 98.2% of Codex-authored PRs receive no recorded comments—including nearly all feat and fix PRs—implying that many agent contributions are merged or closed without any documented review. This means acceptance decisions are frequently made outside visible review channels, and comment-based measures of review effort substantially underestimate actual scrutiny for most agents.
Commit message quality (RQ3)
Commit message quality diverges sharply from the acceptance results. Claude achieves the highest average good commit rate (0.68) but with the highest variability (SD = 0.19), leading on build, docs, feat, fix, perf, and test. Cursor follows at 0.63, excelling on ci and style; Devin averages 0.57 with the most stable performance (SD = 0.07); Copilot averages 0.41. Codex records the lowest rate at 0.32, despite contributing the most PRs and achieving the highest acceptance. The Claude–Cursor difference is significant (p=4.55×10−31).
| Task |
Claude |
Copilot |
Cursor |
Devin |
Codex |
| build |
0.80 |
0.39 |
0.52 |
0.52 |
0.19 |
| chore |
0.48 |
0.42 |
0.53 |
0.56 |
0.34 |
| ci |
0.40 |
0.46 |
0.74 |
0.58 |
0.20 |
| docs |
0.85 |
0.50 |
0.72 |
0.74 |
0.65 |
| feat |
0.70 |
0.34 |
0.52 |
0.57 |
0.19 |
| fix |
0.75 |
0.38 |
0.48 |
0.53 |
0.22 |
| perf |
1.00 |
0.41 |
0.78 |
0.50 |
0.20 |
| refactor |
0.52 |
0.45 |
0.59 |
0.63 |
0.49 |
| style |
– |
0.23 |
0.68 |
0.53 |
0.28 |
| test |
0.77 |
0.52 |
0.76 |
0.62 |
0.41 |
This is the paper's central contradiction: Codex's PRs are merged at the highest rates while carrying the lowest-quality commit messages—for feat and fix specifically, only 0.19 and 0.22 of commits qualify as high quality, yet these task types account for the bulk of its accepted PRs. The authors conclude that PR acceptance does not strongly depend on commit message quality, implying that maintainers' merge decisions weight other signals (e.g., diffs, CI status) over documentation hygiene, and that high integration throughput can coexist with poor change traceability.
Threats to validity
The authors identify several limitations that bound the interpretation of these results. The three metrics capture observable outcomes but not full quality: an accepted PR may still contain defects, and comment volume does not distinguish positive from negative feedback. Task complexity varies within task types, and reviewer practices differ across projects, although the shared-dataset design partially controls for repository context. Findings are specific to open-source GitHub repositories and may not generalize to proprietary settings or close human–AI collaboration. Finally, the data reflect agent behavior up to August 2025; later model versions may behave differently, so the rankings should be read as a snapshot rather than stable properties of these tools.
Conclusion
This study provides a task-aware comparison of five autonomous coding agents across three stages of the PR lifecycle using 33,549 real-world PRs. Its key result is that the three evaluation dimensions rank the agents differently: Codex leads in acceptance (0.83) but trails in commit message quality (0.32); Copilot generates the most review discussion (2.56 comments/PR) with the lowest acceptance (0.45); Claude produces the highest-quality commit messages (0.68) with greater task-level variability. These divergences demonstrate that no single metric or aggregate ranking adequately characterizes agent performance, and that evaluation must be conditioned on task type. The near-total absence of recorded review comments—particularly for Codex—remains an open question the authors plan to address by examining commit diffs, reviewer identities, and repository-specific practices to understand how acceptance decisions are made without explicit feedback, alongside comparisons of AI-authored versus human-authored PRs and code-level defect analysis.
