- The paper analyzes 33,596 pull requests from five coding agents across 2,807 popular repositories, finding that 71.5% merged but merge rates and decision times varied sharply by agent.
- The paper finds reviewer engagement was the strongest merge correlate, increasing the odds of integration nearly fourfold, while larger changes and force pushes reduced success after accounting for repository clustering.
- The paper’s qualitative analysis shows that actionable review loops—not commit volume alone—help agents converge on reviewer expectations, highlighting the need for coordination-aware AI development workflows.
Overview
This paper presents a large-scale empirical study of how pull requests (PRs) authored by autonomous coding agents integrate into human-driven code review workflows on GitHub. Drawing on version 3 of the AIDev dataset, the authors analyze 33,596 agent-authored PRs from five coding agents (OpenAI Codex, Copilot, Devin, Cursor, and Claude Code) across 2,807 repositories, each with at least 100 GitHub stars. The study addresses two research questions: the baseline integration outcomes and resolution latency of agent-authored PRs (RQ1), and which review-time collaboration signals correlate with successful integration (RQ2). The methodological approach combines descriptive statistics with Wilson score confidence intervals for RQ1, and logistic regression with repository-clustered standard errors for RQ2, complemented by a qualitative analysis of a purposive sample of 60 PRs.
Integration Outcomes and Decision Latency
The headline result is that 71.5% of agent-authored PRs were merged (95% CI [0.710, 0.720]), with 21.6% closed without merge and 6.9% remaining open. This aggregate figure, however, conceals sharp heterogeneity across agents. OpenAI Codex achieves an 82.6% merge share, whereas Copilot merges only 43.0% of its PRs; Devin falls in between at 53.8%. The open fraction also differs substantially, with Copilot exhibiting a notably higher share of unresolved PRs, suggesting divergent triage practices or resolution dynamics.
Decision latency varies even more dramatically than merge rates. For resolved PRs, OpenAI Codex reaches a decision with a median under one hour and a mean of approximately 19.4 hours, while Copilot and Devin have median times of 13 and 9 hours respectively, with means exceeding 80–100 hours. The authors interpret this as evidence that review effort and alignment with repository norms differ across agents, implying that human reviewers bear materially different interaction costs depending on the submitting agent.
The authors deliberately exclude reverts as an outcome measure, acknowledging that reverts are an imperfect proxy for post-merge defects since they may stem from refactoring or redesign rather than failure.
Collaboration Signals Predicting Merge Success
RQ2 models merging as a binary outcome using logistic regression with repository-clustered standard errors, controlling for agent identity (Claude Code as reference). Two model specifications are estimated: Model A operationalizes iteration intensity through commit-count buckets, while Model B uses log-transformed commit counts; consistency between the two supports robustness. The central quantitative findings are:
- Reviewer engagement dominates. Receiving at least one review is the strongest correlate of merging, with an odds ratio of approximately 3.83–3.88 (95% CI roughly [2.78, 5.31]). This suggests reviewers selectively invest attention in changes they perceive as viable.
- Force pushes are penalized. History rewriting during active review is associated with lower merge likelihood (odds ratio ≈ 0.80), consistent with the interpretation that coordination-disrupting actions increase reviewer burden.
- Larger changes face higher rejection risk. Both log-transformed churn and file count show negative associations (churn odds ratio ≈ 0.89), indicating that reviewers apply similar size-based risk heuristics to agent- and human-authored contributions.
- Iteration intensity and test additions show limited independent effects. Neither is significantly associated with merging once reviewer engagement and coordination stability are controlled for — a finding that contradicts any assumption that simply producing more revisions or adding tests improves acceptance odds.
- Longer time to first review correlates positively with merging, conditional on receiving one, which the authors attribute to repository-level prioritization dynamics rather than a causal benefit of delay.
A caveat applies throughout: the design is observational, so all reported associations are correlational rather than causal, and unobserved project norms may confound estimates.
Qualitative Evidence on Review Convergence
To explain how the regression-level signals manifest, the authors coded 60 randomly and purposively sampled PRs into primary outcome drivers. The dominant success mechanism is the actionable review loop: of 32 PRs coded as such, 30 merged, accounting for essentially all merged cases in the sample. In these threads, reviewers provided concrete feedback and agents responded with targeted revisions that converged toward reviewer expectations.
All non-merged PRs map onto failure mechanisms: design or architectural disagreement (10), incomplete solutions (7), process or policy blockers (3), coordination breakdowns such as force pushes (2), failing CI (2), and other issues including stalled discussion (4) — none of these categories yielded a single merge. This qualitative pattern directly explains why iteration volume lacks independent explanatory power: additional commits help only when they resolve specific reviewer concerns rather than merely increasing activity. It also grounds the strong positive association between reviewer engagement and merging in a concrete interactional mechanism.
Threats to Validity
The authors identify three limitations. Construct validity concerns arise because collaboration signals are inferred from GitHub artifacts, which may not capture reviewer intent or interaction quality; the qualitative coding partially mitigates this. Internal validity is constrained by the observational design, precluding causal claims despite controls for agent identity and repository clustering. External validity is bounded by the focus on large public repositories; generalization to private projects, smaller communities, or future agent behaviors is not established. A further implicit limitation is that the dataset does not distinguish autonomous submissions from human-invoked agent runs, so all agent-authored PRs are analyzed uniformly, potentially conflating distinct usage modes.
Conclusion
This study establishes that most agent-authored PRs on GitHub do merge, but that integration success depends primarily on review-time collaboration signals rather than iteration volume or activity measures. Reviewer engagement — specifically engagement producing actionable feedback loops that converge toward reviewer expectations — is the dominant positive signal, while large change sizes and coordination-disrupting behaviors such as force pushes reduce merge likelihood. The practical implication is that effective agentic software engineering requires agents to conform to established review and coordination norms, not merely to generate correct patches. An important open question the authors explicitly defer is whether the same collaboration signals govern the integration of fully human-authored pull requests at comparable scale, and whether their findings hold beyond popular public repositories.