- The paper utilizes empirical data to examine the use of autonomous coding agents across the PR lifecycle by core and peripheral developers, identifying key differences in delegation, review, and validation practices. Core developers tend to exert more rigorous oversight and verification, especially for CI checks, while peripheral developers are more prone to accepting agent-generated code without independent validation.
- This study found a high unmodified acceptance rate of coding agent-generated PRs, with 74.1% of PRs accepted without any modifications, after essentially untouched at the system entry
- The use of coding agents is transforming the role of developers, with core developers acting as gatekeepers to maintain code quality. Despite the increase of AI time-saving tools, ongoing monitoring and development workflow validation by professionals are still essential
Motivation and research questions
Autonomous coding agents such as Claude Code, GitHub Copilot, OpenAI Codex, and Cursor differ from chat-based assistants in that they operate in ephemeral environments, iterate against compilers and linters, and submit pull requests (PRs) with minimal human intervention. Prior work on AI-assisted development has concentrated on copilot-style tools, and its findings on the role of developer experience are contradictory: Peng et al. reported that less-experienced developers completed tasks 55.8% faster with Copilot, while industry studies found larger gains for senior developers, and Vaithilingam et al. observed no experience-based differences. Moreover, most agent assessments derive from benchmarks like SWE-Bench rather than real-world workflows. This paper addresses the resulting gap by asking how core and peripheral developers differ across the full PR lifecycle when delegating to autonomous agents: delegation frequency and purpose (RQ1), review intensity and issue types (RQ2), modification intensity and types (RQ3), and CI verification outcomes at merge time (RQ4).
Methodology and dataset construction
The authors mined GitHub for PRs attributable to four agents using distinctive markers: branch-name prefixes (head:copilot/, head:codex/, head:cursor/) and attribution trailers (Co-Authored-By: Claude). After restricting to repositories with at least 100 stars, excluding developers whose first PR postdates agent introduction (January 2025) to avoid confounding experience with agent-induced activity, removing multi-repository contributors to prevent role ambiguity, and dropping open PRs, the final corpus comprises 9,427 resolved agentic PRs from 1,701 developers across 1,391 repositories, of which 6,819 were accepted.
Developer classification follows Kononenko et al.'s PR-based experience metric — the normalized sum of authored and reviewed closed PRs within a repository — with core developers defined as those above the 80th percentile of the experience distribution. This yields 390 core and 1,311 peripheral developers. The choice of a PR-based rather than commit-count metric is well justified given known biases of commit counts toward individual committing styles and the reduced commit activity of maintainers who shift toward review and coordination.
Qualitative analyses use established taxonomies: Zeng et al.'s ten-category commit taxonomy for PR purposes and modification types, and a code-review comment taxonomy synthesized from Beller et al. and Mäntylä et al., grouped into functional, evolvability, and discussion categories. GPT-4 performed large-scale PR purpose labeling, validated on a stratified sample of 364 PRs (Cohen's κ = 80.3%). Manual coding of review comments (n=340) and commit messages (n=360) achieved Cohen's κ values of 82.5% and 81.9%, respectively. Statistical comparisons use Mann–Whitney–Wilcoxon tests with Cliff's Delta effect sizes and Chi-Squared tests of independence.
RQ1: Delegation frequency and purpose
Median agent usage per developer is identical across groups (2.0 PRs), but the mean is substantially higher for peripheral developers (6.08 vs. 3.73), indicating that a subset of peripheral contributors generates a disproportionate share of agentic PRs. The difference is statistically significant (p<0.05) yet practically negligible (Cliff's ∣d∣=0.06). Overall acceptance rates are nearly identical (72.3% peripheral vs. 72.8% core), but core developers' accepted agentic PRs reach the main/master branch more often (85.8% vs. 77.8%), suggesting that socio-technical gatekeeping around contributor reputation persists even when both groups delegate to agents.
Task delegation differs meaningfully. Peripheral developers distribute their agentic PRs almost uniformly across bug fixing, feature addition, documentation, and testing (~18.7–19.8% each), reflecting an end-to-end delegation strategy. Core developers concentrate on documentation and testing (42.7% combined; 21.8% testing, 20.9% documentation) and delegate fewer bug fixes than peripheral developers (-3.8 percentage points), mirroring the "many eyes" pattern seen in human-authored contributions. The association between developer group and PR type is statistically significant (p<0.0001).
RQ2: Review dynamics
Core developers engage more intensively in review discussions on agentic PRs, with a median of 3.6 comments per reviewed PR versus 2.0 for peripheral developers (p<0.001, Cliff's ∣d∣=0.28, small but non-negligible). Both groups' comments are dominated by evolvability concerns (52.8% core, 59.3% peripheral), consistent with findings on human-authored PRs. Within this category, peripheral reviewers emphasize code organization (17.6%), while core reviewers more frequently propose alternative solution approaches (18.2%), such as different algorithms or API choices. Notably, neither group raised resource- or timing-related functional issues on agentic PRs. The Chi-Squared test found no statistically significant association between comment category and developer group, so the qualitative differences should be read as tendencies rather than categorical distinctions.
RQ3: Modifications to agent-generated code
On average, 74.1% of agentic PRs are accepted without any developer modification, a striking figure that indicates agents' outputs frequently enter the codebase untouched. When modifications do occur, core developers modify a larger share of their accepted agentic PRs (28.3%) than peripheral developers (23.5%), and their edits are larger in magnitude (mean added LOC 55.4% vs. 40.4%; deleted LOC 38.2% vs. 24.5%), though the Mann–Whitney–Wilcoxon test did not detect a statistically significant difference between groups. A subset of core contributors ("heroes") performs substantially larger rewrites, as illustrated by a case where a developer made 19 follow-up commits (1,154 lines added, 1,126 deleted) after a Copilot structural change failed checks.
Thematically, refactoring is the most common modification for both groups. Peripheral developers additionally fix bugs in agent output (36.2% combined bug-fix/refactor), while core developers also improve documentation (38.5% combined refactor/docs). No statistically significant association was found between modification category and developer group.
RQ4: CI verification outcomes
The verification results are the most consequential finding for software quality. Core developers run slightly more check suites per PR (9.3 vs. 8.1) and achieve higher all-checks-pass rates (51.2% vs. 43.1%). More importantly, peripheral developers are nearly twice as likely to merge an agentic PR without running any checks (19.1% vs. 11.2%). A documented case shows a peripheral contributor accepting a Copilot test-migration PR while two integration checks were still failing after multiple failed agent attempts, whereas a core developer corrected formatting and cleaned up tests before merging only once all checks passed. These patterns indicate that core developers continue to act as quality gatekeepers under agentic workflows, enforcing the "all checks must pass" principle, while some peripheral contributors accept agent assurances ("I've resolved the issue") without independent verification — a behavior consistent with automation-bias risks documented in the human–AI collaboration literature.
Lessons and implications
The authors distill six lessons. First, socio-technical inequities persist: main-branch gatekeeping still favors core contributors even for agent-generated work, motivating standardized review rubrics that evaluate the contribution rather than the contributor. Second, agents are absorbing "toil": documentation and testing dominate delegation for both groups, aligning with survey evidence that developers most want AI help with these low-satisfaction tasks; enterprises with strict AI policies can exploit this pattern deliberately. Third, the high unmodified acceptance rate (74.1%) is concentrated in routine task types, motivating platforms to surface agent competence indicators (e.g., confidence based on historical performance per task type) at the point of use to calibrate trust. Fourth, heavy reliance on agents by a subset of peripheral contributors raises deskilling concerns, since newcomers who do not engage deeply with submitted changes may fail to develop long-term skills; the authors suggest communities encourage human-authored contributions before agentic ones. Fifth, core "heroes" performing large refactoring interventions underscore that experienced oversight remains necessary given evidence that AI-generated code can violate quality and security standards. Sixth, stricter validation practices — monthly audits, quality-gate reviews led by core developers — are recommended to safeguard CI reliability as agents increasingly handle testing and build verification themselves.
Limitations and open questions
The authors acknowledge several constraints. External validity is bounded by the four studied agents, GitHub-only data, the 100-star repository filter (biasing toward mature projects), and the exclusion of multi-repository contributors and post-agent newcomers. Internal validity threats include GPT-4 labeling errors (mitigated by manual validation), the single-repository experience metric, force-push modifications that overwrite history and escape detection, heuristic agent-commit identification that may need refinement as agent practices evolve, and reliance on GitHub's check-run conclusion field, which may not reflect true integration status when failures are unrelated to the change. Construct validity is addressed through effect sizes alongside p-values, but several quantitative contrasts (e.g., modification magnitude) did not reach statistical significance despite visible descriptive differences. Open questions include how rejected agentic PRs differ from accepted ones, how collaboration patterns evolve longitudinally, and how prompt design and organizational factors shape the success of agentic contributions.
Conclusion
This study provides the first large-scale empirical account of how developer experience shapes collaboration with autonomous coding agents across the PR lifecycle. Its central findings — near-uniform acceptance rates masking divergent delegation profiles, evolvability-dominated review discourse, largely untouched agent output (74.1% unmodified), and a pronounced verification gap in which peripheral developers merge unchecked agentic PRs at roughly twice the rate of core developers — collectively demonstrate that experience-based collaboration dynamics persist, and in the case of CI enforcement may be amplified, in agentic software development. The results offer actionable guidance for OSS communities and enterprises seeking reliable human–agent integration workflows.