- The paper provides a large-scale, empirical study comparing 24,014 AI-authored and 5,081 Human-authored GitHub PRs that reveals differences in structural characteristics and consistency in PR descriptions, findings relationships such as commit counts for AI-authored PRs exceeding those of humans (Agnetic: 440,295 commits, 20-commit median; Human: 23,242commits, 7-commit median).
- PR Analysis showed a significant difference between Human PRs using a median of 158 files and 119 commits, versus AI agents altering fewer files and commits in less widely distributed structures. Agent PRs also showed more localized edits, suggesting optimization strategies that human authors could adopt. The study highlighted that commit count and files touched have the most impact on reviewers.
- The comparative lexical similarity between natural language PR descriptions and diffs ranked lower with both Human and AI using natural vocabulary while semantic similarity using embeddings like CodeBERT and GraphCodeBERT measured tight coordinate distances of about 0.8 to 0.93 for AI-penned descriptions, significantly higher than lexical alignments. This linguistic consistency suggests opposed interpretations between descriptions and slang model-shared word relationships for AI dictionary language models.
This paper presents a large-scale empirical study comparing pull requests (PRs) authored by autonomous AI coding agents ("Agentic PRs") with human-authored PRs, using the AIDev dataset distributed for the MSR 2026 Mining Challenge. The study addresses two research questions: how structural characteristics of code changes differ between the two groups (RQ1), and how well PR descriptions align with the underlying diffs (RQ2). The analysis covers 24,014 merged Agentic PRs comprising 440,295 commits and 5,081 merged Human PRs comprising 23,242 commits (2601.17581).
Methodology
The methodology proceeds in four steps. First, the authors collect the MSR 2026 Mining Challenge version of the AIDev dataset (retrieved November 1, 2025), which contains 932,791 Agentic PRs and 6,618 Human PRs across 116,211 repositories. Because Human PRs in AIDev lack commit-level information, the authors reconstruct it via the GitHub REST API, gathering commit metadata, per-file additions and deletions, and unified diffs. Files touched are counted as unique file paths across all commits to avoid double-counting repeated edits.
Second, filtering retains only merged PRs with complete, valid patch text: entries lacking non-empty unified diffs for all modified files were excluded, as were Human PRs with missing repositories or incomplete API retrievals. This filtering is consequential: only 24,014 of roughly 932,791 Agentic PRs survive, a reduction driven largely by the requirement for complete patch text. Readers should note that this yields a filtered population skewed toward smaller, complete patches, since GitHub truncates diffs for very large changes — a threat the authors acknowledge explicitly.
Third, analysis uses Mann–Whitney U tests (all metrics violate normality) with Cliff's δ effect sizes. Fourth, RQ2 evaluates description-to-diff alignment along lexical dimensions — TF-IDF cosine similarity and Okapi BM25 — and semantic dimensions using CodeBERT and GraphCodeBERT embeddings with cosine similarity. Diffs are stripped of metadata and normalized; descriptions concatenate titles and body text. The paper is careful to state that BM25 is unbounded and sensitive to tokenization, so its values (which range from about −13,777 to over 68,000 for Agentic PRs) must be read as relative lexical signals rather than calibrated similarities.
Structural differences: agentic PRs are more localized
The clearest finding concerns commit structure. Commit count exhibits a large Cliff's δ effect (δ=0.5429), files touched (δ=0.4487) and deletions (δ=0.4462) show medium effects, while additions (δ=0.2836) and total line changes (δ=0.3158) show small effects; all Mann–Whitney comparisons are significant at δ0. The interpretation is that Agentic and Human PRs differ less in the raw quantity of code introduced than in how edits are organized and distributed across commits and files. Human PRs have higher medians and long upper tails on additions and deletions, touch more files, and involve more commits, indicating broader and more distributed modifications.
The paper also documents substantial heterogeneity among agents. Claude Code and OpenAI Codex exhibit wider variability in additions, deletions, commits, and files touched, overlapping the lower range of human behavior, whereas Devin, Cursor, and especially Copilot produce consistently small, localized edits. This stratified result undercuts any assumption that "Agentic PR" describes a homogeneous class; agent identity appears to be a meaningful covariate for downstream analyses such as review effort or defect likelihood.
A practical implication drawn by the authors is that review triage heuristics should weight commit count and files touched over LOC-based measures when assessing an Agentic PR's scope. They also flag deletion-heavy or wide-scope Agentic PRs as candidates for closer scrutiny, while conceding that linking these structural patterns to concrete risk outcomes remains future work rather than a demonstrated result.
Description–diff alignment: high semantic consistency
RQ2 reveals a sharp dissociation between lexical and semantic similarity. Lexical scores cluster near zero for both groups (TF-IDF medians of 0.0937 for Agentic vs. 0.0680 for Human), reflecting minimal surface vocabulary overlap between natural-language descriptions and code diffs. Semantic scores, in contrast, form tight peaks between 0.9 and 1.0 (CodeBERT medians of 0.9375 vs. 0.9347; GraphCodeBERT medians of 0.8302 vs. 0.8067).
| Metric |
Agentic median |
Human median |
| TF-IDF cosine |
0.0937 |
0.0680 |
| Okapi BM25 |
1.0169 |
0.0000 |
| CodeBERT cosine |
0.9375 |
0.9347 |
| GraphCodeBERT cosine |
0.8302 |
0.8067 |
Agentic PRs score slightly higher on central tendency across all four metrics, though absolute differences are modest. The substantive claim is that both description types capture the meaning of their patches despite differing wording, and that agent-generated descriptions align at least as well as human ones. This supports downstream applications the authors identify — release-note generation, changelog construction, reviewer routing — and suggests automated low-alignment checks could flag potentially ambiguous or misleading Agentic PRs. Two caveats apply here: similarity measures alignment, not correctness or reviewer comprehension, and the semantic models' uniformly high scores may partly reflect embedding-space properties rather than genuine communicative quality, a limitation the paper does not probe directly.
Threats to validity
The authors identify threats across three categories. Internally, results depend on AIDev completeness and on reconstructed Human-PR data; GitHub API retrieval can introduce gaps from rate limits, deleted repositories, rewritten histories, and diff truncation for large changes. Construct validity is limited because structural metrics approximate scope without capturing intent or correctness, and similarity metrics do not measure code quality. Externally, findings generalize only to open-source GitHub projects represented in AIDev and to the specific agent versions included; private or industrial settings with different review cultures may differ, as will results as agents evolve.
Conclusion
The study provides a large-scale empirical characterization showing that AI coding agents produce structurally distinct PRs — most notably in commit count and file breadth — yet describe their edits with semantic consistency comparable to or slightly exceeding human contributors. Its main open questions concern whether the observed structural patterns translate into measurable differences in review effort, defect risk, or maintainability, and how the heterogeneity across agent types conditions those outcomes.
