- The paper compares the prevalence and impact of agent-authored documentation pull requests to human-author requests in SE 3.0, finding agents both contribute a 60% higher volume changes and a 70% lower rate overall.
- The analysis shows 66.1% of documentation files are edited exclusively by agents, 30.2% by humans, and only 3.7% co-edited, highlighting agent dominance. Almost 90% of documentation addition aligns with zero-human editing suggesting limited human review or alteration of AI writes.
- Exploration of the data raises concerns about scrutiny levels over agent-generated documentation, and document authorship attribution even when mixed with coding edits.
Motivation and scope
As software engineering moves toward the SE 3.0 model—in which autonomous AI agents act as teammates that propose changes through standard collaboration mechanisms such as pull requests (PRs)—the authors argue that documentation is a critical but underexamined surface for agent contributions. Prior work on LLM-based documentation generation has largely evaluated output quality in controlled settings, while repository-scale studies of agentic PRs have focused on coding-centric activities. This paper addresses two practical questions: how prevalent agent-authored documentation changes are relative to human-authored ones, and whether agent-proposed documentation edits receive meaningful human follow-up after integration. The study is published at MSR '26 (2601.20171).
Data collection
The analysis is built on AIDev, a dataset of agent- and human-authored PRs. The authors extracted documentation-related PRs from AIDev's pull_request table (33,596 PRs across 2,807 repositories) and human_pull_request table (6,618 PRs across 818 repositories), restricting both to repositories with more than 500 stars to ensure comparable conditions. This yielded 1,478 documentation-related Agentic-PRs and 519 Human-PRs. Commit-level details were retrieved via the GitHub API for both groups, producing 3,653 commits and 35,428 file-change records for agents versus 1,889 commits and 17,013 records for humans.
Documentation files were identified with a lightweight heuristic following prior work: files with extensions such as .md and .txt, or paths containing tokens like /docs/ or README. A notable consequence of this labeling is discussed below under limitations.
RQ1: Prevalence of agent documentation contributions
Agent-authored documentation PRs substantially outnumber human ones (1,478 vs. 519). Within the agentic subset, OpenAI Codex accounts for 45% of PRs, a distribution closely mirroring the overall agentic PR breakdown reported for AIDev. At the file level, 66.1% of changed files are edited only by agents, 30.2% only by humans, and only 3.7% (948 files) are co-edited by both—meaning 96.3% of files are handled exclusively by one party rather than collaboratively.
A more surprising finding concerns packaging: among documentation-related PRs, 29.0% of agent-authored PRs modify no documentation files at all (only non-docs files), compared with 16.6% for human PRs. Under conventional commit conventions, docs-type changes are intended to be documentation-only; mixed-scope PRs violate this expectation and are associated with higher reviewer effort and greater risk that changes are overlooked. The share of PRs touching docs-only files is nonetheless similar between agents (48.7%) and humans (47.8%).
The implication is twofold: documentation is already a primary entry point for agent contributions, but task labels and actual file changes are frequently misaligned, particularly for agent workflows.
RQ2: Integration and human follow-up
To assess integration, the authors identified 119 file-level cases where an agent commit is followed by the next human commit on the same file, then compared lines added by the agent against lines deleted by the human. The results indicate that agent documentation edits are largely retained:
| Signal |
Value |
| Files where agent additions exceed or match human deletions |
85.7% (102/119) |
| Files with zero human deletions after agent additions |
34.5% (41/119) |
| Files where human deletions exceed agent additions |
14.3% (17/119) |
| Mean retention of agent-added lines (when additions dominate) |
86.8% |
| Median retention of agent-added lines |
98.7% |
These figures suggest that agent-authored documentation changes tend to be accepted with limited follow-up modification. The authors interpret this cautiously: high line retention combined with frequent zero-deletion cases raises questions about the level of scrutiny agent edits receive, since existing review practices do not automatically mitigate reliability risks of agent-generated documentation. Notably, this interpretation rests solely on deletion counts—the analysis does not yet account for human additions, modifications, or review discussions, which could change the picture considerably.
Limitations and open questions
The paper concedes three main threats. First, external validity is bounded by AIDev's construction: the dataset contains only repositories that already host agent-authored PRs, so repositories where agents are not used at all are unrepresented, even though Agentic-PRs and Human-PRs are compared under identical repository conditions. Second, internal validity is affected by the extension- and path-based classification of documentation files; the fact that 29.0% of agent "documentation-related" PRs touch no docs files introduces uncertainty into all prevalence estimates and motivates better alignment between task labels, file classification, and actual edits. Third, the "limited follow-up" finding relies on quantitative signals (deletions and line retention) without considering review comments, approval patterns, or subsequent human additions—an explicitly open question the authors plan to address through qualitative examination of low-follow-up cases.
Conclusion
This study provides the first PR-level, large-scale comparison of agent- and human-authored documentation work, showing that agents dominate documentation PR volume, that co-editing of individual files is rare, and that accepted agent edits typically persist with minimal human revision. The results position documentation quality assurance—and accountable human–AI review workflows—as concrete open problems for SE 3.0, while the released replication package supports reproducibility and follow-up investigation.
