- The paper introduces AIDev, a large-scale dataset of 932,791 agent-authored pull requests across 116,211 repositories and 72,189 developers, covering five commercial coding agents through August 2025.
- The dataset combines broad PR metadata with a curated collection of 33,596 pull requests containing reviews, diffs, issues, and event timelines for studying adoption, code quality, testing, review dynamics, and security risks.
- AIDev enables ecosystem-level research beyond benchmarks, but users should account for its limited agent coverage, popularity bias in the curated subset, unvalidated GPT-generated task labels, and incomplete visibility into off-platform workflows.
Motivation and scope
Coding agents such as OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code have moved from token-level completion tools to autonomous contributors that open pull requests (PRs) in real repositories. Prior evidence about their behavior derives largely from controlled benchmarks (e.g., SWE-bench), small-scale deployments, or self-reported studies, leaving the community without a population-scale record of how agents actually participate in collaborative development. AIDev addresses this gap by aggregating 932,791 agent-authored PRs ("Agentic-PRs") spanning 116,211 GitHub repositories and 72,189 developers, with a data cutoff of August 1, 2025 (2602.09185).
The dataset is framed against the SE 3.0 vision of AI Teammates embedded in core development workflows. The authors illustrate this with a representative workflow in which an agent is assigned an issue, submits a patch as a PR with a detailed description, receives human review feedback, and responds with follow-up commits and replies — though they note that not all agents currently support addressing review comments.
Dataset structure and contents
AIDev is organized into two tiers:
| Tier |
Coverage |
Artifacts |
| Full dataset |
All 932,791 Agentic-PRs |
PR metadata, repository metadata, user metadata |
| Curated subset |
33,596 Agentic-PRs from 2,807 repos with >100 stars |
Review comments, inline review comments, commit diffs, related issues, full event timelines |
The curated subset contains 39,122 discussion comments, 28,875 review verdicts, 19,450 inline code review comments with diff-hunk context, 88,576 commits with 711,923 file-level diffs, 4,923 PR-to-issue mappings, and 325,500 timeline events. An annotation table assigns each curated PR a task-type label (bug fix, feature, documentation, etc.) following Conventional Commits categories, using GPT-based automated classification. This labeling choice is worth noting for downstream users: the annotations are model-generated rather than manually validated, so analyses relying on them inherit that noise.
The full relational schema is distributed via Hugging Face (with interactive SQL exploration through Data Studio), Zenodo, and a companion GitHub repository containing Jupyter notebooks with Google Colab links for reproducibility.
Positioning relative to prior work
AIDev sits at the intersection of three research threads. First, work on agentic SE systems (SWE-agent, OpenHands, AutoCodeRover) has established architectures coupling LLMs with planning, tool invocation, and repository-level reasoning, but evaluation remains benchmark-centric; AIDev supplies field data. Second, studies of development bots show that bot-generated PRs historically suffer lower acceptance rates, notification fatigue, and brittle patches — whereas LLM-assisted PRs are associated with larger changes, heavier review, and substantially slower closure times (ChatGPT-involved PRs were reported to close roughly five times more slowly than comparable PRs). Third, empirical studies of AI adoption report mixed productivity effects, with perceived gains often not matching measured task-completion improvements. AIDev enables these threads to be studied on a common, large-scale corpus rather than through isolated case studies.
Research directions enabled by the data
The paper enumerates five families of research questions the dataset supports:
- Adoption and practices: who adopts coding agents, which practices correlate with Agentic-PR quality, and what distinguishes highly effective agent users.
- Code patch characteristics: change profiles of Agentic-PRs versus Human-PRs, originality versus snippet reuse, and conformance to project conventions and CI checks.
- Testing behavior: test contribution rates, test-to-code churn ratios, and whether humans intervene when initial Agentic-PRs lack tests.
- Review dynamics: which aspects of agent-authored code attract reviewer attention, and which comment types agents fail to resolve.
- Failure patterns and risks: rejection predictors, security vulnerability introduction or mitigation, and persistence of merged Agentic-PRs in production.
These questions are framed as opportunities rather than answered by the resource paper itself; AIDev's contribution is the evidentiary substrate, not empirical findings.
Limitations and open questions
Several constraints bear on the dataset's use. The coverage is limited to five commercial agents, so findings may not generalize to open-source agent frameworks or to agents released after the August 2025 cutoff. The enriched artifacts (reviews, diffs, timelines) exist only for the star-filtered subset, introducing a popularity bias toward well-maintained projects; conclusions drawn from the curated tier may not transfer to the long tail of the 116,211 repositories. Task-type annotations are automatically generated and unvalidated. Finally, because the data captures only what is visible on GitHub, it cannot distinguish agent usage patterns from developer attribution practices, nor measure off-platform workflow context.
Conclusion
AIDev provides the first large-scale, multi-agent corpus of real-world Agentic-PRs, combining broad coverage (932,791 PRs across five agents) with a deeply annotated high-star subset suitable for fine-grained study of review dynamics, patch characteristics, and risk. Its principal value lies in enabling empirical, ecosystem-level investigation of human-AI collaboration in software engineering beyond benchmarks and surveys, with the caveats of agent coverage, popularity filtering, and automated labeling that users should account for in study design.
