AIDev Dataset Corpus: Agentic PRs & CRAFT

• AIDev Dataset Corpus is a large-scale, curated collection capturing agent-authored pull requests and synthetic datasets using the CRAFT workflow.
• It enables empirical studies on AI coding agents, benchmark analyses, and quality control within Software Engineering 3.0.
• The CRAFT pipeline systematically retrieves and augments documents for fine-tuning foundation models and adapting to domain-specific tasks.

The AIDev Dataset Corpus refers to a suite of large-scale, systematically curated, and extensible corpora for AI development (“AIDev”), emphasizing datasets that either (1) capture the behavior and output of AI coding agents in real-world contexts, or (2) are constructed for synthetic data-driven advancement of foundation models and agentic systems. Most notably, “AIDev Dataset Corpus” is associated with the AIDev collection of agent-authored pull requests and the methodology for creating synthetic, task-specific corpora via the CRAFT workflow. These corpora support analysis of agentic AI, facilitate benchmarking and fine-tuning, and enable studies of human–AI collaboration, quality control, and adoption patterns in “Software Engineering 3.0” (Li et al., 9 Feb 2026, Ziegler et al., 2024).

1. Motivation and Conceptual Scope

The AIDev Dataset Corpus addresses two intersecting needs in contemporary AI development:

• Enabling empirical study of AI coding agents and their integration in real-world collaborative software development environments, as formalized by the AIDev initiative and related dataset releases (Li et al., 9 Feb 2026).
• Providing mechanisms and methodologies—exemplified by CRAFT (Corpus Retrieval and Augmentation for Fine-Tuning)—for scalable generation of task-specific, high-quality synthetic datasets in AI development, software engineering, and related domains. This aligns with the need for reproducible, customizable, and high-coverage data for training, benchmarking, and domain adaptation (Ziegler et al., 2024).

2. AIDev: Dataset of Agentic Pull Requests

AIDev presents the first large-scale corpus of agent-authored pull requests (“Agentic-PRs”) mined from public GitHub repositories. The dataset comprises 932,791 PRs written by five agentic systems (OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code), spanning 116,211 repositories and 72,189 unique developers up to August 1, 2025. A curated subset focuses on 33,596 PRs from 2,807 repositories with at least 100 stars, enriching the base with review artifacts, comments, issues, and changelogs (Li et al., 9 Feb 2026).

The dataset captures extensive metadata, including agent origin, language, PR state, code diffs, review events, and developer annotations. The aim is to facilitate systematic research into the adoption rate ($R = N_{agentic}/N_{total}$), diversity (normalized Shannon entropy), review dynamics, security impact, and workflow patterns inherent in Software Engineering 3.0 settings.

Key distributional statistics:

Statistic	Value (Full)	Value (High-Star)
Agentic-PRs	932,791	33,596
Repositories	116,211	2,807
Developers	72,189	1,796
Top Language (Full)	JavaScript 28%	—
Median PR size	35 lines	—
Median time-to-merge	2.1 days	4.3 days

The data model is highly normalized, with 15 interrelated tables for PRs, users, comments, reviews, commits, issues, timeline events, and automated annotations (including task type via regex or LLM tagging). Supported export formats include Parquet, CSV, and JSONLines.

3. Synthetic Corpus Generation via CRAFT

CRAFT (“Corpus Retrieval and Augmentation for Fine-Tuning”) defines an extensible pipeline for generating synthetic, task-specific training corpora based on a small few-shot seed, large-scale public corpora, and LLM-based transformation (Ziegler et al., 2024). The pipeline comprises two primary phases:

3.1 Corpus Retrieval

• Embedding: Encode $n$ few-shot task exemplars $Q = \{q_1,...,q_n\}$ and $N$ corpus documents $D = \{d_1,...,d_N\}$ in $\mathbb{R}^d$ using a SentenceTransformer such as MiniLM.
• Similarity Search: Rank documents by cosine similarity to either individual $E(q_i)$ or the mean embedding $\bar{q}$.
• Candidate Pooling: Retrieve top-$k_i$ per $q_i$, and top-$n$0 for $n$1, combining disjoint pools.
• Pre-filtering: Remove documents with $n$2 or $n$3 (e.g., $n$4, $n$5 characters), deduplicate with Jaccard threshold $n$6, and optionally filter by language or metadata.

3.2 Augmentation

• Prompt Construction: Wrap each candidate $n$7 in a template injection that includes sampled few-shots. The LLM is prompted to generate a new sample matching the few-shot's style, format, and task target but grounded in $n$8's content.
• LLM Sampling: Parameters include $n$9, $Q = \{q_1,...,q_n\}$0, $Q = \{q_1,...,q_n\}$1.
• Validation and Filtering: Ensure outputs are well-formed (e.g., JSON), enforce task key schema (e.g., four options A–D for QA), deduplicate with fuzzy matching, drop malformed or deficient records.

A high-level pseudocode outline is as follows:

$Q = \{q_1,...,q_n\}$7

Typical output is in JSONL format, compatible with standard LLM fine-tuning workflows.

4. Fine-Tuning, Evaluation, and Metrics

The AIDev synthetic corpus supports fine-tuning of foundation models such as Mistral-7B or LLaMA-8B, typically using LoRA adapters (all linear layers, $Q = \{q_1,...,q_n\}$2, $Q = \{q_1,...,q_n\}$3, $Q = \{q_1,...,q_n\}$4), AdamW optimizer (weight decay $Q = \{q_1,...,q_n\}$5, $Q = \{q_1,...,q_n\}$6), batch size 16, and 16-bit BrainFloat compute. Three-epoch schedules with 10% warm-up are standard.

Evaluation relies on:

• QA Tasks: Exact-match accuracy (greedy decode).
• Text Generation: ROUGE (1/2/L), METEOR, human or LLM-based preference win-rate evaluations.
• Contamination Checks: Test set and training corpus overlap under 5-gram weighted Jaccard < 0.5%.

Iterative refinement of few-shots, retrieval parameters, or prompts is recommended if target performance lags human-curated baselines. Comparative performance for QA and summarization shows parity or superiority relative to existing LLMs when training on the synthetic corpus (Ziegler et al., 2024).

5. Extensibility and Specialization

CRAFT generalizes to multiple AIDev tasks beyond QA and summarization:

• Code Generation: Using code corpora (GitHub, StackOverflow), code-aware embeddings (e.g., CodeBERT), and prompts targeting code synthesis (“Given this function signature …”).
• API Documentation: Retrieval and augmentation of code blocks/docstrings into OpenAPI specs.
• Multimodal Extension: CLIP image/text embeddings, producing captioning or grounded summaries.
• Domain Customization: Biomedical (BioBERT, ICD-10 prompts), legal (legislation retrieval, brief generation), finance (FinBERT, summaries).

6. Accessibility, Licensing, and Reproducibility

The AIDev dataset (Li et al., 9 Feb 2026) is distributed on Hugging Face and Zenodo with explicit support for open research and industrial use. File formats include Parquet, CSV, and JSONLines. Helper functions and example Jupyter/Colab notebooks are provided for exploration and analysis. The CRAFT corpus recipes do not bundle raw data but provide reproducible pipelines for dataset instantiation, compatible with standard open-source models and toolkits. The design encourages derivative works, further augmentation, and domain transfer, provided attribution is preserved (Ziegler et al., 2024).