--- title: 'AIDev Dataset Corpus: Agentic PRs & CRAFT' url: https://www.emergentmind.com/topics/aidev-dataset-corpus type: topic --- # AIDev Dataset Corpus: Agentic PRs & CRAFT The AIDev Dataset Corpus refers to a suite of large-scale, systematically curated, and extensible corpora for AI development (“AIDev”), emphasizing datasets that either (1) capture the behavior and output of AI coding agents in real-world contexts, or (2) are constructed for synthetic data-driven advancement of foundation models and agentic systems. Most notably, “AIDev Dataset Corpus” is associated with the AIDev collection of agent-authored pull requests and the methodology for creating synthetic, task-specific corpora via the CRAFT workflow. These corpora support analysis of agentic AI, facilitate benchmarking and fine-tuning, and enable studies of human–AI collaboration, quality control, and adoption patterns in “Software Engineering 3.0” [2602.09185][2409.02098]. ## 1. Motivation and Conceptual Scope The AIDev Dataset Corpus addresses two intersecting needs in contemporary AI development: - Enabling empirical study of AI coding agents and their integration in real-world collaborative software development environments, as formalized by the AIDev initiative and related dataset releases [2602.09185]. - Providing mechanisms and methodologies—exemplified by CRAFT (Corpus Retrieval and Augmentation for Fine-Tuning)—for scalable generation of task-specific, high-quality synthetic datasets in AI development, software engineering, and related domains. This aligns with the need for reproducible, customizable, and high-coverage data for training, benchmarking, and domain adaptation [2409.02098]. ## 2. AIDev: Dataset of Agentic Pull Requests AIDev presents the first large-scale corpus of agent-authored pull requests (“Agentic-PRs”) mined from public GitHub repositories. The dataset comprises 932,791 PRs written by five agentic systems (OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code), spanning 116,211 repositories and 72,189 unique developers up to August 1, 2025. A curated subset focuses on 33,596 PRs from 2,807 repositories with at least 100 stars, enriching the base with review artifacts, comments, issues, and changelogs [2602.09185]. The dataset captures extensive metadata, including agent origin, language, PR state, code diffs, review events, and developer annotations. The aim is to facilitate systematic research into the adoption rate ($R = N_{agentic}/N_{total}$), diversity (normalized Shannon entropy), review dynamics, security impact, and workflow patterns inherent in Software Engineering 3.0 settings. Key distributional statistics: | Statistic | Value (Full) | Value (High-Star) | |----------------------------|------------------|------------------------| | Agentic-PRs | 932,791 | 33,596 | | Repositories | 116,211 | 2,807 | | Developers | 72,189 | 1,796 | | Top Language (Full) | JavaScript 28% | — | | Median PR size | 35 lines | — | | Median time-to-merge | 2.1 days | 4.3 days | The data model is highly normalized, with 15 interrelated tables for PRs, users, comments, reviews, commits, issues, timeline events, and automated annotations (including task type via regex or LLM tagging). Supported export formats include Parquet, CSV, and JSONLines. ## 3. Synthetic Corpus Generation via CRAFT CRAFT (“Corpus Retrieval and Augmentation for Fine-Tuning”) defines an extensible pipeline for generating synthetic, task-specific training corpora based on a small few-shot seed, large-scale public corpora, and LLM-based transformation [2409.02098]. The pipeline comprises two primary phases: ### 3.1 Corpus Retrieval - **Embedding**: Encode $n$ few-shot task exemplars $Q = \{q_1,...,q_n\}$ and $N$ corpus documents $D = \{d_1,...,d_N\}$ in $\mathbb{R}^d$ using a SentenceTransformer such as MiniLM. - **Similarity Search**: Rank documents by cosine similarity to either individual $E(q_i)$ or the mean embedding $\bar{q}$. - **Candidate Pooling**: Retrieve top-$k_i$ per $q_i$, and top-$k_{avg}$ for $\bar{q}$, combining disjoint pools. - **Pre-filtering**: Remove documents with $len(d)L_{max}$ (e.g., $L_{min}=200$, $L_{max}=25k$ characters), deduplicate with Jaccard threshold $>0.85$, and optionally filter by language or metadata. ### 3.2 Augmentation - **Prompt Construction**: Wrap each candidate $d_j$ in a template injection that includes sampled few-shots. The LLM is prompted to generate a new sample matching the few-shot's style, format, and task target but grounded in $d_j$'s content. - **LLM Sampling**: Parameters include $temperature=0.7$, $top_k=40$, $top_p=0.9$. - **Validation and Filtering**: Ensure outputs are well-formed (e.g., JSON), enforce task key schema (e.g., four options A–D for QA), deduplicate with fuzzy matching, drop malformed or deficient records. A high-level pseudocode outline is as follows: ```python # 1. Embed few-shots & corpus E_few = [E(q) for q in few_shots] store_corpus_embeddings(E(d_j) for d_j in raw_corpus) # 2. Retrieve candidates candidates = set() for q_emb in E_few: candidates |= top_k_by_similarity(q_emb, corpus_db, k=k_i) avg_emb = mean(E_few) candidates |= top_k_by_similarity(avg_emb, corpus_db, k=k_avg) candidates = filter_by_length_and_dup(candidates) # 3. Augment each candidate synthetic = [] for d in candidates: prompt = build_prompt(few_shots=sample(few_shots,3), doc=d) out = LLM.generate(prompt, temp=0.7, top_k=40, top_p=0.9) if valid_format(out): synthetic.append(parse_JSON(out)) # 4. Deduplicate, save synthetic = dedupe_fuzzy(synthetic, threshold=0.85) save_as_JSONL(synthetic, "aidc_corpus.jsonl") ``` Typical output is in JSONL format, compatible with standard LLM fine-tuning workflows. ## 4. Fine-Tuning, Evaluation, and Metrics The AIDev synthetic corpus supports fine-tuning of foundation models such as Mistral-7B or LLaMA-8B, typically using LoRA adapters (all linear layers, $rank=64$, $\alpha=64$, $dropout=0.1$), AdamW optimizer (weight decay $0.05$, $lr=10^{-4}$), batch size 16, and 16-bit BrainFloat compute. Three-epoch schedules with 10% warm-up are standard. Evaluation relies on: - **QA Tasks**: Exact-match accuracy (greedy decode). - **Text Generation**: ROUGE (1/2/L), METEOR, human or LLM-based preference win-rate evaluations. - **Contamination Checks**: Test set and training corpus overlap under 5-gram weighted Jaccard < 0.5%. Iterative refinement of few-shots, retrieval parameters, or prompts is recommended if target performance lags human-curated baselines. Comparative performance for QA and summarization shows parity or superiority relative to existing LLMs when training on the synthetic corpus [2409.02098]. ## 5. Extensibility and Specialization CRAFT generalizes to multiple AIDev tasks beyond QA and summarization: - **Code Generation**: Using code corpora (GitHub, StackOverflow), code-aware embeddings (e.g., CodeBERT), and prompts targeting code synthesis (“Given this function signature …”). - **API Documentation**: Retrieval and augmentation of code blocks/docstrings into OpenAPI specs. - **Multimodal Extension**: CLIP image/text embeddings, producing captioning or grounded summaries. - **Domain Customization**: Biomedical (BioBERT, ICD-10 prompts), legal (legislation retrieval, brief generation), finance (FinBERT, summaries). ## 6. Accessibility, Licensing, and Reproducibility The AIDev dataset [2602.09185] is distributed on Hugging Face and Zenodo with explicit support for open research and industrial use. File formats include Parquet, CSV, and JSONLines. Helper functions and example Jupyter/Colab notebooks are provided for exploration and analysis. The CRAFT corpus recipes do not bundle raw data but provide reproducible pipelines for dataset instantiation, compatible with standard open-source models and toolkits. The design encourages derivative works, further augmentation, and domain transfer, provided attribution is preserved [2409.02098]. Source: https://www.emergentmind.com/topics/aidev-dataset-corpus