Toward Instructions-as-Code: Understanding the Impact of Instruction Files on Agentic Pull Requests

Abstract: AI-agents (e.g., GitHub Copilot) collaborate as teammates in different software engineering tasks, including code generation proposed through pull requests (Agentic-PRs). For better agent efficiency, developers create instruction files that guide the AI-agents, including how to navigate the project, locate the right components, run tests, respect best practices, and more. In this paper, we investigate the relationship between the creation of these instructions and the performance of AI-agents in creating better pull requests, which have a higher chance of success (i.e., the merge rate), address more complex tasks (e.g., code churn), and require less effort to be merged (e.g., time to merge). To this end, we analyze 15,549 agentic PRs from 148 projects in the AIDev dataset. Using the three dimensions, we compare each project before and after the creation of the instruction files. We find that specifying instructions for AI-agents does not necessarily lead to better results. With the instruction files, 27.7\% of the projects increased their merge rate by at least 20\%, while 26.35\% decreased it. The same observation is seen with the amount of changes (e.g., code churn, number of modified files) and with the efforts to merge an agentic PR (e.g., merge time and number of comments). From a first exploration, we find that projects that managed to increase their merge rate have substantially longer instruction files, which are also well structured into a higher number of sections and sub-sections. Our results motivate the need for research to assist practitioners in framing the development of instruction files as a software engineering activity (aka, \textbf{Instructions-as-Code}).

• The paper shows that introducing detailed instruction files leads to mixed effects on agentic pull request merge rates, demonstrating both improvements and declines.
• It employs rigorous statistical methods on 15,549 PRs to analyze changes in complexity metrics like code churn, commit counts, and merge effort indicators.
• The study highlights that verbose, well-structured instruction files correlate with improved merge rates, emphasizing the importance of quality instructions in autonomous code contributions.

Assessing the Impact of Instruction Files on Agentic Pull Requests: An Empirical Analysis

Problem Statement and Motivation

The integration of LLM-based autonomous coding agents (e.g., Copilot, Claude, Cursor, Devin) into software development workflows has led to the emergence of agent-generated pull requests (Agentic-PRs), fundamentally shifting collaboration practices. To optimize agent productivity, developers now provide project-specific instruction files, which encode repository navigation, task guidelines, and best practices. Despite their rising adoption, there is no quantitative consensus on whether these instruction files actually enhance the quality, success, or complexity of Agentic-PRs.

This work conducts a large-scale, quantitative analysis of 15,549 Agentic-PRs across 148 projects from the AIDev dataset, explicitly investigating the effect of instruction file introduction on multiple outcome dimensions: merge rate, PR complexity (code churn, file/commit counts), and merge effort (time-to-merge, discussion count). The study also correlates changes in merge metrics with the verbosity and structure of instruction files, aiming to clarify if “instructions-as-code” leads to empirically measurable improvements.

Methodology

The study systematically filters repositories in AIDev (minimum 20 GitHub stars, non-toy projects, minimum repository and PR activity) and leverages GitHub's GraphQL API to extract PRs co-authored by coding agents. Instruction files are identified with agent-specific filename heuristics and timestamped by their first commit. Projects are included if they have at least five Agentic-PRs both before and after the addition of instruction files, resulting in a substantial but carefully curated dataset.

The analysis is conducted in four axes:

1. Merge Rate: Project-level change in PR merge rate before vs. after instruction file addition.
2. PR Complexity: Statistical assessment (Mann-Whitney U test, Cliff’s delta) of code churn, commit, and file counts in merged PRs.
3. Merge Effort: Statistical analysis of discussion comments and time-to-merge before and after instruction file creation.
4. Instruction File Verbosity: Quantitative comparison of word count and Markdown header sections in instruction files for projects with strong merge rate shifts.

Merge Rate Analysis

The introduction of instruction files is not universally beneficial for Agentic-PR merge rates.

Figure 1: Distribution of projects by merge rate improvement interval after the first instruction file; both improvement and decline are common.

For PRs closed after the first instruction file, 27.7% of projects showed a ≥20% increase in merge rate; however, 26.35% saw a ≥20% decrease. The effect remains when considering the last-added instruction file, with 31.93% of projects exhibiting a substantial improvement and 27.73% experiencing substantial degradation.

Figure 2: Merge rate interval distribution characterized before the first and after the last agent instruction file; again, both positive and negative outcomes are prominent.

This bidirectional outcome suggests that the mere presence of instructions does not guarantee improved agentic contribution acceptance. The impact is heterogeneous and context-dependent.

Task Complexity Evolution

Instruction file introduction has a mixed but generally upward effect on PR complexity metrics.

For projects passing the merged-PR count threshold, 35.35% demonstrated significant increases in PR description length, 10.10% in code churn, and 12.12% in commit counts following instruction file addition.

Figure 3: Number of projects exhibiting significant increases ($+$) or decreases ($-$) in PR complexity metrics after first/last agent file creation.

In contrast, between 4.04% and 7.07% of projects saw significant declines in these metrics. The increase in description length may reflect agents leveraging richer context, while increased code churn and file modification imply more complex or ambitious PRs. However, for a non-trivial minority, instruction files appear to constrain or confuse agentic PR generation.

Effects on Merge Effort

The effect of instruction files on PR merge effort is similarly ambiguous.

Following the introduction of instruction files, 13.13% and 15.15% of projects saw statistically significant increases in time-to-merge and discussion comments, respectively; 8.8% saw a decrease in these metrics.

Figure 5: Projects showing significant increases or decreases in merge effort metrics (time-to-merge, comments) after first/last instruction file addition.

This suggests that more detailed or restrictive instructions can both catalyze and hinder agent-human interaction around PRs, potentially increasing review/discussion requirements in some contexts. The variation in outcomes may be attributable to variability in instruction precision, agent support, or project practices.

Instruction File Verbosity and Structure

Projects with substantial merge rate improvements (≥20%) provide instruction files that are quantitatively richer and more structured.

Figure 4: Projects with higher merge rate increases have significantly longer instruction files (median 976 vs. 569 words).

Statistical analysis shows instruction files in these projects are nearly twice as long (median 976 vs. 569 words, p = 0.0029) and contain more header sections at all Markdown header levels, especially H3 subsections, reflecting more granular guidance.

Figure 7: Higher-performing projects (by merge rate) employ more section headers, especially at H2/H3, indicating more detailed structure in agent instructions.

In contrast, projects with declining merge rates typically have shorter and flatter instruction files, supporting the hypothesis that depth, detail, and precise organization in guidance are correlated with improved agentic PR success.

Implications

The findings have several theoretical and practical implications:

• Instructions are not a panacea: The heterogeneous effect of instruction file introduction mandates nuanced engineering practices—simply adding instructions is insufficient and, in some contexts, counterproductive.
• Instruction quality/structure is critical: Verbose, well-structured, and context-rich instruction files are positively associated with PR success, supporting the formalization of “instructions-as-code” as a new SE discipline requiring design, review, and evolution.
• Research direction: There is a clear opportunity for automated or assisted frameworks that help create high-quality, effective instruction files, including standards for content density, sectioning, and project-context alignment.
• Human-agent co-evolution: Increasing merge effort metrics in some projects highlights the non-trivial integration of autonomous agents into social-technical workflows, requiring further work on contextual adaptation and human-agent collaboration strategies.

Conclusion

This empirical study provides rigorous evidence that the development and management of agent instruction files has a complex, non-monotonic effect on autonomous pull request success, task complexity, and integration effort. Projects where instruction files are deployed as thoughtfully structured, verbose, and conceptually rich artifacts benefit most—the “instructions-as-code” paradigm should be recognized as a first-class concern in SE, meriting dedicated guidelines, quality metrics, and research focus to unlock the full potential of autonomous coding agents.

Reference: "Toward Instructions-as-Code: Understanding the Impact of Instruction Files on Agentic Pull Requests" (2606.13449).