Do Coding Agents Follow AGENTS.md? The Effect of Policy Wording on Instruction Compliance
Abstract: Coding agents do not always follow the policies in an Agent Context File (ACF). We inserted one policy, in four styles, into the ACFs of 86 benchmark tasks across four harness-model combinations. Conditional phrasing was followed less often than Prescriptive and Explanatory. The better of those two varied by combination.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
1. What is this paper about?
This paper studies whether the way an instruction is written affects how well an AI coding agent follows it.
Many AI coding tools read a file such as AGENTS.md before working on a software project. This file contains rules, for example:
- “Always run the tests before finishing.”
- “Do not edit generated files.”
- “Never include passwords in the code.”
The researchers wanted to know whether small changes in the wording of these rules make AI agents more or less likely to obey them.
2. Main research questions
The paper asks two main questions:
- Does the writing style of a rule change how often an AI agent follows it?
- Does the best writing style stay the same when using different AI models and coding tools?
In other words, the researchers wanted to find out whether there is one “best” way to write instructions for all coding agents.
3. How did the researchers investigate this?
The test rule
The researchers used one simple rule:
Whenever the agent adds or changes a Python function, it must place the comment
# @trackeddirectly below the function’sdefline.
For example, a correct change would look like this:
1 2 3 |
def calculate_total(items): # @tracked return sum(items) |
This rule was useful because a computer could easily check whether the comment was in the correct place.
Four writing styles
The same basic rule was written in four different ways:
| Style | Example idea |
|---|---|
| Prescriptive | Directly tells the agent what to do. |
| Explanatory | Gives the instruction and explains why it matters. |
| Conditional | Says what to do if a certain condition happens. |
| Prohibitive | Says what the agent must never do. |
The researchers added each version to real project instruction files. They then asked AI coding agents to solve programming tasks.
Projects, tools, and models
The experiment used:
- 86 programming tasks
- Projects from 10 software repositories
- 4 combinations of coding tools and AI models
- 3 repeated attempts for each task and writing style
The researchers then checked whether every Python function changed by the agent included the required # @tracked comment.
They calculated a compliance rate, which simply means the percentage of tasks where the AI followed the rule correctly.
The researchers also used statistical tests. These are mathematical methods that help determine whether a difference is probably real rather than caused by random chance.
4. What did they find?
Conditional instructions were usually the least effective
The clearest result was that conditional instructions were followed less often.
For example, the conditional version said:
“If you add or modify a Python function, add the comment…”
This was less successful than a direct instruction or an instruction with an explanation.
For one model, the compliance rates were:
| Writing style | Compliance rate |
|---|---|
| Prescriptive | 78.0% |
| Explanatory | 77.2% |
| Conditional | 64.2% |
| Prohibitive | 70.7% |
The researchers think conditional instructions may be harder for an AI agent because the agent must first decide whether the condition applies to its action. That extra decision may cause the rule to be missed.
There was no single best style for every AI system
The direct and explanatory styles did not always perform in the same way.
For one model, the prescriptive style worked best. For another model, the explanatory style worked better. This means that an instruction that works well for one AI model may not work as well for another.
The results were especially different for the local models. One model followed the rules less than 21% of the time in every style. Another followed them less than 1.5% of the time.
This shows that the model and the coding tool themselves can have a very large effect on instruction-following.
The important pattern
The main pattern was:
- Conditional wording was usually worse than direct or explanatory wording.
- Prescriptive and explanatory wording competed with each other.
- The better of those two styles depended on the particular AI model and coding tool.
5. Why are these results important?
The study shows that writing instructions for AI agents is not just about what the rule means. The exact wording also matters.
For software developers, this suggests two practical lessons:
- Avoid relying on conditional wording when possible. A direct rule may be easier for an AI agent to understand and follow.
- Test instructions with the specific AI tool being used. Developers should not assume that a rule written successfully for one model will work equally well with another.
This matters because files such as AGENTS.md are often used across many programming tasks. If an agent ignores an important rule, it might introduce bugs, change files it should not touch, or accidentally create security problems.
6. Limitations of the study
The researchers also explain that their results are not the final answer.
For example:
- They tested only one rule.
- Each writing style used only one exact sentence, so the results might partly depend on the particular wording.
- The tasks came from only 10 repositories.
- The experiment focused on Python functions and one special comment.
- Some sessions where the agent did not change a Python function were left out.
- The researchers could not completely separate the effects of the AI model from the effects of the coding tool.
Because of these limits, the findings may not apply to every type of instruction or every AI coding agent.
7. Simple conclusion
The paper shows that AI coding agents do not always follow instructions in their context files. The way a rule is written can change the chance that the agent obeys it.
The safest general advice from this study is to avoid complicated conditional wording. Direct instructions, or instructions that include a short explanation, usually worked better. However, the best choice depends on the particular AI model and coding tool.
In the future, researchers may test many more kinds of rules and develop systems that automatically choose the most effective wording for each AI agent.
Practical Applications
Immediate Applications
The findings support practical changes to how organizations configure and supervise repository-level coding agents. These uses are deployable with existing agent harnesses, code repositories, and CI tooling, although they should be validated for the specific model–harness combination.
- Software engineering: rewrite ACF policies to avoid conditional phrasing. Developers can replace policies such as “If you modify a function, remember to add a tracking comment” with direct formulations such as “Add the tracking comment below every modified function.” The study found that conditional wording produced lower compliance than Prescriptive and Explanatory wording in every combination where style differences were statistically significant. Dependency: The result was demonstrated using one Python-comment policy, so teams should test equivalent wording for security, testing, documentation, and repository-specific rules.
- Software engineering: use direct or explanatory policies as the initial default.
- a direct command: “Run the complete test suite before submitting changes”; or
- a direct command plus rationale: “Run the complete test suite before submitting changes because the CI pipeline rejects untested modifications.”
- The better of these two styles depends on the agent environment, but both generally outperformed conditional phrasing in the study.
- Dependency: Prescriptive wording was strongest for some combinations, while Explanatory wording was strongest for others; neither should be treated as universally optimal.
- Software quality assurance: add mechanical compliance checks rather than relying on ACF text alone.
- tests being added or executed;
- generated files remaining unmodified;
- credentials not being committed;
- required audit markers being present;
- formatting and static-analysis checks passing.
- This follows directly from the paper’s mechanical compliance criterion and reduces the risk that an agent silently ignores an ACF instruction.
- Dependency: Policies must be converted into objectively checkable rules. Natural-language-only requirements remain difficult to verify.
- AI-agent operations: maintain model–harness-specific policy templates.
- model;
- agent harness;
- prompt-injection or context-loading mechanism;
- repository type.
- A lightweight evaluation can run a representative task suite under alternative Prescriptive and Explanatory formulations and select the better-performing version.
- Dependency: The study could not fully separate model effects from harness effects, and the optimal style varied across combinations.
- Software procurement and deployment: include instruction-compliance testing in agent evaluations.
Before adopting an autonomous coding tool, engineering teams can evaluate not only issue-resolution accuracy but also whether the tool follows repository policies. A benchmark should measure:
- compliance with security and testing rules;
- compliance across repeated runs;
- performance on modified and newly created files;
- behavior across different policy formulations. This is particularly relevant for enterprise software, regulated codebases, and open-source contribution workflows. Dependency: The reported compliance levels are specific to 86 tasks from 10 repositories and should not be interpreted as universal performance estimates.
Cybersecurity and compliance: convert high-risk instructions into explicit, testable controls. Security teams can rewrite vague ACF rules such as “Be careful with secrets” into direct requirements such as “Never add credentials, API keys, or private keys to tracked files.” They can then reinforce the requirement with secret-scanning tools and blocked commits. Similar workflows apply to license compliance, data-handling rules, and restrictions on production-system access. Dependency: The paper measured a low-risk marker-comment task, not actual secret handling; safety-critical deployment requires independent enforcement.
- Open-source project governance: strengthen agent contribution rules. Maintainers can place explicit contribution policies in repository context files, such as “Run the project’s test command before opening a pull request” or “Do not modify generated files; update the source definition instead.” CI can verify these requirements automatically. This can reduce inconsistent agent-generated pull requests and make agent contributions easier to audit. Dependency: Prior work cited by the paper indicates that some contribution-rejection rules may have extremely low compliance, so ACF wording should not replace human review or repository protections.
- Education and training: teach policy writing as part of AI-assisted programming. Universities and professional-training programs can use the study as a practical exercise: students write Prescriptive, Explanatory, Conditional, and Prohibitive versions of the same coding rule, run them through an agent, and compare compliance. This develops skills in prompt engineering, software verification, and empirical evaluation. Dependency: Results may vary with the selected model, harness, programming language, and task complexity.
- Academic research: reproduce and extend the experiment with broader policy categories. Researchers can use the paper’s replication package and methodology to study policies involving testing, security, documentation, dependency management, generated code, licensing, and data privacy. A stronger design should use multiple phrasings per style, larger repository samples, and independent models and harnesses. Dependency: The original experiment confounds style with the exact wording and length of one policy realization, so replication is needed before making general linguistic claims.
- Daily life and individual use: write clearer rules for personal coding assistants. Individual developers can improve local agent configuration files by using short, direct instructions instead of conditional reminders. For example, “Run the formatter on every changed Python file” is preferable to “If you change a Python file, you might want to run the formatter.” Dependency: Users should still inspect changes because the paper found substantial noncompliance, especially for some open-weight model–harness combinations.
Long-Term Applications
These applications require additional research, larger-scale validation, or development of new agent-management infrastructure.
- Automatic policy-style optimization for each model and harness. A future tool could generate several semantically equivalent versions of an ACF rule, execute a standardized test suite, measure compliance, and recommend the best formulation for a particular model–harness pair. The system could maintain separate policy profiles for commercial and local agents. Dependencies: This requires reliable semantic-equivalence checking, representative task suites, stable model behavior, and protection against optimizing for narrow benchmark artifacts.
- Adaptive ACFs that select wording dynamically. Agent platforms could detect the active model and harness and load an appropriate policy variant. For example, one environment might receive a Prescriptive formulation while another receives an Explanatory formulation. A central policy specification would preserve the intended rule while allowing environment-specific rendering. Dependencies: Dynamic rendering must not introduce contradictions, weaken security requirements, or create inconsistent behavior across team members.
- Policy compilers and typed agent configuration languages.
- natural-language instructions for the agent;
- CI checks;
- pre-commit hooks;
- audit logs;
- human-readable documentation.
- Such a system could represent conditions formally while presenting the agent with a direct operational instruction, addressing the finding that conditional language is often overlooked.
- Dependencies: A policy language must capture ambiguous software practices and remain compatible with existing agent harnesses.
- Continuous compliance monitoring for autonomous software agents. Enterprise platforms could record which ACF rules were applicable to each agent action, whether the rule was followed, and whether a human or automated checker detected a violation. This would support auditability in healthcare, finance, government, and other regulated sectors. Dependencies: Monitoring must distinguish genuine violations from inapplicable rules, handle multi-file changes, protect source-code confidentiality, and avoid excessive logging overhead.
- Agent safety gates for high-impact workflows.
- healthcare software requiring tests and traceability;
- financial systems requiring security scans and approval;
- industrial or robotic systems prohibiting unreviewed control-logic changes;
- energy systems requiring change-impact analysis.
- Dependencies: The paper does not establish that ACF wording alone is sufficient for safety. Independent verification, access control, sandboxing, and human approval remain necessary.
- Benchmarking standards for instruction compliance. The field could develop shared benchmarks that report compliance separately from task success. Such benchmarks should include multiple policy types, policy conflicts, long-horizon tasks, repeated runs, different programming languages, and both commercial and open-weight systems. This would help organizations compare agents on operational reliability rather than code-generation quality alone. Dependencies: Benchmarks must avoid rewarding superficial marker insertion and should test whether the agent understood and consistently applied the underlying policy.
- Research on why conditional policies fail.
- imperfect self-monitoring;
- instruction salience;
- context-window placement;
- planning failures;
- differences in model alignment or harness prompt construction.
- Findings could lead to improved agent planners that explicitly track applicable policies before editing files.
- Dependency: Causal conclusions require experiments that isolate wording, sentence length, policy complexity, and context position.
- Agent planning systems with explicit policy checklists. Coding agents could create an internal or externally visible checklist before making changes, update it as actions occur, and run a final compliance pass before completion. This would operationalize policies rather than leaving them as passive context. Dependencies: The checklist itself could be ignored or incorrectly completed, so it should be coupled with executable validators and adversarial testing.
- Cross-domain adaptation beyond coding. The central insight—that equivalent rules may be followed differently depending on wording and system implementation—could be studied in document automation, customer-service agents, robotics, laboratory automation, and administrative workflows. For example, a robot-control agent might receive direct operational constraints while a planning assistant receives explanatory safety requirements. Dependencies: Transfer is speculative: the study evaluated repository-grounded coding agents, not physical agents or high-stakes decision systems.
- Policy-aware agent marketplaces and enterprise governance tools. Future commercial products could offer policy libraries, compatibility ratings, compliance dashboards, and tested templates for common requirements such as security, privacy, testing, licensing, and data retention. Organizations could select a policy package based on their agent model and harness. Dependencies: Ratings would need regular revalidation because model updates, harness changes, and prompt-construction changes may alter compliance behavior.
- Improved human–agent workflows for everyday users. Personal productivity tools could translate informal preferences into explicit, verifiable agent rules—for example, “never overwrite my original files” becoming a direct instruction plus automatic backup and diff checks. This could make autonomous assistants safer for coding, writing, spreadsheet manipulation, and file organization. Dependencies: User preferences are often ambiguous, and automatic translation must preserve intent without creating overly restrictive or conflicting policies.
Glossary
- Agent Context File (ACF): A natural-language configuration file that provides an AI agent with project context and behavioral instructions. “An ACF is a Markdown file written in natural language, such as AGENTS.md, that describes a project’s background and instructions.”
- alignment methods: Techniques used to train or configure AI models to follow intended human instructions and behavioral objectives. “Because models and harnesses may differ in their alignment methods and training data, a writing style that works in one environment may not work in another.”
- benchmark task: A standardized software-engineering problem used to evaluate the performance of coding systems. “As a case study, we took benchmark tasks from repositories that already contain an ACF and placed a single target policy into each repository’s ACF.”
- compliance rate: The proportion of evaluated sessions or tasks in which an agent satisfies a specified policy. “For every session we judged whether the agent complied with the policy, and calculated the compliance rate for each writing style.”
- confounding: A methodological problem in which the effects of two or more factors cannot be distinguished. “First, we used one text per style, so a style is confounded with its single realization.”
- conditional phrasing: Wording that makes an instruction applicable only when a stated condition is met. “A conditional policy such as ‘if you add or modify a function’ requires the agent to first judge whether its own action meets the condition.”
- effect size: A statistical measure indicating the magnitude of a difference or relationship. “We use the matched-pairs rank-biserial correlation as the effect size.”
- factorial study: An experimental design that investigates the effects of multiple factors and, often, their interactions. “Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variables”
- harness: The software framework that executes an AI model, supplies it with task context, and manages its interaction with a repository. “In addition, the effect of writing style may not be consistent across models and harnesses.”
- LLM: A machine-learning model trained on large amounts of text to generate and analyze natural language and code. “Previous studies have shown that LLMs often generate outputs while ignoring policies contained in prompts.”
- matched-pairs rank-biserial correlation: A nonparametric effect-size statistic for paired observations that expresses the direction and magnitude of differences between two conditions. “We use the matched-pairs rank-biserial correlation as the effect size.”
- multiple-comparisons correction: A statistical adjustment that reduces the risk of false-positive findings when several hypothesis tests are performed. “If significant, we perform the [PERSON_NAME] signed rank test on all pairs of writing styles and correct for multiple comparisons using the [PERSON_NAME] method.”
- open-weight model: An AI model whose trained parameters are publicly available for use or deployment. “For the evaluation of local models, we compare gpt-oss-120b and the code-specialized Qwen3-Coder-Next on the qwen-code harness.”
- policy: An instruction that a developer expects an AI agent to follow. “We call an instruction that a developer wants an AI agent to follow a Policy.”
- Pratt method: A variant of the Wilcoxon signed-rank procedure that accounts for zero differences and tied observations. “Because the values per task are proportions over at most three repetitions, they take only a few levels and produce many ties, so we adopt the Pratt method to handle ties.”
- prescriptive: A writing style that states a direct instruction describing what an agent should do. “Prescriptive refers to a direct policy.”
- rank-biserial correlation: A nonparametric statistic used to quantify the strength and direction of differences between ranked paired or independent samples. “Table 4 presents the matched-pairs rank-biserial correlation for each pair of writing styles.”
- replication package: The materials, data, code, or procedures released to enable others to reproduce a study. “Our replication package is available at https://doi.org/10.5281/zenodo.23148524.”
- SWE-bench: A benchmark for evaluating whether LLMs can resolve real-world software issues in repositories. “We use CTXbench [8], a SWE-bench[9] like benchmark built from repositories with an ACF.”
- threats to validity: Limitations or potential sources of error that may affect the credibility or generalizability of research findings. “Four limitations qualify our results.”
- Wilcoxon signed-rank test: A nonparametric statistical test for comparing paired observations when distributional assumptions required by a paired -test may not hold. “If significant, we perform the [PERSON_NAME] signed rank test on all pairs of writing styles and correct for multiple comparisons using the [PERSON_NAME] method.”
