Do Coding Agents Follow AGENTS.md? The Effect of Policy Wording on Instruction Compliance
This presentation examines a controlled study of how linguistic formulation affects coding-agent compliance with repository policies. Using 86 tasks from CTXbench, researchers tested four policy phrasings—prescriptive, explanatory, conditional, and prohibitive—against a mechanically verifiable marker-comment requirement across four harness-model combinations. The results reveal that conditional wording consistently underperformed direct instructions, reducing compliance by 14 to 56 percentage points depending on the environment, while the advantage of explanatory over prescriptive phrasing varied by agent. The findings suggest that AGENTS.md files are not merely documentation but active control interfaces where word choice matters, though optimal formulation depends on the deployment environment.Script
When you drop a policy into an AGENTS.md file, does the way you phrase it actually change whether a coding agent follows the rule? This study finds that writing style can swing compliance by more than 50 percentage points.
The authors used CTXbench, a repository task benchmark, and inserted one mechanically verifiable policy: every Python function added or modified must include a marker comment directly below its definition line. Compliance was all-or-nothing—every function had to follow the rule or the session failed.
Across the Claude Opus 5 environment, prescriptive and explanatory phrasings achieved 78 and 77 percent compliance, but conditional wording fell to 64 percent. The conditional penalty was even steeper with Sonnet 5, where compliance dropped from 72 percent for prescriptive to just 36 percent for conditional.
The authors interpret the conditional penalty as an extra classification step. A direct instruction tells the agent what to do, while a conditional instruction first asks the agent to decide whether its current operation matches the triggering condition. That intermediate decision becomes a failure point.
The effect generalized partially. Conditional wording underperformed in every environment that showed significant style differences, but the best alternative was environment-dependent: prescriptive wording won for one agent, explanatory wording for another, and one local-model combination complied less than 2 percent of the time under every formulation.
The takeaway is precise: if you want a coding agent to follow a repository rule, avoid phrasing it as a conditional when a direct statement will do. But the optimal formulation depends on which agent reads it, so test your wording in your actual environment—and explore more studies like this one at EmergentMind.com, where you can create your own research video.
