Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix
Abstract: Purpose: Coding agents follow a project's instruction file, such as CLAUDE.md or AGENTS.md, even when it disagrees with the existing code and even when they notice the disagreement. We measure this behaviour and test whether one sentence in the user's own instruction file makes agents follow the code instead. Methods: We built 40 Python projects in which the instruction file asks for one convention while the existing code uses another, for example print instead of logging. Claude Code and Codex CLI added a feature to each project, with and without a sentence in the user-level instruction file that gives the existing code precedence, in 320 runs. Results: Without the sentence, 90.1% of the agents' choices followed the stale file. Claude Code pointed out the conflict in 79 of 80 runs and still followed the file. Codex CLI rewrote the existing code to match the file in 20 of 80 runs, and the project's own tests then failed in 17 of them. The sentence raised the share of choices that follow the code from 4.2% to 91.4%, stopped the rewrites and raised the test pass rate from 89.4% to 100%. Conclusion: We attribute the behaviour to how agents receive the two sources: the instruction file reaches the model as an instruction and the code as tool output, which models are trained to rank lower. Users can add the sentence today, and makers of agents and models can build the same rule into their products.
Summary
- the paper introduces a one-sentence fix to guide coding agents to follow existing code over stale instruction file conventions, significantly improving decision accuracy to 91.4% in 320 validated runs.
- the inconsistency between stale instruction files and current codebases often causes incoherence debt, leading to recurring issues and test failures in 21.2% of uncontrolled Codex CLI runs.
- two agents, Claude Code and Codex CLI, handling instruction files differently show that the problematic behavior is systematically related to the instruction file and code authoritativeness.
Research question and central claim
The paper investigates a specific failure mode in repository-scale coding agents: an instruction file such as CLAUDE.md or AGENTS.md becomes stale, conflicts with the current codebase, and nevertheless continues to control the agent’s implementation choices. The central claim is that this behavior is not merely a failure to inspect the repository. Rather, agents may inspect the relevant code, recognize the inconsistency, and still privilege the stale instruction because the instruction file enters the model context as an instruction whereas the code is encountered as tool output.
The proposed intervention is deliberately narrow. The authors add one sentence to the user-level instruction file:
When a repository's documentation and its existing code disagree, follow the conventions used in the existing code, and in your final reply point out each inconsistency you found between the documentation and the code.
Across 40 synthetic but executable Python projects, two coding agents, two instruction-file conditions, and 320 valid runs, this sentence substantially reverses the observed preference. Without it, 90.1% of convention choices followed the stale instruction file and only 4.2% followed the existing code. With it, 91.4% followed the existing code. The paper therefore links a concrete empirical pathology to a concrete intervention at the instruction-hierarchy level (2610.30571).
How instruction files become stale
Instruction files combine two semantically different artifacts. Some lines are genuine procedural instructions to the agent—for example, commands for running tests or restrictions on modifying particular files. Other lines are descriptive claims about the repository, such as “use print() for progress messages” or “store monetary values as floating-point numbers.” The latter can become false when the implementation evolves, but their imperative syntax remains unchanged.
The resulting inconsistency is not hypothetical. The paper cites evidence that 73.8% of AI configuration files in a sample of 441 company repositories were committed once and never modified, while a consistency checker identified obsolete code references in 23.0% of 356 public repositories (Denisov-Blanch et al., 26 Aug 2026, Treude et al., 8 Jun 2026). The authors also document three cases in actively maintained repositories, including a Supabase instruction file that continued to describe React 18 for 13 days after the codebase had moved to React 19, and an OpenAI Codex instruction file that continued to request an update to a document deleted 24 days earlier.
Figure 1: Examples of instruction files remaining inconsistent with repository changes until a later corrective commit.
The operational consequence is persistent repetition. Because the file is loaded into each agent session, the same stale convention can influence every subsequent task. This converts documentation drift into what the paper characterizes as a recurring source of coherence debt: new code diverges from the established implementation, reviewers must identify and repair the divergence, and potentially correctness-relevant conventions are repeatedly reintroduced. The cited example involving monetary amounts is particularly important: a stale instruction can induce floating-point arithmetic where the existing code uses integer cents, making the problem semantic rather than stylistic (Mohammadi et al., 17 Aug 2026).
Instruction hierarchy as the explanatory mechanism
The paper’s explanation depends on the channel through which each source reaches the model. Claude Code inserts the project instruction file into a user message and explicitly frames it as authoritative. Codex CLI likewise places AGENTS.md in an instruction-bearing context. By contrast, the existing source code is normally returned after a file-reading tool call.
Figure 2: The instruction file reaches Claude Code as an instruction, whereas existing code reaches the model as tool output.
This distinction matters because contemporary models are trained to assign different authority to different input surfaces. The instruction hierarchy prioritizes system and user instructions over tool output, partly to prevent text embedded in web pages, files, or other external artifacts from overriding the agent’s goals. The paper connects this design principle to instruction-hierarchy research such as “The instruction hierarchy: training LLMs to prioritize privileged instructions” (Wallace et al., 2024) and IH-Challenge (Guo et al., 11 Mar 2026).
The failure occurs when a descriptive statement is delivered through the high-priority instruction channel. “Use print()” may have originally documented the repository, but after the code changes it functions as an active instruction. The code that contradicts it remains lower-authority evidence. The agent therefore interprets the conflict not as “documentation versus current implementation,” but as “instruction versus information.”
Figure 3: Claude Code identifies the conflict between print and logging, but follows the stale instruction without the intervention and the existing code with it.
This mechanism generates two testable predictions. First, agents should follow the stale instruction even when they explicitly notice the contradiction. Second, a higher-priority instruction that resolves the conflict in favor of existing code should reverse the outcome without requiring prior detection or manual correction of the stale line.
Experimental design
The authors construct 40 small Python projects containing three to eight working modules, a passing test suite, an instruction file, and a feature request. Each project presents a controlled conflict: the existing code consistently uses one convention while the instruction file requests the alternative. The feature request specifies required behavior but does not mention which convention to use.
Sixteen convention types are represented, including logging, formatting, imports, configuration, monetary representation, and enumeration design. Each type appears in both directions across projects, reducing the risk that a general model preference for one convention explains the results.
Figure 4: The 16 convention conflicts used in the projects, with both directions represented across the study.
The agents are Claude Code 2.1.302 with Claude Sonnet 5.5 and Codex CLI 0.163.0 with GPT-6 Sol. Each project is run under four combinations: agent, with or without the added sentence. Every run begins from a fresh project copy, uses automatic edit approval, and receives only the feature request as the task prompt. This produces 80 runs per agent-condition combination and 320 runs overall.
The primary automated measure classifies each implementation choice as matching the instruction file, matching the existing code, or matching neither. A separate blinded judge examines final responses to determine whether the agent reported the documentation–code conflict. Rewriting is operationalized as removal of at least 20 lines from pre-existing Python files. The authors restore the original test files after each run and evaluate both the original tests and held-out tests for the requested feature.
This design isolates source precedence reasonably well: the stale file, feature request, projects, and agent configurations remain fixed within comparisons, while the added sentence is the principal manipulation. However, the projects are intentionally small and synthetic, and the runs are unattended single-task executions; those properties become important when interpreting external validity.
Agents follow stale instructions despite recognizing the conflict
The baseline result is strong. Without the precedence sentence, 649 of 720 convention choices—90.1%—match the stale instruction file, while only 4.2% match the existing code. Claude Code follows the file in 86.4% of choices, and Codex CLI in 93.9%. The stale file prevails in 14 of the 16 convention categories.
Figure 5: Baseline choices overwhelmingly match the stale instruction file, while the precedence sentence shifts choices toward existing code.
The result is not reducible to simple failure to inspect the repository. Claude Code explicitly reports the discrepancy in 79 of 80 baseline runs. In those runs, 86.2% of its choices still follow the instruction file. A representative response states that several modules use logging, pathlib, str.format(), and absolute imports, “which go against CLAUDE.md.” The wording treats the existing implementation as nonconforming even though it is the codebase being extended.
This is a particularly consequential finding because conflict recognition does not imply conflict resolution. An agent can produce a correct diagnosis in its final report while implementing the lower-quality choice. For evaluation, this means that textual awareness metrics or self-reported repository understanding can substantially overestimate behavioral alignment.
Codex CLI exhibits a different failure pattern. It identifies the conflict in only 2 of 80 baseline replies; in 60 runs it follows the stale file without mentioning the conflict, and in the remaining 20 it modifies pre-existing code to conform to it. Thus, the same precedence structure can manifest either as localized inconsistency in newly added code or as destructive repository-wide normalization.
One sentence reverses the preference
Adding the precedence sentence changes the distribution sharply. The share of choices matching existing code rises from 4.2% to 91.4%, an increase of 87.2 percentage points, with a reported 95% resampling interval of 83.6 to 90.9 percentage points. Claude Code rises from 7.5% to 94.4%, and Codex CLI from 0.8% to 88.3%.
The intervention therefore does not merely reduce adherence to the stale file; it establishes existing code as the dominant convention source. Both agents report the detected inconsistencies in all 160 runs with the sentence. This is consistent with the sentence’s two functions: it specifies precedence for implementation and requires explicit reporting of the inconsistency.
The result supports the proposed channel-based explanation. The sentence itself arrives as a high-priority instruction, so it does not ask the model to infer that code should outrank documentation from lower-authority evidence. Instead, it instructs the model to use the code as the decision criterion whenever the two sources disagree. Other project instructions, such as build commands or file-organization rules, are not generally displaced; the precedence rule is conditional on a documentation–code conflict.
The intervention also reduces operational damage. Runs with the sentence use slightly fewer tool calls and less time, although the paper does not present this as a primary efficiency result. More importantly, no Codex CLI run rewrites existing code under the intervention condition.
Rewriting existing code and test failures
The most severe baseline behavior occurs in Codex CLI. In 20 of 80 runs without the sentence, it rewrites pre-existing code to conform to AGENTS.md. These runs remove a median of 198 lines across nine existing files and occur in 12 of the 40 projects. After restoration of the original tests, 17 of the 20 rewritten runs fail those tests.
Figure 6: A Codex CLI baseline run in which existing implementation code is rewritten to match a stale instruction.
One example replaces a Column enumeration in model.py with string constants because the instruction file requests that convention. The original tests can no longer import Column. This is not simply a mismatch between newly added code and repository style: the agent alters working code in a way that breaks established interfaces.
The corresponding test results quantify the practical effect. Overall, tests pass in 89.4% of runs without the sentence and in 100% of runs with it. For Codex CLI, the pass rate rises from 78.8% to 100%; every baseline failure is associated with a rewrite. Claude Code rewrites no existing code under either condition, so the agent-specific risk is heterogeneous. The paper’s intervention eliminates the observed destructive behavior in this benchmark, but the difference between agents also indicates that instruction precedence interacts with agent planning, editing policies, and model behavior rather than determining a single uniform failure mode.
Practical deployment implications
For users, the proposed sentence is a low-cost mitigation because it can be placed once in a user-level instruction file and applied across projects. It does not require identifying every stale line in advance. The sentence also requests that inconsistencies be reported, preserving an audit trail that can support later documentation repair.
For agent developers, the result supports incorporating an explicit documentation–implementation precedence rule into the agent’s system-level behavior. This would avoid relying on users to maintain a workaround. The paper distinguishes this conflict from conflicts between multiple instruction files: existing documentation for Claude Code addresses instruction-file precedence but does not resolve the case in which an instruction file contradicts the repository it describes.
For model developers, the paper identifies a tension in instruction-hierarchy training. Treating tool output as non-authoritative is necessary for prompt-injection resistance, but treating all instruction-channel text as semantically current can make stale repository descriptions dominate direct implementation evidence. A model that separates normative instructions from descriptive claims, or that validates descriptive claims against the code before applying them, could address this failure without simply assigning unrestricted authority to tool output.
Figure 7: The available mitigation paths for users, agent developers, and model developers.
The paper’s contribution is therefore not that instruction files are intrinsically harmful. Correct instruction files can provide useful repository context and improve agent behavior, as studied in work on AGENTS.md and coding-agent configuration (Gloaguen et al., 12 Feb 2026, Lulla et al., 28 Jan 2026). The narrower conclusion is that descriptive conventions should not retain unconditional instruction authority after the implementation has changed.
Limitations and open questions
The evidence is bounded by 40 deliberately constructed Python projects, 16 convention types, two commercial agents, and one unattended feature-completion protocol. The projects are small relative to industrial repositories, and the conflicts are intentionally clean: the existing code uses one convention consistently and the instruction file requests the other. Real repositories may contain partial migrations, exceptions, generated code, multiple valid conventions, or code that is itself defective. In such cases, “follow existing code” is not always the correct resolution rule.
The study also tests one sentence and one formulation. Its effectiveness may depend on placement, wording, instruction-file format, system-prompt composition, model version, tool permissions, and whether the agent is allowed to ask for clarification. The reported experiments do not establish that the sentence remains effective when the code is insecure, when tests are incomplete, or when the stale instruction encodes a safety or compliance requirement that should legitimately override local implementation precedent.
The causal interpretation is persuasive but not fully isolated from implementation-specific effects. The two agents use different models, command-line interfaces, context wrappers, and tool policies, so the results cannot attribute all variation exclusively to instruction hierarchy. The code-versus-instruction channel distinction is directly documented for Claude Code and inferred for Codex CLI from observed behavior and prompt structure. A controlled experiment that delivers identical content through systematically varied channels would provide a stronger test.
Finally, the benchmark measures convention choices, test outcomes, conflict reports, rewrites, tool calls, and elapsed time. It does not measure long-term repository maintenance, reviewer burden, security regressions, or behavior under multi-agent collaboration. Open questions therefore include whether agents can reliably distinguish descriptive claims from genuine directives, whether automatic consistency checking outperforms a generic precedence sentence, and how precedence should be resolved when both the instruction file and existing code contain safety-critical requirements.
Conclusion
The paper demonstrates that stale instruction files can dominate current repository code because the former are delivered as high-priority instructions while the latter appears as lower-authority tool output. In the benchmark, agents followed stale instructions in 90.1% of convention choices without intervention, even when Claude Code explicitly recognized the conflict. Codex CLI rewrote working code in 20 of 80 baseline runs, causing test failures in 17.
A single conditional sentence changes the outcome: 91.4% of choices then match existing code, destructive rewrites disappear, and the overall test pass rate reaches 100%. The result supports an operational rule for coding-agent systems: descriptive repository guidance must be conditional on consistency with the implementation, and explicit precedence should be encoded at an instruction level that the model will honor.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is the paper about?
This paper studies how coding agents—AI tools that can read and change computer programs—deal with outdated project instructions.
Many software projects have instruction files such as CLAUDE.md or AGENTS.md. These files tell an AI agent things like:
- which programming style to use,
- how the project is organised,
- which commands to run, and
- how to write new code.
The problem is that the instructions may become stale, meaning they are no longer correct. For example, an instruction file might say, “Use print(),” while all the existing code has already changed to use a better system called logging.
The paper asks: Will the AI follow the outdated instructions or the working code?
2. What questions did the researchers ask?
The researchers focused on three main questions:
- Do coding agents follow old instructions even when the existing code disagrees?
- Do the agents notice and report the disagreement?
- Can one extra sentence make the agents trust the existing code instead?
The sentence they tested was:
“When a repository's documentation and its existing code disagree, follow the conventions used in the existing code, and in your final reply point out each inconsistency you found between the documentation and the code.”
In simpler words, the sentence tells the AI:
“If the written instructions and the actual code disagree, follow the code and tell us about the problem.”
3. How did the researchers investigate this?
Creating test projects
The researchers built 40 small Python projects. Each project had:
- several working Python files,
- a test suite,
- an instruction file, and
- a request to add a new feature.
Each project was deliberately designed with a disagreement. For example:
| Existing code used | Instruction file said to use |
|---|---|
logging |
print() |
| one kind of data format | another data format |
| absolute imports | relative imports |
| one error-handling style | a different style |
The projects were like carefully prepared puzzles: the researchers knew exactly which source—the code or the instruction file—was correct.
Testing two coding agents
They tested two AI coding tools:
- Claude Code
- Codex CLI
Each tool worked on every project in two situations:
- Without the special sentence
- With the special sentence
This produced 320 total runs. A run means that an AI agent received a feature request and attempted to change one of the projects.
Checking the results
The researchers examined:
- whether new code followed the instruction file,
- whether it matched the existing code,
- whether the AI noticed the disagreement,
- whether it changed old working files,
- whether the tests still passed, and
- how much time and effort the agent used.
The tests acted like a safety check. If the tests failed, it meant the AI probably damaged something or added incompatible code.
4. What did the researchers discover?
AI agents usually followed the outdated file
When the special sentence was not included:
- 90.1% of the agents’ choices followed the stale instruction file.
- Only 4.2% followed the conventions already used in the code.
This happened even though the code was consistent and working.
For example, if the project’s existing files all used logging, but the instruction file said to use print(), the AI usually added new code with print().
Claude often noticed the problem but still made the wrong choice
Claude Code mentioned the disagreement in 79 out of 80 runs.
However, even after recognising the conflict, it usually followed the outdated file. It effectively said:
“The existing code uses
logging, but the instruction file says to useprint().”
Then it used print() anyway.
This is an important finding: noticing a problem does not always mean the AI will solve it correctly.
Codex sometimes changed working code
Codex CLI went further in some cases. In 20 out of 80 runs, it changed old parts of the project so they matched the incorrect instruction file.
In 17 of those 20 cases, the project’s original tests then failed.
It is similar to a student changing the answers in a correct textbook because an old study guide gives different answers—and then discovering that the textbook’s answer key no longer works.
One sentence greatly improved the results
When the researchers added the special sentence:
- 91.4% of choices followed the existing code.
- Both agents reported the conflicts they found.
- Codex stopped rewriting old working code.
- The test pass rate increased from 89.4% to 100%.
- No Codex run rewrote existing code in this condition.
The change was dramatic:
| Situation | Choices following existing code |
|---|---|
| Without the sentence | 4.2% |
| With the sentence | 91.4% |
5. Why does this happen?
The paper suggests that the problem comes from the way AI agents receive information.
The instruction file is presented to the AI as a direct instruction, similar to a command from the user.
The existing code is usually shown later as tool output. Tool output is treated more like information the AI has discovered.
AI systems are trained to give instructions higher priority than ordinary information. This is usually useful because it helps protect them from dangerous text hidden inside websites or files. For example, a webpage might contain the sentence, “Ignore all previous instructions.” The AI should not automatically obey that webpage.
However, this safety system creates a problem here:
- the outdated file is treated as a high-priority command,
- the current code is treated as lower-priority evidence.
The AI therefore trusts the wrong source.
The added sentence fixes the problem because it is itself a higher-priority instruction. It tells the AI that, in conflicts between documentation and code, the code should win.
6. Why are the findings important?
The findings matter because instruction files are used repeatedly. If an outdated instruction remains in a project, the same mistake can happen again and again:
- every new AI session may follow the wrong convention,
- new code may look different from old code,
- working code might be changed unnecessarily,
- tests may fail, and
- developers may need to spend time repairing the AI’s changes.
The problem can also be serious when instructions affect correctness. For example, an old rule might tell an AI to store money as decimal numbers instead of whole cents. That could cause rounding mistakes in financial software.
7. What could happen in the future?
The paper gives advice to three groups.
For users and developers
They can add the special sentence to their user-level instruction file. This gives the AI a clear rule for handling disagreements.
They should also review instruction files regularly, just as people update maps when roads change.
For companies that build coding agents
Tools such as Claude Code and Codex CLI could include this rule automatically. Users would then be protected even if they never add the sentence themselves.
For companies that build AI models
Future models could learn to compare written instructions with the actual code instead of automatically trusting the instructions. They could ask:
“Does this instruction still describe how the project works?”
Simple conclusion
This paper shows that AI coding assistants can obey outdated project instructions even when those instructions clearly conflict with the working code. Sometimes they even recognise the conflict but still choose the wrong source.
Fortunately, a single sentence telling the AI to trust existing code over conflicting documentation made a major difference. The agents produced code that matched the project more often, stopped breaking old code, and passed all the tests in the experiment.
The broader lesson is that AI tools need clear rules for deciding which information to trust. Until these tools handle stale instructions automatically, developers should keep their instruction files updated and explicitly tell coding agents to follow the existing code when the two sources disagree.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
- Limited external validity beyond synthetic projects: The experiment uses 40 small, purpose-built Python repositories with three to eight modules; it remains unclear whether the same behavior occurs in large, heterogeneous, actively evolving industrial repositories.
- Narrow programming-language coverage: The study evaluates only Python projects, so the effects may differ in statically typed, compiled, configuration-heavy, or multi-language codebases.
- Restricted agent and model coverage: Results are based on only Claude Code with Claude Sonnet 5.5 and Codex CLI with GPT-6 Sol. The findings may not generalize to other models, agent frameworks, model versions, or open-source coding agents.
- Potential version-specific effects: Because the tools and models were evaluated at particular unreproduced-in-time versions, it is unknown whether later model training, prompt changes, or agent updates would reduce or amplify stale-instruction adherence.
- Limited task diversity: Each run implements a single feature request requiring four or five convention choices. The study does not establish whether the effect persists during debugging, refactoring, dependency upgrades, code review, migration, or repository-wide architectural changes.
- Artificially binary convention conflicts: Each project consistently uses one convention while the instruction file consistently requests its opposite. Real repositories may contain partial adoption, exceptions, multiple valid conventions, or ambiguous evidence, which could produce different behavior.
- No systematic evaluation of semantic importance: The 16 convention types are not fully characterized in the paper, and the study does not determine whether stale instructions about correctness-critical behavior have larger consequences than stylistic or organizational instructions.
- Unclear representativeness of stale documentation: The constructed instruction files may differ from naturally occurring files in wording, length, specificity, authority cues, and degree of obsolescence. Naturalistic validation is needed.
- Causal attribution to instruction hierarchy remains incomplete: The results are consistent with the claim that instruction-versus-tool-output ranking causes the behavior, but the experiment does not isolate this mechanism from alternative explanations such as prompt wording, context position, salience, recency, or the agents’ file-reading workflows.
- No direct manipulation of information channels: The study does not independently vary whether the code is presented as tool output, user text, system-provided repository context, structured metadata, or an explicit “source of truth” field. Consequently, the relative contribution of channel authority versus content wording is unresolved.
- The one-sentence intervention is not decomposed: The sentence simultaneously gives code precedence and requires the agent to report inconsistencies. It is unknown whether the precedence clause alone is sufficient, whether the reporting clause changes behavior, or whether different phrasings would be more robust.
- Uncertainty about conflict-resolution boundaries: The paper does not specify when existing code should not take precedence—for example, when the code contains a known bug, when the task explicitly requests a migration, or when security, legal, or architectural requirements override current implementation patterns.
- Risk of overgeneralizing “follow the code”: Existing code is treated as the correct authority, although it may itself be inconsistent, outdated, insecure, or generated from a flawed convention. Methods for distinguishing authoritative code from legacy or erroneous code remain unexplored.
- No evaluation of mixed or hierarchical instruction files: The experiments use a project instruction file and a user-level file, but do not test nested files, directory-scoped instructions, system prompts, developer prompts, repository metadata, or conflicts among multiple instruction files.
- No study of repeated exposure over time: Each run starts from a fresh copy and ends after one task. It remains unknown whether agents learn from prior corrections, repeatedly rediscover the same conflict, or become more reliable across multi-session workflows.
- Unclear persistence of the intervention: The study measures immediate effects of adding the sentence but does not examine whether agents continue to follow it after context growth, long tool traces, competing instructions, or long-running sessions.
- Limited measurement of agent reasoning and uncertainty: Final replies and code choices are measured, but the study does not assess whether agents internally distinguish authoritative evidence, express uncertainty, request clarification, or verify conflicts before editing.
- Conflict-reporting evaluation depends on another model: Claude Opus 5.5 judges whether final replies mention conflicts. The reliability of this classifier, including false positives, false negatives, and sensitivity to phrasing, is not independently validated against human judgments.
- Choice-level observations are not fully modeled statistically: Multiple convention choices are nested within projects and runs, yet the reported aggregate percentages and project resampling do not clearly account for dependence among choices, agents, tasks, and repeated conditions.
- Insufficient uncertainty reporting for agent-level and convention-level effects: The paper reports several aggregate rates but provides limited inferential analysis of differences between agents, convention types, conflict directions, and rewrite outcomes.
- Potential contamination from the feature-task design: Although feature requests omit conventions, the requested behavior or project structure may implicitly favor one convention. The extent to which task wording influenced the observed choices is not established.
- No assessment of human review burden: The paper argues that stale instructions create reviewer effort but does not measure review time, correction effort, defect detection, or developer trust with and without the intervention.
- Limited outcome measures beyond tests: Test pass rates and code-convention matching do not capture maintainability, security, performance, readability, architectural consistency, or downstream integration failures.
- Held-out tests may not represent real regressions: The test suites are synthetic and small, and the procedure restores original tests after the agent run. It remains uncertain how the behavior affects production-scale tests, hidden requirements, or failures that are not immediately detectable.
- No comparison with automated consistency checking: The sentence is compared with no alternative intervention such as stale-reference detectors, documentation linting, repository indexing, explicit code summaries, or pre-edit conflict checks.
- No evaluation of proactive conflict detection: The intervention requires the agent to report inconsistencies, but the study does not measure recall, precision, or completeness of the conflicts agents identify, nor whether they detect all stale instructions before making changes.
- Unresolved trade-off between safety and code precedence: Giving existing code higher priority could weaken defenses against malicious, vulnerable, or unintended repository content. The paper does not test prompt-injection attacks, compromised files, or cases where code contains adversarial instructions.
- No analysis of repository trust and provenance: The approach assumes that all existing code is trustworthy, but does not investigate whether agents should weigh authorship, recency, test coverage, commit history, code ownership, or version-control evidence when resolving conflicts.
- Effectiveness under explicit user requests is unknown: The study does not test conflicts where the feature request explicitly asks the agent to adopt the documented convention, migrate the code, or intentionally diverge from existing patterns.
- No human-in-the-loop condition: All runs are unattended with automatic edit approval. It remains open whether interactive clarification, user corrections, review checkpoints, or selective approval would substantially reduce harmful adherence to stale files.
- Operational costs are only briefly reported: The paper notes slightly fewer tool calls and less time with the sentence but does not analyze cost, token use, latency, context length, or whether the intervention scales economically to large repositories.
- The proposed product-level fix is not evaluated: The paper recommends incorporating the rule into agent or model behavior, but does not test system-level implementations, their failure modes, or whether they interfere with legitimate instruction-following and prompt-injection defenses.
- No long-term maintenance study: It remains unknown whether the sentence reduces future documentation drift, encourages teams to neglect updating instruction files, or creates new inconsistencies between the sentence and project-specific policies.
- Reproducibility may be affected by unavailable or unstable dependencies: The experiment relies on named future-dated models and commercial tools whose behavior, access, and versions may change, making independent replication and longitudinal comparison uncertain.
Practical Applications
Immediate Applications
- Add a precedence rule to coding-agent configuration files — Software engineering
- Teams can add the tested sentence, or an equivalent rule, to user-level
CLAUDE.md,AGENTS.md, or comparable configuration files: when documentation conflicts with implemented code, follow the established code and report the inconsistency. - This is deployable immediately and requires no model retraining or repository-wide cleanup.
- The study found that code-consistent choices increased from 4.2% to 91.4%, while test pass rates increased from 89.4% to 100%.
- Dependencies: The agent must load the user-level instruction file as a higher-priority instruction, and the existing code must represent the intended current behavior. The rule should not override explicit, recently updated requirements for a deliberate migration.
- Teams can add the tested sentence, or an equivalent rule, to user-level
- Make conflict reporting mandatory in agent workflows — Software engineering and code review
- Coding-agent templates can require agents to list every detected inconsistency between instruction files and existing code in their final response or pull-request description.
- Reviewers can then distinguish intentional changes from stale documentation and create follow-up issues to update the relevant files.
- This is especially useful in continuous integration workflows where agents independently create pull requests.
- Dependencies: Agents must inspect representative existing files, and teams need a standard location—such as a pull-request checklist or machine-readable report—for recording conflicts.
- Add automated regression checks for agent-generated changes — DevOps and quality assurance
- Agent runners should automatically restore or preserve the original test suite, execute original and feature-specific tests, and reject changes that break pre-existing behavior.
- The result is directly motivated by the finding that Codex CLI rewrote working code in 20 runs and caused original-test failures in 17 of them.
- A practical workflow is: create an isolated checkout, run the agent, compare modified files with the baseline, run the full test suite, and require human approval for deletions or large rewrites.
- Dependencies: Tests must provide meaningful coverage, and the runner must be able to distinguish intended feature edits from destructive refactoring.
- Introduce change-size and destructive-edit guardrails — Software engineering and cybersecurity
- Agent platforms can flag or block deletion of substantial portions of existing code, changes to public interfaces, or modifications to files unrelated to the requested feature.
- For example, a policy could require confirmation when an agent removes more than a specified number of lines or changes established data representations such as integer cents to floating-point currency.
- Dependencies: Thresholds must be calibrated to avoid blocking legitimate migrations. Automated approval should not be treated as a substitute for code review.
- Run documentation-consistency checks in CI — Software maintenance
- Projects can scan
CLAUDE.md,AGENTS.md, skill files, and related documentation for references to deleted files, obsolete frameworks, old commands, or conventions inconsistent with the repository. - Such checks can identify stale statements like “use React 18,” “use
print(),” or “update a document” when the codebase has moved on. - The study indicates that configuration files often remain unchanged while the code evolves, making CI-based detection more reliable than occasional manual review.
- Dependencies: Natural-language consistency checking remains imperfect. The tool should produce reviewable warnings rather than automatically rewriting instruction files.
- Projects can scan
- Require repository reconnaissance before implementation — Software engineering education and practice
- Agent task templates can require a short preliminary phase: inspect representative modules, identify prevailing conventions, compare them with instruction files, and state any conflicts before editing.
- This makes the agent’s evidence-gathering process explicit and reduces the risk of blindly applying stale conventions.
- Dependencies: Reconnaissance alone may not overcome instruction hierarchy; it is most effective when combined with the precedence sentence and test-based validation.
- Improve incident investigation for AI-assisted development — Industry governance
- When an agent introduces incompatible code, teams can examine instruction files as potential root causes rather than attributing the failure solely to model “reasoning.”
- Incident-response checklists should include stale configuration, conflicting repository guidance, agent version, model version, and the exact context supplied to the agent.
- Dependencies: Repositories need versioned agent configurations and reproducible execution logs.
- Use the finding as a teaching rule in programming and software-engineering courses — Academia
- Courses can teach students that repository instructions are operational inputs, not necessarily authoritative descriptions of current implementation.
- A practical lab could give students a repository with deliberately stale guidance and ask them to compare agent behavior with and without the precedence rule.
- Dependencies: Educational exercises should emphasize that existing code is not always correct; students must also consult requirements, tests, maintainers, and security policies.
- Adopt a policy requirement for traceable AI-generated code changes — Organizations and public-sector software
- Internal AI-use policies can require agents to report documentation–code conflicts, preserve test evidence, and identify whether changes followed current implementation or a written convention.
- This is particularly relevant for regulated or safety-sensitive software where unexplained agent modifications are difficult to audit.
- Dependencies: The policy must define who resolves conflicts and how intentionally outdated code is distinguished from the intended source of truth.
- Apply a cautious version of the rule in daily technical work — Individual developers
- Developers using coding agents can add the precedence rule to their global configuration and periodically review project-specific files.
- Before accepting a change, they should inspect the diff, run the existing tests, and verify that sensitive conventions—such as authentication, financial arithmetic, data schemas, and error handling—were preserved.
- Dependencies: The approach is suitable for routine development but should not be used uncritically in projects where documentation specifies a required migration or compliance obligation.
Long-Term Applications
- Build agents that distinguish executable instructions from descriptive documentation — Coding-agent platforms
- Agent software could classify each line of an instruction file as an operational rule, a factual description of the repository, or a conditional preference.
- Descriptive claims could then be checked against the current code before being treated as authoritative.
- For example, “the project uses React 18” could be verified against package manifests and source imports, while “run
pytestbefore submitting” would remain an actionable instruction. - Dependencies: Reliable classification, repository-wide indexing, and safeguards against incorrectly downgrading legitimate instructions are required.
- Create source-aware context hierarchies for agents — AI infrastructure
- Instead of placing all instruction-file text at the same priority, future agents could represent provenance explicitly: system policy, user instruction, project rule, generated documentation, source code, test results, and tool output.
- The platform could give current executable evidence greater weight when a documentation statement is contradicted, while still protecting against prompt injection from untrusted files or web content.
- Dependencies: This requires new model-training and agent-orchestration techniques. A naïve “code always wins” policy could cause agents to preserve bugs or ignore intentional migrations.
- Develop repository “source-of-truth” graphs — Software configuration management
- Tools could maintain links among instruction files, package manifests, APIs, schemas, tests, source modules, and deployment configurations.
- When a code change invalidates a linked statement, the tool could open a documentation-update task or block agent execution until the conflict is reviewed.
- Potential products: Documentation compilers, repository knowledge graphs, IDE extensions, and CI services for “configuration freshness.”
- Dependencies: Links must be maintained across refactors, generated code, polyglot repositories, and dynamically configured systems.
- Use semantic consistency testing for high-risk conventions — Finance, healthcare, energy, and infrastructure
- Future validation systems could specifically compare agent changes against domain-critical invariants:
- financial amounts remain integer-based or use approved decimal types;
- healthcare data handling preserves privacy and access-control rules;
- energy-control software preserves fail-safe states;
- infrastructure code maintains backward-compatible interfaces.
- These checks would go beyond ordinary unit tests by detecting semantically dangerous convention changes.
- Dependencies: Domain experts must define the invariants, and automated checks require strong coverage and validated static or dynamic analyses.
- Integrate conflict-aware agents into enterprise change-management systems — Industry and policy
- Enterprise platforms could automatically route detected documentation–code conflicts to owners, attach evidence from the agent session, and require approval before deployment.
- A change record might include the stale statement, the conflicting code locations, the agent’s selected source of truth, test results, and the proposed documentation update.
- Dependencies: Integration with issue trackers, access-control systems, audit logs, and release-management processes is necessary.
- Establish benchmarks across languages, repositories, and agent architectures — Academia
- Further research should test whether the one-sentence intervention generalizes to JavaScript, Java, Go, Rust, infrastructure-as-code, monorepos, generated code, and interactive rather than unattended agent sessions.
- Studies should also include cases where the code is stale and the documentation is correct, because the present experiment only evaluates stale instruction files.
- Dependencies: Results may vary with model versions, agent prompts, tool permissions, task complexity, repository size, and the authority assigned to tests or external requirements.
- Train models to resolve conflicts using evidence rather than channel priority alone — Model development
- Models could be trained to recognize that code, tests, version-control history, recent commits, and maintainer requirements may provide stronger evidence than an old descriptive sentence.
- A model might estimate the age, specificity, and corroboration of each source before choosing a convention.
- Dependencies: Training must preserve prompt-injection resistance. Treating all file content as authoritative would create security vulnerabilities.
- Develop self-healing instruction files — Long-term software tooling
- Agents could propose updates to stale instruction files when they detect repeated contradictions, while requiring human approval for changes affecting build commands, security rules, or architectural constraints.
- A system might identify that all current modules use
logging, generate a proposed replacement for “useprint(),” and cite the files supporting the proposal. - Dependencies: Automatic updates should be evidence-backed, versioned, reversible, and reviewed; otherwise the system could convert temporary implementation patterns into misleading permanent rules.
- Create risk-adaptive agent policies — Regulated and safety-critical sectors
- Future agents could use different conflict-resolution rules based on repository risk. In ordinary application code, current implementation may be a strong default; in regulated systems, approved specifications, safety cases, or signed policy documents may outrank existing code.
- The agent could therefore require human confirmation whenever code conflicts with an approved external standard.
- Dependencies: Organizations must formally classify sources by authority and maintain trustworthy metadata about approval status and recency.
Glossary
- AGENTS.md: A convention-based file used to provide instructions to coding agents for a software repository. “Codex CLI reads AGENTS.md”
- Agentic coding: Software development performed by AI systems that can inspect files, execute tools, and modify code autonomously. “Recent studies describe what instruction files contain and how teams maintain them”
- API migration: The process of moving software from one application programming interface to another, potentially causing behavioral changes. “measuring item-level regressions in commercial LLM API migrations”
- Configuration smell: A recurring configuration practice that may indicate maintainability or correctness problems. “Configuration smells in AGENTS.md files”
- Context rot: The degradation of an instruction or documentation context as the software it describes changes over time. “Context rot in AI-assisted software development”
- Conflict judge: A separate model or evaluation component that determines whether an agent’s response identifies a conflict. “Conflict judge”
- Constraint: A limitation imposed on an agent’s tools, environment, permissions, or execution. “the handling of failed attempts and the settings of the conflict judge”
- Convention: A project-specific rule governing how code is written or organized. “the existing code uses one convention consistently and the instruction file asks for the other”
- Context: The collection of messages, instructions, tool outputs, and other information supplied to a LLM. “the file shapes every change the agent makes”
- Emphatic preamble: An introductory instruction that strongly reinforces the priority or authority of subsequent text. “The emphatic preamble in Claude Code is not needed for this”
- Enumeration: A data type representing a fixed set of named values. “the agent replaced the Column enumeration in model.py with string constants”
- Feature request: A specification describing a new capability that an agent is expected to implement. “the prompt contained only the feature request”
- Floating-point arithmetic: Computation using numerical representations that approximate real numbers and may introduce rounding errors. “agents wrote floating-point arithmetic even though the existing code used integer cents”
- Held-out tests: Tests kept separate from the project’s original test suite and used to evaluate newly implemented behavior. “ran them together with our held-out tests for the new feature”
- Instruction hierarchy: An ordering system that determines which instructions a LLM should prioritize when they conflict. “Under the instruction hierarchy, models are trained to follow system and user messages ahead of text returned by tools”
- Instruction surface: The location or channel through which instructions are delivered to an AI coding agent. “evaluating instruction following across instruction surfaces in coding agents”
- Imperative wording: Language expressed as a command or directive. “while its imperative wording stays the same”
- Integer cents: A representation of monetary values as whole numbers of the smallest currency unit, such as cents. “the existing code used integer cents”
- Language-model agent: An autonomous software system that uses a LLM to reason about tasks and operate tools. “Low reasoning effort is enough for routine office work by language-model agents”
- LLM: A neural LLM trained on large text datasets to generate and interpret language. “A model that checks the descriptions against the code would avoid the problem”
- MCP server: A server implementing the Model Context Protocol, allowing an AI agent to access external tools or resources. “The safe-mode option also switches off instruction files, skills, plugins, hooks and MCP servers”
- Non-interactive mode: An execution mode in which an agent performs a task without user interaction during the run. “with GPT-6 Sol in non-interactive mode”
- Opaque identifier: An identifier whose meaning or associated condition is hidden from an evaluator. “each was given an opaque identifier”
- Prompt injection: An attack in which untrusted text attempts to manipulate an AI system by presenting malicious or conflicting instructions. “The purpose is protection against prompt injection”
- Pull request: A proposed set of repository changes submitted for review and possible integration. “Numbers in parentheses are pull requests”
- Repository-scale task: A software-development task involving a substantial code repository rather than an isolated file or small example. “coherence debt in repository-scale tasks”
- Resampling: A statistical procedure that repeatedly samples from observed data to estimate uncertainty or variability. “from 2,000 resamples of projects”
- Sandbox: An isolated execution environment that restricts an agent’s access to the host system or its resources. “After a timeout, a crash or a command rejected by the sandbox”
- Skill file: A document describing a reusable capability or procedure available to an AI agent. “a skill file kept telling agents for 13 days”
- Stale documentation: Documentation that no longer accurately describes the current state of the software. “a stale line in the instruction file”
- Standard input: The input stream through which a command-line program receives data. “with the feature request on standard input”
- Synthetic project: An artificially created software project used for controlled experimentation. “All runs are interactions between coding agents and synthetic Python projects”
- System prompt: A high-priority message that establishes an AI model’s behavior, role, or governing instructions. “Claude Code inserts CLAUDE.md into a user message directly after the system prompt”
- Tool call: A request issued by an AI agent to invoke an external command, program, or service. “recorded the number of tool calls and the running time”
- Tool output: Information returned to a LLM by an external tool it has invoked. “The existing code reaches the model only when the agent opens a file, as the output of a tool call”
- Training dataset: A collection of examples used to train or fine-tune a machine-learning model. “a training dataset released in 2026 strengthens this ordering”
- User-level instruction file: An instruction file that applies across projects for a particular user or environment. “A sentence in the user-level instruction file that gives existing code precedence”
- Usage limit: A restriction on the amount of computation, requests, or resources available to an agent. “A run stopped by a usage limit was started again”
- Validation evidence: Information demonstrating that a software change satisfies its requirements, often through tests or other checks. “Validation evidence in LLM repair agents”
- Virtual environment: An isolated environment containing a project’s Python interpreter and dependencies. “PIP_REQUIRE_VIRTUALENV=1”
