Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse

Abstract: Code LLMs are now trusted collaborators in production workflows for debugging, refactoring, and iterative repair, and every benchmark that evaluates them assumes the instructions they act on are correct. We study what happens when that assumption breaks. We evaluate code LLMs across four experiments designed to assess whether models resist or obey incorrect instructions in single-pass and iterative repair settings, using the RunBugRun dataset of algorithmic Python problems with deterministic test cases. Our findings reveal a striking behavioral pattern: models correctly identify an incorrect instruction as wrong, then follow it anyway. This compliance unknowingly introduces errors beyond the original bug, and the corrupted code state cannot be recovered through subsequent self-guided iterative repair, which fails to converge across passes. We term this Blind Obedience, characterize the Ghost (Unknown) Errors it introduces, quantify the proportion of cases where semantic corruption proves irrecoverable, and show that extended reasoning cannot reverse it. These findings surface behavioral properties invisible to pass-rate evaluation, with direct consequences for code LLMs deployed in production settings.

• The paper demonstrates that code LLMs blindly obey incorrect, confidently-phrased instructions, causing irreversible semantic collapse in code repair tasks.
• It uses a four-phase experimental framework on 538 deterministic Python repair instances across five models, leveraging test oracles to quantify damage.
• Results indicate that iterative self-repair rapidly saturates and fails to recover from 'ghost errors', urging the need for architectural countermeasures in autonomous agents.

Blind Obedience and Semantic Collapse in Code LLMs: Systematic Irrecoverability under Incorrect Guidance

Problem Context and Motivation

Current production workflows in software engineering increasingly employ LLM-based coding agents for debugging, refactoring, and program repair, moving beyond single-shot code synthesis to real modification of codebases. Benchmarks and research overwhelmingly assume instruction correctness—models are measured by their adherence to user-provided prompts, with resistance to such instructions implicitly penalized. This paper (2607.04537) systematically interrogates that assumption, asking: What happens when code LLMs receive confidently-phrased but incorrect instructions?

The inquiry is crucial. In real settings, developers regularly misdiagnose bugs; code LLMs must arbitrate between potentially incorrect instructions and objective execution evidence provided by test failures. The central concern is whether models can resist being led astray, and if not, whether the resultant code drift is reversible.

Experimental Framework and Methodology

The authors developed a four-phase empirical protocol using the RunBugRun dataset of 538 deterministic Python program repair instances, ensuring unambiguous binary feedback from test oracles. Five code LLMs were evaluated: GPT-5.3 Codex, Claude Sonnet 4.6 (commercial), Qwen3-Coder, GLM-5, and Kimi K2.5 (open-source). Each experiment stage isolates a phase in the interaction loop: (1) basic obedience vs. resistance, (2) iterative self-repair, (3) error compounding under incorrect instruction, (4) attempted recovery from a semantically corrupted state.

Figure 1: The four experimental settings form a progressive chain: RQ1 establishes Blind Obedience, RQ2 bounds recovery, RQ3 measures compounding Ghost Errors, and RQ4 tests for irrecoverability.

The stages are described as follows:

• RQ1 (Blind Obedience Assessment): Models receive, in randomized order, a correct instruction, an incorrect but plausible human-generated instruction, or are prompted to self-diagnose (self-thinking). Pass rate and instruction-classification awareness are measured.
• RQ2 (Self-Guided Iterative Repair): Without any external instructions, models self-generate corrections from test feedback, iterating up to five times, to establish an upper recovery bound on difficult instances.
• RQ3 (Compounding Damage from Obedience): Models are forced to follow a new dynamically generated (but erroneous) instruction at each iteration, with the hypothetical human reviewer denied access to execution feedback. The experiment probes if models learn resistance as evidence of semantic damage accumulates.
• RQ4 (Irrecoverability Measurement): Starting from code corrupted by RQ3's accumulated Ghost Errors, models attempt self-guided iterative repair; failure to cross the original buggy baseline quantifies the irrecoverable damage rate.

Empirical Results

Blind Obedience Is Systematic, Not Incidental

Models overwhelmingly followed incorrect instructions even when able to classify these instructions as wrong when directly queried. McNemar's test demonstrates a systematic gap: for all five models, the number of instances where a model passed only with a correct instruction but failed under an incorrect one (“blind obedience”) vastly exceeded cases where lucky passes occurred under wrong instructions. B/C ratios ranged from 3.5 (Qwen3-Coder) to 13.5 (Claude Sonnet), all with $p \ll 0.001$, confirming non-randomness.

Figure 2: T2 (Incorrect Instruction) leads to the steepest performance drop, especially for open-source models; T1 (Correct Instruction) always outperforms T3 (Self-Thinking).

Classification-role awareness does not transfer to generation: models detect instruction error but do not resist during compliance.

Figure 3: Models detect incorrect instructions in the evaluator role (left) but propagate more errors under those same instructions in the generator role (right). Detection ≠ resistance.

Iterative Repair Exhibits Early Saturation

When left to “self-think” and iteratively repair, most successful recoveries occur within just two passes. After that, code and reasoning swirl, with little further progress—either the right patch is found quickly, or the problem remains unsolved due to inability to escape local optima.

Figure 4: Recovery is heavily front-loaded; most problems resolving within two passes, with a prominent plateau. Unresolved problems are not harder in an objective sense—the model's reasoning, not problem hardness, is the barrier.

Ablations at different reasoning depths (zero, low, medium effort) confirm that, except for GPT-5.3 Codex, increasing token budget for “thinking” degrades rather than improves outcome rates, as output is diverted into self-justification chains rather than code modifications.

Figure 5: Elevated reasoning shifts output from code to thought-chains, with decreasing patch correctness. Only GPT-5.3 Codex maintains stable output under more “thinking,” but without actually exceeding its no-reasoning baseline.

Compounding and Irrecoverable Semantic Damage

When models are made to follow new, incorrect instructions at each pass (RQ3), they accumulate structural “Ghost Errors”—patches that introduce logical bugs at the wrong part of the code, compounding with each iteration.

Figure 6: Problems entering RQ3 with damage never escape. Blind obedience under iterated incorrect guidance is sustained—damage does not self-correct.

Critically, the rate at which a model “escapes” erroneous behavior remains flat even as context accumulates more contradicting evidence (i.e., persistent test failures). Models do not become more resistant as evidence of harm mounts.

Figure 7: Escape rates remain constant regardless of how many incorrect passes are applied; no accumulation of resistance.

Structural Barriers to Recovery

When presented with self-repair opportunities from ghost-error-damaged code (RQ4), models plateau within two passes—identical in structure to RQ2, but with drastically diminished success rates.

Figure 8: Most problems carrying Ghost Errors after RQ3 never recover across five repair passes. More obedience earlier results in more irrecoverable problem sets.

Recovery ceilings are invariant to reasoning configuration; elevated chain-of-thought or higher token allocation offers no escape.

Figure 9: Recovery plateaus for all reasoning configurations—irrecoverable states are defined by prior code drift, not model reasoning capacity or compute.

The “irrecoverable damage rate” is thus a persistent phenomenon, highly sensitive to a model’s initial susceptibility to erroneous instructions.

Discussion: Implications and Theoretical Impact

Instruction Following as an Unconditional Directive

A central finding is that LLMs—regardless of capacity, source, or configuration—treat incoming instructions in the generative setting as unconditional directives. The models are able to epistemically recognize an instruction as erroneous but lack an arbitration mechanism to act on this knowledge during code modification; the “obedient” path is systemically followed even when test oracles disagree.

This represents a structural flaw at the control plane of instruction-following agents, echoing wider findings on LLM sycophancy and instruction adjunction [sharma2025sycophancy]. The model’s “awareness” of error is partitioned, never surfacing in the generative context.

Limitations of Iterative and Reasoned Self-Repair

Iterative self-repair “saturates” early, with little gain from increased depth or token budget for “reasoning” steps. The process hits structural local minima, unable to reconstruct or revert the accumulated semantic drift imposed by earlier obedience to bad guidance.

Blind obedience induces a form of semantic collapse, whereby part of the code’s logical intent is irreversibly displaced. In the dataset’s relatively simple function-scale patches, where test feedback is deterministic and immediate, this effect is already pronounced—implicating that the phenomenon will likely be amplified in real-world multi-file or architectural settings with partial or noisy feedback.

Evaluation and Benchmarking Consequences

Standard code LLM benchmarks measuring only pass rate or edit distance are insufficient, as they do not register code drift or irrecoverable semantic corruption induced by intermediate obedient steps. The damage persists even when the original bug could have been repaired by self-guided inference from the buggy baseline. Benchmarking frameworks must be revised to incorporate resistance-to-incorrect-instructions and semantic recoverability as first-class evaluation metrics.

Mitigation Directions and Future Research

Pragmatically, any system deploying autonomous LLM-based repair agents must admit the possibility of bad guidance and employ cross-validation with multiple evidence sources, program analysis, or test-based arbitration before acting on an instruction.

There is a strong need for architectural innovations that couple model awareness of instruction error and its generative process—e.g., integrating explicit arbitration layers or fine-tuning on adversarial dialogues where withholding compliance is rewardable when evidence contradicts guidance.

Future research should also probe the interplay of this failure mode with task complexity, codebase size, non-deterministic feedback, and collaborative settings mixing human and agent edits.

Conclusion

This work demonstrates, with strong statistical and empirical rigor, that current LLMs for code lack robust resistance to plausible-seeming but incorrect instructions. Blind obedience is systemic, produces compounding and often irrecoverable semantic drift (even in simple settings), and is undetectable by standard evaluation. Awareness of instruction error is present at the epistemic level but structurally dissociated from the generative process. Elevated reasoning or rounds of iterative repair do not mitigate these effects. These findings have direct implications for the engineering and deployment of autonomous code agents: safe integration requires architectural countermeasures against semantically irreversible obedience.