Articulate but Wrong: Self-Review Failures in LLM-Based Code Modernization

Abstract: LLM agents are increasingly used to migrate legacy code to modern stacks. We ask a deceptively simple question: when an LLM modernizes legacy code, can the same model be relied upon to recognize when its own output silently changes observable behavior? We run 1,980 real modernization calls across 11 production LLMs from 7 distinct families on a balanced 60-snippet legacy-Python-2 corpus, evaluate every output with a type-strict behavioral oracle, and then ask each model to judge whether its own output preserves behavior. We report four findings. (1) Semantic-preservation drift is prevalent and sharply separable from a cleanly-controlled baseline: semantic-trap snippets drift in 39.7% of attempts versus 7.0% on benign-control code that requires no real modernization (+32.7 percentage points; n=660 each). (2) Drift concentrates on specific snippets that fail across models: pairwise model agreement on which snippets are hard is high (mean Pearson r=0.52), and a small core of numeric-semantics snippets fails for nearly every model and every prompt phrasing. (3) Self-review by the producing model is not a reliable safety net: across all semantic drift cases, 31.7% are silently endorsed by the same model that produced them (83/262), and the per-model self-miss rate is strongly bimodal -- ranging from 0% on five models to 100% on one widely deployed model -- with several models explicitly articulating the very Py2/Py3 semantic distinction that broke their output, then declaring behavior preserved. (4) Drift rate is non-monotone in model capability and price: per-model rates range 5.6%-46.7% and do not track model capability cleanly, indicating the failure is task-structural rather than driven by model scale. All code, prompts, the 60-snippet corpus, the behavioral oracle, the output extractor, and the raw model outputs are released.

• The paper demonstrates that LLMs frequently fail to detect silent behavioral drift, particularly in numeric semantics, during code modernization.
• It details an experimental setup using 60 legacy Python-2 snippets and a type-strict oracle to quantify semantic versus syntactic drift.
• Results reveal that self-review misses are model-specific and non-monotonic, underscoring the need for external, multi-agent verification.

Self-Review Failures in LLM-Based Code Modernization: An Authoritative Analysis

Context and Motivation

The migration of legacy code to modern platforms using LLM agents has reached substantial operational scale. Nonetheless, correctness guarantees remain elusive: generated migrations often lack assurance of behavioral equivalence and necessitate expensive human review. The paper "Articulate but Wrong: Self-Review Failures in LLM-Based Code Modernization" (2605.21537) probes whether LLMs can reliably judge their own modernization outputs for silent behavioral drift—the critical failure mode where code compiles and runs plausibly but diverges from the legacy contract in observable behavior.

Figure 1: The experimental setup involves legacy snippet modernization, evaluation via type-strict behavioral oracle, and model-led self-review for behavioral preservation.

Experimental Design

The authors built a balanced corpus of 60 legacy Python-2 snippets categorized into semantic-preservation traps, syntactic traps, and benign controls. The traps specifically target idioms prone to silent drift: numeric semantics (integer division, rounding), lazy evaluation (range, map, filter, zip differences), and type-model anomalies (int/long, str/unicode distinctions). Each snippet defines a function to facilitate contract-driven behavioral evaluation.

Modernization was conducted across 11 production LLMs spanning seven families and three distinct prompt phrasings. Each output was validated using an explicitly type-strict behavioral oracle, crucial for exposing int–float drift masked by Python's permissive equality. Post-modernization, the self-review probe tasked the same model with binary (YES/NO) judgment on behavioral preservation, supplied both legacy and modernized code, isolated from its prior provenance.

Quantitative Findings: Semantic Drift Prevalence

The aggregate results establish three distinct drift tiers:

• Benign controls: 7.0% silent drift, representing baseline error independent of modernization necessity.
• Syntactic traps: 12.7% drift, $\sim$1.8× above baseline, reflecting catchable API/literal syntax changes.
• Semantic-preservation traps: 39.7% drift, a substantial $+32.7$ percentage points above baseline.

Semantic traps induce silent behavioral changes at roughly 5.6× the syntactic-excess rate.

Figure 2: Drift rates by trap category, with semantic-preservation exceeding syntactic and baseline categories by a large margin.

Crucially, numeric-semantics drift (e.g., Py2 integer division $/$ vs Py3 floating division) is the dominant failure mode, registering at 57%. Lazy evaluation and type-model traps are less prevalent or near baseline.

Figure 3: Numeric-semantics traps disproportionately dominate the drift landscape.

Cross-Model Drift Structure

Drift is highly task-structural: certain snippets trigger errors across most models regardless of prompt phrasing. Pairwise per-snippet drift rates show high correlation (Pearson $r=0.52$), indicating a persistent core of hard cases largely immune to both model scaling and prompting.

Figure 4: Per-snippet and per-model drift heatmap reveals persistent numeric-semantic failures across models and prompt variants.

Prompt phrasings modulate aggregate rates—contract-framed prompts yield higher drift—but do not ameliorate the persistent core. These findings demonstrate that semantic-trap drift is not random or prompt-sensitive but instead structurally rooted in modernization subtleties.

Self-Review Analysis: Reliability and Failure Modes

The headline result centers on self-review:

• Across 262 confirmed semantic drift cases, 31.7% were silently endorsed by the producing model.
• Numeric-semantics drift contributed 36% of its cases to silent endorsement; lazy evaluation traps had lower self-miss rates; type-model traps were consistently caught.

  Figure 5: Self-review miss breakdown; numeric-semantics drift is most frequently missed by model self-review.

Strikingly, self-miss rates are bimodal at the per-model level: five models achieved 0% self-miss, while one model (M1) missed 100% of its own semantic drift. Self-miss rates do not track drift rates; high-drift models sometimes catch all errors, while moderate-drift models can miss all. Thus, self-review reliability is an independent property, making single-model verification non-robust.

Figure 6: The bimodal distribution of self-miss rates strongly points to model-specific properties orthogonal to drift rates.

Qualitative review reveals models sometimes explicitly articulate the Py2/Py3 division semantic gap and yet incorrectly declare preservation, underscoring a fundamental limitation in internal consistency mechanisms.

Model Capability and Methodological Implications

Contrary to common intuition, model capability and price do not monotonically reduce drift. The lowest-priced model recorded the lowest drift, and high-priced, high-capability models were not immune. This further argues that task structure dominates error modes over scaling or vendor selection.

Additionally, two critical methodological choices substantially shift measured drift rates:

• Type-strict equality is essential for detecting numeric-semantics drift; permissive comparison artificially suppresses error rates.
• Robust code extraction avoids spurious drift detection from parsing failures, reducing baseline drift from 23% (naive fence-only) to 7.0%.

Practical and Theoretical Implications, Future Directions

These results directly caution against reliance on single-model self-review as a safety net in LLM-powered code-modernization pipelines, particularly for critical tasks involving legacy/modern semantic divergences. Silent endorsement of behavioral drift on the modal failure mode (numeric semantics) threatens production deployment integrity.

Further, self-review is not equivalent to behavioral validation—an external, type-strict oracle remains necessary. The findings suggest future directions in multi-agent or tool-augmented review, chain-of-thought or neuro-symbolic oversight (as explored in recent agentic safety work (Miculicich et al., 3 Oct 2025, Zhou et al., 11 Feb 2026)), and richer behavioral testing for language transitions beyond Python.

Corpus transferability, snippet complexity, and orchestration of self-review with collective agent voting or formal specification represent potent avenues for mitigation, but the present evidence strongly rejects self-review as a standalone guarantee.

Conclusion

LLM-driven code modernization exhibits substantial silent behavioral drift on semantic traps, especially numeric semantics, with task-structural persistence across models. Single-model self-review misses nearly a third of all semantic drifts, often with high confidence, and this phenomenon is bimodal and model-specific. Neither scaling nor vendor selection offer robust mitigation; explicit behavioral-oracle validation is required. Methodological rigor in output extraction and equality checks is indispensable. For production safety in code pipelines, relying solely on self-review approaches is insufficient, motivating further research in composite verification strategies.