- The paper shows that foundation models can detect real Java refactoring errors with up to 99.6% accuracy, including compilation failures and behavioral changes, using zero-shot prompts and executable test generation.
- Repeated sampling, metamorphic perturbations, and statistical comparisons indicate that larger models provide strong, relatively stable performance, while model quality varies sharply across families and smaller models may fail completely.
- The findings position foundation models as useful complements to deterministic tools such as javac and SafeRefactor, offering explanations and language-evolution robustness but lacking formal guarantees and requiring careful handling of false positives, cost, and context limits.
Motivation and problem statement
Automated refactoring in IDEs such as IntelliJ-IDEA, Eclipse, and NetBeans remains unreliable: faulty implementations can silently alter program behavior or produce code that no longer compiles. Traditional detection relies on handcrafted preconditions, static analyses, and dynamic oracles such as SafeRefactor (2605.02096). These techniques are labor-intensive to build, difficult to adapt to new refactoring types, and constrained by the adequacy of underlying test suites. The authors investigate whether foundation models, used zero-shot as oracle-like components, can classify whether a given transformation introduces a compilation error (CE) or a behavioral change (BC), thereby complementing rather than replacing existing verification mechanisms.
Benchmark construction
The evaluation builds on the dataset of Wang et al., extended with reconstructed executable artifacts for 226 real refactoring bugs spanning 47 refactoring types across the three major Java IDEs, collected over more than a decade (2605.02096). The dataset comprises 185 CE instances and 41 BC instances; programs average 10.8 LOC. Validation is executable rather than anecdotal: all programs compile under OpenJDK Temurin 21.0.7+6, and each of the 41 BC cases ships a JUnit test that passes on the original program and fails on the refactored one. A client-observable equivalence notion is adopted: behavior is preserved only if common public methods yield identical return values, printed output, exceptions, and externally visible state.
A notable construct-validity limitation is acknowledged directly: the benchmark contains only positive bug instances, so accuracy coincides numerically with recall, and false-positive behavior on correct refactorings is measured only in a small complementary study of 50 JDolly-generated transformations validated by SafeRefactor.
Methodology
Models receive both program versions via a structured zero-shot prompt requiring JSON output with a verdict (YES, NO - COMPILATION ERROR, NO - BEHAVIOR CHANGE), an explanation, and—for BC verdicts—a complete JUnit test that must compile against both versions and expose the difference. Correctness is assessed mechanically by compiling and executing the model-provided tests. GPT-OSS-20B runs locally via Ollama on consumer hardware; GPT-5.4 runs via API. Each model is run five times to measure accuracy (mean accuracy, pass@k) and stability (accuracy spread, tar@k agreement, cons@k consensus).
To address data contamination, metamorphic testing applies six randomized, semantics-preserving source perturbations (unused fields, comments, imports, local declarations, auxiliary classes) built on Spoon to all 226 instances (2605.02096). If predictions rested on memorized surface patterns, these perturbations should cause broad degradation.
Main results
The headline numbers show strong performance from frontier models but substantial variance across model families:
| Model |
Overall |
BC |
CE |
| Gemini-3.1-Pro-Preview |
99.6 (225/226) |
97.6 |
100.0 |
| Claude-4.6-Sonnet |
94.7 |
92.7 |
95.1 |
| GPT-5.4 |
93.8 |
92.7 |
94.1 |
| Gemma-4-31B (open) |
96.5 |
97.6 |
96.2 |
| Qwen-3.6-35B |
75.7 |
90.2 |
72.4 |
| GPT-OSS-20B |
80.5 |
92.7 |
77.8 |
| Phi-4-14B |
27.9 |
14.6 |
30.8 |
| Llama-3.2-3B |
0.0 |
0.0 |
0.0 |
Gemini-3.1-Pro-Preview classified every CE case correctly, missing only one BC case due to a non-compiling generated test. Gemma-4-31B's 96.5% is a striking result for an open-weight model, exceeding GPT-5.4's first-run score under the single-run configuration. At the other extreme, Llama-3.2-3B solved nothing, failing even at output-format compliance—evidence that this task demands nontrivial instruction-following and reasoning capability.
Repeated sampling improves cumulative coverage: GPT-OSS-20B rises from 80.5% to 92.9% over five attempts (reaching full BC coverage by three attempts), while GPT-5.4 rises from 93.8% to 97.3%, indicating most of its effectiveness is realized in the first attempt. Majority voting (cons@k) is less effective than cumulative coverage because some instances alternate between correct and incorrect verdicts.
Failure modes differ by category. For GPT-OSS-20B, sixteen CE instances are misclassified in all five attempts, concentrated in transformations affecting name resolution, inheritance, member visibility, and signatures (Push Down Method, Move Method, Inline Variable, Rename Method, Pull Up Method). For GPT-5.4, persistent CE failures stem from treating uncompilable transformations as semantically analyzable—a recurring pattern where the model reasons about behavior instead of recognizing that the code cannot compile. A concrete illustration is an Inline Variable bug from NetBeans involving conditional-expression typing: GPT-OSS-20B correctly identifies the primitive-type error in all five runs, while GPT-5.4 incorrectly assumes autoboxing applies and declares behavior preserved in all five runs.
On the complementary true-positive study, GPT-5.4 correctly accepted 48/50 valid refactorings (96%) at USD 0.72 total cost, with its two errors caused by reflective field access outside the intended equivalence notion—an instructive false-positive mode showing sensitivity to the precise specification of observable behavior.
Data-leakage analysis
Metamorphic testing yields only marginal changes: GPT-OSS-20B moves from 80.5 to 80.1 overall, GPT-5.4 from 93.8 to 94.2, with many hard CE cases remaining hard after transformation. The authors conclude this is inconsistent with exact-match memorization but explicitly caution that it does not rule out contamination through related code, refactoring patterns, or abstract problem structures. This is appropriately stated as evidence against one leakage mechanism rather than proof of leakage-freedom.
Statistical comparison
Wilson confidence intervals and paired tests separate the models cleanly: Cochran's Q rejects homogeneity (Q=30.60, p<0.001), and Holm-corrected McNemar exact tests show GPT-5.4 and Claude-4.6-Sonnet significantly outperform GPT-OSS-20B (gains of 13.3% and 14.2%), while the GPT-5.4 versus Claude-4.6-Sonnet difference (<1%) is not statistically significant. An OR-based mixture-of-experts analysis shows residual-error diversity: the four models jointly solve 163 of 226 instances, and cross-model complementarity recovers most individual failures—though the authors note this is an optimistic upper bound assuming an oracle selector, not a deployable routing procedure.
Cost, latency, and large-project feasibility
Latency varies widely: Llama-3.2-3B averages 1.76 s per instance (with zero utility), GPT-5.4 3.25 s (~12 minutes per full run at ~USD 0.79), Gemini-3.1-Pro-Preview 11.41 s, while GPT-OSS-20B needs ~1.4 hours locally (~EUR 0.31 infrastructure cost) and Gemma-4-31B ~7 hours. The best observed accuracy–latency combinations come from Gemini-3.1-Pro-Preview and GPT-5.4.
For projects exceeding context windows, a diff-only feasibility study evaluates 44 real IntelliJ refactorings across five open-source projects using an UNKNOWN-augmented prompt. GPT-5.4 returns YES for 24 (all judged supported by visible evidence), NO - BEHAVIOR CHANGE for 2 (neither confirmed by adjudicators), and UNKNOWN for 18, mostly because decisive evidence lies outside the diff. The authors are explicit that this experiment cannot be mechanically verified and serves as a feasibility study only; two-author agreement was 42/44 before adjudication.
Comparison with traditional baselines
Against traditional tooling, javac trivially detects all 185 CE cases, and SafeRefactor achieves 97.8% overall—but fails on five CE instances solely due to Java 8/Randoop environment constraints when programs use newer features like var and switch expressions. This comparison supports the paper's positioning argument: deterministic tools provide stronger guarantees when applicable, while foundation models offer robustness to language evolution, natural-language explanations, and applicability without project configuration—at the price of non-determinism and absence of formal guarantees.
Limitations
The paper concedes several limitations at the point they bear on results. Prompt sensitivity was not systematically ablated. Ground-truth reconstruction from historical bug reports is manual and potentially error-prone. Proprietary-model evaluations are snapshots subject to provider-side revision; Claude-4.6-Sonnet was evaluated manually through a web interface. The benchmark is Java-specific, imbalanced toward CEs (185 vs. 41), and omits features such as assert, volatile, and native. False-positive characterization rests on only 50 validated-correct instances. Diff-only reasoning inherits the fundamental limitation that behavior preservation may depend on non-local semantic relationships invisible in the patch—a problem shared by human reviewers. The MoE analysis assumes an ideal selector.
Conclusion
This study provides systematic evidence that foundation models, prompted zero-shot with executable test-oracle generation, can detect real refactoring bugs with high accuracy—up to 225/226 for Gemini-3.1-Pro-Preview—and that performance is robust under semantics-preserving perturbations, though not provably free of contamination. Open-weight models span an enormous quality range, from Gemma-4-31B at 96.5% to complete failure at 3B scale. The paper leaves open concrete questions: how to resolve persistently misclassified CE cases involving name resolution and inheritance, how to design cost-aware cascades beyond OR-based complementarity, and how retrieval-augmented context can reduce the high UNKNOWN rate in diff-only assessment of large projects.