RefModel: Automated Refactoring Detection
- RefModel is a foundation model-based approach to automated refactoring detection that replaces handcrafted static-analysis rules with natural-language prompts.
- It processes full program versions or commit diffs alongside natural-language refactoring definitions to identify and explain code transformations.
- Empirical evaluations on synthetic Java and real-world Java, Python, and Golang demonstrate high recall and precision, often rivaling established static-analysis tools.
Searching arXiv for the specific paper and closely related work on refactoring detection. arXiv.search query: "2ti:\2 OR 2ti:\2 Refactorings using Foundation Models\""
RefModel is a foundation-model-based approach to automated refactoring detection that replaces handcrafted static-analysis rules with prompt-driven reasoning over code changes. Rather than deriving refactorings from AST differencing, entity matching, and language-specific detection logic, it supplies a model with either the full original and transformed program or a commit diff, together with a natural-language list of allowed refactoring definitions, and asks it to identify the transformation and justify the decision. In the reported formulation, RefModel is evaluated primarily as a zero-shot detector and explanation generator for single refactoring instances, with the central claim that modern foundation models can be competitive with, and sometimes outperform, established tools such as RefactoringMiner, RefDiff, and ReExtractor+ (&&&2ti:\2&&&).
2 OR ti:\2. Conceptual basis and scope
RefModel addresses a longstanding problem in software maintenance: automatic recognition of refactorings that improve code quality without altering behavior. Existing detectors are effective, but the paper characterizes them as dependent on complex rule definitions and language-specific static analysis, which makes them difficult to extend and harder to generalize across languages. RefModel reframes the task as a code-understanding problem for foundation models. Each refactoring type is defined by a single natural-language sentence, and the model must decide which definition best matches the observed transformation (&&&2ti:\2&&&).
This design gives RefModel three defining properties. First, it is prompt-driven rather than rule-driven. Second, it is definition-based: new refactoring types can be introduced by adding natural-language descriptions instead of new detection algorithms. Third, it is explanatory: the output is not just a label, but also a short justification tied to the changed code. The paper therefore positions RefModel less as a new static-analysis engine than as a flexible detection interface built on top of general-purpose foundation models.
The prompt vocabulary covers a master list of 2 OR ti:\29 refactoring types inspired by Fowler’s catalog: Add Method Parameter, Encapsulate Field, Extract Class, Extract Interface, Extract Superclass, Inline Class, Inline Method, Move Field, Move Method, Pull Up Field, Pull Up Method, Push Down Field, Push Down Method, Remove Method Parameter, Rename Field, Rename Method, Rename Package, Rename Class, and Replace Magic Number with Constant. Not all of these appear in every experiment. The synthetic Java study evaluates 2 OR ti:\2ti:\2^ types, while the real-world Java study evaluates 2 OR ti:\22, and the prompt deliberately includes additional definitions beyond those present in the test sets to expose false positives (&&&2ti:\2&&&).
2. Task formulation, prompting, and model instantiations
RefModel uses two input modes. For small synthetic Java programs, the model receives the entire original version and transformed version. For real-world programs, it receives the GitHub diff of a commit. In both cases, the prompt instructs the model to identify the refactoring type or types from a predefined list, begin with a bullet-point list of detected refactorings, and then briefly justify each one with reference to the specific code changes. The paper reports no few-shot examples; the setup is explicitly zero-shot (&&&2ti:\2&&&).
The prompting strategy is simple in form but not static in development. After preliminary errors, the authors refined the prompt to better distinguish confusable categories such as Move Method, Pull Up Method, Push Down Method, Extract Class, and Extract Superclass. They also used metaprompting with o4-mini-high to revise and optimize the prompt. This is significant because the reported performance depends not only on model capability but also on careful natural-language separation of nearby refactoring concepts.
The paper instantiates RefModel with several foundation models. The small-program synthetic Java study uses Phi4-2 OR ti:\24B and Claude 3.5 Sonnet. The real-world Java study adds Gemini 2.5 Pro and o4-mini-high. The baseline tools are RefactoringMiner 3.2ti:\2, RefDiff 2.2ti:\2, and ReExtractor+ 2.4.4. Execution conditions differ across systems: Phi4-2 OR ti:\24B is run locally via Ollama on a MacBook Pro with M3 and 2 OR ti:\28 GB RAM at temperature 2ti:\2.6; Claude 3.5 Sonnet is accessed through the API with default settings; Gemini 2.5 Pro and o4-mini-high are executed manually through official web interfaces; and the traditional tools use default configurations. Model outputs are manually assessed by at least two authors, with disagreements resolved by a third (&&&2ti:\2&&&).
A notable methodological omission is that the paper reports no formal confidence score, output parser, calibration scheme, or post-processing heuristic. RefModel therefore depends directly on the first answer produced by the model, and evaluation centers on pass@2 OR ti:\2^, defined as whether that first answer is correct (&&&2ti:\2&&&).
3. Evaluation corpora and ground-truth construction
The empirical design is based on controlled single-operation transformations rather than arbitrary mixed commits. This matters because the reported results are about recognition of isolated refactorings, not general-purpose mining of noisy software history.
| Study | Input form | Instances |
|---|---|---|
| Synthetic Java | Full original and transformed programs | 858 |
| Real-world Java | GitHub diffs | 44 |
| Python/Golang feasibility | Diffs up to 2 OR ti:\249 LOC | 22ti:\2^ |
The synthetic Java corpus begins with 2 OR ti:\2,2ti:\2ti:\2ti:\2^ candidates generated from JDolly programs and transformed automatically with Eclipse JDT 4.2 OR ti:\26. Invalid cases are removed if they introduce compilation errors, behavioral changes, or faulty mechanics. Validation uses the Java compiler, SafeRefactor, and filtering procedures drawn from prior work, yielding 858 valid single-refactoring transformations (&&&2ti:\2&&&).
The real-world Java corpus contains 44 manually applied single refactoring instances created with IntelliJ 22ti:\224.2 OR ti:\2.4 across five open-source projects: Lettuce, Apache Gobblin, Google Maps Services, Spring Boot, and RefMiner. The paper is explicit that each transformation is a single refactoring instance, even though the prompt is phrased in terms of commit diffs. The distribution across refactoring types sums to 44 and includes Extract Class, Extract Interface, Extract Superclass, Inline Method, Move Method, Pull Up Field, Pull Up Method, Push Down Field, Push Down Method, Rename Class, Rename Field, and Rename Method (&&&2ti:\2&&&).
The cross-language feasibility study adds 22ti:\2^ real-world transformations, split evenly between Python and Golang, across three open-source projects. These were applied with PyCharm 22ti:\225.2 OR ti:\2.2 OR ti:\2.2 OR ti:\2^ and IntelliJ 22ti:\224.2 OR ti:\2.4. The study is smaller and does not include static-analysis baselines, but it is used to probe whether a prompt-and-definition detector trained conceptually on refactoring semantics can transfer beyond Java (&&&2ti:\2&&&).
4. Empirical performance
On the 858 synthetic Java transformations, Claude 3.5 Sonnet achieves 98.5% recall and 88.5% precision, slightly exceeding RefactoringMiner in recall while trailing it marginally in precision. RefactoringMiner reaches 95.8% recall and 89.3% precision. Phi4-2 OR ti:\24B is substantially weaker, with 79.4% recall and 56.4% precision. Category-level results reveal that Push Down Method is the hardest case for all systems: Phi4 detects 2ti:\2^ of 2 OR ti:\2 OR ti:\2, Claude 9 of 2 OR ti:\2 OR ti:\2, and RefactoringMiner 3 of 2 OR ti:\2 OR ti:\2. Claude is especially strong on Rename Method, detecting 2 OR ti:\2ti:\2ti:\2^ of 2 OR ti:\2ti:\2ti:\2^ cases compared with RefactoringMiner’s 74 of 2 OR ti:\2ti:\2ti:\2^ (&&&2ti:\2&&&).
On the 44 real-world Java instances, the strongest overall results come from the larger frontier models. Gemini 2.5 Pro and o4-mini-high both reach 93.8% recall; Claude 3.5 Sonnet follows closely at 92.2%. Their precisions are 82.2%, 82 OR ti:\2.2 OR ti:\2%, and 77.6%, respectively. Among the static-analysis tools, RefactoringMiner achieves the best recall at 88.6%, while RefDiff has the best precision at 72ti:\2.8%. RefactoringMiner and ReExtractor+ show notably low precision on this corpus, at 42 OR ti:\2.9% and 42ti:\2.2%, respectively, indicating substantial false-positive pressure under the real-world evaluation protocol (&&&2ti:\2&&&).
The paper also reports robustness by diff size. Claude 3.5 Sonnet, Gemini 2.5 Pro, and o4-mini-high remain strong across diff sizes from 2ti:\2^ to 359 LOC. Phi4-2 OR ti:\24B degrades sharply on 2 OR ti:\22ti:\2–2 OR ti:\259 LOC diffs, while RefDiff degrades on both very small and larger diffs. This suggests that the strongest RefModel instantiations are not merely memorizing minimal edit patterns, but can often reason across moderate diff contexts (&&&2ti:\2&&&).
A further claim in the paper is that Claude 3.5 Sonnet and Gemini 2.5 Pro jointly identified 97% of all real-world refactorings. The surrounding discussion indicates that this is best understood as joint coverage or union recall, not precision, and likely aggregates the 44 Java real-world instances with the 22ti:\2^ Python/Golang feasibility cases into 64 total non-synthetic instances (&&&2ti:\2&&&).
5. Generalization behavior, explanation quality, and error modes
One of RefModel’s main attractions is generalization across languages and refactoring definitions. In the Python/Golang feasibility study, Phi4-2 OR ti:\24B reaches 82ti:\2%, Claude 3.5 Sonnet 92ti:\2%, o4-mini-high 95%, and Gemini 2.5 Pro 2 OR ti:\2ti:\2ti:\2%. The paper interprets this as encouraging evidence that the prompt-and-definition paradigm can transfer to languages for which no specialized static detector was engineered in the study (&&&2ti:\2&&&).
Explanation quality is another differentiator. The models are instructed not only to name the refactoring but also to justify it briefly. The paper reports that the stronger models typically provide clear explanations, refer to relevant code elements, and accurately describe the corresponding transformation. This matters because RefModel is intended not only for batch detection but also for interactive use in code review and program comprehension, where explanations can be as valuable as labels.
The error analysis shows consistent failure modes. Models sometimes understand the structural change but output an invalid or near-miss label, such as “Pull Down Member” instead of Push Down Method. Confusions cluster around directional and hierarchy-sensitive categories: Pull Up versus Push Down, Move versus Rename, and Extract Class versus Extract Superclass. These errors are especially likely when the relevant inheritance information is not fully visible in the diff. Phi4-2 OR ti:\24B is most prone to overgeneralizing hierarchy-sensitive operations as Move Method. Claude 3.5 Sonnet occasionally confuses directionality, while Gemini 2.5 Pro can misclassify Extract Class as Extract Superclass when hierarchy cues are limited. The paper notes that o4-mini-high is comparatively good at inferring class-hierarchy structure from restricted diff context (&&&2ti:\2&&&).
6. Significance, constraints, and position within refactoring detection
RefModel’s principal significance lies in its change to the extension model of refactoring detection. In traditional detectors, adding a new refactoring type usually requires new structural logic. In RefModel, the paper’s core claim is that a new refactoring can be introduced with a natural-language sentence. This lowers the barrier to experimentation, encourages interactive querying, and makes cross-language portability more plausible than in language-bound static-analysis pipelines (&&&2ti:\2&&&).
At the same time, the study defines clear boundaries. RefModel is not yet a replacement for deterministic static analysis in all settings. The evaluated corpora focus on single-operation transformations, not multi-refactoring commits or refactorings interleaved with bug fixes and feature edits. The largest real-world diff is 359 LOC, so the paper does not establish performance on very large commits or repository-scale context. Prompt wording matters materially, and the best-performing systems are proprietary models, which introduces cost, privacy, latency, and reproducibility concerns. Even the strongest model outputs remain subject to prompt sensitivity and manual interpretation (&&&2ti:\2&&&).
A further misconception the paper implicitly rejects is that RefModel should be viewed as a black-box substitute for all refactoring infrastructure. Static tools still retain advantages where reproducibility, deterministic offline analysis, and local execution are paramount. RefModel is strongest where flexibility, natural-language extensibility, explanation, and language portability matter most. In that sense, it is best understood as a new detection paradigm rather than merely another benchmarked tool.
The overall empirical picture is therefore specific but consequential. On controlled synthetic Java data, Claude 3.5 Sonnet is comparable to RefactoringMiner and stronger on some categories. On real-world Java, Gemini 2.5 Pro, o4-mini-high, and Claude 3.5 Sonnet surpass the best static-analysis baselines in recall and precision. Across Python and Golang, the same prompting framework shows encouraging transfer. RefModel thus establishes that foundation models can perform refactoring detection at a level that is already competitive with established detectors, while additionally providing natural-language justifications and a radically simpler mechanism for extending the refactoring vocabulary (&&&2ti:\2&&&).