UTRefactor: LLM-Driven Java Test Refactoring

• UTRefactor is a context-enhanced, LLM-driven framework for automated refactoring of Java unit tests, systematically eliminating test smells.
• It integrates rule-based smell detection, explicit static context extraction, a DSL of refactoring rules, and chain-of-thought prompting for consistent code transformation.
• Empirical evaluations show up to an 89% reduction in test smells with high compilation and execution pass rates across multiple Java Maven projects.

Searching arXiv for UTRefactor and related refactoring work to ground the article in current literature. Searching arXiv for "Automated Unit Test Refactoring UTRefactor (Gao et al., 2024)". UTRefactor is a context-enhanced, LLM-driven framework for automatically refactoring Java unit tests to remove test smells while preserving behavior. It was introduced for Java Maven projects in response to two recurring difficulties in automated test maintenance: rule-based tools are often tied to rigid syntax patterns and a small set of predefined rules, while direct prompting of LLMs can produce hallucinations, inconsistent smell interpretations, and incomplete handling when multiple smells coexist. UTRefactor addresses these issues by combining rule-based smell detection, explicit static context extraction, a knowledge base of canonical smell definitions, a DSL of refactoring rules, chain-of-thought prompting, and a smell-wise checkpoint mechanism that systematically verifies whether each targeted smell has been removed (Gao et al., 2024).

1. Problem domain and formal objective

Test smells are poor design patterns in test code, including Assertion Roulette, Eager Test, Duplicate Assert, and Magic Number Test. They decrease test readability and maintainability, make debugging and evolution harder, and are widespread in real projects. The motivating premise behind UTRefactor is that manually refactoring such smells is time-consuming and error-prone, while automated support remains limited in both robustness and coverage (Gao et al., 2024).

The framework is defined against two baseline families. The first is rule-based refactoring, exemplified by TESTAXE, which uses hard-coded AST transformation rules tied to a specific framework transition, JUnit4 to JUnit5. In the reported characterization, TESTAXE covers only five rules, depends on rigid syntax patterns, and cannot generalize well to complex patterns such as exceptions inside try/catch with custom handling. The second is direct LLM-based refactoring, in which the model is given a test and a list of smells and asked to rewrite the test directly. The paper identifies three recurrent problems in that setting: hallucinations that invent or delete logic, inconsistent understanding of smell definitions, and multi-smell incompleteness, in which only a subset of the detected smells is actually removed (Gao et al., 2024).

UTRefactor’s stated goal is therefore twofold: to guide LLMs to refactor in an expected, rule-conformant way while avoiding hallucinations and preserving behavior, and to eliminate multiple smells in a single test systematically, with an order and checks that ensure completeness. Formally, given a project $P$ and test suite $T$, with a smell detector $D$ yielding a set of smells $S$ per test, the framework aims to compute a refactored test suite $T'$ such that $T'$ compiles and its tests pass, the number of smells reported by $D$ is minimized, and no new smells of handled types are introduced (Gao et al., 2024).

A broader empirical context makes this focus technically consequential rather than incidental. An analysis of refactoring discussions on Stack Overflow identified Unit Testing as one of five major refactoring areas, comprising 1,086 questions, or 11.46% of the refactoring-related questions studied, with “unit test” among the prominent bigrams in question bodies (Peruma et al., 2021). This suggests that automated test refactoring addresses a persistent maintenance problem rather than a narrow benchmark niche.

2. System architecture and end-to-end workflow

UTRefactor is organized into three main steps: preprocessing, construction of a test refactoring knowledge base, and refactored test generation. The input is a Java Maven project; the output is a modified project in which smelly tests have been refactored, plus a report listing each refactored test, its smells, and how they were eliminated (Gao et al., 2024).

In preprocessing, the framework extracts test files and individual test methods, detects smells with tsDetect, and collects contextual information for each smelly test. A notable design choice is the shift from file-level smell detection to method-level refactoring. Because tsDetect operates at file level, UTRefactor splits each test file into temporary sub-files, each containing a single @Test method together with the minimal supporting context such as class declaration, imports, and setup code. It then runs tsDetect on those sub-files to map smells to specific test methods, and after refactoring merges the updated sub-files back into complete test files (Gao et al., 2024).

The second stage constructs a knowledge base that standardizes smell definitions and examples based on tsDetect’s 19 smell types and defines a DSL of refactoring rules for 13 function-level smells. The framework also prioritizes smell categories in order to determine refactoring order. This turns smell semantics and remediation steps into externalized artifacts rather than latent assumptions embedded in the LLM’s pretraining distribution (Gao et al., 2024).

The third stage generates refactored tests through a CoT-style prompt containing extracted context, the detected smell set for the test, canonical definitions, and the relevant DSL rules. The LLM is instructed to understand the test, reason about each smell, apply the DSL steps, and generate refactored code. A checkpoint mechanism is then applied per smell type: after a refactoring pass, the model explicitly checks whether the smell remains; if it does, the test is refined again before the system moves to the next smell (Gao et al., 2024).

The resulting architecture is explicitly hybrid. Smell detection remains rule-based and external to the LLM; the LLM is used only for code transformation. This division of labor is central to the framework’s design: detection provides deterministic smell labels, while generation uses richer contextual reasoning to instantiate refactorings that rigid AST rewrite systems often miss (Gao et al., 2024).

3. Context extraction, smell semantics, and the DSL

A defining feature of UTRefactor is its explicit treatment of context. For each smelly test method, static analysis extracts the package name of the focal class, the focal class name, the focal method signature, the focal method comment, and the signatures of other invoked methods. The mapping from test to focal class is performed by stripping the Test prefix or suffix, as in ParserTest to Parser, and searching the main source tree. No retrieval scoring function is introduced; selection is deterministic and static-analysis-based (Gao et al., 2024).

This extracted context is represented directly in the prompt together with the test code snippet, the list of detected smell types for the test, canonical smell definitions, and the DSL rule blocks. The intended effect is to make the LLM’s interpretation of focal behavior, auxiliary scaffolding, and refactoring constraints explicit. This is particularly important for operations such as splitting an Eager Test into multiple tests without changing semantics (Gao et al., 2024).

The knowledge base uses Peruma et al.’s 19 tsDetect smell definitions as the authoritative semantics. The motivation is that LLMs’ internal notions of test smells are inconsistent. The paper provides a concrete example: LLaMA-70B interprets Mystery Guest as dependence on external resources, while the detector defines it as unused variables in setup or teardown. Injecting canonical definitions into the prompt therefore serves as a semantic normalization layer (Gao et al., 2024).

The DSL is structured rather than free-form. Each entry contains SmellType, Description, Steps, Example, and Variables. Although the paper does not present a full formal grammar in BNF, the structure is uniform across smell types. For Assertion Roulette, the DSL instructs the model to add descriptive messages to assertions that lack them. For Magic Number Test, it instructs the model to replace numeric literals with named constants or variables. For Duplicate Assert, it provides a three-step plan: replace @Test with @ParameterizedTest, add a @CsvSource or similar annotation listing scenarios, and refactor the method body so that only one assertion remains, using parameters (Gao et al., 2024).

The DSL is not executed by an external engine. Instead, it is injected into the prompt as structured and unambiguous instructions, and the LLM is told to follow these steps exactly. This design preserves generative flexibility for naming variables and writing concrete Java code while imposing deterministic guidance on what transformation should occur and in what order (Gao et al., 2024).

A plausible implication is that UTRefactor operationalizes a transformation-oriented view of refactoring at a finer granularity than traditional text rewriting. That perspective aligns with broader arguments that software refactoring can be treated as compositions of small code transformations, each narrow enough to validate semi-formally through tests, build checks, or intermediate artifacts (Liu, 2023).

4. Prompting strategy and checkpoint-based multi-smell handling

UTRefactor explicitly models human refactoring behavior in four chain-of-thought phases: understand test intent, identify and reason about smells, plan refactoring using DSL rules, and generate the refactored test code. The prompt also contains system-style constraints such as acting as a Java testing expert and not changing external behavior or tested logic (Gao et al., 2024).

This design is aimed at reducing the ad hoc nature of direct LLM refactoring. The model is not asked merely to rewrite a test; it is asked first to explain what the test verifies, then to ground each detected smell in the concrete test, then to plan transformations in terms of the DSL steps, and only then to emit code. The intent is accuracy and consistency rather than opportunistic rewriting (Gao et al., 2024).

The checkpoint mechanism is the framework’s primary response to multi-smell incompleteness. For a test with smell set $S=\{s_1,s_2,\dots,s_n\}$, smells are ordered by priority. For each smell $s_i$, the current version of the test is presented together with the corresponding DSL rule, and the model is asked whether the smell is still present according to the supplied definition. If yes, it refactors further; if no, it leaves the test unchanged and advances to the next smell. After all smells are processed, the resulting test is taken as final (Gao et al., 2024).

The paper’s example of a Jsoup test containing Assertion Roulette, Eager Test, Duplicate Assert, and Magic Number Test illustrates why this machinery is needed. Even when all smells are enumerated in a single prompt, a naive one-shot edit often fixes only some of them. By contrast, the checkpoint chain treats each smell as a guarded subtask whose elimination must be checked before the pipeline proceeds (Gao et al., 2024).

This sequencing also shapes how combined refactorings are realized. A Duplicate Assert refactoring may first convert a test into a parameterized test, after which Assertion Roulette still requires the single remaining assertion to receive a descriptive message. The framework therefore does not assume that smells are independent; it assumes they interact and must be revisited in an ordered process (Gao et al., 2024).

5. Smell model, target refactorings, and empirical performance

UTRefactor builds on the 19 test smells from tsDetect and uses 13 function-level smells for automated refactoring. Explicitly referenced function-level smells include Assertion Roulette, Magic Number Test, Duplicate Assert, Eager Test, Exception Catching Throwing, Conditional Test Logic, Redundant Assertion, Mystery Guest, Resource Optimism, and Sensitive Equality. Some smells are not refactored but removed instead: Default Test, Empty Test, Unknown Test, and Ignored Test are discarded because they do not contribute meaningful testing behavior (Gao et al., 2024).

The refactoring strategies encoded in the DSL vary by smell type. Eager Test is addressed by identifying groups of assertions tied to different production behaviors and splitting the original test into multiple tests. Duplicate Assert is converted into a parameterized test with a single core assertion. Exception Catching Throwing is transformed into JUnit exception assertions such as assertThrows or assertDoesNotThrow. Sensitive Equality is handled by preferring domain-specific equality or dedicated comparison methods when such alternatives are available. Across all smell types, the stated constraints are to preserve line and branch coverage, avoid behavior-changing additions or removals of production API calls, and perform structural changes only (Gao et al., 2024).

The evaluation uses six open-source Java Maven projects: commons-cli, commons-compress, commons-math-legacy, gson, jfreechart, and jsoup. Across these projects there are 9,149 tests, 879 tests with smells, and 2,375 total smells. The baselines are TESTAXE and a direct LLM baseline using Llama3-70B with a prompt that includes only the Java testing expert role, the unit test code, and the smell list, without context, smell definitions, DSL rules, or checkpoints (Gao et al., 2024).

Quality is reported using Compilation Pass Rate, Execution Pass Rate, smell counts before and after refactoring, smell reduction rate, per-smell counts, time, and coverage with Jacoco for tests that still pass. Across 1,522 refactored test instances, the average CPR is 95% and the average EPR is 89%, with the highest CPR in Jsoup at 97% and the highest EPR in Gson and Jsoup at 92%. The reported line and branch coverage for each project remain unchanged after refactoring for passing tests (Gao et al., 2024).

The headline effectiveness result is a reduction from 2,375 smells to 265 in tests that still pass, corresponding to an 89% reduction. The direct Llama3-70B baseline reduces smells from 2,375 to 1,080, or 55%, and TESTAXE reduces them to 2,315, or less than 1%. The paper reports a 61.82% relative improvement over direct LLM-based refactoring in smell elimination. Per-project reduction under UTRefactor reaches 94% for Commons-compress, 91% for Jsoup, and approximately 86% to 89% for the others (Gao et al., 2024).

Per-smell results further indicate that Assertion Roulette is completely eliminated in all refactored tests, Exception Catching Throwing is reduced from 526 to 54, and Eager Test is reduced from 509 to 20. Sensitive Equality remains difficult because successful elimination depends on the project already offering non-fragile comparison methods; where the codebase lacks such alternatives, the framework cannot remove the smell without inventing behavior (Gao et al., 2024).

Time efficiency is also quantified. Across all six projects and 879 smelly tests, preprocessing takes 536 seconds, knowledge-base and DSL construction 24 seconds, and LLM-based refactoring 3,369 seconds, for a total of approximately 4,379 seconds, or about 3.8 seconds per test on average. TESTAXE is faster at approximately 600 seconds total because it uses purely local AST transformations with no network calls; the direct Llama3-70B baseline takes approximately 3,291 seconds. UTRefactor is therefore slower than direct prompting, but the paper attributes the difference to context extraction, knowledge-base loading, and checkpoint iterations (Gao et al., 2024).

6. Limitations, related work, and research significance

UTRefactor’s limitations are explicit. DSL rules may not cover all corner cases, and custom assertion APIs can violate the assumptions encoded in a generic rule. The reported Gson failure case is illustrative: the custom assertion assertStrictError accepts exactly two parameters, but the Assertion Roulette rule prompts the model to add a descriptive message parameter, yielding a signature mismatch and a compilation error. The paper also notes that LLM hallucinations can still occur, though they are mitigated by the DSL and checkpoint mechanism (Gao et al., 2024).

External-validity constraints are also narrow. tsDetect supports Java only up to version 13; newer constructs may cause parsing errors. The evaluation covers only Java projects with JUnit, and only six open-source projects. The framework is described as detector-agnostic in principle, but no cross-language or industrial-scale evaluation is reported (Gao et al., 2024).

In the broader landscape of automated refactoring, UTRefactor occupies a distinctive position. It differs from detection-oriented work such as tsDetect, TASTE, and PYNOSE by targeting automated repair rather than smell identification alone, and it differs from rigid rule-based tools such as TESTAXE by combining externalized semantics with generative rewriting (Gao et al., 2024). Relative to transformation-centric views of refactoring, UTRefactor can be read as a method-level instantiation of the idea that refactoring should be decomposed into small, validated transformations rather than performed as a monolithic rewrite (Liu, 2023).

Later work on automated refactoring with reinforcement learning suggests one possible extension path. For Java Extract Method refactoring, a sequence-to-sequence model aligned with PPO and code-centric rewards improved BLEU by 11.96% and CodeBLEU by 16.45% over supervised fine-tuning alone, and increased successful unit tests from 41 to 66 on a suite of 122 tests for the best CodeT5-based model (Palit et al., 2024). This suggests that UTRefactor’s existing detector, DSL, and checkpoint pipeline could plausibly be combined with code-centric or test-centric RL alignment, although such integration is not part of the reported system.

The paper also reports that UTRefactor works with multiple LLMs. On a sampled subset, GPT-4o with a naive prompt achieves approximately 57% smell reduction, whereas GPT-4o integrated into UTRefactor achieves approximately 91% reduction, mirroring the improvement observed with LLaMA-3-70B (Gao et al., 2024). This suggests that the framework’s gains are attributed primarily to orchestration, context, and structured guidance rather than to a single model family.

UTRefactor’s principal contribution is therefore not merely the use of an LLM for code rewriting, but the construction of a reliable refactoring process around the model: deterministic smell detection, explicit context extraction, canonical smell semantics, structured DSL rules, CoT prompting, and smell-wise checkpoints. Within the reported Java/JUnit setting, that process yields high smell reduction, high compilation and execution pass rates, and unchanged coverage for passing tests, positioning UTRefactor as a concrete synthesis of static analysis and controlled generative refactoring (Gao et al., 2024).