YATE: Repair-Oriented Java Test Generation

• YATE is a repair-oriented Java unit-test generation approach that leverages LLMs to convert syntactically and semantically near-miss tests into valid tests.
• It employs a multi-stage workflow combining chain-of-thought test planning, static analysis for compilation repair, oracle adjustment, and coverage augmentation.
• Empirical results demonstrate that YATE significantly improves line, branch, and mutation scores over existing LLM-based testing methods in diverse Java projects.

YATE is a Java unit-test generation approach that uses LLMs but treats incorrect generated tests as repairable assets rather than outputs to discard. Its central premise is that many LLM-generated tests are “near misses”: they often already target relevant program logic, yet fail because of syntax, compilation, or oracle errors. YATE therefore combines initial chain-of-thought-driven test generation, rule-based static analysis for compilation repair, rule-based and LLM-assisted oracle repair, and coverage-guided augmentation into a single repair-oriented workflow for automated test generation (Konstantinou et al., 24 Jul 2025).

1. Conceptual basis and problem setting

YATE addresses a recurring weakness of LLM-based unit test generation: LLMs frequently produce tests that are syntactically malformed, uncompilable, semantically wrong, or equipped with failing assertions. Existing assured workflows often detect such tests and discard them, but YATE characterizes this as a “missed opportunity” because these tests may already invoke the right methods, instantiate the right objects, and target useful paths in the class under test (Konstantinou et al., 24 Jul 2025).

The approach is grounded in two claims. First, incorrect tests may still have direct testing value if repaired, since they can increase line coverage, branch coverage, and mutation score. Second, they have indirect value because repaired tests can serve as seeds for further augmentation. The paper emphasizes that this matters especially in Java, where realistic unit tests often require constructors in other classes, interfaces and implementing classes, external method calls, inherited behavior, or mockable dependencies that are not present in a minimal prompt containing only the class or method under test (Konstantinou et al., 24 Jul 2025).

A common misconception is that invalid generated tests are merely noise. YATE rejects that view and instead treats them as partially useful artifacts whose defects can often be recovered through lightweight software-engineering analysis plus targeted re-prompting. This suggests a shift from generation-only pipelines toward generation-and-repair pipelines (Konstantinou et al., 24 Jul 2025).

2. End-to-end workflow

YATE’s workflow has four main components: initial test generation with chain-of-thought, compilation error fixing, oracle fixing, and coverage-based test augmentation (Konstantinou et al., 24 Jul 2025).

The initial generation stage is not a single direct prompt for code. YATE first asks the model to analyze the target code, at either class level or method level, with the goal of achieving 100% code coverage. Before producing tests, the model is asked for a report containing the list of tests needed, descriptions of those tests, and the total number of tests required per targeted method. If the class contains private or protected methods, YATE issues an additional prompt to expose them and update the report. It then asks the model to reflect on whether the report will achieve 100% branch coverage, and only after this planning phase does it request the actual unit test code (Konstantinou et al., 24 Jul 2025).

After generation, YATE compiles and executes the tests, distinguishing compilation failures from runtime or oracle failures. Non-compiling tests enter the compilation-fixing pipeline. Tests that compile but fail due to assertions enter oracle fixing. Coverage augmentation is performed in a second generation cycle using information about uncovered branches; the process stops at the end of that second cycle (Konstantinou et al., 24 Jul 2025).

The overall design is therefore hybrid rather than purely prompt-based. Deterministic repair is used where mechanical fixes are feasible, while the LLM is re-engaged only after YATE has gathered repository-derived context that was missing during the original generation step (Konstantinou et al., 24 Jul 2025).

3. Repair mechanisms

Compilation repair is organized as a breadth-first process that prioritizes issues affecting the entire file, then shared issues, then more local defects. YATE uses Spoon for code analysis and method-call-graph construction before compilation fixing (Konstantinou et al., 24 Jul 2025).

The first repair stage concerns imports. YATE statically analyzes the repository and classpath to find required imports for referenced classes, inserts missing import statements, checks whether referenced symbols actually exist, and removes or corrects hallucinated imports. It then retrieves external class implementations when generated tests instantiate objects defined outside the class under test. For such classes, YATE collects constructor signatures, and for interfaces or abstract classes it also retrieves implementing classes; this information is fed back to the LLM (Konstantinou et al., 24 Jul 2025).

A further repair stage handles wrong method invocations. If a generated test calls a method that does not exist, YATE explicitly informs the model; if the method exists but has been invoked with the wrong signature or arguments, YATE supplies the valid method signatures. If errors remain, YATE analyzes the project’s method call graph and retrieves method bodies, signatures, and relation information for methods called by the class under test but defined elsewhere. Complete compiler error logs can also be injected into the repair prompt. The paper notes a recurring ambiguity caused by constructor calls using null; in such cases YATE prompts the model to provide a stricter-typed object instead of null (Konstantinou et al., 24 Jul 2025).

Oracle repair is partly rule-based. If an assertion compares an explicit value, such as a number, string, or boolean, YATE replaces the expected value with the actual runtime value observed during execution. For non-explicit assertions such as assertTrue(...) or assertNotNull(...), it can reverse the assertion to match the observed outcome. If a test fails because it throws an exception, YATE can replace the last oracle with an exception oracle asserting that exception; conversely, if an exception oracle expects an exception but none occurs, it can invert the oracle, for example replacing assertThrows(...) with assertDoesNotThrow(...) in JUnit 5. The paper also states that YATE maintains a configurable denylist of exception types to avoid, especially for failures caused by incorrect mock usage. When assertions are too complex for rule-based parsing, the failing oracle line and failure information are sent to the LLM for modification. If an oracle repair causes the test to stop compiling, the last change is reverted. LLM-based oracle repair repeats for at most $N=3$ iterations by default (Konstantinou et al., 24 Jul 2025).

This repair strategy implies that YATE’s main novelty lies less in novel prompting syntax than in structured prompt enrichment. The LLM is re-prompted with compiler logs, constructor signatures, dependency class code, method signatures, external method bodies, relation information, uncovered branches, and failing assertion information, depending on the repair stage (Konstantinou et al., 24 Jul 2025).

4. Evaluation methodology and empirical results

YATE was evaluated on 6 open-source Java projects, totaling 393 classes under test and 3657 mutants (Konstantinou et al., 24 Jul 2025). The projects were Binance connector 2.0.0, Chesslib, ConfigMe, Event-ruler, Windward, and Batch-processing-gateway. The study distinguishes seen and unseen projects using a cutoff date of October 1, 2023, relative to likely LLM training exposure (Konstantinou et al., 24 Jul 2025).

The evaluation used both class-level and method-level prompting. Coverage was measured with JaCoCo using line coverage and branch coverage, mutation was measured with PiTest using default configuration, and cost was measured by number of LLM requests. Statistical analysis used the Mann–Whitney U test with significance threshold $p < 0.05$ and Vargha–Delaney effect size (Konstantinou et al., 24 Jul 2025).

Relative to a plain LLM baseline, YATE substantially improved both test validity and test effectiveness. At class level, total compilation rate increased from 88.64% to 94.60%, and total passing rate increased from 50.53% to 90.18%. At method level, total compilation rate increased from 84.34% to 90.02%, and total passing rate increased from 53.40% to 82.96% (Konstantinou et al., 24 Jul 2025).

The aggregate adequacy results are central:

Configuration	Coverage / score
Plain-C	line 33.53%, branch 21.05%, mutation 22.08%
Plain-M	line 49.95%, branch 35.33%, mutation 33.82%
YATE-C	line 60.71%, branch 44.27%, mutation 41.39%
YATE-M	line 64.82%, branch 51.06%, mutation 47.99%
YATE-Comb	line 70.13%, branch 57.01%, mutation 52.04%

At average per-project level, YATE(Class) achieved 64.94% line coverage, 52.26% branch coverage, and 43.98% mutation score, while YATE(Method) achieved 69.33% line coverage, 60.35% branch coverage, and 51.10% mutation score (Konstantinou et al., 24 Jul 2025). The paper reports that class-level YATE improved over plain class-level generation by 32.06 percentage points in line coverage, 28.69 percentage points in branch coverage, and 21.77 percentage points in mutation score. Method-level YATE improved over plain method-level generation by 18.36 percentage points in line coverage, 22.03 percentage points in branch coverage, and 16.98 percentage points in mutation score (Konstantinou et al., 24 Jul 2025).

Ablation results show that all four components matter. Removing coverage augmentation had a smaller effect on line coverage but a larger effect on branch coverage, while replacing compilation fixing caused the largest drop in branch coverage and mutation score. The paper quantifies the absence of compilation fixing as causing a 19.30% drop in total branch coverage and a 10.45% drop in total mutation score. This supports the interpretation that compilation-relevant feedback is more important than coverage feedback in this workflow (Konstantinou et al., 24 Jul 2025).

5. Position relative to other LLM-based testing systems

YATE was compared with four other LLM-based methods: HITS, SymPrompt, TestSpark, and CoverUp (Konstantinou et al., 24 Jul 2025). For fairness, the main comparison used GPT-4o-mini across methods. The results show that both YATE(Class) and YATE(Method) exceeded all compared systems in line coverage, branch coverage, and mutation score (Konstantinou et al., 24 Jul 2025).

Method	Total result
HITS	line 42.4%, branch 29.5%, mutation 28.0%
SymPrompt	line 41.0%, branch 27.2%, mutation 28.4%
TestSpark	line 23.7%, branch 16.5%, mutation 13.9%
CoverUp	line 38.9%, branch 25.1%, mutation 26.3%
YATE (Class)	line 60.7%, branch 44.3%, mutation 41.6%
YATE (Method)	line 64.8%, branch 51.1%, mutation 47.9%

The paper summarizes this as approximately 22 percentage points higher line coverage, 20 percentage points higher branch coverage, and 20 percentage points higher mutation score than prior methods at comparable LLM-call cost (Konstantinou et al., 24 Jul 2025). Relative to HITS specifically, YATE(Method) achieved 22.36 percentage points higher line coverage, 21.57 percentage points higher branch coverage, and 19.97 percentage points higher mutation score (Konstantinou et al., 24 Jul 2025).

Cost analysis is also significant. YATE(Class) required 4296 total LLM calls, fewer than HITS, SymPrompt, TestSpark, CoverUp, and YATE(Method). YATE(Method) required 21249 calls and was among the most expensive configurations, but also the most effective. A notable practical finding is that class-level prompting gave only about 5% lower coverage than method-level prompting while requiring about 5× fewer LLM calls (Konstantinou et al., 24 Jul 2025). This positions class-level YATE as an efficiency-oriented configuration and method-level YATE as a performance-oriented configuration.

6. Limitations, interpretation, and significance

YATE’s evaluation is limited to Java and JUnit5-style testing, and the paper does not provide a deep treatment of flaky or nondeterministic tests (Konstantinou et al., 24 Jul 2025). Its repair mechanisms are heuristic and bounded: exception repair uses a configurable denylist, call-graph exploration is recursive but bounded in depth, and oracle repair is capped at $N=3$ LLM-assisted iterations (Konstantinou et al., 24 Jul 2025). The implementation also depends on Spoon and on the accuracy of the derived method call graph (Konstantinou et al., 24 Jul 2025).

Another limitation is that YATE is stronger at fixing tests that fail than at detecting tests that pass with a wrong oracle. This follows from its regression-style oracle repair strategy, which aligns failing assertions to observed behavior, rather than solving the broader problem of latent oracle invalidity (Konstantinou et al., 24 Jul 2025).

Even with those constraints, the system has broader methodological significance. It demonstrates that repository-aware static analysis can supply the missing program facts that raw compiler-error feedback does not provide, and that this context materially changes the utility of LLM-based test generation (Konstantinou et al., 24 Jul 2025). A plausible implication is that future LLM-based testing systems may increasingly resemble software-engineering toolchains with retrieval, repair, validation, and augmentation stages, rather than single-shot prompt-to-test generators.

YATE’s central contribution is therefore not merely higher adequacy metrics. It is a reframing of invalid generated tests as recoverable intermediate artifacts. In the paper’s formulation, repair is not an afterthought but a first-class mechanism for extracting latent value from LLM-generated tests, yielding stronger compile rates, stronger pass rates, stronger coverage, and stronger mutation performance than both a plain LLM baseline and the compared LLM-based state of the art (Konstantinou et al., 24 Jul 2025).