Round-Trip Mutation Testing: Translating Code to Natural Language Intent and back

Abstract: This paper presents Round-Trip Mutation Testing (RTM), a novel approach that generates mutants from LLM mistranslations between a program code and its intent. Leveraging the generative capability of LLMs from and to programming and natural language, and given an input program, our approach predicts its intent, that is used to generate programs, which when different from the original one, constitute the output mutants. The approach produces additionally mutants, stemming from artificially provoked mistranslations, by mutating the intent prior to the final programs (mutants) generation. Originating from the propagation of small changes in the intent to the code, our intuition is that these programs would present subtle semantic differences from the original one, simulating likely-to-occur faults that could result from specification misunderstandings, and enabling mutation testing. To evaluate RTM, we run it on 40 real buggy methods and evaluate its effectiveness and cost-efficiency in guiding testing towards detecting the bugs. Our results demonstrate the potential of round-trip mutation testing to produce syntactically more diverse mutants, potentially exposing faults that traditional mutation operators fail to reveal. More interestingly, RTM outperforms traditional pattern-based mutation in producing smaller and stronger test-suites, detecting on average over 4 and 1.7 times more faults when selecting only 4 and 30 tests respectively.

• The paper introduces an LLM-powered round-trip mutation testing pipeline that translates code to natural language intent and back to generate diverse mutants.
• It leverages intent mutation to simulate specification misunderstandings, yielding mutants with greater syntactic diversity than traditional operator methods.
• Empirical results show that RTM enhances early fault detection by 1.7x–4.3x with small test suites compared to conventional mutation testing.

Round-Trip Mutation Testing via Code/Intent Translation: A LLM-based Approach

Introduction

Mutation testing is conventionally based on introducing synthetic faults—mutants—into a program and analyzing the capacity of test suites to distinguish mutants from the original implementation. State-of-the-art strategies have historically relied on syntactic code-level manipulation using predefined mutation operators. However, such operator-based schemes are limited in their ability to generate semantically diverse mutants that effectively simulate real-world specification misunderstandings or deeper semantic faults, especially in the absence of rich documentation.

The paper "Round-Trip Mutation Testing: Translating Code to Natural Language Intent and back" (2607.03223) introduces an alternative paradigm: leveraging LLM-powered round-trip translation between code and inferred natural language intent to generate mutation candidates (mutants). This approach generalizes intent-based mutation to scenarios where explicit documentation is scarce, enabling widespread applicability.

Round-Trip Mutation Testing Methodology

Unlike traditional operator-based mutation, the Round-Trip Mutation Testing (RTM) pipeline translates the program into a natural language description (intent) using an LLM, then either mutates the intent or not, and translates the (possibly mutated) intent back to code via the same or another LLM. In effect, noise and ambiguity in LLM translation, as well as explicit mutations in the intent, introduce subtle or diverse program changes that may correspond to likely specification misunderstandings and corner case defects.

The pipeline is parameterized over:

• Precision of intent extraction: The prompt can emphasize either a broad (general goal) or precise (algorithmic/operational detail) extraction.
• Application of explicit intent mutation ($\mu$): Mutations (word replacements with semantic drift) can be injected in the intent before back-translation.

By decoupling mutant generation from code syntax and mutation operator libraries, RTM unlocks new diversity not reachable by pattern-based mutation.

Empirical Study: Data and Mutant Generation

RTM is empirically evaluated on 40 real-world buggy Python methods from the BugsInPy dataset, across 5 diverse open-source projects. Mutants are generated via 4 RTM variants (Broad, Precise, $\mu$Broad, $\mu$Precise) and compared against a reimplementation of MutPy's traditional (pattern-based) operators. For each method, 10 mutants per approach are synthesized.

The evaluation is designed to simulate actual developer processes: tests are incrementally selected to "kill" mutants, and the ability to detect real faults (i.e., bug-revealing tests) is empirically measured at various test suite sizes.

The distribution of available documentation across these projects demonstrates the scarcity of docstrings in real-world methods, motivating the necessity for automatic intent inference:

(Figure 1)

Figure 1: Distribution of documented methods across the 5 projects analyzed, highlighting limited explicit intent annotation.

Mutation Validity and Syntactic Diversity

RTM produces a substantial proportion of valid, killable mutants across all variants, though the classic operator-based approach still slightly leads in total valid mutants. A crucial finding is that RTM mutants—especially those resulting from broad intent extraction—exhibit significantly greater syntactic diversity compared to traditional mutants, as quantified by BLEU-based code distance metrics.

Figure 2: Distribution of distances (1 - BLEU score) for code mutations, contrasting the proximity of operator-based mutants to the original code with the increased syntactic dispersion yielded by RTM.

Notably, mutants resulting from broad intent translation are more syntactically distant both from the source and from each other, while precise intent and intent mutation steps modulate this effect. This diversity is hypothesized to reflect the introduction of specification-level deviations and implementation innovations not possible with operator mutation.

Guided Test Selection and Fault Detection Efficacy

Test suites were incrementally built by killing surviving mutants using the pool of available (developer and LLM-generated) tests, recording the detection rate of real faults as a function of test budget.

RTM demonstrates a strong advantage in cost-effective fault detection: for small test suite sizes (especially up to 30 tests), RTM mutants produce test selections that detect 1.7x--4.3x more faults than traditional operator mutants. This advantage attenuates as more tests are selected, as traditional mutants eventually reach parity or even surpass RTM in aggregate fault detection—a finding attributed to higher redundancy among RTM mutants (i.e., tests that kill one mutation often kill many).

(Figure 3)

Figure 3: Mutation testing simulation; each curve traces the mean fault detection rate as a function of selected test suite size, for both RTM and pattern-based schemes.

Analysis and Discussion

The results indicate that RTM mutants are particularly effective for guiding the efficient early discovery of specification-revealing tests. Precise intent translation (without explicit intent mutation) yields the highest early fault-detection rates, despite producing fewer runnable mutants. Contrarily, explicit intent mutation increases the number of non-runnable mutants, slightly impairing practical efficacy.

However, RTM mutants are less semantically diverse in the sense that the set of tests required to kill distinct mutants rapidly plateaus. This limits the incremental benefit as the test suite size grows. The findings suggest future directions:

• Contextualization in back-translation to reduce non-runnable mutants
• Integration of semantics-driven or feedback-guided mutation to enhance behavioral diversity
• Application to other architectures and languages by exploiting the language-agnostic nature of the LLM pipeline

Implications and Outlook

RTM does not require operator libraries or a-priori knowledge of programming language specifics, and does not depend on developer-written documentation—unlike operator-based and prior intent-based methods. The LLM-centric approach is highly portable and applicable in continuous integration/deployment ecosystems where code and tests coevolve rapidly with sparse annotations.

In the broader context, the study anticipates new hybrid mutation testing pipelines that exploit LLMs for both specification inference and mutant synthesis. As LLM capabilities mature, fine-grained control over semantic drift in intent mutation and better calibration of translation fidelity can allow precise targeting of specification-level test objectives. Enhancing test suites guided by RTM can expand the detection of nuanced or emergent faults rooted in misunderstandings of requirements, rather than simple implementation errors.

RTM ultimately exemplifies the feasibility and practical benefit of using generative models for advanced software testing methodologies, bridging the gap between natural-language-driven development and formal mutation testing practices.

Conclusion

Round-Trip Mutation Testing, as proposed in this work, constitutes a significant advancement beyond traditional syntactic mutation approaches by operationalizing code-intent-code translation via LLMs. The ability to synthesize highly diverse mutants without dependency on documentation or operator libraries, and the empirical demonstration of increased early-stage fault detection, mark RTM as a compelling direction for mutation testing research and practice. Scaling and refining RTM with improved LLMs, intent mutation strategies, and domain-aware constraints represents an open trajectory for future work in test adequacy and software quality assurance.