Intent-Based Mutation Testing: From Naturally Written Programming Intents to Mutants

Abstract: This paper presents intent-based mutation testing, a testing approach that generates mutations by changing the programming intents that are implemented in the programs under test. In contrast to traditional mutation testing, which changes (mutates) the way programs are written, intent mutation changes (mutates) the behavior of the programs by producing mutations that implement (slightly) different intents than those implemented in the original program. The mutations of the programming intents represent possible corner cases and misunderstandings of the program behavior, i.e., program specifications, and thus can capture different classes of faults than traditional (syntax-based) mutation. Moreover, since programming intents can be implemented in different ways, intent-based mutation testing can generate diverse and complex mutations that are close to the original programming intents (specifications) and thus direct testing towards the intent variants of the program behavior/specifications. We implement intent-based mutation testing using LLMs that mutate programming intents and transform them into mutants. We evaluate intent-based mutation on 29 programs and show that it generates mutations that are syntactically complex, semantically diverse, and quite different (semantically) from the traditional ones. We also show that 55% of the intent-based mutations are not subsumed by traditional mutations. Overall, our analysis shows that intent-based mutation testing can be a powerful complement to traditional (syntax-based) mutation testing.

• The paper introduces a novel approach that mutates natural-language programming intents using LLMs to generate semantically diverse and behaviorally meaningful mutants.
• It employs intent masking and MLM-based prediction to synthesize code artifacts, achieving over 80% syntactic validity and substantial semantic divergence from traditional mutants.
• Empirical results show that intent-based testing enhances test suite effectiveness by detecting faults that are typically missed by syntax-based mutation techniques.

Intent-Based Mutation Testing: Mutating Programming Intents via LLMs

Motivation and Limitations of Traditional Mutation Testing

Mutation testing is regarded as a premier methodology for assessing the adequacy of software test suites. Standard approaches rely on the syntactic perturbation of code through mutation operators, such as token replacements or operator inversions, generating mutants that emulate small, local code defects. While highly effective for certain fault categories, such models are inherently limited to logic explicitly implemented in code, and predominantly induce shallow, syntax-level changes. This leaves critical gaps: they miss omission faults, misunderstandings of specification, or behavioral deviations rooted in intent rather than explicit code structure.

To address this, "Intent-Based Mutation Testing: From Naturally Written Programming Intents to Mutants" (2607.05149) introduces a fundamentally distinct paradigm—mutating the programming intent itself, as expressed in natural language, to derive code mutations that embody plausible misunderstandings or alternative interpretations of a specification. This approach is positioned to induce more substantial, behaviorally meaningful faults, spanning entire method implementations, and to drive the generation of more semantically diverse mutants.

Methodology: Mutating Intents with LLMs

The intent-based workflow leverages LLMs for both mutating the natural-language intent and for synthesizing corresponding code implementations. The process is orchestrated in several interconnected steps:

1. Intent Masking: Each token in the original intent (natural language description) is sequentially masked, generating a set of masked intents.
2. MLM-Based Prediction: For each masked intent, a pre-trained LLM (such as BERT) predicts plausible token replacements, yielding variant intents with minimal but meaningful differences.
3. Code Synthesis: These mutated intents are then provided as prompts to a generative LLM (GPT-3.5-turbo), which synthesizes corresponding code artifacts—these implementations form the intent-based mutants.
4. Implementation Diversity Generation: Additionally, the system prompts the LLM to generate numerous alternative implementations for the original intent, capturing variance introduced by the stochastic nature and creative capacity of the model itself.

   Figure 1: Overview of intent-based mutation testing workflow, where intent mutation and implementation synthesis are orchestrated via LLMs.

This process is agnostic to programming language grammar and historical bug data, relying solely on the capabilities of state-of-the-art LLMs for contextual comprehension and code synthesis.

Evaluation Framework and Research Questions

The authors evaluate the effectiveness and uniqueness of the intent-based paradigm along three axes:

• RQ1 (Diversity): Does intent-based mutation testing generate syntactically and semantically diverse mutants?
• RQ2 (Overlap): Are intent-based mutants distinct from those generated by syntax-based methods?
• RQ3 (Effectiveness): How effective are tests designed using intent-based mutants in detecting faults compared to other methods?

For evaluation, 29 Java programs from the HumanEval-x dataset (augmented with HumanEval+ test cases) are used, ensuring that only tasks with non-trivial, killable mutant sets are selected.

Empirical Results and Key Findings

Validity and Diversity of Mutants

Intent-based mutation testing demonstrates a high rate of syntactically valid code generation (over 80%), a stark improvement over several baselines. The generated mutants exhibit pronounced syntactic and semantic distance from the original code—BLEU scores and Jaccard/cosine distances indicate that LLM-generated intent mutants are substantially more divergent than those from classical syntax-based mutation.

Regarding behavioral diversity, subsumption analyses reveal that a large proportion of intent-based mutants are not subsumed by others. The percentage of subsuming mutants for intent-based approaches is significantly higher (60.2% and 76.29% for intent-mutation and implementation-diversity modes, respectively) compared to syntax-based baselines, indicating broad behavioral coverage.

Figure 2: Percentages of subsuming and subsumed mutants produced by the intent-based approach, highlighting high semantic diversity.

Semantic Novelty and Overlap with Syntactic Mutants

Intent-based mutants are found to be largely orthogonal to syntactic mutants. Only 55% of intent-based mutants are subsumed by traditional mutations; over 23% of the intent-based subsuming mutants are uniquely exposed and cannot be detected by tests designed for syntactic mutants. Venn analyses of test suite effectiveness demonstrate that intent-based mutations produce a richer spectrum of behavioral anomalies, many of which do not overlap with those induced by syntax-based mutation tools.

Figure 3: Proportion of subsuming mutants among different approaches, quantifying the unique and complementary potential of intent-based testing.

Test Suite Effectiveness: Subsuming Mutation Scores

A direct comparison of test suite quality—using subsuming mutation scores as the metric—shows that tests derived via intent-based mutants consistently kill a broader set of strong, subsuming mutants (objective score 0.817) compared to syntactic (0.77) and implementation-diversity (0.66) approaches. Notably, none of the approaches fully subsume the others, but intent-based methods capture behavioral faults missed by traditional techniques, as evidenced by the fact that 23% of faults detected by the unified reference set are not captured by the strongest individual method.

Figure 4: Distribution of subsuming mutation scores achieved by test suites for each mutant generation strategy, with intent-based approaches achieving higher coverage on average.

Implications and Future Research Directions

This work substantiates that intent-based mutation testing enables the systematic creation of complex and semantically novel mutants that serve as proxies for specification misunderstandings, omissions, and high-level behavioral bugs. From a practical standpoint, this leads to:

• Comprehensive fault detection, with significant coverage of otherwise elusive behavioral defects.
• Expansion of mutation testing beyond what is possible with current grammar-bound, syntax-based operators.
• Replacement or augmentation of traditional mutant selection criteria with intent-centric strategies to improve test suite quality.

On the theoretical front, LLM-driven intent mutation bridges the gap between natural-language specifications and code-level testing, proposing a new pipeline for specification-driven test generation and augmenting semantic adequacy metrics in automated software testing.

Looking forward, open research questions include: optimizing the selection of intent mutation strategies, integrating intent-based mutants with symbolic execution or formal methods for proof- or refinement-oriented testing, and applying this methodology to large-scale, real-world software and continuous integration pipelines. Further, the expansion to complex system-level intents and integration with requirements engineering could lead to robust specification-based quality assurance frameworks.

Conclusion

Intent-based mutation testing fundamentally enhances the scope and efficacy of mutation analysis by leveraging LLMs to mutate and implement program intents directly. Empirical evidence shows that this approach generates mutants that are more syntactically and semantically diverse, uncovers faults missed by standard mutation tools, and results in higher-quality test suites. These findings position intent-based mutation as a crucial complement to syntax-based mutation testing, with significant practical and theoretical implications for the automation and robustness of software testing processes (2607.05149).