PITMuS: A Tool for Automated Bug Dataset Generation via Source-Level Mutant Reconstruction

Abstract: LLM-based software engineering increasingly depends on executable, context-rich bug artifacts: paired correct and buggy code, methods under test (MUTs), documentation, and metadata. These artifacts support the training and evaluation of automated bug localization and repair techniques, testing and test oracle generation methods, and documentation-driven automation. Although curated benchmarks (e.g., Defects4J) remain valuable, they are static and increasingly vulnerable to contamination as code models are trained on large public corpora. A complementary strategy is to generate fresh, cutoff-aware datasets by selecting real system versions and injecting controlled bugs at the source level. Mutation testing is a natural basis for this strategy: it applies predefined mutation operators to programs and records whether the existing test suite detects each injected change. PIT is a state-of-the-practice mutation testing tool for Java that performs mutation at the bytecode level. This design makes mutation testing fast and practical, but PITMuS reports mutants primarily through XML, making them difficult to inspect, replay, or reuse as structured source-level dataset records. To address this gap, we present PITMuS, which combines PITMuS XML metadata with debug information from compiled Java class files to localize and reconstruct the source edit corresponding to each mutant. PITMuS then automatically produces structured datasets containing source-level buggy and fixed code pairs, documentation context, and metadata for downstream training and evaluation. Although we evaluate PITMuS on eight open-source Java systems, it can be applied to any Java system where PITMuS can be integrated.

• The paper presents PITMuS, a tool that reconstructs source-level bugs from PIT mutation reports, bridging bytecode-level data with source code analysis.
• It leverages XML metadata, bytecode debug info, and source parsing to generate paired original-mutant code artifacts with a 99.96% reconstruction rate.
• Empirical evaluation on eight Java systems demonstrates robust performance, enhancing datasets for LLM-driven bug localization, repair, and testing.

PITMuS: Automated Bug Dataset Generation via Source-Level Mutant Reconstruction

Motivation and Context

The development and evaluation of LLM-based software engineering methods—such as automated bug localization, program repair, test generation, and documentation-aware automation—require rich, structured datasets comprising paired correct and buggy code, relevant context (e.g., Javadoc), and metadata. Traditional curated benchmarks, exemplified by Defects4J, are fixed and static, leading to increased vulnerability to benchmark contamination as code models ingest broad public corpora. This limitation is exacerbated in the current era of LLM-centric software engineering, creating demand for fresh, dynamically generated datasets.

Mutation testing, particularly in the Java ecosystem through the PIT tool, offers a principled approach for injecting controlled bugs. However, PIT operates at the bytecode level and reports mutants through XML metadata, hindering access to source-level mutant artifacts required for empirical studies, model training, and tool development.

PITMuS Tool Overview

PITMuS addresses a critical gap in mutation testing by reconstructing source-level mutants from PIT's bytecode-oriented reports. By integrating XML metadata, debug information from compiled class files, and javalang-based source parsing, PITMuS precisely localizes mutations and generates datasets capturing original and mutated code pairs, enriched with context and metadata. PITMuS offers two main functionalities: (i) source-level mutant injection, enabling replay at granularities ranging from single statements to system-wide; (ii) method-level dataset extraction, producing artifacts inclusive of Javadoc comments and detailed mutation information for downstream applications.

The workflow orchestrated by PITMuS is depicted in the tool's overview figure:

Figure 1: PITMuS workflow integrating PIT XML metadata, Java bytecode debug info, and source parsing for mutant reconstruction and dataset generation.

Technical Approach

PITMuS processes three inputs: PIT XML reports identifying mutant metadata; compiled .class files supplying bytecode-location mapping essential for disambiguation; and the original Java source files. For each mutation, PITMuS matches the mutation description and line number to candidate source tokens via javalang parsing, resolving ambiguities using bytecode indices. The mutant is reconstructed by editing the corresponding source line and extracting the enclosing method and preceding Javadoc. This is formalized in an algorithm that iterates over PIT reports, reconstructs the mutant edit, and outputs method-level code pairs enriched with metadata.

PITMuS supports injection at mutation-level, statement-level, class-level, and system-level granularity, facilitating both fine-grained empirical analysis and large-scale dataset construction.

This reconstruction process for a single source line is illustrated below:

Figure 2: Example showing a Java source line, the applied mutation, and the resulting mutated source line.

Empirical Evaluation

PITMuS is evaluated on eight open-source, real-world Java systems spanning diverse domains and codebase sizes. Across 1,913 source files, PITMuS processes 69,229 PIT mutations and successfully reconstructs 69,198 aligned original-mutant pairs (99.96% reconstruction rate). Notably, 45,913 (66.35%) of these pairs retain non-empty Javadoc context, enabling documentation-aware evaluation.

Preservation rates by PIT mutation operator are uniformly high, with complete reconstruction for VoidMethodCall, NullReturns, TrueReturns, FalseReturns, Increments, and InvertNegatives. RemoveConditionals saw a minor loss, with 14 unreconstructed mutations out of 35,739. Failure cases are dominated by source-position mismatches (multi-line expressions or duplicate bytecode-level mutants mapping to the same source edit), and account for less than 0.05% of total mutations.

Implications and Future Directions

PITMuS transforms the utility of bytecode-level PIT mutants, producing empirically valuable datasets that support LLM-driven research in bug localization, repair, testing, and documentation integration. Strong numerical results demonstrate that nearly all generated PIT mutants can be reconstructed at the source level, unlocking large-scale, context-rich datasets for automated and LLM-based software engineering tasks.

Practically, PITMuS enables contamination-free, cutoff-aware dataset generation by leveraging arbitrary system versions and controlled bug injection. The tool permits extensibility to any Java system where PIT is integrated. Theoretically, this methodology enables new lines of inquiry into program repair and test generation, including model robustness, mutation operator efficacy, and the impact of documentation context.

Potential future developments include extension to additional programming languages, support for more granular mutation operator discrimination, and integration with advanced code analysis and annotation pipelines to improve mapping for complex multi-line mutations.

Conclusion

PITMuS enables automated reconstruction of source-level buggy code artifacts from PIT mutation reports, yielding high-fidelity, context-rich datasets for LLM-based and automated software engineering studies. By bridging the gap between bytecode-level mutation testing and source-level empirical requirements, PITMuS facilitates rigorous, scalable evaluation and training pipelines, minimizing manual effort and benchmark contamination. The tool's open-source release under the Apache 2.0 license further supports broad adoption and practical impact in empirical software engineering and AI-driven development workflows.