PITMuS: Automated Java Bug Dataset Generation

• PITMuS is a Java tool for automated bug dataset generation by reconstructing source-level mutants from PIT's XML reports.
• It combines debug information from compiled classes with original source files to produce method-level bug pairs enriched with Javadoc and metadata.
• Evaluated on multiple open-source projects, PITMuS achieves a 99.96% reconstruction rate, supporting empirical studies in bug localization and repair.

Searching arXiv for PITMuS and closely related references to ground the article in the relevant paper. PITMuS is a Java tool for automated bug dataset generation via source-level mutant reconstruction. It addresses a specific limitation of PIT, a state-of-the-practice mutation testing tool for Java that performs mutation at the bytecode level: PIT reports mutants primarily through XML metadata, which is sufficient for mutation analysis but not for directly producing editable buggy source files, method-level buggy/fixed pairs, or structured artifacts for downstream machine-learning and empirical software-engineering workflows. PITMuS bridges that gap by combining PIT XML metadata with debug information from compiled Java class files and the original Java source to reconstruct the corresponding source edit for each mutant, then packaging the result as a dataset record containing code, documentation context, and metadata (Tasnim et al., 21 May 2026).

1. Scope, motivation, and problem setting

PITMuS is motivated by the increasing dependence of LLM-based software engineering on executable, context-rich bug artifacts. The targeted artifact structure includes paired correct and buggy code, the method under test, documentation such as Javadoc, and metadata describing the mutation site and type. The intended downstream uses named for these artifacts are bug localization, program repair, test generation, test oracle generation and assessment, and documentation-driven automation (Tasnim et al., 21 May 2026).

The tool is positioned as a complement to curated benchmarks such as Defects4J rather than a replacement for them. The stated rationale is twofold. First, curated benchmarks are static. Second, they are increasingly vulnerable to contamination as code models are trained on large public corpora. PITMuS instead supports the generation of fresh, cutoff-aware datasets from selected versions of real Java systems by injecting controlled bugs through mutation testing and then reconstructing those bugs at source level (Tasnim et al., 21 May 2026).

A central distinction in this design is that PITMuS does not perform mutation testing itself. PIT remains the mutation producer, and PITMuS acts as a reconstruction-and-packaging layer on top of PIT’s output. This means that PITMuS inherits PIT’s practical mutation workflow while targeting a different output regime: inspectable source-level artifacts rather than primarily XML mutant reports (Tasnim et al., 21 May 2026).

2. Inputs, operating assumptions, and system organization

PITMuS requires three inputs for a target Java system: a PIT XML report, compiled .class files with debug metadata, and the original Java source files. Its implementation requirements are Python 3.8+, javalang, and a Java JDK providing javap. The tool exposes two commands: python gen_dataset.py <system-path> for dataset generation and python inject.py <system-path> for source-level mutant injection (Tasnim et al., 21 May 2026).

The system organization has two major phases. The first is dataset generation. The second is source-level injection. In the first phase, PITMuS reads PIT’s XML entries, identifies the corresponding source-level edit, extracts the enclosing method and preceding Javadoc, and writes the resulting record into a structured dataset. In the second phase, PITMuS materializes reconstructed mutations as concrete Java source files by copying the original file, replacing the target statement with the reconstructed mutated statement, preserving indentation, and checking the result via javalang tokenization (Tasnim et al., 21 May 2026).

This organization depends on a specific reconciliation problem. PIT reports metadata such as class, source file, line number, mutator, mutation description, and bytecode index, but it does not directly specify the mutated source text. A single reported line may contain multiple syntactically compatible mutation sites. The example given is a line containing both index + 1 and length + index: a line number plus a mutation description such as “Replaced integer addition with subtraction” is not by itself sufficient to identify which + operator was mutated. PITMuS resolves such ambiguity by combining source parsing with bytecode/debug-level disambiguation (Tasnim et al., 21 May 2026).

3. Reconstruction model and algorithm

PITMuS defines the dataset for a Java system $P$ with PIT report $R_P$ as

$$\mathcal{D}_P = \{\, \langle \mathrm{orig}_i, \mathrm{mut}_i, \mathrm{doc}_i, \mathrm{meta}_i \rangle \mid m_i \in R_P \,\}.$$

Each record corresponds to one PIT mutation $m_i$. The fields are defined as the original enclosing method, the mutated version of that method, the preceding Javadoc when present, and metadata such as mutant ID, line number, file path, bytecode index, and mutation description (Tasnim et al., 21 May 2026).

The algorithmic core is presented as “PITMuS extraction and injection.” For each parsed PIT XML entry $(c,\mu,\ell,d,i)$, PITMuS loads the relevant source representation $(L,\Sigma)$, where $L$ denotes source lines and $\Sigma$ structural information. It then computes a candidate set

$C \gets \textsc{MatchDescription}(L[\ell], d),$

which identifies candidate source occurrences on the reported line that are textually compatible with the mutation description. If $|C| = 1$, the occurrence is selected directly. Otherwise PITMuS invokes bytecode-based disambiguation:

$R_P$0

It then applies the mutation,

$R_P$1

finds the enclosing method span $R_P$2, extracts the original method $R_P$3, constructs the mutated method $R_P$4, extracts documentation $R_P$5, and appends the resulting record (Tasnim et al., 21 May 2026).

The source-level injection phase uses a filter predicate

$R_P$6

where $R_P$7 is the selected mode and $R_P$8 the target. For each matching mutant, PITMuS repeats source loading, candidate matching, optional bytecode disambiguation, and mutation application, then reindents the result, substitutes it into the source file, computes an output path, and writes the mutated Java file. In plain terms, PITMuS is a mapping-and-rewrite pipeline: parse PIT metadata, localize the source occurrence, resolve ambiguity using class-file debug information when necessary, reconstruct the source edit, and package the enclosing source context (Tasnim et al., 21 May 2026).

4. Produced artifacts and replay modes

The primary output artifact is a method-level dataset record. Each record contains the original enclosing method, the mutated enclosing method, the preceding Javadoc when available, and mutation metadata. The metadata is described as including mutant ID, line number, file path, bytecode index, and mutation description. This organization makes the output directly usable for training and evaluation tasks that need both code and natural-language context (Tasnim et al., 21 May 2026).

A second output class consists of materialized source files. PITMuS can generate replayable mutated Java files at four granularities: mutation-level, statement-level, class-level, and system-level. Mutation-level injection targets one specific mutant by mutant ID. Statement-level injection materializes all reconstructed mutants for a selected statement. Class-level injection covers all reconstructed mutants for a selected Java class. System-level injection covers all reconstructed mutants across the entire system (Tasnim et al., 21 May 2026).

A recurring misconception is that PITMuS merely converts XML into another textual representation. Its actual output regime is richer. It preserves method boundaries, extracts Javadoc context, supports filtered replay, and collapses bytecode-level mutant information into source-level artifacts suitable for empirical study. The paper emphasizes inspectability, replayability, and dataset usability as the practical gains of source-level reconstruction (Tasnim et al., 21 May 2026).

The evaluated mutation categories reflect the PIT operators preserved by the reconstruction pipeline: VoidMethodCall, NullReturns, TrueReturns, FalseReturns, Increments, InvertNegatives, PrimitiveReturns, RemoveConditionals, ConditionalsBoundary, EmptyReturns, Math, and ExperimentalSwitch (Tasnim et al., 21 May 2026).

5. Empirical evaluation

The evaluation studies PITMuS on eight real-world open-source Java projects: bcel, commons-beanutils, commons-dbutils, commons-jexl3, commons-lang3, http-request, joda-time, and jsoup. These systems span bytecode engineering, reflection utilities, JDBC support, expression-language processing, Java utility libraries, HTTP clients, date/time handling, and HTML parsing. They also vary substantially in size, from http-request with 4 source files to commons-lang3 with 401 source files (Tasnim et al., 21 May 2026).

Across all eight systems, the evaluation reports 1,913 source files, 316,352 SLOC, 69,229 PIT mutants processed, and 69,198 reconstructed original–mutant method pairs. This corresponds to a 99.96% overall reconstruction rate. The resulting datasets also preserve substantial documentation context: 45,913 records have non-empty Javadoc, corresponding to 66.35% Javadoc coverage (Tasnim et al., 21 May 2026).

At project level, several results are notable. commons-lang3 yields the largest generated dataset, with 17,992 reconstructed pairs and 94.01% Javadoc coverage. commons-beanutils, commons-dbutils, commons-lang3, http-request, and jsoup each achieve 100.00% preservation on the reported evaluation. commons-jexl3 reports 14,030 mutants and 14,003 preserved cases, corresponding to 99.81%, while joda-time reports 13,119 mutants and 13,117 preserved cases, corresponding to 99.98% (Tasnim et al., 21 May 2026).

Operator-level results are similarly high. VoidMethodCall, NullReturns, TrueReturns, FalseReturns, Increments, and InvertNegatives are reconstructed perfectly at 100.00%. The remaining categories also remain near-complete: PrimitiveReturns at 99.97%, RemoveConditionals at 99.96%, ConditionalsBoundary at 99.89%, EmptyReturns at 99.86%, Math at 99.82%, and ExperimentalSwitch at 99.69% (Tasnim et al., 21 May 2026).

The paper uses the terms “preserved” and “reconstructed” to denote successful conversion of a PIT mutant into an aligned source-level original–mutant method pair. The success criterion is therefore stronger than merely parsing the PIT XML: it requires producing a usable source-level artifact (Tasnim et al., 21 May 2026).

6. Failure modes, relation to PIT, and broader significance

Only 31 of 69,229 mutants were not reconstructed, and the failure analysis identifies two main patterns. The first is duplicate bytecode-level mutants mapping to the same source edit. The example given is the statement fp = tinter.fp - 1;, for which PIT reports six mutations for a single - operator. Once PITMuS reconstructs that source-level edit once, the remaining bytecode-distinct instances collapse to duplicates at source level and are skipped. The second pattern involves multi-line expressions in which PIT reports a line number that is not the source line containing the operator. Because the current rewriter searches within a single source line, such cases are not reconstructed (Tasnim et al., 21 May 2026).

These limits clarify the operating assumptions of the current implementation. PITMuS assumes a consistent triple of PIT XML report, original source, and compiled classes with debug metadata. It also assumes that the relevant source edit can be localized within a single source line. This suggests that the present tool is highly effective for the dominant cases produced by PIT, while some source-position mismatches and multi-line rewrites remain outside its current scope (Tasnim et al., 21 May 2026).

Within the broader PIT ecosystem, PITMuS occupies a different layer from mutation engines such as Descartes. Descartes is a PIT mutation engine plugin implementing extreme mutation and using PIT’s infrastructure for project inspection, dependency discovery, mutant/test execution, and integration with Java build workflows (Vera-Pérez et al., 2018). PITMuS, by contrast, does not extend PIT’s mutation semantics or operator set. It operates after mutation analysis, taking PIT’s bytecode-level results and reconstructing source-level artifacts for dataset generation and replay. This distinction is important: PITMuS is not a new mutation engine, but a source-level reconstruction and packaging system built around PIT’s output model (Tasnim et al., 21 May 2026).

The broader significance of PITMuS lies in making mutation-generated faults reusable at source level for contemporary software-engineering research. By transforming bytecode-level PIT mutants into original–mutant method pairs with documentation and metadata, it enables fresh, scalable, cutoff-aware benchmark construction on arbitrary Java systems integrated with PIT. In that sense, PITMuS connects mutation testing to the data requirements of modern bug localization, repair, test-generation, and documentation-aware learning pipelines while preserving the practical advantages of PIT’s bytecode-level mutation workflow (Tasnim et al., 21 May 2026).