On the Reproducibility of Quantum Software Defect Datasets: A Case Study of Bugs4Q

Abstract: The reproducibility of software defect datasets is essential for obtaining reliable and comparable research results. Zhu et al. have shown that defect datasets such as Defects4J suffer from reproduction failures (i.e., reported bugs become non-reproducible) as time passes since their creation. However, it remains unclear whether these findings generalize to quantum software defect datasets. We therefore conduct a replication study of the prior work using Bugs4Q, a widely used dataset of real-world bugs in quantum programs. Our analysis includes 77,700 quantum program executions of 37 Bugs4Q artifacts across 21 core-library versions. The experimental results showed that the reproducibility of Bugs4Q dropped from 62.2% on Qiskit v0.20.1 to 16.2% on v2.3.1, the latest version as of April 1, 2026. A manual inspection of the root causes further indicated that 93.6% of the failures were dependency-related. While these findings are consistent with those of the prior work, we also observed differences. In particular, most reproduction failures in Bugs4Q cannot be resolved merely by adjusting dependency versions; instead, they require source-code modifications such as migrating import paths and API invocations. Based on this observation, we curated Bugs4Q-Robust, a patched version of Bugs4Q to restore reproducibility. Bugs4Q-Robust increases reproducibility from 16.2% to 78.4% on Qiskit v2.3.1. Our findings highlight the importance of continuous dataset maintenance in the rapidly evolving quantum software ecosystem.

• The paper demonstrates that Bugs4Q exhibits significant reproducibility decay, dropping from 62.2% to 16.2% with major Qiskit upgrades.
• It employs a rigorous longitudinal design across 37 quantum programs and 77,700 runs to identify API changes as the primary root cause.
• The study introduces Bugs4Q-Robust, a patched benchmark that recovers reproducibility to 59.5%, underscoring the need for proactive code migration.

Reproducibility of Quantum Software Defect Datasets: An Expert Analysis of "On the Reproducibility of Quantum Software Defect Datasets: A Case Study of Bugs4Q"

Introduction and Motivation

Ensuring the reproducibility of software defect benchmarks is pivotal for the rigorous assessment and comparison of testing, fault localization, and automated repair techniques. With the growing prominence of quantum programming paradigms and corresponding software engineering research, the stability of quantum-centric defect datasets like Bugs4Q underpins the integrity and comparability of empirical investigations. This paper systematically interrogates the temporal reproducibility of Bugs4Q, situating its findings in the context of prior studies on classical software defect collections such as Defects4J and BugSwarm. The research involves a comprehensive experimental protocol spanning 37 real-world quantum program artifacts across 21 Qiskit releases (encompassing three major version series), resulting in a dataset of over 77,700 executed runs.

Figure 1: An overview of the longitudinal assessment design, including artifact selection, environment orchestration, and reproducibility criteria evaluation.

Experimental Design and Evaluation Criteria

The longitudinal framework is designed to answer two central research questions: how does Bugs4Q's reproducibility decay over time and what are the root causes for any observed failures? Each snapshot environment targets a specific Qiskit version (major or minor release), and two dependency configurations are considered for each: 'Core-only' (updating only the primary Qiskit library) and 'Pinned-stack' (aligning related ecosystem packages with documented compatibility for the selected core Qiskit version).

Reproducibility is stringently defined at three hierarchical levels:

• Existence: The original buggy behavior is observable and the fixed version passes the test case.
• Type Match: The failure mode (e.g., exception type, assertion failure) matches contemporary documentation.
• Cause Match: The underlying failure evidence (error message, distribution bias) semantically aligns with the original artifact.

Artifacts were executed 30 times per configuration to control for inherent quantum program stochasticity, leveraging statistical oracles where appropriate.

Figure 2: Existence—Reproducibility percentages for each snapshot as Qiskit evolves across 0.x, 1.x, and 2.x version series.

Key Results

Temporal Degradation of Reproducibility

The analysis reveals a dramatic, stepwise loss in reproducibility following major Qiskit version transitions. While 62.2% of Bugs4Q artifacts remain reproducible under Qiskit v0.20.1 (using the strictest Cause Match criterion), this rate plunges to 16.2% under v2.3.1, in spite of best-effort ecosystem-wide dependency pinning. Notably, artifacts' reproducibility never spontaneously recovers at subsequent versions after a breakdown event.

Root Cause Taxonomy

Manual labelling of 543 root cause instances (with substantial inter-rater agreement, κ = 0.67) yields the following high-level breakdown:

• Source-related failures (82.0%): Evolving Qiskit APIs invalidate import paths or invocation signatures, necessitating source code migration (e.g., module or symbol relocations, argument changes).
• Library-related failures (11.6%): Ecosystem package incompatibilities or behavioral changes, partially addressable by dependency management.
• Quantum-specific nondeterminism (5.1%): Statistical oracles overly sensitive to shot variance.
• Test construction defects (1.3%): Artifact test logic insufficiently robust or incorrect, rendering bugs irreproducible even in legacy environments.

Crucially, only 4.6% of all failures admit a remedy purely at the environment/dependency layer, underscoring the necessity for active code migration as APIs evolve.

Figure 3: Number of newly broken artifacts per criterion under Pinned-stack, illustrating breakage clustering around major Qiskit upgrades.

Dataset Maintenance and Patch Strategies

Leveraging observed failure modes, the authors curate a patched benchmark, Bugs4Q-Robust. This involves semantically minimal code edits (guided by Qiskit migration documentation) and, where applicable, test oracle and execution parameter adjustments. Under Qiskit v2.3.1, Bugs4Q-Robust recovers reproducibility for 59.5% of artifacts (Cause Match level), compared to the 16.2% baseline with the unmodified dataset.

Discussion: Classical vs. Quantum Defect Dataset Decay

A direct comparison against classical software defect dataset studies (see, e.g., Zhu et al.) demonstrates both continuity and divergence in root causes and practical remediation scope:

• Both classical and quantum datasets overwhelmingly lose reproducibility due to environment and dependency changes.
• Quantum-specific breakage is predominantly due to API surface volatility. Migrating core libraries like Qiskit deprecate or refactor symbols and behaviors at a much higher velocity than seen in most classical environments.
• Contrary to the Java/Defects4J world, dependency version pinning is grossly inadequate for quantum datasets. Recovery in the quantum context almost always necessitates non-trivial source-level refactoring.
• The rate of permanently unrecoverable breakages is greater and irreversible, especially as major library releases systematically invalidate legacy interfaces.
• There is a clear imperative for continuous, automated code migration tooling (potentially leveraging LLMs or program transformation frameworks) for sustaining the utility of quantum defect corpora.

Implications and Future Directions

Theoretical Impact: The study sharply elucidates limits of reproducibility for software defect research in fast-evolving quantum ecosystems. Evolving user-facing APIs, rather than only dependency rot or environment entropy, emerge as the dominant vector of decay.

Practical Recommendations:

• For benchmark creators/maintainers: Proactive, recurring migration of artifact code is mandatory; one-shot patching is futile. Documentation of environment, dependency, and patch history at release time is essential.
• For users/evaluators: Careful recording of version context and applied fixes is critical when benchmarking or reusing quantum defect artifacts.
• For tool builders: There is a research gap in reliable, automated code refactoring tools tailored for scientific and quantum software stack evolution.

Generalizability: While the study is situated in Qiskit and Bugs4Q, the root articulation and strategies generalize to any domain where rapid platform evolution outpaces environment management practices.

Conclusion

This investigation conclusively establishes that quantum software defect datasets are profoundly susceptible to reproducibility decay, much more so than their classical counterparts. The root cause analysis reveals the primacy of source-level interface rot—remediation hence demands curated code migration, not solely dependency control. The patched Bugs4Q-Robust resource marks a significant practical advance, but its maintenance will require ongoing effort or automated migration as Qiskit continues evolving. The broader implication is that reproducibility, and thus the integrity of quantum software engineering research, now depends as much on active dataset stewardship as it does on sound experimental protocol.