Bugs4Q-Robust: Restoring Quantum Bug Reproducibility

Updated 5 July 2026

Bugs4Q-Robust is a curated, patched release of the Bugs4Q dataset that restores bug reproducibility in modern Qiskit environments using minimal source-level migrations.
It boosts reproducibility metrics from 16.2% to as high as 78.4% by addressing dependency issues and interface drifts while preserving original bug semantics.
The dataset underpins reproducible research in quantum software engineering by maintaining executable fail–pass artifacts for consistent evaluation and repair studies.

Bugs4Q-Robust is a curated, patched release of the Bugs4Q quantum software defect dataset designed to restore and stabilize bug reproducibility under modern Qiskit versions. It was created after a longitudinal replication study showed that the reproducibility of the original Bugs4Q, collected in May 2022, degraded sharply as the Qiskit ecosystem evolved: under Qiskit v2.3.1, reproducibility fell to 16.2% across the study’s criteria, and most failures were dependency-related. Bugs4Q-Robust addresses this decay through minimal source-level migrations and oracle adjustments intended to preserve the original bug semantics while enabling execution on Qiskit v2.3.1, raising reproducibility to 78.4% for Existence, 64.9% for Type Match, and 59.5% for Cause Match (Ohto et al., 25 Jun 2026).

1. Origin in the Bugs4Q benchmark lineage

Bugs4Q-Robust is derived from Bugs4Q, a benchmark of real bugs for quantum programs in the Qiskit ecosystem. The original Bugs4Q paper describes a publicly available benchmark of thirty-six real, manually validated Qiskit bugs from four popular Qiskit elements—Terra, Aer, Ignis, and Aqua—supplemented with test cases for reproducing buggy behaviors, along with interfaces for accessing buggy and fixed versions and executing the corresponding tests (Zhao et al., 2021).

Subsequent studies used different Bugs4Q subsets rather than a single invariant artifact set. The reproducibility study underlying Bugs4Q-Robust analyzed 37 artifacts after excluding five artifacts lacking tests (Ohto et al., 25 Jun 2026). The ChatGPT-based repair study retested 38 reproducible buggy Qiskit programs from Bugs4Q (Guo et al., 2024), and QChecker evaluated 42 real-world buggy Qiskit programs written in Qiskit (Zhao et al., 2023). These counts are reported by the respective studies and reflect evaluation-specific selections rather than a unified “robust” split.

A central clarification is that Bugs4Q-Robust is not a robustness-focused bug-type subset in the sense of reclassifying bugs by semantic difficulty, nor is it a new benchmark family independent of Bugs4Q. Earlier Bugs4Q-based papers explicitly do not mention any “Bugs4Q-Robust” variant (Guo et al., 2024). Its defining purpose is maintenance of reproducibility under contemporary Qiskit, not redefinition of the underlying defects.

2. Reproducibility decay and the need for a patched release

The motivating study conducted 77,700 quantum program executions of 37 Bugs4Q artifacts across 21 Qiskit core-library versions spanning the 0.x, 1.x, and 2.x series. Execution used Docker-isolated environments, with Python 3.9.25 for 0.x and 1.x and Python 3.10.20 for 2.x, and compared two dependency configurations: Core-only and Pinned-stack (Ohto et al., 25 Jun 2026).

The study formalized reproducibility using three increasingly strict criteria. Existence requires that the buggy version fails and the fixed version succeeds. Type Match additionally requires that the failure type match the source. Cause Match further requires that runtime evidence—such as traceback, message, or output distribution—match the cause described in the original source. Reproducibility rate and related quantities were defined as

$R = 100 \times \frac{n_{\text{reproduced}}}{n_{\text{total}}}, \qquad F = 100 - R, \qquad \Delta R = R_{\text{Robust}} - R_{\text{Original}}.$

On Qiskit v0.20.1, the original Bugs4Q achieved 75.7% for Existence, 64.9% for Type Match, and 62.2% for Cause Match. Under later major upgrades, these rates dropped sharply. In the 1.x series, Existence peaked at 21.6% and Cause Match at 18.9%. In the 2.x series, Existence, Type Match, and Cause Match converged to 16.2% on v2.3.1. Overall, 31 of 37 artifacts, or 83.8%, experienced at least one reproduction failure over the three-year window, and once an artifact became non-reproducible, it never returned to a reproducible state in subsequent snapshots (Ohto et al., 25 Jun 2026).

The paper identifies a point of contrast with prior observations on classical defect datasets. Only 4.6% of failures were fixable by dependency pinning alone; most required source-code modifications. This is a defining rationale for Bugs4Q-Robust: version pinning was insufficient to recover the benchmark.

3. Failure modes induced by Qiskit ecosystem evolution

Manual analysis classified 543 snapshot–artifact failures into four main classes and seven leaf categories. Source-related failures accounted for 82.0%, library-related failures for 11.6%, quantum-specific failures for 5.1%, and original test issues for 1.3%. The paper further states that dependency-related failures dominated overall, with

$f_{\text{dep}} = 93.6\%.$

This aggregates Source-related 82.0% and Library-related 11.6% (Ohto et al., 25 Jun 2026).

Within source-related failures, Import incompatibility represented 48.1% and API invocation incompatibility 33.9%. The paper gives a representative import migration:

1
2
3

from qiskit.extensions import HGate, CXGate

from qiskit.circuit.library import HGate, CXGate

It also gives a representative API migration from execute to explicit transpilation plus backend execution:

job = execute(qc, backend=Aer.get_backend('qasm_simulator'), shots=1024)

backend = Aer.get_backend('qasm_simulator')
new_circuit = transpile(qc, backend)
job = backend.run(new_circuit, shots=1024)

Library-related failures were subdivided into Library–library incompatibility at 4.6%, Library-internal behavior change at 6.1%, and Output format change at 0.9%. The paper also identifies Quantum-specific failures caused by probabilistic sampling variance, mitigated by tuning shots and statistical thresholds such as the significance level $\alpha$ . Finally, a small fraction of cases corresponded to original test issues that could not be repaired without redefining semantics (Ohto et al., 25 Jun 2026).

At a higher level, the relevant Qiskit transitions were the v1.0 reorganization and the v2.0 consolidation around primitives such as Sampler and Estimator, along with continuing import path migrations, removal of execute, and changed backend invocation patterns. These changes often caused artifacts to fail before reaching the original buggy lines, for example via ImportError, or to fail with a different type or cause than the one originally recorded.

4. Curation principles and construction of Bugs4Q-Robust

Bugs4Q-Robust was curated by starting from failures under the strictest criterion, Cause Match, and determining a concrete root cause for each artifact. The patch policy was to apply only minimal changes necessary to address the identified root cause, preserve the original user-level bug semantics and test intent, and prefer documented migration paths in official Qiskit guides. The paper summarizes this policy with the principle “patch the interface break, not the bug” (Ohto et al., 25 Jun 2026).

The curated patches fall into four main classes. Import path updates migrate legacy modules such as qiskit.extensions to qiskit.circuit.library and qiskit.providers.aer to qiskit_aer. API invocation changes replace removed or renamed functionality, including execute with transpile + backend.run, and, where applicable, opflow-based APIs with primitives such as Sampler and Estimator. Circuit construction and oracle adjustments address signature changes, output-format changes, and probabilistic assertions by tuning shots and significance levels. Dependency adjustments align surrounding packages to mutually compatible versions with Qiskit v2.3.1 and remove deprecated packages that conflict with modern stacks (Ohto et al., 25 Jun 2026).

All patches were authored manually using Qiskit 1.0 and 2.0 migration guides and release notes, and the changes and rationale were recorded with runtime logs. Bugs4Q-Robust includes all 37 artifacts used in the study, and each artifact contains the buggy program, fixed program, standalone test code, and metadata linking to the original reported source.

The replication package is public at https://github.com/kusumotolab/quantum-defect-reproducibility. Its root contains dependency specifications targeting Qiskit v2.3.1, runner scripts, a Dockerfile, a Makefile, and runtime logs. The target environment is Python 3.10.20 with qiskit==2.3.1, and the paper recommends QasmSimulator from qiskit_aer for deterministic availability (Ohto et al., 25 Jun 2026).

5. Evaluation outcomes and residual irreproducibility

The principal quantitative effect of Bugs4Q-Robust is restoration of reproducibility on Qiskit v2.3.1. The study reports the following results (Ohto et al., 25 Jun 2026):

Criterion	Original Bugs4Q	Bugs4Q-Robust
Existence	16.2%	78.4%
Type Match	16.2%	64.9%
Cause Match	16.2%	59.5%

The corresponding improvements are reported as $\Delta R = 62.2$ percentage points for Existence, $\Delta R = 48.7$ percentage points for Type Match, and $\Delta R = 43.3$ percentage points for Cause Match. These values indicate that most recoverable failures were due to interface drift rather than disappearance of the underlying defects.

The remaining shortfall is also characterized. For the 15 artifacts that still do not satisfy Cause Match, the dominant cause is Library-internal behavior change, accounting for approximately 66.7% of the remaining artifacts. The paper notes cases in which new APIs absorb past misuse—for example, Estimator implicitly diagonalizes, eliminating the original failure mode. A few cases remain blocked by import incompatibility where removed modules lack direct replacements, and one artifact has unresolved original test issues (Ohto et al., 25 Jun 2026).

The validation protocol remains test- and runtime-evidence-based rather than semantics-complete. Repeated runs, specifically 30 trials per buggy and fixed artifact per snapshot, were used to control flakiness, especially for probabilistic oracles. This suggests that Bugs4Q-Robust improves operational reproducibility under a fixed software stack, but does not claim a formal semantic-equivalence guarantee for the migrated artifacts.

6. Role in the Bugs4Q research ecosystem

Bugs4Q-Robust changes the practical status of Bugs4Q within quantum software engineering by turning an increasingly decayed benchmark into a usable contemporary baseline. This matters because multiple lines of research already depend on Bugs4Q. The ChatGPT repair study evaluated semi-automated repair on 38 reproducible Qiskit bugs, using Bugs4Q test suites as the sole correctness oracle and reporting that richer descriptions and iterative feedback substantially improve repair outcomes (Guo et al., 2024). QChecker evaluated static bug detection on the standard Bugs4Q and reported aggregate precision, recall, and F1 on Qiskit programs, again without any Bugs4Q-Robust variant (Zhao et al., 2023). For such studies, benchmark decay directly threatens comparability across time.

A recurrent misconception is that reproducible defect datasets can be sustained by environment pinning alone. Bugs4Q-Robust is explicit evidence against that assumption in the Qiskit setting: only 4.6% of failures were fixable through dependency alignment without source modification (Ohto et al., 25 Jun 2026). Another misconception is that the “robust” designation refers to adversarial robustness, mutation-based robustness, or robustness-by-bug-category. Here it refers specifically to restored defect reproducibility under a maintained Qiskit target.

In a broader benchmarking context, the proposal of QBugs emphasizes real bugs mined from history, fail–pass pairs, diverse languages, and explicit experimental infrastructure for repeatable studies (Campos et al., 2021). A plausible implication is that Bugs4Q-Robust operationalizes one concrete aspect of that agenda—continuous maintenance of executable fail–pass artifacts in a rapidly evolving quantum software stack. Its most direct methodological lesson is that quantum defect benchmarks require sustained source-level migration, oracle maintenance, and environment documentation if they are to remain comparable across research cycles.

Bugs4Q-Robust therefore occupies a specific niche: it is neither a replacement for Bugs4Q’s historical bug curation nor a general multi-framework benchmark, but a maintained Qiskit-targeted release that preserves the empirical utility of Bugs4Q under Qiskit v2.3.1. In the rapidly evolving quantum software ecosystem, that maintenance function is itself part of benchmark design.