---
title: 'Bugs4Q-Robust: Restoring Quantum Bug Reproducibility'
url: https://www.emergentmind.com/topics/bugs4q-robust
type: topic
---

# Bugs4Q-Robust: Restoring Quantum Bug Reproducibility

Bugs4Q-Robust is a curated, patched release of the Bugs4Q quantum software defect dataset designed to restore and stabilize bug reproducibility under modern Qiskit versions. It was created after a longitudinal replication study showed that the reproducibility of the original Bugs4Q, collected in May 2022, degraded sharply as the Qiskit ecosystem evolved: under Qiskit v2.3.1, reproducibility fell to 16.2% across the study’s criteria, and most failures were dependency-related. Bugs4Q-Robust addresses this decay through minimal source-level migrations and oracle adjustments intended to preserve the original bug semantics while enabling execution on Qiskit v2.3.1, raising reproducibility to 78.4% for Existence, 64.9% for Type Match, and 59.5% for Cause Match [2606.27124].

## 1. Origin in the Bugs4Q benchmark lineage

Bugs4Q-Robust is derived from Bugs4Q, a benchmark of real bugs for quantum programs in the Qiskit ecosystem. The original Bugs4Q paper describes a publicly available benchmark of thirty-six real, manually validated Qiskit bugs from four popular Qiskit elements—Terra, Aer, Ignis, and Aqua—supplemented with test cases for reproducing buggy behaviors, along with interfaces for accessing buggy and fixed versions and executing the corresponding tests [2108.09744].

Subsequent studies used different Bugs4Q subsets rather than a single invariant artifact set. The reproducibility study underlying Bugs4Q-Robust analyzed 37 artifacts after excluding five artifacts lacking tests [2606.27124]. The ChatGPT-based repair study retested 38 reproducible buggy Qiskit programs from Bugs4Q [2401.14913], and QChecker evaluated 42 real-world buggy Qiskit programs written in Qiskit [2304.04387]. These counts are reported by the respective studies and reflect evaluation-specific selections rather than a unified “robust” split.

A central clarification is that Bugs4Q-Robust is not a robustness-focused bug-type subset in the sense of reclassifying bugs by semantic difficulty, nor is it a new benchmark family independent of Bugs4Q. Earlier Bugs4Q-based papers explicitly do not mention any “Bugs4Q-Robust” variant [2401.14913]. Its defining purpose is maintenance of reproducibility under contemporary Qiskit, not redefinition of the underlying defects.

## 2. Reproducibility decay and the need for a patched release

The motivating study conducted 77,700 quantum program executions of 37 Bugs4Q artifacts across 21 Qiskit core-library versions spanning the 0.x, 1.x, and 2.x series. Execution used Docker-isolated environments, with Python 3.9.25 for 0.x and 1.x and Python 3.10.20 for 2.x, and compared two dependency configurations: Core-only and Pinned-stack [2606.27124].

The study formalized reproducibility using three increasingly strict criteria. **Existence** requires that the buggy version fails and the fixed version succeeds. **Type Match** additionally requires that the failure type match the source. **Cause Match** further requires that runtime evidence—such as traceback, message, or output distribution—match the cause described in the original source. Reproducibility rate and related quantities were defined as
$$
R = 100 \times \frac{n_{\text{reproduced}}}{n_{\text{total}}},
\qquad
F = 100 - R,
\qquad
\Delta R = R_{\text{Robust}} - R_{\text{Original}}.
$$

On Qiskit v0.20.1, the original Bugs4Q achieved 75.7% for Existence, 64.9% for Type Match, and 62.2% for Cause Match. Under later major upgrades, these rates dropped sharply. In the 1.x series, Existence peaked at 21.6% and Cause Match at 18.9%. In the 2.x series, Existence, Type Match, and Cause Match converged to 16.2% on v2.3.1. Overall, 31 of 37 artifacts, or 83.8%, experienced at least one reproduction failure over the three-year window, and once an artifact became non-reproducible, it never returned to a reproducible state in subsequent snapshots [2606.27124].

The paper identifies a point of contrast with prior observations on classical defect datasets. Only 4.6% of failures were fixable by dependency pinning alone; most required source-code modifications. This is a defining rationale for Bugs4Q-Robust: version pinning was insufficient to recover the benchmark.

## 3. Failure modes induced by Qiskit ecosystem evolution

Manual analysis classified 543 snapshot–artifact failures into four main classes and seven leaf categories. Source-related failures accounted for 82.0%, library-related failures for 11.6%, quantum-specific failures for 5.1%, and original test issues for 1.3%. The paper further states that dependency-related failures dominated overall, with
$$
f_{\text{dep}} = 93.6\%.
$$
This aggregates Source-related 82.0% and Library-related 11.6% [2606.27124].

Within source-related failures, **Import incompatibility** represented 48.1% and **API invocation incompatibility** 33.9%. The paper gives a representative import migration:
```python
# Before
from qiskit.extensions import HGate, CXGate

# After
from qiskit.circuit.library import HGate, CXGate
```
It also gives a representative API migration from `execute` to explicit transpilation plus backend execution:
```python
# Before
job = execute(qc, backend=Aer.get_backend('qasm_simulator'), shots=1024)

# After
backend = Aer.get_backend('qasm_simulator')
new_circuit = transpile(qc, backend)
job = backend.run(new_circuit, shots=1024)
```

Library-related failures were subdivided into **Library–library incompatibility** at 4.6%, **Library-internal behavior change** at 6.1%, and **Output format change** at 0.9%. The paper also identifies **Quantum-specific failures** caused by probabilistic sampling variance, mitigated by tuning shots and statistical thresholds such as the significance level $\alpha$. Finally, a small fraction of cases corresponded to original test issues that could not be repaired without redefining semantics [2606.27124].

At a higher level, the relevant Qiskit transitions were the v1.0 reorganization and the v2.0 consolidation around primitives such as `Sampler` and `Estimator`, along with continuing import path migrations, removal of `execute`, and changed backend invocation patterns. These changes often caused artifacts to fail before reaching the original buggy lines, for example via `ImportError`, or to fail with a different type or cause than the one originally recorded.

## 4. Curation principles and construction of Bugs4Q-Robust

Bugs4Q-Robust was curated by starting from failures under the strictest criterion, Cause Match, and determining a concrete root cause for each artifact. The patch policy was to apply only minimal changes necessary to address the identified root cause, preserve the original user-level bug semantics and test intent, and prefer documented migration paths in official Qiskit guides. The paper summarizes this policy with the principle “patch the interface break, not the bug” [2606.27124].

The curated patches fall into four main classes. **Import path updates** migrate legacy modules such as `qiskit.extensions` to `qiskit.circuit.library` and `qiskit.providers.aer` to `qiskit_aer`. **API invocation changes** replace removed or renamed functionality, including `execute` with `transpile + backend.run`, and, where applicable, opflow-based APIs with primitives such as `Sampler` and `Estimator`. **Circuit construction and oracle adjustments** address signature changes, output-format changes, and probabilistic assertions by tuning shots and significance levels. **Dependency adjustments** align surrounding packages to mutually compatible versions with Qiskit v2.3.1 and remove deprecated packages that conflict with modern stacks [2606.27124].

All patches were authored manually using Qiskit 1.0 and 2.0 migration guides and release notes, and the changes and rationale were recorded with runtime logs. Bugs4Q-Robust includes all 37 artifacts used in the study, and each artifact contains the buggy program, fixed program, standalone test code, and metadata linking to the original reported source.

The replication package is public at `https://github.com/kusumotolab/quantum-defect-reproducibility`. Its root contains dependency specifications targeting Qiskit v2.3.1, runner scripts, a Dockerfile, a Makefile, and runtime logs. The target environment is Python 3.10.20 with `qiskit==2.3.1`, and the paper recommends `QasmSimulator` from `qiskit_aer` for deterministic availability [2606.27124].

## 5. Evaluation outcomes and residual irreproducibility

The principal quantitative effect of Bugs4Q-Robust is restoration of reproducibility on Qiskit v2.3.1. The study reports the following results [2606.27124]:

| Criterion | Original Bugs4Q | Bugs4Q-Robust |
|---|---:|---:|
| Existence | 16.2% | 78.4% |
| Type Match | 16.2% | 64.9% |
| Cause Match | 16.2% | 59.5% |

The corresponding improvements are reported as $\Delta R = 62.2$ percentage points for Existence, $\Delta R = 48.7$ percentage points for Type Match, and $\Delta R = 43.3$ percentage points for Cause Match. These values indicate that most recoverable failures were due to interface drift rather than disappearance of the underlying defects.

The remaining shortfall is also characterized. For the 15 artifacts that still do not satisfy Cause Match, the dominant cause is **Library-internal behavior change**, accounting for approximately 66.7% of the remaining artifacts. The paper notes cases in which new APIs absorb past misuse—for example, `Estimator` implicitly diagonalizes, eliminating the original failure mode. A few cases remain blocked by import incompatibility where removed modules lack direct replacements, and one artifact has unresolved original test issues [2606.27124].

The validation protocol remains test- and runtime-evidence-based rather than semantics-complete. Repeated runs, specifically 30 trials per buggy and fixed artifact per snapshot, were used to control flakiness, especially for probabilistic oracles. This suggests that Bugs4Q-Robust improves operational reproducibility under a fixed software stack, but does not claim a formal semantic-equivalence guarantee for the migrated artifacts.

## 6. Role in the Bugs4Q research ecosystem

Bugs4Q-Robust changes the practical status of Bugs4Q within quantum software engineering by turning an increasingly decayed benchmark into a usable contemporary baseline. This matters because multiple lines of research already depend on Bugs4Q. The ChatGPT repair study evaluated semi-automated repair on 38 reproducible Qiskit bugs, using Bugs4Q test suites as the sole correctness oracle and reporting that richer descriptions and iterative feedback substantially improve repair outcomes [2401.14913]. QChecker evaluated static bug detection on the standard Bugs4Q and reported aggregate precision, recall, and F1 on Qiskit programs, again without any Bugs4Q-Robust variant [2304.04387]. For such studies, benchmark decay directly threatens comparability across time.

A recurrent misconception is that reproducible defect datasets can be sustained by environment pinning alone. Bugs4Q-Robust is explicit evidence against that assumption in the Qiskit setting: only 4.6% of failures were fixable through dependency alignment without source modification [2606.27124]. Another misconception is that the “robust” designation refers to adversarial robustness, mutation-based robustness, or robustness-by-bug-category. Here it refers specifically to restored defect reproducibility under a maintained Qiskit target.

In a broader benchmarking context, the proposal of QBugs emphasizes real bugs mined from history, fail–pass pairs, diverse languages, and explicit experimental infrastructure for repeatable studies [2103.16968]. A plausible implication is that Bugs4Q-Robust operationalizes one concrete aspect of that agenda—continuous maintenance of executable fail–pass artifacts in a rapidly evolving quantum software stack. Its most direct methodological lesson is that quantum defect benchmarks require sustained source-level migration, oracle maintenance, and environment documentation if they are to remain comparable across research cycles.

Bugs4Q-Robust therefore occupies a specific niche: it is neither a replacement for Bugs4Q’s historical bug curation nor a general multi-framework benchmark, but a maintained Qiskit-targeted release that preserves the empirical utility of Bugs4Q under Qiskit v2.3.1. In the rapidly evolving quantum software ecosystem, that maintenance function is itself part of benchmark design.

Source: https://www.emergentmind.com/topics/bugs4q-robust