Bugs4Q: Quantum Software Bug Benchmark
- Bugs4Q is an empirically grounded quantum software bug benchmark, providing a taxonomy of reproducible defects and artifacts across various QC frameworks.
- It curates real-world bugs from Qiskit and related platforms to support studies in debugging, reproducibility, and automated repair using large language models.
- The benchmark employs rigorous validation protocols and performance metrics, highlighting challenges in dependency management and API transitions in evolving quantum systems.
Bugs4Q denotes a line of empirically grounded quantum-software bug corpora and benchmarks centered on reproducible defects, their manifestations, and the evaluation of debugging, testing, reproducibility, and repair methods for quantum programs. In the literature represented here, the term covers both a taxonomy-oriented collection of real-world quantum-program bugs distilled from Huang and Martonosi’s study of canonical quantum algorithms across multiple QC frameworks and simulators, and a later public benchmark of real Qiskit bugs with buggy/fixed versions and executable tests. Subsequent work used Bugs4Q to study longitudinal reproducibility under Qiskit evolution and to evaluate automated program repair with LLMs (Huang et al., 2018, Zhao et al., 2021, Ohto et al., 25 Jun 2026, Guo et al., 2024).
1. Origins and scope
The earliest Bugs4Q-style corpus in this literature is the curated collection extracted from Huang and Martonosi’s "QDB: From Quantum Algorithms Towards Correct Quantum Programs," where implementations were written for three canonical domains: quantum chemistry via H ground-state energy estimation using iterative phase estimation, integer factorization via Shor’s order-finding for $15$, and unstructured search via Grover’s amplitude amplification in a small Galois-field example. These implementations were coded and debugged in at least two QC frameworks, including Scaffold OpenQASM QX simulator, Qiskit, ProjectQ, and Q#, with hardware backends such as IBM Q Experience exercised when available (Huang et al., 2018).
A later and more standardized use of the name appears in "Bugs4Q: A Benchmark of Real Bugs for Quantum Programs," which defines Bugs4Q as a benchmark of thirty-six real, manually validated Qiskit bugs from Terra, Aer, Ignis, and Aqua, supplemented with test cases for reproducing buggy behaviors and interfaces for accessing buggy and fixed versions of the programs (Zhao et al., 2021).
Subsequent studies used closely related artifact sets rather than a single immutable release. The 2026 reproducibility study analyzes $37$ Bugs4Q artifacts out of $42$ in the original release, excluding $5$ due to missing test code, across $21$ Qiskit snapshots and executions (Ohto et al., 25 Jun 2026). The 2024 repair study describes Bugs4Q as $38$ reproducible buggy Qiskit programs drawn from real-world GitHub histories (Guo et al., 2024). This suggests that the label “Bugs4Q” is stable, while the exact artifact count depends on the curation and inclusion protocol used by a given study.
| Corpus or use | Scope | Salient count |
|---|---|---|
| QDB-derived Bugs4Q | Real-world quantum-program bugs organized by bug-pattern across multiple languages and simulators | 78 distinct bug instances |
| Public Qiskit Bugs4Q benchmark | Real, manually validated Qiskit bugs with buggy/fixed versions and tests | 36 bugs |
| Reproducibility case study | Bugs4Q artifacts replayed across Qiskit snapshots | 37 artifacts, 77,700 executions |
| APR evaluation | Reproducible buggy Qiskit programs used for LLM-based repair | 38 programs |
2. Benchmark construction and artifact model
The public Qiskit benchmark was constructed from GitHub issues labeled “bug” in four Qiskit elements: Terra, Aer, Ignis, and Aqua. The initial set contained $15$0 quantum-related issues, both closed and open. Screening then imposed four requirements: the issue had to be a source-code bug rather than a test or build-system issue, had to involve a quantum effect such as gate operation, measurement, or simulation noise, had to be reproducible by at least one failing test case or script, and had to be isolated so that the fix touched only the minimal set of source files needed to repair the bug (Zhao et al., 2021).
Manual validation reduced the corpus further. Of the $15$1 candidate issues, $15$2 had original test cases; these were executed in a controlled Qiskit environment. Non-reproducibility arose from factors including multi-file fixes, missing dependencies, and non-matching error messages. The final curated set contained $15$3 bugs. In the initial v1.0 distribution, Terra contributed the largest share at approximately $15$4, followed by Aer at approximately $15$5, Aqua at approximately $15$6, and Ignis at approximately $15$7 (Zhao et al., 2021).
Each benchmark entry is structured as a fail-pass pair comprising buggy and fixed source files plus a reproducing test. The benchmark exposes command-line utilities and a Python interface. The script checkout_bug.py --id <BUG_ID> --version {buggy,fixed} replaces only the affected source files in a clean Qiskit clone, while run_test.py --id <BUG_ID> invokes the corresponding tests/test_<BUG_ID>.py. Users may also import the Bugs4Q module and call Bugs4Q.checkout(bug_id, 'buggy') (Zhao et al., 2021).
The test cases encode the oracle directly. A typical test imports the specific Qiskit version, defines registers and builds a circuit that exercises the faulty API or gate, transpiles or simulates it, and then asserts on console output or exception behavior. In the Aqua example supplied in the benchmark description, the test distinguishes buggy and fixed behavior through the predicates “$15$8” before the fix and “$15$9” after the fix (Zhao et al., 2021).
3. Defect taxonomies and classification schemes
The QDB-derived Bugs4Q collection is organized by a formal root-cause taxonomy that partitions bugs into Input, Operation, and Output classes: 0 with seven subtypes: 1 Representative failures include wrong Hamiltonian coefficients in H2 chemistry encodings, endian confusion during basis-state preparation, incorrect decompositions of controlled-3, off-by-one or inverted loop order in QFT-style routines, missing inverse invocations for ancilla uncomputation, symmetry violations in Compute/Uncompute regions, and recursive multi-controlled-NOT constructions that leave ancilla entangled (Huang et al., 2018).
The later Qiskit benchmark adopts a different classification vocabulary. Its 4 faults are grouped into four primary categories: Output Wrong, Noise Simulation Errors, Lost Information, and Exception Throwing. Examples include Aqua’s CircuitSampler coefficient mismatch under Output Wrong, mismatched base gates in Aer’s noise model under Noise Simulation Errors, missing calibration data when composing QuantumCircuit objects in Terra under Lost Information, and API misuse or type errors leading to QiskitError under Exception Throwing (Zhao et al., 2021).
The 2026 reproducibility study further normalizes artifacts by observed test outcome: Wrong Output (5 artifacts), Throw Exception (6 artifacts), and Simulation Failure (7 artifacts), where Simulation Failure denotes a false simulator success flag. Four artifacts with non-deterministic measurement outputs are handled via statistical assertions (Ohto et al., 25 Jun 2026). Taken together, these schemes suggest a distinction between root-cause-oriented taxonomies, as in QDB, and benchmark-oriented or oracle-oriented taxonomies, as in the later Qiskit releases.
4. Validation protocols and reproducibility methodology
In the QDB-derived corpus, bug admission required cross-validation across languages and backends. Every buggy implementation was re-written or ported into at least one other QC framework; for example, a faulty Scaffold 8 QX version could be re-implemented in Qiskit and ProjectQ. Validation then compared measurement distributions, checked intermediate state-vectors for small qubit counts, and used known theoretical results such as the exact period in Shor or the exact ground-state energy in H9. Only bugs that consistently manifested across two different backends or languages were admitted to the final corpus (Huang et al., 2018).
The public benchmark provides explicit quantitative measures for evaluation. During curation it tracked a reproducibility rate,
0
and defines a generic bug-detection coverage metric for testing tools as
1
These formulas standardize comparison across debugging and testing methods (Zhao et al., 2021).
The 2026 case study makes the reproducibility notion more stringent by introducing three criteria adapted from Zhu et al.: Existence, where the buggy version fails and the fixed version succeeds under the same test oracle; Type Match, where the observed failure type matches the original report; and Cause Match, where the failure message or incorrect output matches the original root cause. The study isolates environments using one Docker image per Qiskit snapshot, uses Python 2 for the 3 series and Python 4 for the 5 series, compares a Core-only dependency update against a Pinned-stack strategy, and executes each buggy/fixed pair 6 times to account for quantum non-determinism. An artifact is deemed reproducible under criterion 7 at snapshot 8 only if all 9 runs agree, yielding
$37$0
5. Empirical characteristics and longitudinal findings
Within the QDB-derived taxonomy corpus, Bugs4Q contains seventy-eight distinct bug instances. The subtype distribution is $37$1: $37$2 ($37$3); $37$4: $37$5 ($37$6); $37$7: $37$8 ($37$9); $42$0: $42$1 ($42$2); $42$3: $42$4 ($42$5); $42$6: $42$7 ($42$8); and $42$9: $5$0 ($5$1). Notable severity characteristics are also reported: $5$2 and $5$3 are $5$4 catastrophic, producing no meaningful convergence, whereas $5$5, $5$6, and $5$7 exhibit $5$8–$5$9 partial correctness, meaning some outcomes remain plausible despite the defect (Huang et al., 2018).
The reproducibility case study shows that Bugs4Q degrades substantially as Qiskit evolves. Under the Pinned-stack configuration and the Cause Match criterion, reproducibility drops from $21$0 on Qiskit $21$1 to $21$2 on $21$3. Across the $21$4 series, the rate declines gradually from $21$5 to $21$6; at the major transition to $21$7, it falls to $21$8; and it remains approximately $21$9–0 across later 1 minor updates (Ohto et al., 25 Jun 2026).
Manual inspection of failure causes shows that 2 of failures are dependency-related. Across 3 failing 4 pairs under Cause Match and Pinned-stack, 5 are source-related reproduction failures, subdivided into import incompatibility at 6 and API-invocation incompatibility at 7. Library-related failures account for 8, with library–library incompatibility at 9, library-internal behavior change at $38$0, and output-format change at $38$1. Quantum-specific reproduction failures contribute $38$2, arising from probabilistic sampling outliers in statistical oracles, and original test issues contribute $38$3 (Ohto et al., 25 Jun 2026).
To restore reproducibility, the 2026 study constructs Bugs4Q-Robust for Qiskit $38$4 and an updated surrounding stack by applying import and API migrations, dependency-pin adjustments, statistical-oracle tuning, and output-format fixes, while leaving unpatchable library-internal changes and invalid tests unchanged. Representative migrations include replacing from qiskit.providers import aer with import qiskit_aer, replacing execute with transpile plus backend.run, and increasing shots from $38$5 to $38$6 while decreasing $38$7 from $38$8 to $38$9 for statistical tests (Ohto et al., 25 Jun 2026).
| Dataset | Existence | Type Match | Cause Match |
|---|---|---|---|
| Bugs4Q (orig.) | 16.2% | 16.2% | 16.2% |
| Bugs4Q-Robust | 78.4% | 64.9% | 59.5% |
These results indicate that Bugs4Q is not merely a static benchmark of quantum defects; it is also a case study in benchmark aging under rapid SDK refactoring. A plausible implication is that longitudinal validity in quantum-software evaluation depends as much on maintenance infrastructure as on the initial curation process.
6. Research uses, defenses, and limitations
Bugs4Q has been used both as a descriptive resource for understanding quantum-programming failure modes and as an experimental substrate for tools. The QDB study extracts six defense strategies directly from its bug taxonomy: QC-specific library and unit-testing support for $15$00–$15$01; first-class quantum numeric data types for $15$02; Reverse/Compute–Uncompute language constructs for $15$03 and $15$04; built-in controlled-operation abstractions for $15$05; assertion macros for preconditions, algorithm progress, and postconditions; and cross-language cross-validation harnesses over multiple SDKs. The same study states that by adopting a QC language or framework that provides Compute/Uncompute and built-in MCX gates, and by writing small unit tests around each basic subroutine, one can eliminate approximately $15$06 of the common bugs in the corpus, while assertions on numeric ranges guard against the remaining classes (Huang et al., 2018).
The benchmark has also been used for automated program repair. "On Repairing Quantum Programs Using ChatGPT" evaluates four prompt templates—Chat, Chat_D, Chat_2D, and Chat_3D—under a multi-turn protocol with five independent threads per bug and up to five iterations per thread. Correctness is defined purely by passing all supplied Bugs4Q unit tests. On $15$07 reproducible buggy Qiskit programs, ChatGPT with the best prompt, Chat_2D or Chat_3D, repaired $15$08 of $15$09 bugs, corresponding to approximately $15$10 TOP-5 success. The paper also reports $15$11 TOP-5 success for simple quantum bugs, $15$12 for quantum-algorithm bugs, and $15$13 for classical-style bugs, together with an illustrative teleportation repair involving the insertion of the final classically controlled gates before measuring qubit $15$14 (Guo et al., 2024).
Limitations recur across the literature. The 2021 benchmark is restricted to Qiskit $15$15, four elements, and excludes Ignis C++ portions; it contains only $15$16 bugs and notes that probabilistic tests require seed control, with some nondeterministic cases not yet fully supported (Zhao et al., 2021). The reproducibility study shows that many failures cannot be repaired by dependency adjustments alone and instead require source-code migration, especially for import paths and API invocations (Ohto et al., 25 Jun 2026). The APR study emphasizes that success depends heavily on detailed bug descriptions, that performance is much weaker on novel quantum-algorithm bugs, and that test-based correctness does not provide formal guarantees such as quantum-state fidelity or analytic equivalence (Guo et al., 2024).
Future work identified in these studies includes automated continuous integration for new GitHub issues, enhanced classification schemas, support for other quantum frameworks such as Cirq and Q#, benchmarks on real QPU backends, AST-based codemods for legacy Qiskit migrations, “dataset healers” integrated into CI, and stronger correctness criteria for repair, including quantum-state fidelity and analytic equivalence checks (Zhao et al., 2021, Ohto et al., 25 Jun 2026, Guo et al., 2024). In that sense, Bugs4Q functions simultaneously as a benchmark suite, a defect taxonomy, and an evolving target for research on the engineering of quantum software.