QASMBench: A Low-level QASM Benchmark Suite for NISQ Evaluation and Simulation

Abstract: The rapid development of quantum computing (QC) in the NISQ era urgently demands a low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers and assemblers, and the capability of quantum system simulators in a classical computer. In this work, we fill this gap by proposing a low-level, easy-to-use benchmark suite called QASMBench based on the OpenQASM assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, etc., trading-off between generality and usability. To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. Applications in QASMBench can be launched and verified on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum. For evaluation, we measure the execution fidelity of a subset of QASMBench applications on 12 IBM-Q machines through density matrix state tomography, which comprises 25K circuit evaluations. We also compare the fidelity of executions among the IBM-Q machines, the IonQ QPU and the Rigetti Aspen M-1 system. QASMBench is released at: http://github.com/pnnl/QASMBench.

• The paper introduces QASMBench, a comprehensive low-level QASM benchmark suite that bridges theoretical quantum algorithms with practical NISQ evaluations.
• It categorizes benchmarks by qubit count and spans applications in quantum chemistry, optimization, machine learning, and cryptography to test hardware capabilities.
• The study develops novel circuit metrics—including width, gate density, and entanglement variance—to assess performance variations on NISQ devices like IBM-Q.

QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation

The paper "QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation" addresses the imperative need for a dedicated benchmark suite intended for evaluating the nascent, noisy-intermediate-scale-quantum (NISQ) computing devices. In the current landscape of quantum computing, characterized by rapid yet somewhat premature developments, the design of versatile and effective benchmarking tools has bourgeoned as a focal point for researchers aiming to bridge the gap between theoretical quantum algorithms and physical quantum devices.

This work introduces QASMBench, a diverse collection of quantum algorithms expressed in OpenQASM, an assembly language providing an intermediate representation crucial for execution on various quantum platforms, most notably IBM-Q, Rigetti, IonQ, and Quantinuum. The benchmark includes a wide range of applications spanning domains such as quantum chemistry, optimization, arithmetic, machine learning, and cryptography, designed to strike a balance between generality and domain-specific functionality.

QASMBench encompasses an array of quantum routines with varying scales, categorized by the number of qubits involved. Benchmarks range from small (2-5 qubits) to medium (6-15 qubits), and large-scale implementations (more than 15 qubits), facilitating a heterogeneous evaluation environment to test the capabilities and limitations of NISQ devices. The benchmarks were purposely selected to encompass both standard and emergent quantum operations, such as the well-known quantum Fourier transform and variational quantum eigensolver.

The paper outlines several novel circuit metrics developed to provide deeper insight into the characteristics and performance of quantum circuits on NISQ hardware. These metrics include:

1. Circuit Width and Depth: These provide foundational dimensions of quantum circuits, specifying the number of qubits and the series of computational layers, respectively.
2. Gate Density: Refers to the fraction of gates executed relative to potential execution slots, which impacts the hardware occupancy rate.
3. Retention Lifespan: Assesses the coherence time requirements by evaluating the qubit's role longevity, given the constraints of T1 and T2 decay times.
4. Measurement Density: A quantitative measure of the circuit's dependence on measurement operations relative to its width and depth, which is crucial in ancillary-based quantum computations such as SWAP test algorithms.
5. Entanglement Variance: A metric elucidating the distribution of entanglements across qubits, thus shedding light on potential mapping efficiencies when deploying circuits on physically topologized quantum devices.

The authors systematically evaluate a subset of QASMBench applications on multiple IBM-Q machines, extracting fidelity metrics based on density matrix tomography. This empirical analysis unveils the variabilities and limitations inherent in current NISQ machines, notably reflecting on the fidelity decays associated with deeper circuits. The research verifies general trends, such as the fidelity correlation with circuit depth and coherence, with IBM-Q devices displaying notable variance in fidelity that is only partially mitigated by optimizations like effective transpilation and logical-physical qubit mapping.

The QASMBench suite is made publicly accessible to encourage collaborative development and benchmarking in the quantum computing community. By offering a comprehensive selection of benchmarks alongside evaluative metrics, the work contributes to the foundational tools needed to drive advancements in quantum algorithm performance and understanding NISQ limitations.

Moving forward, the refinement of benchmarks like QASMBench is pivotal for developing increasingly sophisticated quantum evaluation techniques, coupling algorithmic advances with ongoing hardware improvements. The concerted efforts in integrating robust benchmarks with detailed metrics provide the groundwork for not only improving current quantum platforms but also guiding the design of future quantum systems beyond the NISQ era.