Quality, Speed, and Scale: three key attributes to measure the performance of near-term quantum computers

Abstract: Defining the right metrics to properly represent the performance of a quantum computer is critical to both users and developers of a computing system. In this white paper, we identify three key attributes for quantum computing performance: quality, speed, and scale. Quality and scale are measured by quantum volume and number of qubits, respectively. We propose a speed benchmark, using an update to the quantum volume experiments that allows the measurement of Circuit Layer Operations Per Second (CLOPS) and identify how both classical and quantum components play a role in improving performance. We prescribe a procedure for measuring CLOPS and use it to characterize the performance of some IBM Quantum systems.

• The paper introduces a benchmarking framework that uses quality (quantum volume) for circuit accuracy alongside speed (CLOPS) and scale (qubit count).
• The methodology employs empirical analyses on IBM Quantum systems, revealing that circuit delays and classical-quantum interfacing significantly impact performance.
• The study’s insights guide future optimizations by highlighting areas such as reducing latency and enhancing compilation efficiency in quantum hardware.

An Analysis of Performance Metrics for Near-Term Quantum Computers

The paper "Scale, Quality, and Speed: three key attributes to measure the performance of near-term quantum computers," presents a structured approach to evaluate quantum computing systems. The authors, affiliated with IBM Quantum, propose three fundamental metrics: quality, speed, and scale, each playing a crucial role in assessing the overall performance of quantum computers. This analysis of performance metrics aims to establish benchmarks that capture the comprehensive capabilities of quantum computing systems.

Key Metrics for Quantum Performance

1. Scale: The number of qubits is the defining metric for scale, reflecting the system’s capacity to handle complex problem sets. With a greater number of qubits, a quantum system can solve larger computational problems, encoding more information. The scalability of quantum systems is affected by technological advancements in qubit integration and coherence maintenance. Notably, superconducting qubits show significant progress in this area.
2. Quality: Quantum volume (QV) is employed to indicate the ability to execute quantum circuits accurately. It incorporates factors like gate fidelity, connectivity, coherence, and compilation efficiency. The QV metric is established by determining the largest circuit, width and depth, that generates accurate outputs. This holistic approach captures system-wide synergies, requiring multiple system parameters to be optimized simultaneously.
3. Speed: The Circuit Layer Operations Per Second (CLOPS) metric is introduced to measure the execution speed of quantum circuits comprehensively. It takes into account not only gate execution time but also delays in setup, initialization, and data transfer between quantum and classical systems. The CLOPS benchmark is particularly insightful as it emulates usage patterns seen in real-world quantum applications like variational algorithms.

Experimental Insights and Implications

The study provides empirical measurements of these metrics on several IBM Quantum systems with varying qubit counts but a constant quantum volume. Notably, the differences in CLOPS values across systems revealed the dominance of factors such as circuit delays and data transfer overheads over mere gate execution time. This indicates potential areas for future optimization, particularly in minimizing interaction latency between quantum and classical components.

The analysis emphasizes the importance of using holistic benchmarks over simplistic metrics, which may overlook crucial real-world performance bottlenecks. By considering all system components holistically, CLOPS serves as a substantial indicator of a system’s practical utility in handling dynamic quantum-classical workloads.

Future Directions

Future developments in quantum computing performance measurement should focus on reducing elements like circuit delay, enhancing runtime compilation efficiency, and optimizing data transfer protocols. System architectures featuring advanced technologies for rapid parameter updates, efficient qubit repurposing, and improved classical-quantum interfacing are likely to enhance CLOPS significantly.

Overall, the methodologies presented in this work provide a robust framework for evaluating and comparing quantum computing systems’ performance. By addressing both theoretical implications and practical application relevance, these metrics support the ongoing evolution toward achieving quantum advantage over classical computing for specific computational tasks. As the field progresses, adaptations to these metrics will be necessary to capture performance advances accurately and efficiently reflect new quantum computing capabilities.