CLOPS: Benchmarking System Speed at Utility Scale

Abstract: As quantum processors scale to hundreds of qubits, execution speed is a critical performance dimension alongside scale and quality. While substantial progress has been made in benchmarking circuit fidelity, existing speed metrics often fail to reflect the sustained, end-to-end throughput experienced by users running utility-scale workloads. This shortfall is especially pronounced for layered, parameterized circuits executed repeatedly within classical-quantum workflows, such as variational algorithms and error-mitigated simulations. In this work, we formalize CLOPS_h (Circuit Layer Operations Per Second) as a holistic speed benchmark defined over layered, hardware-aware circuits. CLOPS_h measures the sustained rate at which the system executes physical layers, parallel slices of qubit-disjoint two-qubit gates separated by synchronization barriers. Because each such layer is one time slice of an N-qubit circuit, this rate maps directly to the execution rate of layered $N$-qubit circuits, connecting CLOPS_h to published device capability claims, and to the device-level throughput ceiling we formalize as Max Circuits Per Second (MCPS). CLOPS_h is obtained under layer-fidelity operating conditions, binding the speed measurement to an independently verified quality envelope, and it shares its layer decomposition with scalable layer-fidelity (LF) quality benchmarks, enabling coherent interpretation of speed and quality without conflating the two.

• The paper introduces CLOPS, a system-level benchmark quantifying physical layer execution, proving its validity through multi-year analysis across IBM Quantum hardware
• CLOPS reveals striking improvements (~330×) in practical speed primarily driven by parametric compilation and stack optimizations.
• CLOPS is applicable for realistically quantizing systems with validated physical layer execution.

CLOPS is a formalized system-level speed benchmark for utility-scale quantum computers, defined as the sustained rate at which a system executes physical layers—parallel, qubit-disjoint slices of two-qubit gates terminated by global synchronization barriers—under operating conditions anchored to an independently measured quality benchmark (2608.18044). The paper positions CLOPS within a performance hierarchy of scale, quality, and speed, argues that lower-level speed metrics cannot predict system throughput, and validates the metric through multi-year paired measurements on IBM Quantum hardware.

Motivation: the gap between device and system metrics

The central argument is that quantum computing benchmarks, like classical ones, form an abstraction hierarchy—qubit, device, system, application—and that cross-level inference fails. Device-level speed indicators such as MCPS (Max Circuits Per Second), defined from rep delay, state preparation/measurement time, and single-qubit gate duration, are valid ceilings but are opaque to the software stack, parametric compilation, data movement, and control overheads that dominate real iterative workloads. The paper draws the analogy to LINPACK displacing peak FLOPS in classical HPC. Existing application-oriented efforts (QED-C suite, Metriq) are characterized as falling short: they operate below utility scale and exclude end-to-end classical overhead.

Formal definition

A physical layer comprises a matching of two-qubit gates on the coupling graph, fully twirled single-qubit gates on all $N$ qubits (including idlers), and a synchronization barrier forbidding gate reordering across boundaries. The circuit template $C(\boldsymbol{\theta}) = L_1 \cdots L_D$ cycles the fixed sublayer decomposition (for a 1D chain, two sublayers per EPLG layer). The canonical operating point is $N = D = S = 100$; any deviation must be reported as $\mathrm{CLOPS}(N, D, S)$.

The measurement protocol is deliberately implementation-agnostic: after pipeline saturation (marked by the first result batch, which is excluded), throughput is computed over steady-state batches as

$$\mathrm{CLOPS} = \frac{K_{\mathrm{ss}} \cdot D \cdot S}{t_n - t_1}$$

with all per-execution costs included—parameter binding, runtime compilation where recompilation is forced, payload construction, control loading, execution, and result retrieval. One-time template transpilation amortized outside the timing window is excluded by design; on platforms requiring full recompilation per parameter set, that cost is naturally included. This cleanly separates CLOPS from software-stack benchmarks such as Benchpress (Nation et al., 2024).

The most consequential constraint is the layer-fidelity operating conditions (LFOC) requirement: gates, durations, scheduling, error suppression, and calibration cycles must match those used for the companion LF/EPLG measurement. Without this, an unconstrained speed benchmark can be trivially inflated by degrading fidelity. The paper notes plainly that CLOPS does not verify these conditions internally; compliance is external and declaratory—a genuine enforcement limitation.

The structural alignment with layer fidelity (McKay et al., 2023) is the key conceptual contribution: the same physical sublayers serve as the CLOPS counting unit and as the constituents of an EPLG quality object, enabling joint interpretation ("LF measures how faithfully a layer executes; CLOPS how fast") without conflating the two dimensions.

Case study results

Paired measurements spanning roughly 3.5 years, using the Kim et al. Trotterized dynamics experiment [kim2023] as a fixed application probe, yield strong numbers:

Date	CLOPS	Gain	App. runtime gain
June 2022	950	baseline	baseline
Mar. 2025	250,000	~250×	~75×
Nov. 2025	330,000	~330×	~110×

The decomposition is informative. From June 2022 to March 2025, hardware accounted for only ~4× (rep delay 1,000→250 µs, two-qubit gate time 73→12 µs); the remaining ~60× came predominantly from parametric compilation, which eliminated per-parameter-set recompilation and enabled near-continuous QPU utilization. From March to November 2025, with hardware unchanged, CLOPS rose from 68% to 90% of the ~365,000-layer/s device ceiling purely through stack optimization—an Amdahl's Law diagnostic the paper uses to argue that CLOPS identifies shifting bottlenecks invisible to device-level metrics. Notably, MCPS (~3.9× gain) and gate-time reduction (~6× gain) each mischaracterize the relevant ceiling individually; only their combination bounds system throughput correctly.

The persistent ~3:1 ratio between CLOPS gains and application gains is attributed to coordinated application-layer optimization rather than fixed structural overhead—the authors concede this ratio is not guaranteed and would worsen under unmanaged Amdahl effects.

Limitations

The paper is candid about scope. CLOPS says nothing about correctness or accuracy of outputs; it excludes irregular control flow, mid-circuit measurement with real-time feedback, and routing-heavy topologies. Results depend strongly on $(N, D, S)$—larger $N$ lowers CLOPS, larger $D$ raises it via amortization—creating a headline-reporting hazard that the pinned canonical operating point mitigates but does not eliminate. Cross-system comparison is meaningful only when CLOPS and LF/EPLG are reported together under equivalent calibrations. Attributing changes to specific components requires care since the metric deliberately reflects the whole stack. The framework's projected extensions—to syndrome-cycle throughput in the early fault-tolerant era and logical-layer throughput thereafter—are explicitly forward-looking specifications, not completed definitions.

Conclusion

CLOPS provides a principled, gaming-resistant system-level speed benchmark whose counting unit coincides with the structure of existing scalable quality benchmarks, and whose empirical record—~330× improvement traced to identifiable hardware and software contributions—demonstrates its value both as a progress tracker and as a bottleneck diagnostic. Two problems remain open: a gaming-resistant application-level speed benchmark (blocked until quantum advantage provides a quality anchor) and concrete instantiation of the syndrome-cycle and logical-layer variants of the framework.