Metriq: Open-Source Quantum Benchmarking

• Metriq is an open-source collaborative platform for reproducible, cross-platform quantum benchmarking that standardizes execution, data collection, and public presentation.
• It combines system-level metrics (e.g., entanglement quality, gate performance) with application-inspired protocols to provide comprehensive evaluations of quantum hardware.
• The platform integrates an end-to-end workflow—including JSON benchmark definitions, asynchronous execution via metriq-gym, and interactive web visualization—for transparent and scalable performance assessment.

Metriq is an open-source collaborative platform for reproducible cross-platform quantum benchmarking. It integrates benchmark definition and execution, data collection, and public presentation into a unified workflow, and its benchmark suite spans both system-level metrics—such as entanglement quality, gate performance, and circuit speed—and application-inspired protocols for quantum machine learning, optimization, and quantum simulation. The platform was used to collect and publicly release results from more than ten quantum computers across multiple hardware vendors, and it introduces the Metriq Score as a composite index aggregating benchmark outcomes (Cosentino et al., 9 Mar 2026).

1. Benchmarking context and scope

Metriq emerged from a benchmarking landscape characterized by vendor-specific tooling, non-standardized methodologies, isolated case studies, and a lack of neutral third-party comparisons. The underlying difficulty is structural: quantum hardware is heterogeneous, native gate sets and connectivities differ, compilation choices materially affect outcomes, provider metadata are non-uniform, and queueing and cloud-runtime variability complicate repeated measurements (Cosentino et al., 9 Mar 2026).

The platform is explicitly organized around cross-platform, reproducible benchmarking rather than a single-vendor characterization stack. Its stated design principles are openness, schema-driven transparency, cross-platform execution, versioned reproducibility, continuous updating, and community-driven development. Code, benchmark definitions, and data are public; the code is released under Apache-2.0 and the data under CC-BY-4.0 (Cosentino et al., 9 Mar 2026).

A broader benchmarking ecosystem for quantum computing also includes initiatives such as BACQ, a project within MetriQs-France dedicated to application-oriented benchmarks for quantum computing and the establishment of performance evaluation criteria of reference that are meaningful for industry users (Barbaresco et al., 2024). A plausible distinction is that BACQ emphasizes application-oriented benchmark specification and multi-criteria evaluation, whereas Metriq is described as a third-party, end-to-end platform coupling execution, dataset curation, and public presentation (Barbaresco et al., 2024, Cosentino et al., 9 Mar 2026).

2. Platform architecture and execution model

Metriq consists of three main components: the Python runner metriq-gym, the version-controlled JSON dataset metriq-data, and the TypeScript/Vega front-end metriq-web (Cosentino et al., 9 Mar 2026).

The runner, metriq-gym, treats benchmarking both as a framework and as a curated suite. Each benchmark is defined by a JSON schema for configuration parameters and by two handlers: dispatch, which constructs circuits, submits jobs, and returns a BenchmarkData object, and poll, which retrieves results, analyzes them, and returns a BenchmarkResult containing a BenchmarkScore. The user-facing command-line interface is

$$\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.$$7

where the resource is job or suite and actions include dispatch, poll, upload, and estimate. Execution is asynchronous: dispatch returns immediately, and polling later fetches results from the provider (Cosentino et al., 9 Mar 2026).

Provider integration is mediated through qBraid-SDK. The platform supports combinations of cloud providers and hardware including IBM, IonQ, IQM, OriginQ, Quantinuum, and Rigetti across IBM Quantum, AWS Braket, Azure, Quantinuum NEXUS, Origin Quantum, and local simulators. Local simulators are treated as a provider "local" device, so the same benchmark logic can run on hardware and on simulation backends (Cosentino et al., 9 Mar 2026).

The dataset is stored as schema-validated JSON artifacts with a layout of the form

$$\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.$$8

Each file includes configuration, raw measurements, derived metrics, and metadata such as device name, provider, and seeds. The web front-end reads these JSON records, provides interactive plots and filtering, and exposes provenance links back to raw records (Cosentino et al., 9 Mar 2026).

This architecture makes compilation, backend choice, and timing part of the recorded benchmark stack rather than hidden implementation details. The paper treats this as a methodological requirement, because transpilation and runtime behavior are themselves part of practical device performance (Cosentino et al., 9 Mar 2026).

3. Benchmark suite and taxonomy

The Metriq suite is intentionally heterogeneous. It combines system-level benchmarks that target intrinsic device properties with application-inspired benchmarks that probe structured workloads (Cosentino et al., 9 Mar 2026).

Benchmark	Category	Primary target
BSEQ	System-level	Device-wide entangling capability
EPLG	System-level	Effective two-qubit error per layered gate
Mirror Circuits	System-level	Scalable circuit fidelity
CLOPS	System-level	Throughput in circuit layers per second
QML Kernel	Application-inspired	Quantum kernel inner products
LR-QAOA	Application-inspired	MaxCut performance under linear-ramp QAOA
WIT	Application-inspired	Coherent teleportation-style workload
QFT	Application-inspired	Fourier-transform-style coherent phase processing

The suite is designed to scale with processor size, and the framework incorporates cost and resource estimation to support practical evaluation (Cosentino et al., 9 Mar 2026). System-level benchmarks include BSEQ for Bell-state-based entanglement capability, EPLG for effective error per layered gate, Mirror Circuits for efficiently verifiable circuit fidelity, and CLOPS for throughput. Application-inspired protocols include QML Kernel, LR-QAOA, QFT, and WIT (Cosentino et al., 9 Mar 2026).

Metriq also integrates existing benchmark sources where appropriate. For example, the QFT benchmark is wrapped from the QED-C suite so that it can be dispatched through the same machinery while retaining QED-C’s circuit generation and analysis (Cosentino et al., 9 Mar 2026).

A methodological feature of the suite is that benchmarks are evaluated at multiple widths or scales where possible. This scaling enters the later aggregation procedure through explicit width weights, so that a benchmark evaluated at larger circuit widths contributes more strongly to the composite index (Cosentino et al., 9 Mar 2026).

4. Benchmark definitions and scoring primitives

Several of Metriq’s individual benchmarks are defined by explicit physical or algorithmic quantities.

BSEQ

BSEQ quantifies how many qubits can form high-quality entangled pairs that violate a Bell inequality, and how connected those pairs are. For each edge in the device connectivity graph, Metriq prepares Bell states and estimates the CHSH parameter

$$S = |\langle ZZ \rangle + \langle ZX \rangle + \langle XZ \rangle - \langle XX \rangle|.$$

Pairs with $S > 2$ are retained, and the induced subgraph is summarized by the size of its largest connected component (LCCS) and by the connection fraction. The benchmark score is then

$$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$

with each component normalized against a baseline device (Cosentino et al., 9 Mar 2026).

EPLG

EPLG captures effective error per two-qubit gate in a layer of simultaneously executed entangling gates along a chain. For sublayers of a chain decomposition, simultaneous direct randomized benchmarking is used to estimate process fidelities. The layer fidelity is

$\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$

and the effective error per layered gate is

$$\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.$$

Smaller EPLG is better. Metriq then constructs an aggregate EPLG Score across chain lengths using a weighted harmonic mean with a coverage penalty, so that longer chains and missing large-scale data have stronger influence (Cosentino et al., 9 Mar 2026).

Mirror Circuits

Mirror Circuits implement a Loschmidt-echo-style protocol: a random product state is prepared, a circuit $C$ is applied, then a Pauli layer, then a quasi-inverse $\tilde C^{-1}$, after which measurement is compared with the unique classically predictable output. If $S$ is the success probability and the random baseline is $2^{-w}$ for width $w$, the polarization is

$S > 2$0

truncated at $S > 2$1. Metriq aggregates these polarizations across a fixed panel of widths using width-based weights (Cosentino et al., 9 Mar 2026).

QML Kernel

The QML Kernel benchmark evaluates quantum kernel inner products. For feature map $S > 2$2,

$S > 2$3

Metriq uses a ZZ feature map $S > 2$4 and estimates accuracy by the probability of measuring the all-zero bitstring in an inner-product circuit that ideally returns $S > 2$5. The overall QMLK score is a width-weighted average over benchmark sizes $S > 2$6 (Cosentino et al., 9 Mar 2026).

LR-QAOA

LR-QAOA is an application-inspired MaxCut benchmark that avoids inner classical parameter optimization by fixing linear-ramp parameters. For weighted MaxCut,

$S > 2$7

Metriq uses a depth-$S > 2$8 QAOA ansatz with linearly ramped $S > 2$9 and $$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$0, then defines an approximation ratio relative to the optimal classical solution and a random baseline. The effective approximation ratio is

$$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$1

and the LR-QAOA Score is a width-weighted aggregate over $$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$2 (Cosentino et al., 9 Mar 2026).

QFT and WIT

For QFT, Metriq adopts QED-C benchmark methods for structured Fourier-transform circuits and aggregates per-width scores at $$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$3. For WIT, a 6- or 7-qubit circuit derived from wormhole-inspired teleportation is used, with the principal metric being a single expectation value $$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$4 whose ideal value is 1.0 (Cosentino et al., 9 Mar 2026).

These benchmark definitions illustrate an important feature of Metriq: the suite does not reduce quantum performance to a single hardware primitive. It instead samples distinct operational regimes—entanglement, layered gate quality, coherent reversibility, throughput, kernel estimation, optimization, and structured interference—within a common collection and scoring framework (Cosentino et al., 9 Mar 2026).

5. Metriq Score and comparative analysis

To summarize heterogeneous benchmark outcomes, Metriq defines a composite index called the Metriq Score. Its construction has three stages: within-benchmark aggregation across widths, baseline normalization, and across-benchmark aggregation (Cosentino et al., 9 Mar 2026).

For each benchmark component $$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$5, with associated width $$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$6, the width weight is

$$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$7

The raw value for benchmark $$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$8 is then

$$\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},$$9

For larger-is-better benchmarks, the normalized benchmark score is

$\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$0

whereas for smaller-is-better quantities the inverse normalization is used. Missing benchmark values are assigned $\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$1 (Cosentino et al., 9 Mar 2026).

Across benchmarks, Metriq defines an effective width

$\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$2

and benchmark weights

$\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$3

The final Metriq Score is

$\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$4

In the reported suite version, the baseline device is ibm_torino (Cosentino et al., 9 Mar 2026).

The resulting dataset supports cross-benchmark analysis. Spearman rank correlations among most quality-oriented benchmarks are high, ranging from $\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$5 to $\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$6. Mirror Circuits and QML Kernel have $\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$7, QML Kernel and QFT have $\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$8, and BSEQ and LR-QAOA have $\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,$9. The Metriq Score is most correlated with Mirror Circuits, with $$\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.$$0 (Cosentino et al., 9 Mar 2026).

The paper also reports a PCA on log-scores, with the first principal component explaining about $$\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.$$1 of the variance among devices with complete data. In additional regression analysis, system metrics such as BSEQ, EPLG, and Mirror Circuits predict application-inspired benchmarks strongly: for example, log-linear ridge regression gives $$\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.$$2 for predicting QML Kernel from the three system metrics, while Mirror Circuits alone gives $$\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.$$3 (Cosentino et al., 9 Mar 2026).

Metriq further compares benchmark scores with publicly reported calibration quantities. The Metriq Score has Spearman $$\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.$$4 with average two-qubit gate fidelity, Mirror Circuits has $$\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.$$5, and QML Kernel has $$\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.$$6. The paper presents this as a strong, though caveated, monotonic relationship between calibration quality and benchmark outcomes (Cosentino et al., 9 Mar 2026).

6. Interpretation, limitations, and future directions

Metriq’s central interpretive caution is that the Metriq Score is not a fidelity or a single physical figure. It is a composite index aggregating heterogeneous, normalized metrics, and rankings depend on the choice of baseline device, benchmark weights, and the set of included benchmarks. The linear structure is deliberate and favors interpretability over latent-variable modeling (Cosentino et al., 9 Mar 2026).

Several limitations are explicit. Provider metadata inconsistencies constrain speed benchmarks, especially outside IBM, where timing and runtime exposure are less uniform. Some providers impose job limits or optimize away circuit structure unless verbatim compilation is enforced. On AWS Braket, for example, barrier handling originally required Metriq to transpile into a native gate set, wrap the circuit into a verbatim_box, and disable qubit rewiring to preserve benchmark structure (Cosentino et al., 9 Mar 2026).

CLOPS is presented as particularly sensitive to timing ambiguities and is therefore best interpreted as a way to track a platform’s own time evolution rather than as a universal cross-architecture ranking. Missing data are treated as zeros in the scoring pipeline, which creates a hard coverage penalty. This makes incomplete participation visible, but it also means that composite scores can reflect missingness as well as measured performance (Cosentino et al., 9 Mar 2026).

Cost is another constraint. Metriq reports post-run resource figures such as tasks, shots, Hardware Quantum Credits, or device runtime, and it provides pre-run estimation through mgym job estimate, but it does not collapse all providers into a single monetary model. This suggests a pragmatic notion of reproducibility: raw resource use is standardized as far as possible, while provider pricing remains external to the core score (Cosentino et al., 9 Mar 2026).

Future directions include logical-level benchmarking, integration of quantum error mitigation methods such as ZNE, PEC, and CDR, support for dynamic circuits and mid-circuit measurement, better accommodation of heterogeneous platforms such as neutral atoms and photonics, cost-aware and HPC-QC benchmarking, and uncertainty propagation for the Metriq Score itself (Cosentino et al., 9 Mar 2026).

A recurring misconception is that Metriq provides a final or canonical ordering of quantum computers. The platform instead presents itself as a living infrastructure whose benchmark suite, weighting choices, and collected data are open to revision through public RFCs and pull requests. In that sense, the platform’s most durable contribution is not a fixed leaderboard but a reproducible, inspectable, and continuously extensible benchmarking workflow (Cosentino et al., 9 Mar 2026).