---
title: 'Metriq: Open-Source Quantum Benchmarking'
url: https://www.emergentmind.com/topics/metriq
type: topic
---

# Metriq: Open-Source Quantum Benchmarking

Metriq is an open-source collaborative platform for reproducible cross-platform quantum benchmarking. It integrates benchmark definition and execution, data collection, and public presentation into a unified workflow, and its benchmark suite spans both system-level metrics—such as entanglement quality, gate performance, and circuit speed—and application-inspired protocols for quantum machine learning, optimization, and quantum simulation. The platform was used to collect and publicly release results from more than ten quantum computers across multiple hardware vendors, and it introduces the Metriq Score as a composite index aggregating benchmark outcomes [2603.08680].

## 1. Benchmarking context and scope

Metriq emerged from a benchmarking landscape characterized by vendor-specific tooling, non-standardized methodologies, isolated case studies, and a lack of neutral third-party comparisons. The underlying difficulty is structural: quantum hardware is heterogeneous, native gate sets and connectivities differ, compilation choices materially affect outcomes, provider metadata are non-uniform, and queueing and cloud-runtime variability complicate repeated measurements [2603.08680].

The platform is explicitly organized around cross-platform, reproducible benchmarking rather than a single-vendor characterization stack. Its stated design principles are openness, schema-driven transparency, cross-platform execution, versioned reproducibility, continuous updating, and community-driven development. Code, benchmark definitions, and data are public; the code is released under Apache-2.0 and the data under CC-BY-4.0 [2603.08680].

A broader benchmarking ecosystem for quantum computing also includes initiatives such as BACQ, a project within MetriQs-France dedicated to application-oriented benchmarks for quantum computing and the establishment of performance evaluation criteria of reference that are meaningful for industry users [2403.12205]. A plausible distinction is that BACQ emphasizes application-oriented benchmark specification and multi-criteria evaluation, whereas Metriq is described as a third-party, end-to-end platform coupling execution, dataset curation, and public presentation [2403.12205] [2603.08680].

## 2. Platform architecture and execution model

Metriq consists of three main components: the Python runner **metriq-gym**, the version-controlled JSON dataset **metriq-data**, and the TypeScript/Vega front-end **metriq-web** [2603.08680].

The runner, **metriq-gym**, treats benchmarking both as a framework and as a curated suite. Each benchmark is defined by a JSON schema for configuration parameters and by two handlers: `dispatch`, which constructs circuits, submits jobs, and returns a `BenchmarkData` object, and `poll`, which retrieves results, analyzes them, and returns a `BenchmarkResult` containing a `BenchmarkScore`. The user-facing command-line interface is

```text
mgym <resource> <action>
```

where the resource is `job` or `suite` and actions include `dispatch`, `poll`, `upload`, and `estimate`. Execution is asynchronous: dispatch returns immediately, and polling later fetches results from the provider [2603.08680].

Provider integration is mediated through qBraid-SDK. The platform supports combinations of cloud providers and hardware including IBM, IonQ, IQM, OriginQ, Quantinuum, and Rigetti across IBM Quantum, AWS Braket, Azure, Quantinuum NEXUS, Origin Quantum, and local simulators. Local simulators are treated as a provider `"local"` device, so the same benchmark logic can run on hardware and on simulation backends [2603.08680].

The dataset is stored as schema-validated JSON artifacts with a layout of the form

```text
{source}/{version}/{provider}/{device}/{timestamp}_{benchmark-type}_{hash}.json
```

Each file includes configuration, raw measurements, derived metrics, and metadata such as device name, provider, and seeds. The web front-end reads these JSON records, provides interactive plots and filtering, and exposes provenance links back to raw records [2603.08680].

This architecture makes compilation, backend choice, and timing part of the recorded benchmark stack rather than hidden implementation details. The paper treats this as a methodological requirement, because transpilation and runtime behavior are themselves part of practical device performance [2603.08680].

## 3. Benchmark suite and taxonomy

The Metriq suite is intentionally heterogeneous. It combines system-level benchmarks that target intrinsic device properties with application-inspired benchmarks that probe structured workloads [2603.08680].

| Benchmark | Category | Primary target |
|---|---|---|
| BSEQ | System-level | Device-wide entangling capability |
| EPLG | System-level | Effective two-qubit error per layered gate |
| Mirror Circuits | System-level | Scalable circuit fidelity |
| CLOPS | System-level | Throughput in circuit layers per second |
| QML Kernel | Application-inspired | Quantum kernel inner products |
| LR-QAOA | Application-inspired | MaxCut performance under linear-ramp QAOA |
| WIT | Application-inspired | Coherent teleportation-style workload |
| QFT | Application-inspired | Fourier-transform-style coherent phase processing |

The suite is designed to scale with processor size, and the framework incorporates cost and resource estimation to support practical evaluation [2603.08680]. System-level benchmarks include **BSEQ** for Bell-state-based entanglement capability, **EPLG** for effective error per layered gate, **Mirror Circuits** for efficiently verifiable circuit fidelity, and **CLOPS** for throughput. Application-inspired protocols include **QML Kernel**, **LR-QAOA**, **QFT**, and **WIT** [2603.08680].

Metriq also integrates existing benchmark sources where appropriate. For example, the QFT benchmark is wrapped from the QED-C suite so that it can be dispatched through the same machinery while retaining QED-C’s circuit generation and analysis [2603.08680].

A methodological feature of the suite is that benchmarks are evaluated at multiple widths or scales where possible. This scaling enters the later aggregation procedure through explicit width weights, so that a benchmark evaluated at larger circuit widths contributes more strongly to the composite index [2603.08680].

## 4. Benchmark definitions and scoring primitives

Several of Metriq’s individual benchmarks are defined by explicit physical or algorithmic quantities.

### BSEQ

BSEQ quantifies how many qubits can form high-quality entangled pairs that violate a Bell inequality, and how connected those pairs are. For each edge in the device connectivity graph, Metriq prepares Bell states and estimates the CHSH parameter

$$
S = |\langle ZZ \rangle + \langle ZX \rangle + \langle XZ \rangle - \langle XX \rangle|.
$$

Pairs with $S > 2$ are retained, and the induced subgraph is summarized by the size of its largest connected component (LCCS) and by the connection fraction. The benchmark score is then

$$
\mathrm{BSEQ~Score}(d) = \frac{7}{8} S_{\text{LCCS}(d)} + \frac{1}{8} S_{f_{\text{conn}(d)}},
$$

with each component normalized against a baseline device [2603.08680].

### EPLG

EPLG captures effective error per two-qubit gate in a layer of simultaneously executed entangling gates along a chain. For sublayers of a chain decomposition, simultaneous direct randomized benchmarking is used to estimate process fidelities. The layer fidelity is

$$
\mathrm{LF} = \prod_{m=1}^{M} \mathrm{LF}_m,
$$

and the effective error per layered gate is

$$
\mathrm{EPLG} = 1 - \mathrm{LF}^{1/n_{2\mathrm{Q}}}.
$$

Smaller EPLG is better. Metriq then constructs an aggregate EPLG Score across chain lengths using a weighted harmonic mean with a coverage penalty, so that longer chains and missing large-scale data have stronger influence [2603.08680].

### Mirror Circuits

Mirror Circuits implement a Loschmidt-echo-style protocol: a random product state is prepared, a circuit $C$ is applied, then a Pauli layer, then a quasi-inverse $\tilde C^{-1}$, after which measurement is compared with the unique classically predictable output. If $S$ is the success probability and the random baseline is $2^{-w}$ for width $w$, the polarization is

$$
P = \frac{S - 2^{-w}}{1 - 2^{-w}}
= \frac{2^{w} S - 1}{2^{w} - 1},
$$

truncated at $P \ge 0$. Metriq aggregates these polarizations across a fixed panel of widths using width-based weights [2603.08680].

### QML Kernel

The QML Kernel benchmark evaluates quantum kernel inner products. For feature map $U(x)$,

$$
\ket{\Phi(x)} = U(x)\ket{0}^{\otimes n}, \qquad
k(x_i,x_j) = \left|\langle \Phi(x_i)\,|\,\Phi(x_j)\rangle\right|^2.
$$

Metriq uses a ZZ feature map $U_{\mathrm{ZZ}}(x)$ and estimates accuracy by the probability of measuring the all-zero bitstring in an inner-product circuit that ideally returns $\ket{0}^{\otimes n}$. The overall QMLK score is a width-weighted average over benchmark sizes $n \in \{10,20,30,50\}$ [2603.08680].

### LR-QAOA

LR-QAOA is an application-inspired MaxCut benchmark that avoids inner classical parameter optimization by fixing linear-ramp parameters. For weighted MaxCut,

$$
C(x) = \sum_{(i,j)\in E} w_{ij} [x_i \oplus x_j].
$$

Metriq uses a depth-$p=10$ QAOA ansatz with linearly ramped $\gamma_j$ and $\beta_j$, then defines an approximation ratio relative to the optimal classical solution and a random baseline. The effective approximation ratio is

$$
r_{\mathrm{eff}} = \frac{r - r_{\mathrm{random}}}{1 - r_{\mathrm{random}}},
$$

and the LR-QAOA Score is a width-weighted aggregate over $N \in \{10,20,50,100\}$ [2603.08680].

### QFT and WIT

For QFT, Metriq adopts QED-C benchmark methods for structured Fourier-transform circuits and aggregates per-width scores at $n \in \{4,8,12,20\}$. For WIT, a 6- or 7-qubit circuit derived from wormhole-inspired teleportation is used, with the principal metric being a single expectation value $E$ whose ideal value is 1.0 [2603.08680].

These benchmark definitions illustrate an important feature of Metriq: the suite does not reduce quantum performance to a single hardware primitive. It instead samples distinct operational regimes—entanglement, layered gate quality, coherent reversibility, throughput, kernel estimation, optimization, and structured interference—within a common collection and scoring framework [2603.08680].

## 5. Metriq Score and comparative analysis

To summarize heterogeneous benchmark outcomes, Metriq defines a composite index called the **Metriq Score**. Its construction has three stages: within-benchmark aggregation across widths, baseline normalization, and across-benchmark aggregation [2603.08680].

For each benchmark component $i$, with associated width $n_i$, the width weight is

$$
\alpha_i := \frac{n_i}{\sum_{j\in\mathcal{C}_{b_i}} n_j}.
$$

The raw value for benchmark $b$ is then

$$
\bar{v}_b(d,s) := \sum_{i\in\mathcal{C}_b} \alpha_i\, v_i(d,s).
$$

For larger-is-better benchmarks, the normalized benchmark score is

$$
\mathrm{BS}_b(d,s) := 100 \cdot \frac{\bar{v}_b(d,s)}{\bar{v}_b(d^{\mathrm{base}},s)},
$$

whereas for smaller-is-better quantities the inverse normalization is used. Missing benchmark values are assigned $\mathrm{BS}_b(d,s):=0$ [2603.08680].

Across benchmarks, Metriq defines an effective width

$$
\mu_b := \sum_{i\in\mathcal{C}_b} \alpha_i\, n_i
= \frac{\sum_{i\in\mathcal{C}_b} n_i^2}{\sum_{i\in\mathcal{C}_b} n_i},
$$

and benchmark weights

$$
w_b := \frac{\mu_b}{\sum_{c\in\mathcal{B}} \mu_c}.
$$

The final Metriq Score is

$$
\mathrm{MS}(d,s) := \sum_{b\in\mathcal{B}} w_b\, \mathrm{BS}_b(d,s).
$$

In the reported suite version, the baseline device is `ibm_torino` [2603.08680].

The resulting dataset supports cross-benchmark analysis. Spearman rank correlations among most quality-oriented benchmarks are high, ranging from $0.664$ to $0.991$. Mirror Circuits and QML Kernel have $\rho = 0.991$, QML Kernel and QFT have $\rho = 0.918$, and BSEQ and LR-QAOA have $\rho = 0.936$. The Metriq Score is most correlated with Mirror Circuits, with $\rho = 0.991$ [2603.08680].

The paper also reports a PCA on log-scores, with the first principal component explaining about $88\%$ of the variance among devices with complete data. In additional regression analysis, system metrics such as BSEQ, EPLG, and Mirror Circuits predict application-inspired benchmarks strongly: for example, log-linear ridge regression gives $R^2_{\log}=0.925$ for predicting QML Kernel from the three system metrics, while Mirror Circuits alone gives $R^2_{\log}=0.941$ [2603.08680].

Metriq further compares benchmark scores with publicly reported calibration quantities. The Metriq Score has Spearman $\rho = 0.982$ with average two-qubit gate fidelity, Mirror Circuits has $\rho = 0.973$, and QML Kernel has $\rho = 0.982$. The paper presents this as a strong, though caveated, monotonic relationship between calibration quality and benchmark outcomes [2603.08680].

## 6. Interpretation, limitations, and future directions

Metriq’s central interpretive caution is that the Metriq Score is not a fidelity or a single physical figure. It is a composite index aggregating heterogeneous, normalized metrics, and rankings depend on the choice of baseline device, benchmark weights, and the set of included benchmarks. The linear structure is deliberate and favors interpretability over latent-variable modeling [2603.08680].

Several limitations are explicit. Provider metadata inconsistencies constrain speed benchmarks, especially outside IBM, where timing and runtime exposure are less uniform. Some providers impose job limits or optimize away circuit structure unless verbatim compilation is enforced. On AWS Braket, for example, barrier handling originally required Metriq to transpile into a native gate set, wrap the circuit into a `verbatim_box`, and disable qubit rewiring to preserve benchmark structure [2603.08680].

CLOPS is presented as particularly sensitive to timing ambiguities and is therefore best interpreted as a way to track a platform’s own time evolution rather than as a universal cross-architecture ranking. Missing data are treated as zeros in the scoring pipeline, which creates a hard coverage penalty. This makes incomplete participation visible, but it also means that composite scores can reflect missingness as well as measured performance [2603.08680].

Cost is another constraint. Metriq reports post-run resource figures such as tasks, shots, Hardware Quantum Credits, or device runtime, and it provides pre-run estimation through `mgym job estimate`, but it does not collapse all providers into a single monetary model. This suggests a pragmatic notion of reproducibility: raw resource use is standardized as far as possible, while provider pricing remains external to the core score [2603.08680].

Future directions include logical-level benchmarking, integration of quantum error mitigation methods such as ZNE, PEC, and CDR, support for dynamic circuits and mid-circuit measurement, better accommodation of heterogeneous platforms such as neutral atoms and photonics, cost-aware and HPC-QC benchmarking, and uncertainty propagation for the Metriq Score itself [2603.08680].

A recurring misconception is that Metriq provides a final or canonical ordering of quantum computers. The platform instead presents itself as a living infrastructure whose benchmark suite, weighting choices, and collected data are open to revision through public RFCs and pull requests. In that sense, the platform’s most durable contribution is not a fixed leaderboard but a reproducible, inspectable, and continuously extensible benchmarking workflow [2603.08680].

Source: https://www.emergentmind.com/topics/metriq