---
title: 'BenchRL-QAS: RL Benchmark for Quantum Architecture Search'
url: https://www.emergentmind.com/topics/benchrl-qas
type: topic
---

# BenchRL-QAS: RL Benchmark for Quantum Architecture Search

BenchRL-QAS is a benchmarking framework for reinforcement-learning-based quantum architecture search (RL-QAS) introduced to provide a unified, systematic, and reproducible evaluation platform for quantum circuit construction across multiple variational quantum algorithm tasks, qubit counts, and noise settings. It benchmarks nine reinforcement learning agents on variational quantum state diagonalization (VQSD), variational quantum eigensolver (VQE), variational quantum classifier (VQC), and GHZ/state preparation tasks over systems ranging from 2 to 8 qubits, and it evaluates performance in both noiseless and realistic noisy regimes through a weighted ranking metric that balances circuit error or accuracy, gate count, circuit depth, and computational efficiency [2507.12189].

## 1. Origin, scope, and motivating problem

BenchRL-QAS was introduced to answer a practical question that earlier RL-QAS studies had not resolved: which reinforcement learning algorithms work best for designing quantum circuits, and under what conditions. The framework is presented as a response to a fragmented literature in which individual studies typically benchmarked only a few agents, on one task, with inconsistent metrics, making cross-paper comparison difficult. In the paper’s formulation, BenchRL-QAS is a systematic and reproducible platform for benchmarking RL methods on quantum circuit construction across VQSD, VQE, VQC, and GHZ/state preparation [2507.12189].

The motivation is rooted in the constraints of noisy intermediate-scale quantum hardware. Quantum circuit design is described as being shaped by limited qubit counts, connectivity restrictions, noise and gate errors, depth limitations, and strong dependence on ansatz choice. Within that setting, RL is treated as promising because it can build circuits sequentially, choosing gates and parameters adaptively. BenchRL-QAS therefore standardizes comparison across several quantum tasks and qubit sizes, evaluates both noiseless and noisy regimes, measures not only accuracy but also circuit efficiency and runtime, and releases code and data for reproducibility [2507.12189].

This benchmark-oriented framing places BenchRL-QAS within a broader shift in QAS research from isolated demonstrations toward controlled comparative evaluation. Related work had already explored RL-assisted ansatz discovery, tensor-based circuit encodings, and noise-aware RL-QAS, but not under a single multi-task benchmark of comparable breadth [2402.13754].

## 2. Task families, qubit ranges, and experimental regimes

BenchRL-QAS covers four task families spanning the 2–8 qubit regime. VQSD is evaluated on 2-qubit systems, where the goal is to diagonalize quantum states in the computational basis. VQC is evaluated on 3-qubit synthetic binary classification data. State preparation is evaluated on 3-qubit GHZ preparation with a non-parameterized gate set. VQE is evaluated on 4-, 6-, and 8-qubit molecular systems, with explicit mention of \(H_2O\) and BeH\(_2\), where the objective is to minimize the Hamiltonian expectation value and achieve chemical accuracy [2507.12189].

The benchmark includes both noiseless and noisy settings. The noiseless experiments cover the benchmark tasks without injected gate noise. The noisy benchmark is applied specifically to VQE, using 0.1% single-qubit depolarizing noise and 0.01% two-qubit depolarizing noise, with noise applied after each gate during ansatz construction. In the noisy setting, the reported results focus on error, gate count, and depth, and the paper emphasizes that the identity of the “best” algorithm becomes even more sensitive to metric choice and qubit size [2507.12189].

The task distribution is significant because it prevents performance claims from being tied to a single problem class. VQSD emphasizes diagonalization, VQE emphasizes energy minimization under molecular scaling, VQC evaluates predictive performance, and GHZ preparation isolates non-parameterized state synthesis. This multi-family coverage is central to the paper’s later conclusion that algorithmic performance is task-dependent, size-dependent, and noise-dependent [2507.12189].

## 3. Benchmarked reinforcement learning algorithms

BenchRL-QAS evaluates nine RL agents spanning value-based, actor-critic, and policy-gradient families. The paper distinguishes these categories because different algorithm classes exhibit different strengths on different QAS tasks.

| Category | Algorithms |
|---|---|
| Value-based methods | DQN, DDQN, DQN\_PER, DQN\_rank, Dueling DQN |
| Actor-critic methods | A2C, A3C |
| Policy-gradient methods | PPO, TPPO |

The value-based methods learn action values \(Q(s,a)\) and select actions by maximizing predicted value. A2C and A3C are described as actor-critic methods combining policy and value estimation. PPO and TPPO are policy-gradient methods with trust-region-like stabilization. The paper’s empirical comparison makes this categorization operational rather than merely taxonomic: value-based methods often perform strongly on noiseless VQE, whereas A3C and TPPO can be stronger on VQSD and VQC [2507.12189].

BenchRL-QAS uses standardized training and optimization settings across these agents. The reported configuration includes vanilla curriculum learning, neural networks with \(L\) layers and 1000 neurons per layer, batch size 1000, replay memory 20,000, learning rate \(3\times 10^{-4}\) with Adam, target network update every 500 steps, epsilon-greedy exploration starting at 1.0 and decaying to 0.05 with decay rate 0.99995 per step, discount factor \(\gamma = 0.88\), and circuit optimization with COBYLA for up to 500 iterations. A3C uses 3 workers, and the reported hardware is an AMD Rome 7H12 CPU with an Nvidia Ampere A100 GPU [2507.12189].

For statistical robustness, each experiment is repeated 5–10 times with independent neural-network initializations, yielding 325 separate optimization instances [2507.12189].

## 4. Formalization of the search environment

BenchRL-QAS represents the RL state as a tensor-based encoding of the current quantum circuit, capturing gate arrangement, circuit depth, parameter values, and achieved accuracy. The state tensor has dimension
\[
[D_{\text{max}} \times ((N+3)\times N)].
\]
Here \(N\) denotes the number of qubits. This encoding is aligned with earlier RL-QAS work that used tensor-based circuit representations to stabilize search over discrete gate spaces [2507.12189].

The action space depends on the task. For parameterized tasks—VQE, VQSD, and VQC—the available actions include gate type \(RX\), \(RY\), \(RZ\), and \(CX\), together with target qubit choices and continuous parameter values for parameterized gates. For the non-parameterized GHZ preparation task, the action set includes \(CX\), \(X\), \(Y\), \(Z\), \(H\), and \(T\), with no continuous parameters [2507.12189].

The framework explicitly masks two illegal action types. The first is redundancy,
\[
G_{m,q} = G_{m-1,q},
\]
meaning the same gate is repeated on the same qubit at consecutive steps. The second is repeated \(CX\) on the same pair,
\[
G_{m,(q_1,q_2)} = CX \wedge G_{m-1,(q_1,q_2)} = CX.
\]
Illegal actions are assigned
\[
Q(a,s) = -\infty,
\]
so they are removed from policy optimization [2507.12189].

Reward design is task-specific. For VQE, VQSD, and VQC, the reward is piecewise:
\[
R = \begin{cases}
5 & \text{if } C_t \leq \zeta,\\
-5 & \text{if } t \geq D_{\text{max}} \text{ and } C_t \geq \zeta,\\
\max\left( \dfrac{C_{t-1} - C_t}{C_{t-1} - E_{\min}}, -1 \right) & \text{otherwise}.
\end{cases}
\]
For state preparation, the reward is fidelity-based:
\[
R = \begin{cases}
\mathcal{R}, & \text{if } F(s_t) \geq 0.98,\\
F(s_t), & \text{otherwise}.
\end{cases}
\]
The benchmark also specifies task-specific thresholds and depth limits, including \(D_{\text{max}}=40\) and \(\zeta=5\times 10^{-2}\) for 2-qubit VQSD, \(\zeta=1.6\times 10^{-3}\) for VQE chemical accuracy, and \(D_{\text{max}}=25\) with training threshold \(\zeta=0.2\) for 3-qubit VQC [2507.12189].

## 5. Weighted ranking metric and comparative methodology

A defining contribution of BenchRL-QAS is its weighted ranking scheme for comparing algorithms across multiple criteria. The benchmark normalizes four metrics to \([0,1]\), with lower being better: average circuit error \(E_{\text{norm}}\), gate count \(G_{\text{norm}}\), circuit depth \(D_{\text{norm}}\), and time per episode \(T_{\text{norm}}\). These are combined in the composite score
\[
S = w_E E_{\text{norm}} + w_G G_{\text{norm}} + w_D D_{\text{norm}} + w_T T_{\text{norm}},
\]
where lower \(S\) is better [2507.12189].

The paper uses different weights in noiseless and noisy regimes. For noiseless evaluation,
\[
[w_E, w_G, w_D, w_T] = [0.5, 0.2, 0.2, 0.1].
\]
For noisy evaluation,
\[
[w_E, w_G, w_D, w_T] = [0.6, 0.1, 0.3, 0.0].
\]
This weighting gives accuracy or error the largest contribution in both regimes, increases the importance of depth under noise, and sets runtime to zero in the noisy ranking used in the reported table [2507.12189].

The ranking methodology is important because BenchRL-QAS does not treat accuracy alone as sufficient. Circuit depth, gate count, and computational efficiency are directly incorporated into model comparison. This contrasts with earlier QAS studies that often reported task-specific success but lacked a common multi-criterion ranking protocol. A comparable benchmarking motivation appears in SQuASH, which standardizes QAS comparison through surrogate-assisted evaluation rather than direct RL benchmarking [2506.06762].

## 6. Empirical findings, interpretation, and place within QAS research

The headline empirical result for VQC is that the RL-based quantum classifier outperforms baseline variational classifiers. On this task, DQN\_rank achieves 99.996% training accuracy and 99.991% test accuracy, outperforming the hardware-efficient ansatz and the net-based methods from Du et al. (2022). The paper further notes that this performance is obtained without retraining [2507.12189].

The benchmark’s central comparative finding is that no single RL algorithm is universally optimal across the evaluated QAS tasks. In noiseless VQE, value-based methods such as DQN, DDQN, and DQN\_rank often perform very well, especially on depth and gate count. In VQSD and VQC, A3C and TPPO can be stronger. In noisy VQE, the best algorithm depends on qubit count and on how error, depth, and gate count are weighted in the ranking. The paper explicitly characterizes performance as highly context-dependent, varying with task structure, qubit count, and noise [2507.12189].

This result is interpreted as empirical support for the no free lunch principle in RL-based quantum circuit design. The evidence cited includes the fact that some algorithms excel at small circuits, others at larger molecular systems, some minimize error, others minimize depth or gate count, and runtime efficiency varies substantially. The benchmark therefore argues that algorithm choice must be matched to the structure of the target quantum problem and to the evaluation objective, rather than inferred from isolated single-task wins [2507.12189].

BenchRL-QAS also has significance as benchmarking infrastructure. The code and all experimental data are publicly available, repeated trials are used, and evaluation criteria are standardized. This suggests a role beyond reporting results: the framework functions as a reproducible benchmark suite for RL-QAS research [2507.12189].

Within the broader QAS literature, BenchRL-QAS occupies a distinct position. Earlier work emphasized noise-aware DDQN-based ansatz discovery for VQAs [2402.13754], quantum-information-theoretic analysis of RL-VQSD and entanglement-guided admissible ansatz construction [2404.06174], predictor-free latent-space QAS with REINFORCE and Bayesian Optimization [2401.11576], and neural-predictor-guided QAS for reducing expensive circuit evaluations [2103.06524]. SQuASH later addressed uniform QAS comparison through surrogate models and fixed search spaces [2506.06762]. BenchRL-QAS differs from these by centering the question of comparative RL performance itself: not whether RL can discover useful circuits, but how distinct RL families behave across heterogeneous quantum architecture search workloads under a common, multi-metric protocol [2507.12189].

Source: https://www.emergentmind.com/topics/benchrl-qas