BenchRL-QAS: RL Benchmark for Quantum Architecture Search

• BenchRL-QAS is a unified benchmark framework for reinforcement learning in quantum architecture search, enabling systematic evaluation of RL agents on tasks like VQE, VQC, and state preparation.
• It integrates metrics such as circuit error, depth, gate count, and runtime to balance performance across both noiseless and noisy quantum environments.
• The framework standardizes experimental settings, ensuring reproducible comparisons and highlighting task- and noise-dependent variations in algorithm performance.

BenchRL-QAS is a benchmarking framework for reinforcement-learning-based quantum architecture search (RL-QAS) introduced to provide a unified, systematic, and reproducible evaluation platform for quantum circuit construction across multiple variational quantum algorithm tasks, qubit counts, and noise settings. It benchmarks nine reinforcement learning agents on variational quantum state diagonalization (VQSD), variational quantum eigensolver (VQE), variational quantum classifier (VQC), and GHZ/state preparation tasks over systems ranging from 2 to 8 qubits, and it evaluates performance in both noiseless and realistic noisy regimes through a weighted ranking metric that balances circuit error or accuracy, gate count, circuit depth, and computational efficiency (Ikhtiarudin et al., 16 Jul 2025).

1. Origin, scope, and motivating problem

BenchRL-QAS was introduced to answer a practical question that earlier RL-QAS studies had not resolved: which reinforcement learning algorithms work best for designing quantum circuits, and under what conditions. The framework is presented as a response to a fragmented literature in which individual studies typically benchmarked only a few agents, on one task, with inconsistent metrics, making cross-paper comparison difficult. In the paper’s formulation, BenchRL-QAS is a systematic and reproducible platform for benchmarking RL methods on quantum circuit construction across VQSD, VQE, VQC, and GHZ/state preparation (Ikhtiarudin et al., 16 Jul 2025).

The motivation is rooted in the constraints of noisy intermediate-scale quantum hardware. Quantum circuit design is described as being shaped by limited qubit counts, connectivity restrictions, noise and gate errors, depth limitations, and strong dependence on ansatz choice. Within that setting, RL is treated as promising because it can build circuits sequentially, choosing gates and parameters adaptively. BenchRL-QAS therefore standardizes comparison across several quantum tasks and qubit sizes, evaluates both noiseless and noisy regimes, measures not only accuracy but also circuit efficiency and runtime, and releases code and data for reproducibility (Ikhtiarudin et al., 16 Jul 2025).

This benchmark-oriented framing places BenchRL-QAS within a broader shift in QAS research from isolated demonstrations toward controlled comparative evaluation. Related work had already explored RL-assisted ansatz discovery, tensor-based circuit encodings, and noise-aware RL-QAS, but not under a single multi-task benchmark of comparable breadth (Kundu, 2024).

2. Task families, qubit ranges, and experimental regimes

BenchRL-QAS covers four task families spanning the 2–8 qubit regime. VQSD is evaluated on 2-qubit systems, where the goal is to diagonalize quantum states in the computational basis. VQC is evaluated on 3-qubit synthetic binary classification data. State preparation is evaluated on 3-qubit GHZ preparation with a non-parameterized gate set. VQE is evaluated on 4-, 6-, and 8-qubit molecular systems, with explicit mention of $H_2O$ and BeH$_2$, where the objective is to minimize the Hamiltonian expectation value and achieve chemical accuracy (Ikhtiarudin et al., 16 Jul 2025).

The benchmark includes both noiseless and noisy settings. The noiseless experiments cover the benchmark tasks without injected gate noise. The noisy benchmark is applied specifically to VQE, using 0.1% single-qubit depolarizing noise and 0.01% two-qubit depolarizing noise, with noise applied after each gate during ansatz construction. In the noisy setting, the reported results focus on error, gate count, and depth, and the paper emphasizes that the identity of the “best” algorithm becomes even more sensitive to metric choice and qubit size (Ikhtiarudin et al., 16 Jul 2025).

The task distribution is significant because it prevents performance claims from being tied to a single problem class. VQSD emphasizes diagonalization, VQE emphasizes energy minimization under molecular scaling, VQC evaluates predictive performance, and GHZ preparation isolates non-parameterized state synthesis. This multi-family coverage is central to the paper’s later conclusion that algorithmic performance is task-dependent, size-dependent, and noise-dependent (Ikhtiarudin et al., 16 Jul 2025).

3. Benchmarked reinforcement learning algorithms

BenchRL-QAS evaluates nine RL agents spanning value-based, actor-critic, and policy-gradient families. The paper distinguishes these categories because different algorithm classes exhibit different strengths on different QAS tasks.

Category	Algorithms
Value-based methods	DQN, DDQN, DQN_PER, DQN_rank, Dueling DQN
Actor-critic methods	A2C, A3C
Policy-gradient methods	PPO, TPPO

The value-based methods learn action values $Q(s,a)$ and select actions by maximizing predicted value. A2C and A3C are described as actor-critic methods combining policy and value estimation. PPO and TPPO are policy-gradient methods with trust-region-like stabilization. The paper’s empirical comparison makes this categorization operational rather than merely taxonomic: value-based methods often perform strongly on noiseless VQE, whereas A3C and TPPO can be stronger on VQSD and VQC (Ikhtiarudin et al., 16 Jul 2025).

BenchRL-QAS uses standardized training and optimization settings across these agents. The reported configuration includes vanilla curriculum learning, neural networks with $L$ layers and 1000 neurons per layer, batch size 1000, replay memory 20,000, learning rate $3\times 10^{-4}$ with Adam, target network update every 500 steps, epsilon-greedy exploration starting at 1.0 and decaying to 0.05 with decay rate 0.99995 per step, discount factor $\gamma = 0.88$, and circuit optimization with COBYLA for up to 500 iterations. A3C uses 3 workers, and the reported hardware is an AMD Rome 7H12 CPU with an Nvidia Ampere A100 GPU (Ikhtiarudin et al., 16 Jul 2025).

For statistical robustness, each experiment is repeated 5–10 times with independent neural-network initializations, yielding 325 separate optimization instances (Ikhtiarudin et al., 16 Jul 2025).

4. Formalization of the search environment

BenchRL-QAS represents the RL state as a tensor-based encoding of the current quantum circuit, capturing gate arrangement, circuit depth, parameter values, and achieved accuracy. The state tensor has dimension

$[D_{\text{max}} \times ((N+3)\times N)].$

Here $N$ denotes the number of qubits. This encoding is aligned with earlier RL-QAS work that used tensor-based circuit representations to stabilize search over discrete gate spaces (Ikhtiarudin et al., 16 Jul 2025).

The action space depends on the task. For parameterized tasks—VQE, VQSD, and VQC—the available actions include gate type $RX$, $RY$, $_2$0, and $_2$1, together with target qubit choices and continuous parameter values for parameterized gates. For the non-parameterized GHZ preparation task, the action set includes $_2$2, $_2$3, $_2$4, $_2$5, $_2$6, and $_2$7, with no continuous parameters (Ikhtiarudin et al., 16 Jul 2025).

The framework explicitly masks two illegal action types. The first is redundancy,

$_2$8

meaning the same gate is repeated on the same qubit at consecutive steps. The second is repeated $_2$9 on the same pair,

$Q(s,a)$0

Illegal actions are assigned

$Q(s,a)$1

so they are removed from policy optimization (Ikhtiarudin et al., 16 Jul 2025).

Reward design is task-specific. For VQE, VQSD, and VQC, the reward is piecewise: $Q(s,a)$2 For state preparation, the reward is fidelity-based: $Q(s,a)$3 The benchmark also specifies task-specific thresholds and depth limits, including $Q(s,a)$4 and $Q(s,a)$5 for 2-qubit VQSD, $Q(s,a)$6 for VQE chemical accuracy, and $Q(s,a)$7 with training threshold $Q(s,a)$8 for 3-qubit VQC (Ikhtiarudin et al., 16 Jul 2025).

5. Weighted ranking metric and comparative methodology

A defining contribution of BenchRL-QAS is its weighted ranking scheme for comparing algorithms across multiple criteria. The benchmark normalizes four metrics to $Q(s,a)$9, with lower being better: average circuit error $L$0, gate count $L$1, circuit depth $L$2, and time per episode $L$3. These are combined in the composite score

$L$4

where lower $L$5 is better (Ikhtiarudin et al., 16 Jul 2025).

The paper uses different weights in noiseless and noisy regimes. For noiseless evaluation,

$L$6

For noisy evaluation,

$L$7

This weighting gives accuracy or error the largest contribution in both regimes, increases the importance of depth under noise, and sets runtime to zero in the noisy ranking used in the reported table (Ikhtiarudin et al., 16 Jul 2025).

The ranking methodology is important because BenchRL-QAS does not treat accuracy alone as sufficient. Circuit depth, gate count, and computational efficiency are directly incorporated into model comparison. This contrasts with earlier QAS studies that often reported task-specific success but lacked a common multi-criterion ranking protocol. A comparable benchmarking motivation appears in SQuASH, which standardizes QAS comparison through surrogate-assisted evaluation rather than direct RL benchmarking (Martyniuk et al., 7 Jun 2025).

6. Empirical findings, interpretation, and place within QAS research

The headline empirical result for VQC is that the RL-based quantum classifier outperforms baseline variational classifiers. On this task, DQN_rank achieves 99.996% training accuracy and 99.991% test accuracy, outperforming the hardware-efficient ansatz and the net-based methods from Du et al. (2022). The paper further notes that this performance is obtained without retraining (Ikhtiarudin et al., 16 Jul 2025).

The benchmark’s central comparative finding is that no single RL algorithm is universally optimal across the evaluated QAS tasks. In noiseless VQE, value-based methods such as DQN, DDQN, and DQN_rank often perform very well, especially on depth and gate count. In VQSD and VQC, A3C and TPPO can be stronger. In noisy VQE, the best algorithm depends on qubit count and on how error, depth, and gate count are weighted in the ranking. The paper explicitly characterizes performance as highly context-dependent, varying with task structure, qubit count, and noise (Ikhtiarudin et al., 16 Jul 2025).

This result is interpreted as empirical support for the no free lunch principle in RL-based quantum circuit design. The evidence cited includes the fact that some algorithms excel at small circuits, others at larger molecular systems, some minimize error, others minimize depth or gate count, and runtime efficiency varies substantially. The benchmark therefore argues that algorithm choice must be matched to the structure of the target quantum problem and to the evaluation objective, rather than inferred from isolated single-task wins (Ikhtiarudin et al., 16 Jul 2025).

BenchRL-QAS also has significance as benchmarking infrastructure. The code and all experimental data are publicly available, repeated trials are used, and evaluation criteria are standardized. This suggests a role beyond reporting results: the framework functions as a reproducible benchmark suite for RL-QAS research (Ikhtiarudin et al., 16 Jul 2025).

Within the broader QAS literature, BenchRL-QAS occupies a distinct position. Earlier work emphasized noise-aware DDQN-based ansatz discovery for VQAs (Kundu, 2024), quantum-information-theoretic analysis of RL-VQSD and entanglement-guided admissible ansatz construction (Sadhu et al., 2024), predictor-free latent-space QAS with REINFORCE and Bayesian Optimization (Sun et al., 2024), and neural-predictor-guided QAS for reducing expensive circuit evaluations (Zhang et al., 2021). SQuASH later addressed uniform QAS comparison through surrogate models and fixed search spaces (Martyniuk et al., 7 Jun 2025). BenchRL-QAS differs from these by centering the question of comparative RL performance itself: not whether RL can discover useful circuits, but how distinct RL families behave across heterogeneous quantum architecture search workloads under a common, multi-metric protocol (Ikhtiarudin et al., 16 Jul 2025).