RL-CQE: Reinforcement Learning Quantum Eigensolver

• RL-CQE is a reinforcement-learning-based hybrid quantum-classical eigensolver that leverages adaptive measurements and feedback-driven operator selection to approximate eigenstates.
• The framework uses a measurement-feedback loop in its original form and integrates ACSE residual minimization with a deep Q-network in the contracted formulation.
• It offers efficient eigenstate approximation and constant-scaling time-evolution ansätze, providing advantages over traditional variational eigensolvers.

Searching arXiv for the cited RL-CQE papers and related context. tool unavailable Reinforcement Learning Contracted Quantum Eigensolver (RL-CQE) denotes a reinforcement-learning-based hybrid quantum–classical eigensolver framework that uses adaptive measurement, feedback, and operator selection to approximate eigenvectors or multi-state wavefunctions of Hermitian operators. In the cited literature, the framework appears in two closely connected forms: a 2019 measurement–feedback protocol for extracting approximate eigenvectors of an arbitrary Hermitian black-box operator, and a 2026 generalization that formulates RL-CQE through the anti-Hermitian contracted Schrödinger equation (ACSE) for electronic excited states and real-time dynamics. The original formulation emphasizes semi-autonomous learning with minimal memory and one measurement per iteration, whereas the later contracted formulation uses ACSE residuals, sign-free two-qubit operators, and a deep Q-network (DQN) agent to build compact ansätze for many-fermion problems (Albarrán-Arriagada et al., 2019, Zhang et al., 18 May 2026).

1. Origin, scope, and nomenclature

The original protocol introduced by Albarrán-Arriagada et al. is a reinforcement-learning procedure for obtaining an approximation of the eigenvectors of an arbitrary Hermitian quantum operator. It is built from two systems: a black box named environment and a quantum state named agent. The environment changes any quantum state by a unitary matrix

$$\hat U_E(\tau)=\exp\big(-i\tau\hat{\mathcal O}_E\big), \quad \tau\in\mathbb R,$$

where $\hat{\mathcal O}_E$ is Hermitian. The agent adapts to some eigenvector of $\hat{\mathcal O}_E$ by repeated interactions with the environment, a feedback process, and semi-random rotations (Albarrán-Arriagada et al., 2019).

The later literature uses the name RL-CQE explicitly and states that it was “previously developed for ground-state problems.” In that formulation, RL-CQE is generalized to electronic excited states and real-time quantum dynamics. The generalization replaces the black-box eigenvector-learning viewpoint with a contracted-eigenvalue framework based on ACSE residual minimization, while retaining the reinforcement-learning principle that a policy adaptively selects the next transformation to apply. A key feature is that a DQN agent selects two-body operators at each iteration, yielding more compact ansätze and improved robustness with respect to critical hyperparameters (Zhang et al., 18 May 2026).

A common source of confusion is that RL-CQE is not a single fixed circuit template. The 2019 construction is a measurement–feedback eigensolver for arbitrary Hermitian operators, while the 2026 construction is a contracted, many-body formulation targeted at electronic excited states and dynamics. The shared structural element is adaptive policy-driven operator selection rather than static variational optimization.

2. Original measurement–feedback formulation

In the original protocol, the environment operator is written in spectral form as

$$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$

acting on a $d$-dimensional Hilbert space. An auxiliary agent system $A$ of the same Hilbert-space dimension is prepared in a fully known reference state $|\phi_{A,0}^{(1)}\rangle$, such as $|0\rangle$ for a single qubit or $|j\rangle$ in higher dimensions. After $k$ iterations, the agent state is denoted $\hat{\mathcal O}_E$0 (Albarrán-Arriagada et al., 2019).

The algorithm proceeds in discrete iterations. Before interaction, the agent is in $\hat{\mathcal O}_E$1. After applying the environment unitary, the state becomes

$\hat{\mathcal O}_E$2

This state is then expressed in the current agent basis $\hat{\mathcal O}_E$3 through a diagonalization operator $\hat{\mathcal O}_E$4 satisfying

$\hat{\mathcal O}_E$5

Operationally, the rotated state is measured in the computational basis. The outcome determines whether the current reference basis vector already approximates an eigenvector or whether a corrective update is required (Albarrán-Arriagada et al., 2019).

The role of the measurement is therefore not merely diagnostic. It defines the reinforcement signal that determines whether the current basis should be preserved or altered. In this sense, the eigenvector-learning task is cast as a closed-loop adaptive control problem on Hilbert space.

3. Policy update, search width, and convergence criterion

If the measurement outcome is $\hat{\mathcal O}_E$6, the protocol assumes that $\hat{\mathcal O}_E$7 may already be an eigenvector, and no corrective rotation is applied. If the outcome is $\hat{\mathcal O}_E$8, the agent state is outside the desired eigen-subspace, and a small random rotation is applied within the two-dimensional subspace $\hat{\mathcal O}_E$9:

$\hat{\mathcal O}_E$0

where, for example,

$\hat{\mathcal O}_E$1

and each angle $\hat{\mathcal O}_E$2 is chosen randomly in $\hat{\mathcal O}_E$3 (Albarrán-Arriagada et al., 2019).

The reinforcement signal is encoded through a search width $\hat{\mathcal O}_E$4 and two real scalars, a reward rate $\hat{\mathcal O}_E$5 and a punishment rate $\hat{\mathcal O}_E$6. The update rule is

$\hat{\mathcal O}_E$7

The diagonalizer evolves according to

$\hat{\mathcal O}_E$8

The agent is then reset to the post-measurement state in the computational basis before the next feedback step. One may take the value function as $\hat{\mathcal O}_E$9, with convergence declared when $$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$0 (Albarrán-Arriagada et al., 2019).

The fidelity metric is defined by

$$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$1

This makes the convergence target explicit: the protocol is designed to learn eigenvectors directly rather than to minimize an energy expectation value as an intermediate surrogate. That distinction is central to its later comparison with VQE (Albarrán-Arriagada et al., 2019).

4. Contracted formulation through ACSE residuals

The 2026 formulation casts RL-CQE as a multi-state contracted eigensolver for a fermionic Hamiltonian

$$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$2

The target is a set of $$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$3 eigenstates $$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$4. The ACSE is written as

$$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$5

At iteration $$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$6, the residual vector is

$$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$7

with fixed positive, descending weights $$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$8 and $$\hat{\mathcal O}_E=\sum_{\ell=0}^{d-1}\lambda_\ell\,|v_\ell\rangle\langle v_\ell|,$$9. The cost function is the residual norm

$d$0

The multi-state wavefunction is updated by a product of two-body unitaries,

$d$1

where one operator $d$2 is selected from an action pool at each step, and the scalar amplitude is obtained by the one-dimensional line search

$d$3

(Zhang et al., 18 May 2026).

A distinguishing feature of this formulation is the use of sign-free qubit operators,

$d$4

with $d$5. The paper states that the expectation value $d$6 differs from $d$7 only by a known, state-independent phase that cancels in the residual norm, so that the iterative sequence of exponentials of $d$8 converges to the same unitary manifold as that generated by the full fermionic operators. This equivalence is important because it allows the RL action pool to be defined directly in terms of sign-free two-qubit mappings rather than the full set of non-local Jordan–Wigner strings (Zhang et al., 18 May 2026).

5. Deep Q-network control and constant-scaling time evolution

In the contracted formulation, the RL state at step $d$9 is the flattened residual vector $A$0 over all orbital indices. Its dimension scales as $A$1 with the one-particle basis size $A$2, but is independent of the number $A$3 of targeted states. The action space is discrete, $A$4, where $A$5 is the number of distinct two-body operators in the pool. The immediate reward is

$A$6

where $A$7 is the ensemble energy drop and $A$8 regularizes residual versus energy. In the reported benchmarks, the DQN is a feed-forward network with 8 linear layers, hidden width 512, GELU activations, replay buffer size $A$9, discount $|\phi_{A,0}^{(1)}\rangle$0, residual weight $|\phi_{A,0}^{(1)}\rangle$1, AdamW optimization with learning rate $|\phi_{A,0}^{(1)}\rangle$2, batch size $|\phi_{A,0}^{(1)}\rangle$3, and total episodes approximately $|\phi_{A,0}^{(1)}\rangle$4; the maximum steps per episode are 5 for ground/excited-state calculations and up to 20 for H$|\phi_{A,0}^{(1)}\rangle$5 time evolution (Zhang et al., 18 May 2026).

The same work introduces a constant-scaling ansatz for time evolution. If the final excited eigenstates are prepared by shared unitaries,

$|\phi_{A,0}^{(1)}\rangle$6

then any superposition $|\phi_{A,0}^{(1)}\rangle$7 can be written as

$|\phi_{A,0}^{(1)}\rangle$8

A time-dependent reference state is prepared from a fixed state $|\phi_{A,0}^{(1)}\rangle$9 by $|0\rangle$0 additional unitaries $|0\rangle$1, so the full ansatz becomes

$|0\rangle$2

with total number of two-body exponentials $|0\rangle$3 independent of simulation time $|0\rangle$4 (Zhang et al., 18 May 2026).

For a time step $|0\rangle$5, the target state is defined as $|0\rangle$6 by classical eigendecomposition. RL-CQE is then run with reward equal to overlap fidelity until $|0\rangle$7, and the number of RL steps per $|0\rangle$8 is bounded by $|0\rangle$9, remaining constant in $|j\rangle$0. This suggests that the reinforcement-learning policy is used not only for state preparation but also as a mechanism for constraining temporal growth of the circuit ansatz (Zhang et al., 18 May 2026).

6. Performance, resource profile, and relation to other eigensolvers

For the original black-box formulation, the reported single-qubit results for random $|j\rangle$1 use reward $|j\rangle$2 and punishment $|j\rangle$3. The average fidelity exceeds $|j\rangle$4 within fewer than about 10 iterations and exceeds $|j\rangle$5 within fewer than about 300 iterations. For two-qubit random operators, all four eigenvectors converge to fidelities above about $|j\rangle$6 in about 8,000 iterations for $|j\rangle$7 and $|j\rangle$8. In the special Bell-basis case, when $|j\rangle$9 is block-diagonal in Bell states, the algorithm discovers each two-dimensional block independently and reaches $k$0 in only about 1,000 iterations. The same description states that each new eigenvector is learned by a fresh copy of the loop in the subspace orthogonal to previously learned eigenvectors, and that in practice the total number of iterations scales roughly linearly with the dimension $k$1, one block at a time, though the per-block cost grows with the subspace size (Albarrán-Arriagada et al., 2019).

The circuit form of one single-qubit loop consists of one black-box gate $k$2, one basis rotation $k$3, one computational-basis measurement, one corrective rotation $k$4, and one re-preparation of $k$5. The description further states that circuit depth per iteration scales as $k$6 two-qubit and single-qubit gates. In its comparison with variational eigensolvers, the same source emphasizes that RL-CQE uses exactly one single-shot measurement per iteration, stores only the current diagonalizer $k$7, absorbs control noise into pseudo-random exploration, and converges toward eigenvectors directly, whereas VQE typically requires many shots to estimate expectation values or overlaps and maintains a classical optimizer history with gradient information (Albarrán-Arriagada et al., 2019).

For the contracted many-body formulation, the measurement cost per iteration is $k$8 because all residual components $k$9 must be estimated. The ansatz size is typically much smaller than $\hat{\mathcal O}_E$00: the benchmarks report 2–5 steps for H$\hat{\mathcal O}_E$01 and approximately 10–20 for H$\hat{\mathcal O}_E$02, with circuit depth scaling as $\hat{\mathcal O}_E$03. On H$\hat{\mathcal O}_E$04 in STO-6G with 4 qubits and 4 singlets over bond lengths $\hat{\mathcal O}_E$05–$\hat{\mathcal O}_E$06 Å, energies are within $\hat{\mathcal O}_E$07 Hartree of FCI across all $\hat{\mathcal O}_E$08 using at most 5 RL steps; at $\hat{\mathcal O}_E$09 Å, even 2 steps give residual norms $\hat{\mathcal O}_E$10 and chemical accuracy. On linear H$\hat{\mathcal O}_E$11 in STO-6G with 6 qubits and 4 singlets over $\hat{\mathcal O}_E$12–$\hat{\mathcal O}_E$13 Å, the energies match exact diagonalization to $\hat{\mathcal O}_E$14 Hartree, and $\hat{\mathcal O}_E$15 decays to $\hat{\mathcal O}_E$16 within 5–10 steps. For real-time dynamics on $\hat{\mathcal O}_E$17 a.u. with $\hat{\mathcal O}_E$18, the reported fidelity is at least $\hat{\mathcal O}_E$19 in at most 5 RL steps for H$\hat{\mathcal O}_E$20 and at most 20 RL steps for H$\hat{\mathcal O}_E$21, constant for all $\hat{\mathcal O}_E$22 (Zhang et al., 18 May 2026).

These results clarify a second common misconception: RL-CQE is not uniformly “measurement-light” across all of its variants. The original semi-autonomous eigensolver uses one single-shot measurement per iteration, but the ACSE-based many-body version pays an $\hat{\mathcal O}_E$23 residual-estimation cost in exchange for compact operator sequences and constant-scaling time-evolution ansätze. The outlook described in the literature includes time-varying reward and punishment rates $\hat{\mathcal O}_E$24, integration of learned block diagonalizers into quantum simulation and quantum chemistry subroutines, continuous-time RL or policy-gradient updates using quantum amplitude estimation, larger molecules beyond STO-6G, hardware-efficient two-body decompositions, open quantum systems, and fully on-device quantum-RL loops for gradient-free autonomous calibration (Albarrán-Arriagada et al., 2019, Zhang et al., 18 May 2026).