Investigation of Automated Design of Quantum Circuits for Imaginary Time Evolution Methods Using Deep Reinforcement Learning

Abstract: Efficient ground state search is fundamental to advancing combinatorial optimization problems and quantum chemistry. While the Variational Imaginary Time Evolution (VITE) method offers a useful alternative to Variational Quantum Eigensolver (VQE), and Quantum Approximate Optimization Algorithm (QAOA), its implementation on Noisy Intermediate-Scale Quantum (NISQ) devices is severely limited by the gate counts and depth of manually designed ansatz. Here, we present an automated framework for VITE circuit design using Double Deep-Q Networks (DDQN). Our approach treats circuit construction as a multi-objective optimization problem, simultaneously minimizing energy expectation values and optimizing circuit complexity. By introducing adoptive thresholds, we demonstrate significant hardware overhead reductions. In Max-Cut problems, our agent autonomously discovered circuits with approximately 37\% fewer gates and 43\% less depth than standard hardware-efficient ansatz on average. For molecular hydrogen ($H_2$), the DDQN also achieved the Full-CI limit, with maintaining a significantly shallower circuit. These results suggest that deep reinforcement learning can be helpful to find non-intuitive, optimal circuit structures, providing a pathway toward efficient, hardware-aware quantum algorithm design.

• The paper demonstrates that deep reinforcement learning can autonomously design compact quantum circuits for VITE, achieving significant reductions in gate count and depth.
• The methodology leverages a DDQN framework with adaptive thresholding to balance energy minimization and circuit complexity under practical hardware constraints.
• Benchmark tests on Max-Cut and the hydrogen molecule showcase improvements over conventional hardware-efficient templates, enabling efficient ansatz discovery.

Automated Quantum Circuit Design for Variational Imaginary Time Evolution via Deep Reinforcement Learning

Introduction

The investigated work presents an automated framework for quantum circuit design in the context of the Variational Imaginary Time Evolution (VITE) method, leveraging Double Deep-Q Networks (DDQN) as a reinforcement learning (RL) strategy (2604.07951). The methodology addresses circuit design for NISQ-era applications under strict gate count and depth constraints, focusing on both combinatorial optimization (Max-Cut) and quantum chemistry (molecular hydrogen), with a multi-objective reward seeking both minimal energy and minimal circuit complexity.

Framework Architecture

The proposed system formulates quantum circuit construction as a sequential decision process, representing the current circuit configuration as the environment state and the addition of quantum gates as discrete agent actions. This is summarized in the automated quantum circuit design workflow.

Figure 1: Schematics of the framework and the process of experience replay when applied to quantum circuit design.

The agent interacts with the environment by iteratively appending gates from a prescribed set and simulating the resultant circuit via VITE to assess energetic performance and circuit complexity, which are combined in a scalar reward signal.

The circuit construction process is further constrained: CNOT gates are limited to nearest-neighbor connectivity, and identical single-qubit gates are not permitted in consecutive positions on a qubit line, reducing the search space and enforcing practical hardware constraints.

Figure 2: Schematics of Quantum Circuit Design Workflow.

RL Algorithmic Details and Circuit Representation

Episodes begin from a uniform superposition prepared by Hadamard gates. The state space is encoded as a vectorized representation of a $4 \times 10$ grid, capturing the occupancy and nature of each gate at each location.

The DDQN-based agent uses an $\varepsilon$-greedy policy with per-episode and per-step adaptive decay, enhancing exploration without the premature convergence risk inherent in traditional $\varepsilon$-decay schemes.

The neural architecture consists of an input layer ingesting the vectorized circuit, three 32-node ReLU hidden layers, and an output corresponding to Q-values for available gate actions. Training is stabilized by experience replay and periodic synchronization of online and target Q-networks.

Figure 3: (a) Example of 4-qubit quantum circuit, (b) List representation of the quantum circuit on (a).

Problem Benchmarks and Hamiltonian Encoding

Two benchmarks were chosen: a 4-node Max-Cut problem mapped to a 4-qubit Ising Hamiltonian and the ground state of the molecular hydrogen (H$_2$), encoded via the Bravyi-Kitaev transformation for logarithmic scaling of Pauli word locality.

Figure 4: Schematics of the Max-Cut problem. A graph with 4 vertices and its maximum cut. All edge weights are set to 1.

For comparison, a hardware-efficient SU$(2)$ ansatz with 2 repetitions, 11 depth and 30 gates (excluding initial Hadamards), was used as a baseline.

Figure 5: Quantum circuit for 4-qubit hardware-efficient SU(2) ansatz (reps=2), referring to the structure following the H gates. The circuit depth is 11 and the gate count is 30; note that the initial H gates are excluded from the count.

Reward Structure and Adaptive Thresholding

The scalar reward integrates energy drop and circuit compactness:

$$R_t = (E_{t-1} - E_t) + c (g_{\max} - g) \Theta(E_{\mathrm{threshold}} - E_t)$$

To prevent reward-scale mismatches and reliance on unknown ground truth energies, a normalized and lower-bound-based reward was also introduced, with $E_{\mathrm{bound}} = -\sum_\alpha |\lambda_\alpha|$ as the lower limit.

The energy threshold $E_{\mathrm{threshold}}$ is dynamically adjusted by two mechanisms: (1) incremental tightening every 200 episodes or upon 20 consecutive successes, and (2) a relaxation to incentivize exploration. For the $H_2$ problem, this enabled progressive convergence toward the Full-CI limit.

Numerical Results

Max-Cut Optimization

In 100-trial assessments, the RL agent rapidly learned to reduce both the expectation value and circuit complexity.

Figure 6: Episode-dependence of (a) the expectation value of the Hamiltonian $\langle H \rangle$, (b) cumulative reward $\varepsilon$0, (c) gate count $\varepsilon$1, and (d) circuit depth $\varepsilon$2 averaged over 100 trials for the Max-Cut problem.

After $\varepsilon$3 episodes, average gate counts and depth were reduced by $\varepsilon$4 and $\varepsilon$5 below the hardware-efficient baseline, with many solutions reaching the theoretical minimum energy using circuits as small as $\varepsilon$6, $\varepsilon$7, marking a $\varepsilon$8 reduction in gate count from the baseline.

Figure 7: The smallest quantum circuit designed for the Max-Cut problem found in the RL method.

Hydrogen Molecule Ground State

The $\varepsilon$9 case proved more challenging due to a more complex energy landscape. Initial runs with the default reward found circuits averaging $\varepsilon$0 and $\varepsilon$1, but few instances ($\varepsilon$2) achieved Full-CI accuracy, saturating at the Hartree-Fock limit.

Figure 8: Episode-dependence of (a) the expectation value of the Hamiltonian $\varepsilon$3, (b) cumulative reward $\varepsilon$4, (c) gate count $\varepsilon$5, and (d) circuit depth $\varepsilon$6 averaged over 100 trials for the hydrogen molecular Hamiltonian.

Extended training and stricter exploration improved circuit compactness but did not substantially increase Full-CI achievements.

Figure 9: Episode-dependence of (a) the expectation value of the Hamiltonian $\varepsilon$7, (b) cumulative reward $\varepsilon$8, (c) gate count $\varepsilon$9, and (d) circuit depth $_2$0 averaged over 10 trials for the hydrogen molecular Hamiltonian up to 5,000 episodes.

Incorporating the normalized reward and adaptive threshold (excluding prior knowledge of $_2$1) increased the proportion of Full-CI-achieving circuits to $_2$2. Chemical accuracy was reached with average gate count/duration reductions of $_2$3 compared to the baseline.

Figure 10: Episode-dependence of (a) the expectation value of the Hamiltonian $_2$4, (b) cumulative reward $_2$5, (c) gate count $_2$6, and (d) circuit depth $_2$7 averaged over 10 trials for the hydrogen molecular Hamiltonian up to 5,000 episodes. RL agent with normalized reward and adaptive energy threshold.

Analysis of the Full-CI circuits enabled the extraction of a minimal skeleton structure, culminating in a circuit that required only $_2$8 gates and $_2$9 depths (a $(2)$0 and $(2)$1 reduction, respectively, from baseline) while reaching the Full-CI energy.

Figure 11: (a)-(c) Examples of circuits reaching $(2)$2 obtained in the RL method; (d) the essential skeleton circuit, further reduced in depth/gate count, while also reaching $(2)$3.

Theoretical and Practical Implications

The RL-based methodology offers several implications:

• Non-intuitive Ansatz Discovery: RL algorithms identified non-trivial, hardware-efficient circuits that standard hardware-efficient and chemical intuition-based ansätze typically miss, demonstrating the power of model-free global search.
• Guidelines for Ansatz Design: The extraction of skeleton structures from RL discoveries facilitates the development of heuristics for problem-specific circuit templates.
• Scalability and Application: While current studies are restricted to four qubits due to the computational overhead of RL-VITE hybridization, the framework serves as a generator of optimal circuit data for downstream supervised pipelines, and as a meta-algorithm for archetype discovery relevant to larger chemical/optimization problems.

However, practical deployment for real-time circuit design remains computationally prohibitive for large system sizes. The most immediate application is in the offline, automated generation of compact, high-quality circuit templates for inclusion in heuristic libraries or training sets for meta-learning and automated algorithm construction.

Conclusion

This work establishes that deep reinforcement learning, specifically double DQN, can autonomously design highly efficient quantum circuits for variational imaginary time evolution. The approach enforces both energetic and circuit-structural objectives, exceeding standard hardware-efficient templates in gate count and depth minimization, and provides a procedural pathway for elucidating essential, minimal ansätze via posthoc analysis. The future directions include extending adaptive thresholding to jointly optimize circuit size, scaling to multi-qubit systems, and systematic cataloging for quantum algorithm design heuristics.