Reinforcement learning with learned gadgets to tackle hard quantum problems on real hardware (2411.00230v2)

Abstract: Designing quantum circuits for specific tasks is challenging due to the exponential growth of the state space. We introduce gadget reinforcement learning (GRL), which integrates reinforcement learning with program synthesis to automatically generate and incorporate composite gates (gadgets) into the action space. This enhances the exploration of parameterized quantum circuits (PQCs) for complex tasks like approximating ground states of quantum Hamiltonians, an NP-hard problem. We evaluate GRL using the transverse field Ising model under typical computational budgets (e.g., 2- 3 days of GPU runtime). Our results show improved accuracy, hardware compatibility and scalability. GRL exhibits robust performance as the size and complexity of the problem increases, even with constrained computational resources. By integrating gadget extraction, GRL facilitates the discovery of reusable circuit components tailored for specific hardware, bridging the gap between algorithmic design and practical implementation. This makes GRL a versatile framework for optimizing quantum circuits with applications in hardware-specific optimizations and variational quantum algorithms. The code is available at: https://github.com/Aqasch/Gadget_RL

• The paper introduces Gadget Reinforcement Learning (GRL), a novel approach leveraging learned composite gates (gadgets) to build efficient parameterized quantum circuits (PQCs).
• GRL iteratively learns gadgets from solving easy quantum problems and integrates them into the RL agent's action space to effectively tackle more complex tasks like finding the TFIM ground state.
• Experiments show GRL achieves significantly higher accuracy and more compact circuits compared to pure RL, demonstrating viability for real-world quantum hardware like IBM Heron by reducing transpilation and increasing fidelity.

Tackling Quantum Problems with Gadget Reinforcement Learning

The paper “From Easy to Hard: Tackling Quantum Problems with Learned Gadgets For Real Hardware” by Akash Kundu and Leopoldo Sarra presents an innovative approach to building quantum circuits using reinforcement learning (RL) to enhance the efficiency and practicality of quantum algorithms on current hardware. The focus is on a novel algorithm, Gadget Reinforcement Learning (GRL), which addresses the limitations of finding parameterized quantum circuits (PQCs) for simulating quantum systems, specifically targeting the NP-hard problem of obtaining the ground state of the transverse field Ising model (TFIM).

Methodological Advances

The GRL algorithm improves upon traditional RL algorithms by introducing learned composite gates, called “gadgets,” as additional actions within the RL framework. This extension to the action space is designed to overcome the exponential scaling issue typical in quantum state preparation and optimization tasks. The method begins with solving simple quantum problems and extracting frequently occurring sequences of gates or “gadgets” from these solutions. These gadgets are then iteratively integrated into the action space of the RL agent to tackle more complex instances, effectively creating a dynamically evolving decision-making space for the agent.

The architecture harnesses a double deep Q-network with an epsilon-greedy policy to sequentially build a quantum circuit by adding gates from a defined action space. The reward scheme is tailored to focus the agent’s search by providing feedback through a stringent cost function tied to the estimation of the TFIM ground state’s energy.

Experimental Results and Impact

The paper's results highlight a significant improvement in achieving compact PQCs that approximate the TFIM ground state with increased accuracy and reduced error margins, up to $10^7$ times better than pure RL approaches. The experiments demonstrate that GRL maintains scalability to larger systems and more complex quantum problems, establishing its viability for real-world quantum hardware applications.

Further, the work underscores the advantage of crafting circuits with a native gateset of the IBM Heron processor, enabling more straightforward implementation on quantum hardware. The GRL algorithm produces circuits that require less transpilation, enhancing fidelity and execution efficiency by minimizing unnecessary quantum gate operations. This practical optimization has tangible implications for the deployment of quantum algorithms under constrained, noise-prone quantum environments.

Implications and Future Directions

This research introduces a promising strategy for the automated design of VQAs by leveraging learned knowledge through gadgets. Its applications extend beyond the immediate problem space, potentially influencing methods in adaptive algorithm design where similar computational challenges exist. Moreover, refining GRL can lead to enhanced integration with noise-aware quantum computing models, thus contributing to more resilient quantum applications across various platforms.

The paper opens avenues for further exploration, such as integrating noise models directly into the learning protocol to assess the robustness of derived algorithms. Another promising direction is the exploration of hierarchical RL frameworks where gadgets form the basis for modular, scalable long-term strategies.

In summary, the paper presents a significant methodological contribution to machine learning for quantum circuits, displaying a thorough evaluation of GRL in optimizing the PQC search process. Its strategic innovation in utilizing learned gadgets illustrates the nuanced way to better align the potential of RL with the requirements and limitations of current quantum hardware, marking a step toward pragmatic quantum computing solutions.