- The paper introduces an RL-assisted quantum eigensolver that employs a deep Q-network to optimally select two-body operators for excited-state simulations.
- The authors achieve chemical accuracy for molecular simulations by leveraging an ACSE residual-based scalable representation and a constant-scaling ansatz for time evolution.
- Empirical evaluations on H2 and H3 demonstrate robustness to hyperparameter choices and efficient real-time dynamics with fixed operator sequences.
Reinforcement Learning-Assisted Quantum Simulation for Many-Body Excited States and Real-Time Dynamics
Introduction
The paper develops and extends the reinforcement learning contracted quantum eigensolver (RL-CQE) framework to address the computational challenges inherent in the quantum simulation of many-fermion excited states and real-time dynamics. Building on previous applications of CQE for ground-state problems, this work generalizes the approach to excited states using the purified ensemble formalism and introduces a deep Q-network-based RL agent for adaptive operator selection. The algorithm features a scalable, state representation based on ACSE residuals, avoids dependence on the number of targeted excited states, and attains robustness against critical hyperparameter choices. The methodology further provides a constant-scaling ansatz for time-dependent quantum simulation, circumventing conventional depth limitations associated with Trotterization. Empirical evaluation on prototypical molecular systems (H2 and linear H3) demonstrates the capacity to achieve chemical accuracy with significantly reduced operator counts across a range of bond lengths.
RL-CQE Methodology for Excited States
The core algorithmic innovation is the mapping of the excited-state CQE update process to a Markov decision process using RL. Within this formalism, the RL agent selects exponential two-body operators (specifically sign-free qubit operators) from an operator pool at each iterative step, leveraging a deep Q-network to optimize the policy. The chosen state representation is the ACSE residual vector, which encodes the quantum system’s deviation from the contracted Schrödinger equation and directly determines the RL agent’s environment with dimensions scaling only with the orbital number, not the number of excited states.
Key advances include:
- Reduced Measurement Overhead: The RL-based approach outputs more compact ansätze due to its sequential one-operator-per-step strategy, reducing the measurement burden compared to conventional CQE, where all operator coefficients are updated simultaneously.
- Hyperparameter Robustness: Empirical evidence demonstrates that RL-CQE is markedly less sensitive to the ensemble weight vector, a critical hyperparameter in purified ensemble CQE, enabling broader applicability.
- Operator Equivalence: The paper proves the equivalence of sign-free qubit operators in the excited-state setting, expanding prior results for ground-state systems.
- Scalable Representation: The ACSE-based state descriptor enables scalability, facilitating simultaneous targeting of multiple excited states without incurring representation overhead.
Deep Q-Network Architecture and Training
The RL agent utilizes a value-based deep Q-network (DQN) to approximate the Q-function, with operator actions selected based on the state-action value and immediate reward defined by the Euclidean norm reduction of the ACSE residual. The DQN architecture consists of 8 linear layers and GELU activations, trained via experience replay and AdamW, with a batch size of 256 and 3000 episodes. The reward function implements residual regularization with adjustable weights, enabling the RL agent to focus on both energy minimization and residual suppression.
RL-CQE for Time Evolution
For quantum simulation of time-dependent dynamics, the algorithm formulates a constant-scaling ansatz by expanding the time-evolved wavefunction in a common eigenstate basis, such that preparation of both time-independent and time-dependent states requires only a fixed number of unitary transformations, independent of total simulation time t. The approach contrasts with Trotterization, where circuit depth grows linearly with simulation time due to sequential short-time propagators. RL is employed to determine both sets of required unitaries efficiently, with the reward replaced by fidelity-based measures for time-evolution tasks.
Importantly, the number of unitaries needed is determined by the number of eigenstates with nonzero initial overlap, and remains constant for the entire simulation, demonstrating practical viability for near-term quantum hardware.
Numerical Evaluation
Benchmarking on H2 and linear H3 molecules covered the calculation of excited-state energies and real-time dynamics, with Hamiltonians mapped to 4 and 6 qubits via Jordan-Wigner transformation. Across various bond lengths, RL-CQE attained energies within 10−3 Hartree of FCI results using at most five operators for H2. Two-step sequences sufficed for representative H2 geometries, and RL-CQE demonstrated insensitivity to the ensemble weight vector as long as it maintained strict descending order. For H3, operator sequences adapted efficiently to local electronic regimes, maintaining accuracy and compactness.
For time-evolution, RL-CQE achieved fidelity convergence to unity within a fixed number of steps for both H2 (5 steps) and H3 (20 steps), independent of simulation duration, thus confirming the theoretical constant-scaling prediction.
Implications and Future Perspectives
The demonstrated efficiency and robustness of RL-CQE provide a promising approach for quantum simulation tasks requiring excited-state resolution and scalable real-time dynamics. The adaptive operator selection by RL mitigates the need for gradient-based optimization and prior domain knowledge regarding system symmetries or optimal hyperparameter choice, paving the way for generalized quantum state preparation protocols. The ACSE-based scalable state encoding and sign-free operator equivalence in excited-state contexts are particularly impactful for practical deployment on near-term quantum devices.
Future trajectories include extending RL-CQE to arbitrary quantum state preparation, open quantum systems, non-equilibrium quantum dynamics, and hybridized fermion-boson systems. Given increasing system sizes or complexity, RL-CQE’s resource minimization and adaptability position it as an algorithmic candidate for overcoming intractability in conventional quantum simulation paradigms. Integration with correlated efficient time-evolution frameworks (e.g., CETE) and extending to external field-driven phenomena are anticipated avenues.
Conclusion
The RL-CQE generalized algorithm substantiates the utility of reinforcement learning-driven approaches for quantum computational chemistry, offering compact, efficient, and robust solutions for many-body excited states and real-time dynamics. By leveraging deep Q-networks, scalable state descriptions, and purified ensemble representations, the framework achieves chemical accuracy with minimal quantum resources and constant circuit depth for time evolution. These advances have implications for the practical simulation of molecular systems on NISQ devices and inform future research in RL-assisted quantum algorithms for increasingly complex quantum systems (2605.18569).
