Intelligent Optimal Control of Rydberg Gates with Incremental-Update Deep Reinforcement Learning

Abstract: Deep reinforcement learning (DRL), acting as a novel and powerful paradigm for quantum optimal control, offers transformative opportunities for advancing neutral-atom quantum computing. In this work, we theoretically demonstrate a DRL-based framework for realizing Rydberg controlled-NOT gates that achieve both high speed and high fidelity through the synchronous modulation of multiple pulse parameters without any prior heuristic ansatz. By introducing an incremental-update learning policy, our framework effectively regularizes the exploration of the control landscape, ensuring the generation of smooth, experimentally feasible pulse profiles while significantly reducing computational overhead compared to conventional schemes. Crucially, the framework autonomously discovers an early-cutoff policy by optimally reconciling operation speed with high-precision coherent control. Our optimized protocol achieves a peak average fidelity of 0.9991, significantly outperforming conventional methods and surpassing the critical fault-tolerant threshold. This work establishes a generalizable, AI-driven pathway for designing high-performance quantum gates and provides a robust paradigm for autonomous control field optimization across diverse qubit platforms.

• The paper introduces a model-free, incremental-update DRL framework that achieves gate fidelities above 0.999 while reducing gate duration by 38.9%.
• The methodology regularizes pulse updates, generating smooth, synchronously-modulated pulse profiles that minimize decoherence through early cutoff strategies.
• The IU-DRL approach demonstrates strong thermal robustness and resilience against Doppler-induced errors, ensuring viability for fault-tolerant quantum computing.

Intelligent Optimal Control of Rydberg Gates with Incremental-Update Deep Reinforcement Learning

Introduction and Motivation

The synthesis of high-fidelity, fast, and robust quantum gates is a central requirement for scalable quantum computation using neutral atom platforms. Traditional quantum optimal control (QOC) techniques, such as GRAPE or genetic algorithms, are constrained by their need for gradient information, reliance on analytical pulse ansatze, or reduced controllability in high-dimensional parameter spaces. These bottlenecks are prominent in engineering Rydberg atom CNOT gates, where complex many-body interactions and decoherence processes rapidly limit achievable gate fidelities and durations.

This work introduces a model-free, deep reinforcement learning (DRL) framework based on an incremental-update (IU) policy to address these key challenges. The approach enables direct, synchronous optimization of both control and target pulse profiles, discovering non-intuitive control trajectories that efficiently reconcile the gate-fidelity/speed trade-off. The IU-DRL agent is demonstrated to autonomously develop early-cutoff strategies, minimizing integrated decoherence and achieving fidelities well above the fault-tolerant threshold.

Framework: Incremental-Update DRL for Rydberg CNOT Gates

Synchronous control of Rydberg CNOT gates is formalized as a Markov Decision Process, where a DRL agent adapts the time-dependent control vector $\vec{\mathcal{A}}(t_i)$ consisting of pulse amplitude and phase increments. The IU scheme regularizes pulse updates, ensuring the generation of smooth experimental profiles and efficient exploration in the high-dimensional control landscape.

This choice of incremental updates over direct parameterization sharply constrains the accessible action-space volume per time step, regularizing the optimization and preventing stagnation in non-smooth, suboptimal control regions. The agent operates without restricting prior knowledge or control ansatze, leveraging stochastic policy improvement (via TRPO) to explore the non-convex control landscape.

Benchmarks against traditional update (TU) DRL agents and gradient-based or piecewise-adiabatic protocols demonstrate the efficacy and accelerated convergence of the IU-DRL framework. The reward function utilizes terminal average gate fidelity (log-scaled) with intermediate penalties on Rydberg and excited-state populations, biasing policy learning toward rapid, low-error evolution and exposing an emergent early-cutoff mechanism.

(Figure 1)

Figure 1: Impact of atomic thermal motion on gate infidelity $\delta F$ for the IU-DRL protocol (red squares) and the traditional adiabatic scheme (blue triangles). The IU-DRL gate demonstrates superior robustness, particularly to Doppler-induced errors; interaction fluctuation errors remain negligible.

Results: Fidelity, Robustness, and Control Pathways

The IU-DRL agent autonomously generates synchronously-modulated pulse sequences for control and target atoms, deviating from the fixed, sequential structure underlying EIT-based or non-adiabatic piecewise protocols. The resulting population dynamics exhibit suppressed and transient Rydberg-state occupation, especially for the control atom, leading to a significant reduction in integrated decay errors ($\gamma_r T_r$). Notably, the agent halts all drives (early-cutoff) immediately upon achieving the target operation, further minimizing dissipation.

Under realistic decoherence parameters, a peak average fidelity $\mathcal{F}_{\text{avg}} = 0.9991$ is attained at a truncated gate duration of $0.336\,\mu$s—a 38.9% reduction over optimized piecewise non-adiabatic schemes. Most importantly, this fidelity is well above the typical $0.99$ threshold demanded by surface code error correction, ensuring practical utility in fault-tolerant architectures.

Smoothing coefficients critically determine control flexibility and learning dynamics. Overly aggressive coefficients induce pulse oscillations and slow convergence; however, extended training epochs yield highly regularized, smooth pulses and reinforce early-cutoff behavior. Comparative analysis with TU-based DRL underscores the inability of unconstrained policy spaces to consistently discover early-cutoff strategies or stabilize at high fidelity, even with increased compute budget.

(Figure 2)

Figure 2: DRL-optimized pulse amplitudes ($\Omega_t$) and phases ($\phi_t$), with different smoothing coefficients. These modulated pulses yield robust population trajectories and minimal occupation of lossy excited and Rydberg states.

Thermal robustness is a key practical metric. The IU-DRL-optimized gates demonstrate strong resilience to Doppler-induced dephasing, with gate infidelity $\delta F < 3 \times 10^{-4}$ at temperatures up to $10\,\mu$K—an order of magnitude better than adiabatic protocols. Fluctuations in Rydberg interaction due to atomic position uncertainty are shown to be negligible in the strong blockade regime.

(Figure 3)

Figure 3: DRL-optimized adiabatic pulse sequences and associated state populations. The IU-DRL approach enables faster protocols (via larger smoothing coefficients) without significant loss of adiabatic following.

Theoretical and Practical Implications

By constraining the DRL policy to incremental updates, the control landscape is efficiently regularized, facilitating the autonomous synthesis of smooth, experimentally feasible pulse sequences. The agent naturally discovers time-optimal trajectories (early cutoff) that static-horizon or direct-update policies miss due to their parametrization or reward design.

The model-free characteristic of this DRL framework is essential. The agent requires no knowledge of the system Hamiltonian or explicit noise models, implying direct extensibility to closed-loop experimental deployment, where learning occurs in situ from raw measurement outcomes. This datapath also positions the framework for autonomous calibration, error suppression (e.g., crosstalk), and adaptation to time-dependent hardware drift.

Beyond Rydberg arrays, the methodological advances in IU-DRL control optimization are broadly transferable to other quantum gate platforms (e.g., superconducting qubits or trapped ions), and quantum optimal control tasks involving non-trivial system dynamics or experimental constraints.

Future Directions

Potential extensions include:

• Noise-adaptive learning: Training the IU-DRL agent directly in the presence of realistic experimental noise, enabling active robustness to hardware drift and environmental variations.
• Resource-constrained optimization: Direct incorporation of hardware constraints, e.g., bandwidth and modulator nonlinearities, during policy learning.
• Autonomous calibration/enhancement: Deployment of the framework for closed-loop gate calibration, hardware crosstalk suppression, and real-time error mitigation.
• Scaling to many-qubit operations: Application to large-format neutral atom arrays and integration into error-correcting logical gate synthesis.

Conclusion

The IU-DRL framework represents a significant methodological advance for quantum gate engineering, resolving the longstanding trade-off between gate speed and fidelity in Rydberg neutral atom systems. By autonomously discovering time-optimal, high-fidelity control solutions without reliance on analytical ansatze or explicit modeling, this approach delivers both theoretical and practical advances in robust quantum optimal control. The combination of high computational efficiency, strong numerical performance against decoherence and thermal noise, and experimental viability portends broad impact for the design and autonomous calibration of quantum logical gates in diverse architectures (2605.04628).