Quantum Reinforcement Learning Controller

• Quantum Reinforcement Learning (QRL) controllers are decision-making systems that incorporate quantum elements, such as variational quantum circuits and quantum feedback, into classical RL frameworks.
• They employ hybrid architectures—combining quantum components with classical algorithms—to overcome challenges like vanishing gradients and barren plateaus.
• Applications span benchmark tasks to quantum control, achieving parameter efficiency and robustness through techniques like QLSTM-based memory and constraint-first designs.

Quantum Reinforcement Learning (QRL) controller denotes a reinforcement-learning decision mechanism in which quantum resources enter the policy, value, memory, or control-update loop. In recent literature, the term covers actor-critic controllers that replace neural policy/value modules with variational quantum circuits (VQCs), recurrent controllers built around quantum long short-term memory (QLSTM), direct unitary feedback laws for quantum tasks such as eigenstate preparation, and classical RL agents that optimize controls for quantum dynamical systems. Taken together, these works indicate that a QRL controller is not a single standardized architecture but a family of sequential-control designs coupling RL with quantum state evolution, quantum parametrization, or both (Kölle et al., 2024, Chen, 2022, Yue et al., 28 Aug 2025, Ernst et al., 24 Jan 2025).

1. Conceptual scope and definitions

A QRL controller is usually the learned decision subsystem that maps an observed state, history, or internal memory state to an action or control update. In the actor-critic literature, this can mean the policy/value subsystem of an RL agent in which the actor, critic, or both are replaced by VQCs; in the A2C formulation studied on Cart Pole, the actor maps $s_t$ to an action distribution and the critic maps $s_t$ to a scalar value estimate (Kölle et al., 2024). In continuous-control PPO formulations, the controller similarly consists of actor and critic VQCs whose measurements are decoded into Gaussian policy parameters and value estimates (Kruse et al., 2023). In recurrent designs, the controller may instead be a hybrid memory-bearing core: QLSTM-DRQN uses a QLSTM recurrent module to produce action values in partially observable environments, while QLSTM-reservoir RL freezes the quantum recurrent parameters and trains only surrounding classical layers (Chen, 2022, Chen, 2023).

The same label is also used for rather different control objects. In the eigensolver scheme of "Noise-Resilient Quantum Reinforcement Learning," the policy is represented directly as a unitary $\hat D^{(k)}$, updated from single-shot measurement feedback rather than by optimizing an explicit neural policy or Q-function. There, the controller’s state is effectively the current controllable quantum state $|\psi^{(k)}(0)\rangle$, equivalently the current action history $\hat D^{(k)}$, together with the exploration scale $w^{(k)}$, and the action is whether to keep the current control or apply a random unitary rotation $\hat R^{(k)}$ (Yue et al., 28 Aug 2025). In simulator-based quantum control, by contrast, RL may remain entirely classical while the controlled plant is quantum: TRPO-based universal quantum control, Q-learning for protocol optimization in different phases of quantum control, and enhanced RL for discrete unitary switching all fit this pattern (Niu et al., 2018, Bukov et al., 2017, Liu et al., 2023).

This diversity resolves a common misconception. A QRL controller is not necessarily a fully quantum agent. In several influential formulations, the environment is quantum but the policy learner is classical; in others, the policy is hybrid quantum-classical; in still others, the controller is a directly updated quantum unitary. A plausible implication is that the defining feature of the field is not full quantization of the RL stack, but the insertion of quantum structure at the level of representation, dynamics, or control law.

2. Control formulations and objectives

The dominant formalism is still classical RL, with quantum components inserted into policy, value, or environment models. In A2C-based QRL, the objective is stated as

$J(\theta) = \mathbb{E}_{\pi_\theta} [R],$

with one-step temporal-difference updates

$$y = r_t + \gamma V_{\omega}(s_{t+1}), \qquad A = y - V_{\omega}(s_t),$$

actor loss

$$L_\text{actor}(\theta) = -A \log \pi_{\theta}(a_t),$$

and critic loss

$s_t$0

This is the operative controller update used in the variational A2C study on Cart Pole (Kölle et al., 2024).

Continuous-action QRL commonly adopts stochastic actor-critic or deterministic actor-critic formulations. In the PPO-style VQC controller for continuous environments, the actor produces Gaussian parameters $s_t$1 for each action dimension from measured observables, with

$s_t$2

$s_t$3

followed by sampling

$s_t$4

The critic is a VQC-based scalar regressor $s_t$5 trained by squared error to returns (Kruse et al., 2023). In the VQC-based DDPG and DQN analogues for dynamic portfolio optimization, the Bellman targets remain classical: $s_t$6 for DDPG, with deterministic policy-gradient actor updates and soft target-network updates (Gurgul et al., 20 Jan 2026). In the earlier quantum DDPG work for continuous action spaces, the actor similarly maps $s_t$7 to a continuous parameter vector $s_t$8, which defines the next control unitary $s_t$9 (Wu et al., 2020).

Other QRL controllers depart more sharply from standard MDP formulations. In the eigensolver setting, the core update is

$\hat D^{(k)}$0

with reward/punishment encoded directly by the binary measurement $\hat D^{(k)}$1, and exploration width updated by

$\hat D^{(k)}$2

Here the controller objective is to maximize overlap with an eigenstate while driving $\hat D^{(k)}$3 (Yue et al., 28 Aug 2025). In constrained quantum-control PPO, the main problem is a continuous-action bandit over pulse parameterizations, with the constraint implemented through a hard penalty when the adaptive ODE solver exceeds a threshold $\hat D^{(k)}$4, and with a shaped reward that sharply rewards fidelity near unity while penalizing smoothness violations, excess pulse area, and occupation of lossy intermediate states (Ernst et al., 24 Jan 2025).

Taken together, these formulations suggest two broad controller classes. One class retains standard RL value and policy equations and substitutes quantum function approximators. The other class uses RL language for adaptive quantum feedback or pulse search where the “policy” is itself a quantum operation or a constrained waveform generator.

3. Architectural patterns

The literature converges on a small number of controller archetypes.

Controller archetype	Quantum component	Representative papers
Hybrid actor-critic	Quantum actor, critic, or both; classical readout optional	(Kölle et al., 2024, Kruse et al., 2023, Zhang et al., 25 Jan 2026)
Recurrent QRL controller	QLSTM or QLSTM reservoir as memory core	(Chen, 2022, Chen, 2023)
Quantum feature extractor + classical head	VQC features mapped by classical MLP/logit/value head	(Kölle et al., 2024, Chen et al., 17 Sep 2025)
Direct unitary feedback controller	Policy represented by updated unitary $\hat D^{(k)}$5	(Yue et al., 28 Aug 2025)
Classical RL for quantum control	Classical RL chooses quantum control operations or topology edits	(Niu et al., 2018, Bukov et al., 2017, Pozza et al., 2021)

Within hybrid actor-critic designs, a recurring result is that raw quantum outputs are often not sufficient. In the A2C Cart Pole study, pure quantum actor or critic variants performed poorly, whereas hybrid configurations using either a quantum actor or a quantum critic with classical post-processing showed a substantial performance increase; the strongest results came from hybrid architectures such as HA2Q and HQ2Q (Kölle et al., 2024). The most reusable design pattern emerging from that work is a VQC as a feature-generating front end with a small classical readout layer.

For continuous action spaces, the VQC architecture itself matters strongly. The PPO study on Pendulum and LunarLander identifies normalized linear angle encoding, repeated data re-uploading, CNOT-chain entanglement, and a simple readout configuration $\hat D^{(k)}$6 based on single-qubit $\hat D^{(k)}$7 observables for the actor and summed $\hat D^{(k)}$8 observables for the critic as the most robust controller design among the configurations tested (Kruse et al., 2023). The paper’s broader claim is not merely that VQCs can replace neural networks, but that encoding, observable choice, and post-processing dominate trainability.

Recurrent memory is the defining architectural addition for partially observable or temporally extended problems. QLSTM-DRQN inserts five VQCs into the classical LSTM gate structure,

$\hat D^{(k)}$9

with $|\psi^{(k)}(0)\rangle$0, whereas the reservoir version freezes the QLSTM parameters and trains only classical preprocessing and readout layers (Chen, 2022, Chen, 2023). This indicates that the controller’s quantum role may be either adaptive sequence modeling or fixed recurrent feature generation, depending on trainability constraints.

A different architectural lesson comes from physics-constrained pulse control. There the controller is not organized around state encoding into a VQC, but around PPO over pulse parameterizations subject to output rescaling, interpolation, Gaussian smoothing, fixed zero endpoints, and a hard simulation-step threshold. This suggests a broader editor’s term, “constraint-first QRL,” for controllers where experimentally plausible action structure is built into the controller/environment interface rather than imposed after optimization (Ernst et al., 24 Jan 2025).

4. Training, optimization, and implementation

Most QRL controllers are trained in hybrid loops where quantum forward passes are embedded inside classical optimization. Adam is common in A2C, PPO, QA3C, and variational DDPG/DQN implementations (Kölle et al., 2024, Chen, 2023, Zhang et al., 25 Jan 2026, Gurgul et al., 20 Jan 2026). PPO appears in both constrained analog quantum control and active flow control, while A3C/QA3C appears in recurrent QLSTM-based controllers, and DDPG/DQN-style training dominates continuous-action off-policy settings (Ernst et al., 24 Jan 2025, Zhang et al., 25 Jan 2026, Chen, 2023, Gurgul et al., 20 Jan 2026).

Where explicit quantum gradients are documented, parameter-shift is the standard choice. In active flow control, the paper gives

$|\psi^{(k)}(0)\rangle$1

and the portfolio-optimization paper states the same rule for VQC outputs $|\psi^{(k)}(0)\rangle$2 (Zhang et al., 25 Jan 2026, Gurgul et al., 20 Jan 2026). By contrast, several papers omit exact differentiation details, which suggests that gradient estimation remains a reproducibility bottleneck in parts of the literature.

A recurrent implementation pattern is simulator-first training. The A2C Cart Pole study was executed on a quantum simulator rather than actual hardware (Kölle et al., 2024). The VQC-PPO work on continuous environments used statevector simulators (Kruse et al., 2023). The dynamic portfolio controller trained entirely on a noiseless statevector simulator, deploying only inference to IBM Eagle r3 hardware, where cloud-system latency dominated end-to-end runtime despite intrinsically fast circuit execution (Gurgul et al., 20 Jan 2026). The active flow-control controller likewise used PennyLane simulation rather than a physical QPU (Zhang et al., 25 Jan 2026). A plausible implication is that many reported gains concern representation and parameter efficiency rather than demonstrated hardware-time advantage.

Trainability constraints recur across disparate implementations. The A2C study reports an average gradient of about $|\psi^{(k)}(0)\rangle$3 for the pure quantum approach and interprets this as severe gradient decay (Kölle et al., 2024). The continuous-control VQC design study reports that around 8–9 qubits training quality degrades markedly, with variance and gradient effects becoming problematic (Kruse et al., 2023). The portfolio paper explicitly prefers shallow circuits and enriched input features to mitigate barren plateaus (Gurgul et al., 20 Jan 2026). These results indicate that controller design is often limited more by trainability than by nominal expressivity.

In quantum control rather than benchmark RL, reward and action constraints are unusually explicit. The physics-constrained PPO controller imposes a hard threshold on the number of adaptive ODE solver steps and uses a reward with fidelity, smoothness, and pulse-area terms; this yields both higher fidelity and improved computational scalability (Ernst et al., 24 Jan 2025). In diffusion guidance control, the reward balances classification confidence, stepwise perceptual improvement, and action regularization, while the action is a bounded additive adjustment to CFG at each denoising step (Chen et al., 17 Sep 2025). This suggests that practical QRL controllers frequently require dense, physically meaningful shaping rather than purely sparse terminal rewards.

5. Applications and empirical behavior

Benchmark control remains a central proving ground. On Cart Pole, pure quantum QA2C variants A2Q, Q2C, and Q2Q did not learn across all runs, while hybrid methods significantly outperformed the classical parameter-matched baseline $|\psi^{(k)}(0)\rangle$4; the paper states that hybrid methods learned the task in almost every run, whereas classical $|\psi^{(k)}(0)\rangle$5 succeeded in only 5 of 10 runs (Kölle et al., 2024). In partially observable Cart-Pole, QLSTM-DRQN achieved more stable and higher average scores than classical DRQN with similar architecture and number of model parameters, and classical LSTM-16 collapsed after 800 training episodes while QLSTM agents remained stable (Chen, 2022).

Memory-intensive navigation produced a more qualified result. In MiniGrid-Empty, QLSTM-reservoir RL often achieved performance comparable to a fully trained QLSTM RL controller with identical architecture and training settings, but the fully trained model was better on the hardest setting, MiniGrid-8x8 fixed-start (Chen, 2023). This suggests that frozen quantum recurrence can be an efficient approximation to fully trainable quantum memory on moderate tasks, but not necessarily on the hardest ones.

Several domain-specific controllers report strong parameter-efficiency results. In the 6G dynamic-spectrum-access case study, the QRL model uses 4 qubits and 5 quantum layers for a total of 94 trainable parameters, versus 4610 for the DNN baseline; it reaches an average throughput of about 700 kbps after around 10,000 training iterations, whereas DRL needs more than 200,000 iterations to reach the same level (Tran et al., 2 Nov 2025). In dynamic portfolio optimization, 60-parameter quantum agents achieve Sharpe ratios in the range of much larger classical models—Quantum DDPG 60 reports $|\psi^{(k)}(0)\rangle$6 and Quantum DQN 60 reports $|\psi^{(k)}(0)\rangle$7, while the best absolute result remains the classical DQN 160k model at $|\psi^{(k)}(0)\rangle$8 (Gurgul et al., 20 Jan 2026). The most defensible interpretation is competitiveness under far smaller parameter budgets, not outright superiority in absolute performance.

In scientific control applications, QRL controllers are often evaluated by fidelity or physically meaningful system observables. The physics-constrained PPO controller reports fidelities exceeding $|\psi^{(k)}(0)\rangle$9 across three open-system tasks and better robustness to time-dependent perturbations and experimental imperfections than previous methods (Ernst et al., 24 Jan 2025). The active flow-control controller applies a VQC-PPO policy to square-cylinder wake control at $\hat D^{(k)}$0, reducing mean drag from about $\hat D^{(k)}$1 to about $\hat D^{(k)}$2 and reducing lift oscillation amplitude from about $\hat D^{(k)}$3 to about $\hat D^{(k)}$4 (Zhang et al., 25 Jan 2026). The diffusion-guidance controller reports improved PSNR and LPIPS relative to a classical RL actor while reducing the actor parameter count from about $\hat D^{(k)}$5K to about $\hat D^{(k)}$6K (Chen et al., 17 Sep 2025).

The field also includes controllers for quantum-native tasks. Classical RL controlling QAOA layer by layer obtains optimal MAXCUT results up to $\hat D^{(k)}$7, with the controller using selected intermediate quantum observables such as $\hat D^{(k)}$8, $\hat D^{(k)}$9, and sometimes edge correlators $w^{(k)}$0 (Garcia-Saez et al., 2019). In quantum transport and maze escape, topology-editing RL improves cumulative exit probability and is especially useful at short time scales in the quantum regime, where coherence allows the walker to exploit useful actions rapidly (Pozza et al., 2021). These examples show that “controller” can mean policy over variational parameters, graph-topology interventions, or sequential gate choices, depending on the problem class.

6. Limitations, controversies, and future directions

The first unresolved issue is definitional. Some papers use “QRL controller” for a hybrid actor-critic with VQC policy/value approximators; others use it for a classical RL controller acting on a quantum environment; still others use it for a directly updated unitary feedback law or even for adaptive search heuristics described in RL language (Kölle et al., 2024, Niu et al., 2018, Yue et al., 28 Aug 2025, Behera et al., 30 Jan 2025). This indicates that the field still lacks a canonical controller abstraction.

A second persistent limitation is simulator dependence. The strongest positive results in A2C, continuous-control PPO, portfolio optimization, active flow control, and recurrent QLSTM-reservoir RL are all simulation-based (Kölle et al., 2024, Kruse et al., 2023, Gurgul et al., 20 Jan 2026, Zhang et al., 25 Jan 2026, Chen, 2023). Where hardware is used, as in inference on IBM Eagle r3, infrastructure overhead rather than native circuit speed dominates runtime (Gurgul et al., 20 Jan 2026). Therefore the literature does not currently establish a general hardware-time advantage for QRL controllers.

Trainability remains the central algorithmic bottleneck. The Cart Pole A2C study links pure-quantum failure to vanishing gradients and barren plateau concerns (Kölle et al., 2024). The continuous-action VQC-design paper shows that larger qubit counts and deeper circuits can harm learning, and explicitly identifies encoding and readout as decisive (Kruse et al., 2023). The portfolio study also highlights barren-plateau risk and incomplete architectural specification as open issues (Gurgul et al., 20 Jan 2026). A plausible implication is that near-term controller design should remain shallow, hybrid, and aggressively regularized.

Noise is not uniformly detrimental, but its beneficial use is highly task-specific. In the eigensolver controller, non-Markovianity alone is not sufficient; performance is restored when the joint agent-noise Hamiltonian develops a bound state satisfying

$w^{(k)}$1

which yields asymptotic suppression of decoherence and recovery of nearly noiseless QRL performance (Yue et al., 28 Aug 2025). In maze transport, some decoherence reproduces a kind of RL-optimal strategy by assisting transport, reducing the margin for additional controller improvement (Pozza et al., 2021). These results suggest that future QRL controllers may have to treat the environment spectrally rather than merely as a source of generic noise.

Future directions are correspondingly diverse. The A2C study explicitly proposes testing different VQC architectures, adding data re-uploading, trying amplitude encoding, using weight re-mapping, adding pre-processing neural networks, studying A3C and PPO, and scaling to more complex tasks (Kölle et al., 2024). The 6G review argues that HQCA is the pragmatic near-term design under NISQ limitations and identifies dynamic resource allocation, UAV coordination, task offloading, and SAGIN control as active or emerging areas (Tran et al., 2 Nov 2025). The constrained quantum-control work suggests that embedding physical priors directly into the action space is a scalable design principle, while the recurrent QLSTM papers imply that memory-efficient quantum reservoirs may be a viable compromise when full QRNN training is too costly (Ernst et al., 24 Jan 2025, Chen, 2023).

Across these directions, the most stable near-term conclusion is conservative rather than maximalist: the most credible QRL controllers are hybrid, shallow, parameter-efficient, and tightly matched to the structure of the controlled task. Purely quantum replacements of entire RL stacks remain more a research target than an established engineering recommendation.