Quantum-Inspired Deep RL: Key Methods

• Quantum-inspired deep reinforcement learning is a framework that integrates quantum circuits and measurement-inspired sampling with classical DRL to enhance parameter efficiency and exploration.
• It combines hybrid and quantum-inspired methods, using variational quantum circuits or qubit-based replay mechanisms to optimize value functions and policy estimation.
• Applications span Atari games, robotics, autonomous driving, and quantum control, though many benefits are currently demonstrated in simulation with noise considerations.

Quantum-inspired deep reinforcement learning (QI-DRL) is a family of reinforcement-learning methods that imports quantum-computing ideas into deep RL at several distinct levels: parameterized quantum circuits as value or policy approximators, quantum-inspired sampling and replay mechanisms implemented on classical hardware, energy-based models whose sampling bottlenecks motivate quantum acceleration, and fully quantum formulations in which the Markov decision process itself is represented in the quantum domain (Chen et al., 2019, Wei et al., 2021, Jerbi et al., 2019, Su et al., 2024). Across these strands, the central technical aims are recurrent: reducing parameter counts, improving exploration and sample reuse, exploiting superposition- or amplitude-amplification-inspired search, and extending DRL to regimes such as large action spaces, partial observability, noisy environments, quantum control, and robotic navigation.

1. Taxonomy and conceptual scope

The literature grouped under QI-DRL is not methodologically uniform. Some works are explicitly hybrid quantum-classical, with quantum circuits embedded inside otherwise conventional DRL pipelines. Others are purely classical but “quantum-inspired,” using qubit-like representations, amplitude amplification, or measurement-inspired sampling rules. A further line studies models whose practical bottleneck is probabilistic sampling rather than feedforward inference, arguing that such architectures are more compatible with genuine quantum advantage in RL (Jerbi et al., 2019), while another line formulates RL entirely within the quantum domain (Su et al., 2024).

Strand	Core mechanism	Representative papers
Hybrid variational QRL	PQCs or VQCs approximate $Q$-values, critics, or actors	(Chen et al., 2019, Hohenfeld et al., 2022, Sinha et al., 2023, Lokossou et al., 14 Sep 2025)
Quantum-inspired DRL	Qubit-like replay states, Grover-inspired prioritization, probabilistic action selection	(Wei et al., 2021, Li et al., 2021, Sannia et al., 2022, Chrisnanto et al., 13 Aug 2025)
Sampling-centric QRL	Energy-based RL, deep projective simulation, quantum Gibbs or walk-based speedups	(Jerbi et al., 2019)
Fully quantum RL	Quantum MDP, quantum arithmetic, Grover trajectory search	(Su et al., 2024)

A common misconception is that QI-DRL always denotes execution on quantum hardware. The cited literature uses the label more broadly. For example, DRL-QER and QiER are classical training paradigms inspired by quantum representations and measurement rules (Wei et al., 2021, Li et al., 2021), whereas Nav-Q and variational quantum DQN use hybrid quantum-classical models during training (Sinha et al., 2023, Chen et al., 2019). By contrast, the 2024 fully quantum framework explicitly aims to eliminate reliance on classical computations in agent-environment interaction and return computation (Su et al., 2024).

2. Variational quantum circuits as DRL function approximators

A foundational line of work replaces classical neural function approximators with variational quantum circuits. The 2019 study “Variational Quantum Circuits for Deep Reinforcement Learning” states that it is the first proof-of-principle demonstration of variational quantum circuits to approximate the deep $Q$-value function for decision-making and policy-selection reinforcement learning with experience replay and target network (Chen et al., 2019). In that construction, each classical state is mapped to qubits by computational basis encoding, the circuit applies entangling gates and parameterized single-qubit rotations, and qubit expectation values are measured as action-value estimates. The input preparation is given by

$R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$

and each trainable single-qubit block is

$$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$

The loss retains the DQN target-network form,

$$L(\theta)=\mathbb{E}\left[(r_t+\gamma \max_{a'}Q(s_{t+1},a';\theta^-)-Q(s_t,a_t;\theta))^2\right].$$

Within this formulation, experience replay and target networks remain structurally classical, but the value approximation is delegated to the VQC (Chen et al., 2019).

That paper also emphasizes parameter scaling. For the discussed setting, classical Q-learning grows as $\mathcal{O}(n^3)$, classical DQN as $\mathcal{O}(n^2)$, and VQ-DQN with computational basis encoding as $\mathcal{O}(n)$; the trainable-parameter count is reported as $n\times(3\times \#\text{layers}+1)$ (Chen et al., 2019). This is the main technical basis for claims of compactness in early QI-DRL.

Subsequent hybrid studies extend the same idea to robotic and actor-critic settings. In robot navigation with DDQN, parameterized quantum circuits with data re-uploading learned optimal policies in multiple navigation scenarios with notably fewer trainable parameters than a classical baseline; however, the same study also reports that the classical neural network consistently showed better results concerning training times and stability, and that in a large and dynamic environment the classical baseline produced more stable and better performing policies overall (Hohenfeld et al., 2022). In Nav-Q for self-driving cars, only the critic is quantum: a hybrid quantum-classical critic receives the LSTM hidden state, encodes it with a PQC, measures qubits in the Pauli-$Z$ basis, and maps the expectation vector to a scalar value. The best critic configuration is reported with 53 parameters versus 2305 in the classical baseline, alongside higher average AUC and lower AUC standard deviation during training (Sinha et al., 2023). In continuous control, a quantum SAC actor for Walker2d-v4 uses a 17-qubit data re-uploading PQC and is reported to have 41 parameters versus 1,250 in the classical actor, achieving an 8% higher average return, 246.40 versus 228.36, after 92% fewer steps (Lokossou et al., 14 Sep 2025).

Taken together, these works establish a recurring hybrid template: classical optimization, replay, and environment simulation are retained, while the representation of $Q$0, $Q$1, or policy parameters is compressed into a trainable quantum layer. This suggests that, within QI-DRL, compactness is often the first claimed advantage, whereas end-to-end quantum execution is usually not the immediate objective.

3. Quantum-inspired replay, prioritization, and memory mechanisms

A second major strand leaves the value network classical or hybrid but re-engineers experience replay using quantum-inspired state representations. In DRL-QER, each transition is associated with a qubit

$Q$2

where $Q$3 denotes accepting the experience and $Q$4 is the replay probability (Wei et al., 2021). New transitions are initialized in the uniform state

$Q$5

A preparation operation rotates this state according to TD-error, and a depreciation operation reduces replay probability as replay count grows. Sampling then follows

$Q$6

The stated purpose is to balance exploration and exploitation by coupling importance to TD-error while preserving transition diversity through replay-count-aware depreciation (Wei et al., 2021).

The empirical scope of this approach is broader than toy environments. DRL-QER was evaluated on 12 Atari 2600 games and is reported to outperform DRL-PER and DCRL on most of these games with improved training efficiency; it was also shown to integrate directly with Double DQN and Dueling DQN, outperforming PER-based counterparts in 3 out of 4 games each (Wei et al., 2021). A closely related QiER framework was later used for cellular-connected UAV path planning, where each experience is associated with a qubit

$Q$7

and one Grover-iteration-based update modulates the measurement probability according to both TD-error and replay count (Li et al., 2021). The stated effect is a better trade-off between sampling priority and diversity than standard ER or PER, and the paper reports that, in late-stage training over episodes 1800–2000, DRL-QiER achieves both the lowest average expected outage duration and the lowest average time cost among the compared baselines (Li et al., 2021).

A more distributed extension is QDQN-DPER, which combines variational quantum Q-learning with prioritized experience replay and asynchronous training (Chen, 2023). Its architecture comprises a global shared quantum policy network and target network, multiple asynchronous agents, and local prioritized replay buffers. PER sampling uses

$Q$8

and the paper replaces single-step TD loss with a matrix loss over an $Q$9-step trajectory:

$R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$0

In CartPoleMod, using an 8-qubit, 2-layer hybrid VQC with 48 quantum parameters, the framework is reported to outperform the baseline distributed quantum Q learning with the same model architecture and to be crucially stabilized by the matrix loss (Chen, 2023).

These replay-oriented papers are important because they shift the meaning of “quantum-inspired” away from hardware and toward algorithm design. Qubit states, amplitude amplification, and measurement-collapse analogies are used here as design principles for data selection, not merely as metaphors.

4. Action selection, recurrence, and large-space sampling

Action selection is another locus of quantum inspiration. A hybrid classical-quantum approach to Q-learning introduces a quantum routine for encoding an arbitrary discrete probability distribution on a quantum register, specifically for selecting actions according to learned action probabilities (Sannia et al., 2022). Starting from a uniform superposition, the algorithm sequentially fixes amplitudes using Grover-based amplitude amplification and an ancilla-based protection mechanism. For large action spaces, actions are grouped into classes according to $R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$1, class sizes can be estimated by quantum counting, and sampling complexity becomes

$R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$2

compared with classical $R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$3 cost. The paper also provides a maximum incremental error bound of $R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$4 and explicitly states that the routine is directly adaptable to deep RL when $R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$5 is produced by a neural network or quantum circuit (Sannia et al., 2022). This makes action sampling, rather than value approximation alone, a primary target for quantum enhancement.

Partial observability motivated the development of quantum recurrent RL. In quantum deep recurrent reinforcement learning, a QLSTM replaces the classical LSTM core of a DRQN with five VQC-based modules:

$R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$6

$R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$7

In Cart-Pole, QLSTM-1 and QLSTM-2 are compared against LSTM-8 and LSTM-16; the parameter counts reported are 150, 270, 634, and 2290, respectively, and the quantum models are described as more stable and higher-scoring in both fully and partially observable settings, with LSTM-16 collapsing after 800 episodes in the partially observable case (Chen, 2022). Here, recurrence is the key contribution: quantum circuits are used not simply as static function approximators but as memory-bearing sequence processors.

A different large-space perspective is provided by energy-based RL. “Quantum enhancements for deep reinforcement learning in large spaces” argues that not all DRL architectures present meaningful quantum bottlenecks, and that models with a sampling bottleneck—such as deep energy-based reinforcement learning and deep projective simulation—are more promising targets for quantum speedups (Jerbi et al., 2019). The policy is written in Boltzmann form,

$R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$8

and the computational bottleneck is sampling or partition-function estimation rather than a single forward pass. The paper emphasizes that classical Gibbs sampling scales as $R_x(\pi b_i)R_z(\pi b_i)\ket{0}=\ket{b_i},$9 while quantum Gibbs sampling can scale as $$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$0, and it positions quantum walks, quantum simulated annealing, and variational Gibbs-state preparation as the appropriate subroutines for RL models whose expressivity comes from difficult inference (Jerbi et al., 2019). This is a conceptually distinct vision of QI-DRL: quantum advantage is pursued by changing the model class, not just by swapping a classical layer for a quantum one.

5. Application domains and reported performance

The empirical spread of QI-DRL is unusually broad, spanning standard DRL benchmarks, robotic navigation, manufacturing optimization, autonomous driving, quantum control, and multi-agent navigation. The reported outcomes are heterogeneous but collectively show that the field is not confined to toy MDPs.

Domain	Method	Reported outcome
Atari 2600	DRL-QER	Outperforms DRL-PER and DCRL on most of 12 games; improved training efficiency (Wei et al., 2021)
Wheeled robot navigation	Hybrid PQC-DDQN	Quantum circuits learn optimal policies with notably fewer trainable parameters, but classical NN is more stable overall in large dynamic settings (Hohenfeld et al., 2022)
Self-driving collision-free navigation	Nav-Q	Higher average AUC, lower AUC standard deviation, and normalized effective dimension 0.20 versus 0.02 for the classical critic (Sinha et al., 2023)
Walker2d-v4	Quantum SAC	Average return 246.40 versus 228.36 and 92% fewer steps; 41 versus 1,250 actor parameters (Lokossou et al., 14 Sep 2025)
Cut order planning	QI-DRL with LSTM and OU noise	Average reward $$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$1, loss $$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$2, and fabric cost savings of up to 13% (Chrisnanto et al., 13 Aug 2025)

Other applications reinforce the same pattern. In ARDNS-FN-Quantum, a 2-qubit circuit for action selection is coupled to a dual-memory system and adaptive exploration; in a $$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$3 grid-world over 20,000 episodes, the paper reports a 99.5% success rate, mean reward 9.0528 across all episodes, average 46.7 steps to goal, and reward variance 5.424, compared with 81.3%, 1.2941, 135.9, and 252.262 for DQN, and 97.0%, 7.6196, 62.5, and 76.583 for PPO (Sousa, 7 May 2025). Q-ARDNS-Multi extends the same design to two agents in a $$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$4 environment, reporting success rates of 99.6% and 99.5%, mean rewards of $$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$5 and $$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$6, 210 steps to goal, and a 2.1% collision rate, outperforming MADDPG and SAC on success rate, stability, navigation efficiency, and collision avoidance (Sousa, 2 Jun 2025).

QI-DRL also appears in quantum-native control problems. In measurement-based quantum feedback control, PPO is used to stabilize two- and three-qubit entangled target states, reducing average stabilization time from 14.36 to 5.64 for the two-qubit Bell-state task and from 43.02 to 33.39 for the three-qubit GHZ task, with robustness to imperfect measurements and feedback delays (Song et al., 2024). In quantum Hamiltonian engineering, DRL discovers pulse sequences that outperform celebrated sequences such as Cory48 and WAHUHA on a solid-state nuclear magnetic resonance quantum simulator, while also revealing a recurrent “yxx” control pattern that the authors then use to restrict the search space for longer robust sequences (Peng et al., 2021). These control papers are not “quantum-inspired” in the narrow sense of replay heuristics or PQC replacements; rather, they show that deep RL itself can become a practical design tool for quantum technologies.

6. Limitations, controversies, and research directions

The literature is explicit that current gains are conditional and domain-specific. One recurring limitation is that many reported advantages are obtained in simulation, not on fault-tolerant quantum hardware. In the wheeled-robot navigation study, circuits are simulated noise-free, the circuits used are described as too deep for current NISQ devices, and classical baselines remain better in training times, stability, and robustness in the large dynamic environment (Hohenfeld et al., 2022). Nav-Q similarly evaluates hybrid critics with noisy quantum simulation and finds that quantum noise deteriorates training performances, even though it enhances the exploratory tendencies of the agent during training (Sinha et al., 2023). A plausible implication is that “better exploration” and “better optimization” do not necessarily coincide under realistic noise.

A second source of ambiguity is terminological. The same umbrella includes qubit-inspired replay rules that run entirely on classical computers (Wei et al., 2021), hybrid quantum-classical critics and actors (Sinha et al., 2023, Lokossou et al., 14 Sep 2025), and fully quantum RL formulations in which state transitions, rewards, return calculation, and trajectory search are all quantum (Su et al., 2024). The fully quantum framework models states and actions as basis vectors in Hilbert space, encodes transition probabilities by controlled $$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$7 rotations with

$$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$8

stores cumulative return in a quantum register, and uses Grover search to retrieve high-return trajectories in $$R(\alpha_i,\beta_i,\gamma_i)=R_z(\alpha_i)R_y(\beta_i)R_z(\gamma_i).$$9 steps (Su et al., 2024). This is materially different from most “quantum-inspired” DRL papers, where replay buffers, optimizers, target-network updates, and environment interaction remain classical. The distinction matters because claims about scalability, hardware readiness, and quantum advantage mean very different things across these formulations.

A third issue is generalization beyond benchmark environments. Several recent high-performing frameworks are evaluated in grid-worlds or simplified simulated domains. ARDNS-FN-Quantum and Q-ARDNS-Multi report strong success-rate and stability results, but their evaluations are confined to $$L(\theta)=\mathbb{E}\left[(r_t+\gamma \max_{a'}Q(s_{t+1},a';\theta^-)-Q(s_t,a_t;\theta))^2\right].$$0 and $$L(\theta)=\mathbb{E}\left[(r_t+\gamma \max_{a'}Q(s_{t+1},a';\theta^-)-Q(s_t,a_t;\theta))^2\right].$$1 environments (Sousa, 7 May 2025, Sousa, 2 Jun 2025). The cut-order-planning study reports up to 13% cost savings, but also states that the simulation model makes several simplifying assumptions (Chrisnanto et al., 13 Aug 2025). This suggests that some of the strongest numerical gains in the current QI-DRL literature are best read as existence proofs of useful inductive biases rather than definitive evidence of broad superiority.

The forward-looking directions identified by the papers are nonetheless coherent. Distributed prioritized replay is proposed as a route to future multiple-QPU training and to more complex tasks (Chen, 2023). Energy-based RL points toward quantum-enhanced sampling, quantum Boltzmann machines, and deep projective simulation in large structured spaces (Jerbi et al., 2019). Fully quantum RL frames quantum arithmetic and trajectory search as a foundation for future end-to-end quantum decision-making (Su et al., 2024). Hybrid actor-critic and variational-circuit studies point toward higher-dimensional control, improved encoding, and more expressive re-uploading architectures (Sinha et al., 2023, Lokossou et al., 14 Sep 2025). Across these lines, the most defensible synthesis is not that QI-DRL has already displaced classical DRL, but that it has opened several technically distinct routes—compact variational approximation, quantum-inspired replay, large-space sampling, and fully quantum trajectory evaluation—by which quantum ideas can reshape sequential decision making.