Grover's Quantum Reinforcement Learning Framework

• Grover's Search-Inspired QRL frameworks are advanced quantum reinforcement learning models that encode classical MDPs into qubit registers to enable simultaneous exploration of trajectories.
• They integrate quantum arithmetic and Grover-based amplitude amplification, using diffusion operators to accelerate policy evaluation and improvement with quadratic speedup.
• Domain-adapted implementations, such as in massive MIMO scheduling, demonstrate practical benefits including improved convergence rates and reduced exploration costs.

Grover's Search-Inspired Quantum Reinforcement Learning (QRL) Frameworks realize quantum speedup for reinforcement learning tasks by embedding amplitude amplification, a core feature of Grover's quantum search algorithm, into the Markov Decision Process (MDP) formalism and algorithmic machinery of RL. These frameworks map classical RL constructs—states, actions, transitions, rewards—onto quantum registers and operations, enabling the preparation, evaluation, and search over exponentially large spaces of trajectories or policies in superposition, with key computational bottlenecks accelerated quadratically relative to classical enumerative approaches. Variants span foundational circuit architectures, dynamic qubit reuse strategies for hardware efficiency, and domain-adapted implementations in wireless communications, all demonstrating the practical integration of Grover-style oracles and diffusion operators as quantum policy improvement or trajectory optimization subroutines.

1. Quantum Representation of Reinforcement Learning

Grover-inspired QRL frameworks convert classical MDP structures into a fully quantum paradigm by encoding the sets of states $S$ and actions $A$ into qubit registers, typically allocating $\lceil \log_2|S| \rceil$ and $\lceil \log_2|A| \rceil$ qubits, respectively. The global agent-environment state at time $t$ is thus prepared as a superposition

$$|\psi_t\rangle = \sum_{s\in S}\sum_{a\in A} \alpha_{s,a}^{(t)}|s,a\rangle,$$

where the amplitude $\alpha_{s,a}^{(t)}$ allows parallel representation and exploration of all state-action pairs. Transition dynamics $P(s'|s,a)$ are loaded onto the quantum register via controlled-rotation gates, with $\theta_{s,a,s'} = 2\arcsin\sqrt{P(s'|s,a)}$ used in $CR_y$ operations. Reward functions $r(s,a)$ are embedded using multi-controlled CNOTs or similar conditional logic, flipping a reward qubit only for the appropriate transitions, thus establishing quantum oracular access to the reward structure (Su et al., 2024, Su et al., 19 Sep 2025, Fan et al., 28 Jan 2026).

Trajectory registers composed of sequences $(s_0,a_0,\dots,s_{H-1},a_{H-1})$ for horizon $H$ aggregate the outcomes over time, with the entire ensemble of possible agent-environment evolutions efficiently representable and manipulable in one quantum state vector. This quantum encoding is foundational to achieving an exponential compression of the classical search space, upon which amplitude amplification acts.

2. Quantum Arithmetic for Return Evaluation

To perform return evaluation directly on quantum hardware, frameworks deploy quantum adders (such as the quantum ripple-carry adder) to accumulate discounted rewards along each sampled trajectory in superposition. For each trajectory $\tau = (r_0, ..., r_{H-1})$, the discounted return

$G(\tau) = \sum_{t=0}^{H-1} \gamma^t r_t$

is computed in a dedicated register $|g\rangle$ using unitary $U_G$, implemented via sequences of CNOT and Toffoli (controlled-controlled-NOT) gates conditioned on reward qubits at each time step. This process produces a joint quantum state

$$\sum_{\tau} \alpha_{\tau}|\tau\rangle|G(\tau)\rangle,$$

such that for any operation targeting the return value (e.g., thresholding by oracle), all trajectory returns are instantly accessible for amplitude amplification, bypassing the need for sequential evaluation (Su et al., 2024, Su et al., 19 Sep 2025, Wiedemann et al., 2022).

3. Grover-Based Amplitude Amplification for Trajectory Optimization

The core computational enhancement arises from leveraging Grover's search and amplitude amplification. The trajectory or policy register is marked using an oracle $U_\omega$ or $O_G$ that implements a conditional phase flip if a quantum-evaluated return exceeds a predefined threshold (e.g., returns $G(\tau) \geq g^*$ or optimal policies with $v(P) > v_{cur}+\epsilon$). This is followed by a diffusion (inversion-about-the-mean) operator $U_s$ or $D$, implementing

$U_s = 2|\psi_s\rangle\langle \psi_s| - I,$

where $|\psi_s\rangle$ is the uniform superposition over all candidates. Repeated application of the Grover iterate $G = U_s U_\omega$ amplifies the amplitude of "good" trajectories or policies to $O(1)$ within $O(\sqrt{N/M})$ oracle queries, where $N$ is the space size and $M$ the number of marked solutions.

This mechanism is embedded variously as a direct search over trajectories (finding optimal episodic returns) (Su et al., 2024, Su et al., 19 Sep 2025), over scheduling vectors in wireless user allocation (Fan et al., 28 Jan 2026), or over deterministic policy label registers in abstract QPI settings (Wiedemann et al., 2022). Quantum policy improvement, in turn, is realized by interleaving Grover search for high-value policies with amplitude estimation-based quantum policy evaluation (Wiedemann et al., 2022).

Quantum Operation	Function in QRL	Example Reference
$CR_y(\theta_{s,a,s'})$	Encodes transition probabilities	(Su et al., 2024)
Multi-controlled CNOT/CZ	Marks rewards or oracle activation	(Su et al., 2024, Fan et al., 28 Jan 2026)
Quantum adder (Toffoli/CNOT)	Computes discounted trajectory return	(Su et al., 2024, Su et al., 19 Sep 2025)
Grover oracle $O_G$	Marks high-return trajectories/policies	(Fan et al., 28 Jan 2026, Wiedemann et al., 2022)
Diffusion $D$/$U_s$	Amplifies "good" solution amplitudes	(Su et al., 2024, Fan et al., 28 Jan 2026)

4. Circuit Architectures and Hardware-Adapted Implementations

State-of-the-art frameworks address the substantial quantum hardware overhead of naively simulating long-horizon RL tasks through dynamic-circuit strategies. In (Su et al., 19 Sep 2025), qubit reuse is achieved via mid-circuit measurement and reset—allowing the same physical set of 7 qubits (for three time steps: 2 for state, 1 for action, 2 for next-state, 1 for reward, 1 for return) to sequentially process $T$ time steps, a substantial reduction from $7T$ in static designs. Propagation of state between steps is enabled by classical measurement/copyback of next-state registers into current state inputs. This dynamic-circuit design preserves the overall unitary and stochastic trajectory evolution, maintaining both computational efficiency and logical fidelity on contemporary superconducting platforms (IBM Heron-class) (Su et al., 19 Sep 2025).

Gate complexity scales as $O(H (n_S+n_A + n_g))$, with depth dominated by quantum addition and multi-controlled oracles. The principal practical limitation remains the fidelity and decoherence time of physical qubits, as multi-qubit controlled phase operations are especially noise-sensitive (Su et al., 19 Sep 2025, Fan et al., 28 Jan 2026).

5. Domain-Adapted Grover-QRL: User Scheduling in Massive MIMO

A direct domain instantiation is the Grover's-search-inspired QRL for massive MIMO user scheduling (Fan et al., 28 Jan 2026). Here, the action space is the combinatorial set $$\{\boldsymbol\theta\in\{0,1\}^T: \sum_i \theta_i \leq A\}$$, representing all possible user allocations under antenna constraints. The quantum circuit implements a layered QRL architecture: initial Hadamards generate uniform schedulings, a quantum oracle marks high-reward (high sum-rate) configurations using the proportional fairness metric, and diffusion amplifies the allocation vectors with maximal RL reward. The resulting algorithm achieves a quadratic speedup in exploration over classical tabular or deep learning baselines. Empirically, amplitude amplification embedded in the Bellman update produces marked throughput improvement (up to 51% over CNN and 43% over quantum deep learning) and near-classical convergence with significantly reduced exploration cost (Fan et al., 28 Jan 2026).

6. Performance, Scalability, and Limitations

Across experimental and simulated benchmarks, Grover-inspired QRLs consistently exhibit quadratic speedup in trajectory or policy search, both in query complexity (oracle calls) and in empirical sample complexity when mapped onto simulators or NISQ devices (Su et al., 2024, Su et al., 19 Sep 2025, Wiedemann et al., 2022, Fan et al., 28 Jan 2026). Full examples include trajectory amplification for finite-state MDPs and hardware-realized multi-step optimization exhibiting robust trajectory fidelity (near-unit trace distance between static and dynamic circuits). Simulation studies in communication settings demonstrate improved convergence, sum-rate scaling, and resilience to increasing problem size.

However, implementation on near-term hardware is hindered by the gate complexity of multi-controlled unitaries and the decoherence budget necessary for $\sqrt{N}$-step Grover procedures in high-dimensional spaces. Efficient construction of quantum oracles for complex reward functions and scalable diffusion operator realizations remain open challenges. Error-mitigation strategies and hybrid quantum-classical updates are viable future directions (Su et al., 19 Sep 2025, Fan et al., 28 Jan 2026).

7. Relations to Broader Quantum RL Paradigms

Grover-accelerated QRL can be contrasted with variational or quantum deep RL approaches, which leverage parameterized quantum circuits for policy approximation but lack oracle-based search and thus do not realize the same quadratic amplification. In frameworks such as (Wiedemann et al., 2022), amplitude estimation is combined with Grover iterations for full quantum policy iteration, providing sample-complexity reduction for both policy evaluation and improvement in small-scale RL tasks. Autonomous quantum agents trained via episodic, reward-based RL are similarly observed to rediscover Grover's search as an optimal solution, highlighting the versatility of the RL framework as a discovery mechanism for quantum algorithms themselves (Kerenidis et al., 9 Oct 2025).

These developments collectively position Grover’s search-inspired QRL as a key route to scalable, quantum-accelerated reinforcement learning for sequential decision-making in the quantum computation era.