Quantum Annealing Enhanced Q-Learning

• QAQL is a reinforcement learning framework that embeds quantum annealing to replace classical value approximators or greedy action selectors, using QBM free energy estimates or one-hot QUBO formulations.
• It incorporates stabilizers like experience replay and target networks to mitigate sampling noise and enhance convergence in both discrete and continuous control tasks.
• QAQL has demonstrated effectiveness in grid-world, multi-agent, CERN beam-line, and predictive maintenance applications, showcasing its potential in real-world and industrial settings.

Quantum Annealing Enhanced Q-Learning (QAQL) denotes a class of value-based reinforcement-learning methods in which quantum annealing, or its finite-temperature surrogate simulated quantum annealing (SQA), is inserted into the Q-learning loop to estimate action values or to realize the state-conditioned action-selection step itself. In the literature considered here, QAQL has two principal instantiations. In the first, the action-value function is approximated by the negative free energy of a clamped quantum Boltzmann machine (QBM), so that $Q(s,a)\approx -F(s,a)$ and temporal-difference learning is driven by annealer-estimated spin expectations. In the second, the greedy action-selection subproblem is encoded directly as a small one-hot QUBO whose low-energy samples provide both the greedy action and structured exploration. These lines of work connect early QBM-based reinforcement learning, later DQN-style stabilization and multi-agent extensions, hybrid actor-critic schemes for continuous control, and recent industrial predictive-maintenance applications on D-Wave hardware (Crawford et al., 2016, Müller et al., 2021, Schenk et al., 2022, Gandhudi et al., 16 Jun 2026).

1. Historical development and conceptual scope

The earliest formulation in this lineage is the QBM-RL framework of “Reinforcement Learning Using Quantum Boltzmann Machines” (Crawford et al., 2016). That work cast reinforcement learning as off-policy TD(0) Q-learning with a Boltzmann-machine value approximator, clamped state and action visibles, and SQA-based sampling from a transverse-field Ising model. It reported that, on clear and windy 3×5 gridworld mazes, QBM-RL outperforms DBM-RL, which in turn outperforms RBM-RL, with fidelity curves showing faster rise and higher plateau for the quantum model. The same work also argued that DBM-like topologies are more expressive than RBMs for these RL tasks and that a significant transverse field improves sampling quality in rugged energy landscapes (Crawford et al., 2016).

The 2021 multi-agent study “Towards Multi-Agent Reinforcement Learning using Quantum Boltzmann Machines” (Müller et al., 2021) explicitly frames the method as Quantum Annealing Enhanced Q-Learning and extends the original QBM-based idea in two directions. First, it imports two DQN stabilizers—experience replay and a target network—into the QAQL loop. Second, it moves from single-agent and very small multi-agent domains to more challenging grid worlds, including fully cooperative two-agent settings and parameter sharing. Its central claim is not that annealing replaces RL, but that annealing replaces the standard parametric Q-value approximator and the associated expectation calculations.

The 2022 CERN beam-line paper “Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines” (Schenk et al., 2022) uses the term free energy-based reinforcement learning (FERL) with clamped QBM rather than QAQL, but the operative construction is the same: a QBM-based critic estimates $Q(s,a)$ through quantum free energy, and annealing-based samples supply the sufficient statistics needed for TD learning. That work extends the approach from discrete state-action spaces to continuous states with discrete actions and then to continuous state-action spaces through a DDPG-inspired hybrid actor-critic in which the actor is classical and the critic is a QBM.

A more recent development changes where the annealer enters the algorithm. “Quantum Annealing Enhanced Reinforcement Learning for Accurate Remaining Useful Lifetime Prediction” (Gandhudi et al., 16 Jun 2026) uses QAQL to denote a tabular Q-learning method in which the value update remains classical but each statewise action-selection step is solved as a one-hot QUBO on D-Wave Advantage. Here the annealer is not estimating QBM free energies; it is sampling low-energy actions from a state-conditioned QUBO, and the resulting empirical distribution supplies exploration inside the RL loop.

These works suggest that QAQL is better understood as a methodological family than as a single canonical algorithm. Its unifying feature is the replacement of a classical inner subroutine of Q-learning—either value approximation or greedy action selection—by annealing-based sampling.

2. Energy-based formulation with quantum Boltzmann machines

In the QBM-based branch of QAQL, the visible layer encodes the state-action pair and the hidden spins mediate the energy landscape. The 2021 multi-agent formulation divides visibles into state units $v_s$ and action units $v_a$, with hidden layers $H=\{h\}$ providing stochastic representations. One example topology uses a visible layer with seven state and five action neurons and three hidden layers with four neurons each. Grid-world observations are one-hot encoded and split across two visible “layers”: one for the agent’s position and its own goal, and one for the positions and goals of other agents. The resulting input shape is $n\times m\times 2$ for an $n\times m$ grid, with an additional layer in the multi-agent case (Müller et al., 2021).

The clamped QBM Hamiltonian used for annealing-based sampling has the transverse-field Ising form

$$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$

with visible variables entering linearly as biases on hidden spins, hidden-hidden couplings capturing correlations, and $\Gamma$ controlling the transverse field. The associated quantum free energy is

$$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$

and the central approximation is

$Q(s,a)$0

In the CERN formulation, the effective free energy includes an entropy term estimated from annealer samples,

$Q(s,a)$1

where $Q(s,a)$2 ranges over sampled configurations (Schenk et al., 2022).

Because direct measurement of the transverse-field term in the $Q(s,a)$3 basis destroys the $Q(s,a)$4-basis information needed for RL updates, both QBM-based lines adopt a Suzuki–Trotter mapping. The transverse-field model is converted into an effective classical Ising system with replica stacking, in which inter-replica couplings encode the quantum contribution. In the CERN paper this coupling is parameterized by

$Q(s,a)$5

and the annealer or SQA sampler estimates moments and empirical configuration frequencies from the replicated system (Schenk et al., 2022).

The 2016 and 2021 papers use the same identification of the value function with negative free energy, but differ in emphasis. The earlier work places the QBM inside an off-policy TD(0) Q-learning rule and compares QBM, DBM, and RBM variants. The 2021 work specializes the same idea to grid-world MDPs and stochastic games, with annealing-based expectations entering a stabilized QAQL loop. It also presents a free-energy expression tied to the MDP transition model,

$Q(s,a)$6

again with the approximation $Q(s,a)$7 (Müller et al., 2021).

State and action encoding are not restricted to one-hot discrete visibles. In the CERN beam-line setting, continuous state components such as BPM positions are fed directly as real-valued visible variables, and continuous actions enter as real-valued visibles in the hybrid actor-critic formulation. This is the principal mechanism by which the QBM-based QAQL line extends beyond discrete tabular domains (Schenk et al., 2022).

3. Learning rules, sampling, and DQN-style stabilization

The QBM-based QAQL update rule follows the gradient identity for free energy: derivatives of the free energy with respect to parameters reduce to expectation values of observables under the clamped quantum state. In the 2016 formulation, the TD(0) error is

$Q(s,a)$8

and the parameter updates are

$Q(s,a)$9

These expectations are obtained from SQA samples of the clamped transverse-field Ising model, rather than from exact summation or classical Gibbs sampling (Crawford et al., 2016).

The 2021 multi-agent QAQL paper replaces the simple online loop by a DQN-like training pipeline. Action selection is $v_s$0-greedy, with $v_s$1 decaying by $v_s$2 per step to $v_s$3. Two QBMs are maintained: a policy QBM and a target QBM. The policy network is updated every training step; the target network is synchronized with hard updates every $v_s$4 steps. Transitions $v_s$5 are stored in a replay buffer $v_s$6, and uniformly sampled mini-batches are used to reduce correlation and non-stationarity. The temporal-difference rule is SARSA-style rather than max-over-actions DQN-style,

$v_s$7

with updates proportional to $v_s$8 times annealer-estimated moments $v_s$9 and $v_a$0 (Müller et al., 2021).

Sampling remains the computational bottleneck and the source of the quantum enhancement. In the 2021 work, for each clamped pair $v_a$1 in a mini-batch, the annealing-based sampler returns $v_a$2 spin configurations across $v_a$3 replicas, and expectations are formed by averaging over those reads. The paper uses SQA rather than a physical QPU and does not report vendor/model, qubit count, or embedding details. In the 2016 work, SQA settings are explicit: the transverse field is linearly decreased from $v_a$4 to either $v_a$5 or $v_a$6, the inverse temperature is fixed at $v_a$7, the number of Trotter slices is $v_a$8, sweeps are $v_a$9 per run, and the number of independent runs per input instance is $H=\{h\}$0 (Crawford et al., 2016, Müller et al., 2021).

The CERN formulation preserves replay-based critic training but generalizes the surrounding RL stack. For discrete action spaces and continuous states, FERL performs Q-learning with experience replay. For continuous state-action spaces, it adopts a DDPG-style hybrid actor-critic in which the critic remains $H=\{h\}$1 and the actor is trained by

$H=\{h\}$2

Because the entropy derivative is not tractable, $H=\{h\}$3 is estimated numerically by finite differences rather than analytically (Schenk et al., 2022).

Across these variants, the central stabilization mechanism is not quantum-specific. Replay buffers, target networks, hard or soft target synchronization, and actor-target pairs are classical RL devices imported into annealing-based value learning. The quantum ingredient lies in the sampling of hidden-spin configurations or low-energy action configurations, not in a replacement of TD learning itself.

4. Multi-agent and continuous-control extensions

The 2021 QAQL extension studies fully cooperative stochastic games $H=\{h\}$4 with homogeneous rewards and simultaneous actions. In the reported experiments $H=\{h\}$5 agents act in 3×3 and 5×3 grid worlds. The architecture does not learn a centralized joint action-value $H=\{h\}$6; instead, each agent approximates its own $H=\{h\}$7, while rewards are shared. Two training regimes are evaluated: independent learners, each with its own QBM and replay buffer, and parameter sharing, in which a single centralized QBM and shared replay buffer are trained from all agents’ transitions and then queried independently by each agent (Müller et al., 2021).

The single-agent results establish the role of DQN-style stabilizers. On the 3×3 grid, the original QAQL/QBM agent without replay and target network learns unstably, with large negative swings of $H=\{h\}$8 and $H=\{h\}$9. Adding experience replay alone or a target network alone stabilizes training; combining both yields the smoothest curve and near-optimal policies. Across $n\times m\times 2$0 test episodes, median rewards approach the optimal level of $n\times m\times 2$1 expected median for 3×3. On the 5×3 grid, instability is more severe, but ERB or target network alone improve stability and median reward, and ERB+target yields the best performance, with median near the task’s optimal $n\times m\times 2$2 expected median (Müller et al., 2021).

The multi-agent picture is more heterogeneous. In the 3×3 two-agent grid, ERB+target training converges and the median test reward is approximately $n\times m\times 2$3, near the summed per-agent optimum. In the 5×3 two-agent grid, the extended QAQL setup does not converge; the median test reward is approximately $n\times m\times 2$4 and training remains unstable. Parameter sharing has a negative effect on the small 3×3 domain but a modest positive effect on the larger 5×3 domain, although the latter remains unsolved. The authors hypothesize that mixing independent optimal behavior distributions may interfere destructively during centralized training (Müller et al., 2021).

The CERN line generalizes QAQL beyond discrete grid worlds. In the TT24–T4 one-dimensional beam-steering task, the state is a continuous BPM horizontal position reading and the action is a discrete choice to increase or decrease dipole deflection by $n\times m\times 2$5, with deflection angle range $n\times m\times 2$6. In that setting, with experience replay, FERL reaches $n\times m\times 2$7 policy optimality in approximately $n\times m\times 2$8 training steps, whereas classical DQN requires approximately $n\times m\times 2$9 steps. The FERL QBM uses $n\times m$0 trainable weights and $n\times m$1 qubits in a $n\times m$2 Chimera-unit-cell design, while the best DQN uses approximately $n\times m$3 weights across two hidden layers. The paper also reports that DQN improves by about $n\times m$4 in sample efficiency when replay is enabled, whereas FERL improves by about $n\times m$5 (Schenk et al., 2022).

For continuous actions, the hybrid actor-critic is evaluated on the 10D AWAKE beam line. The state is a 10D vector of horizontal BPM position deviations relative to a reference trajectory, and the action is a 10D vector of corrector kick angles in $n\times m$6. Under SQA-based training, the hybrid actor-critic reaches near-perfect behavior after about $n\times m$7 agent-environment interactions, while classical DDPG requires about $n\times m$8 interactions. With D-Wave Advantage training and a relaxed reward objective of $n\times m$9 mm, the agent consistently reaches the objective by about episode $$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$0; in evaluation, $$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$1 of cases are solved in $$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$2 step, $$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$3 in $$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$4, and $$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$5 in $$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$6, and $$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$7 achieve $$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$8 mm. The same work also reports successful sim-to-real deployment on the actual AWAKE beam line without further training, with one failed episode for the QA-trained agent in the real environment (Schenk et al., 2022).

A plausible implication is that QBM-based QAQL is strongest when the state-action space remains moderate, the energy landscape is rugged enough that sampling quality matters, and classical RL stabilizers can reduce the variance induced by annealing-based expectation estimates.

5. QUBO-based QAQL for predictive maintenance

The 2026 predictive-maintenance work departs from the QBM/free-energy tradition and inserts the annealer directly into the greedy action-selection step of tabular Q-learning. Remaining useful life (RUL) estimation is formulated as episodic model-free RL, with each run-to-failure trajectory constituting an episode. The action space is a discretized RUL prediction with $$\mathcal{H}(\mathbf{v})= -\sum_{\substack{i\in V,\ j\in H}} w_{ij} v_i \sigma^z_{h_j} -\sum_{j,k\in H} w_{jk}\sigma^z_{h_j}\sigma^z_{h_k} -\Gamma \sum_{j\in H}\sigma^x_{h_j},$$9 bins $\Gamma$0, each with representative center $\Gamma$1, and the reward is the negative per-step squared prediction error,

$\Gamma$2

The Q-learning update remains classical,

$\Gamma$3

but the action $\Gamma$4 is selected by solving a one-hot QUBO on D-Wave Advantage (Gandhudi et al., 16 Jun 2026).

For a state $\Gamma$5, the linear costs are defined by current Q-values,

$\Gamma$6

and the one-hot action-selection QUBO is

$\Gamma$7

with binary decision variables $\Gamma$8, one-hot feasibility $\Gamma$9, and implementation choice $$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$0. The lowest-energy sample identifies the greedy action; the empirical distribution over near-ground-state samples supplies exploration. The paper makes one QPU call per environment step and computes the TD target $$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$1 classically to avoid a second QPU call per step (Gandhudi et al., 16 Jun 2026).

The hardware configuration is explicit: D-Wave Advantage_system6.4 on Pegasus P16 topology with 5,640 qubits, Ocean SDK with EmbeddingComposite over DWaveSampler, heuristic minor embedding by minorminer, $$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$2, $$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$3, $$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$4 with prefactor $$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$5, $$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$6, and $$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$7. For $$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$8 binary variables the embeddings are shallow with short chains, and no custom anneal schedules or offsets are activated (Gandhudi et al., 16 Jun 2026).

The reported benchmark results are summarized below.

Dataset	QAQL RMSE	Selected comparators
C-MAPSS FD001	$$F_q(\mathbf{v};\theta)= -\frac{1}{\beta}\ln \operatorname{Tr}\exp\!\big(-\beta\,\mathcal{H}(\mathbf{v};\theta)\big),$$9	Transformer $Q(s,a)$00; Quantum VQL $Q(s,a)$01
Device-fleet	$Q(s,a)$02	Transformer $Q(s,a)$03; Quantum VQL $Q(s,a)$04

On FD001, QAQL reports $Q(s,a)$05, $Q(s,a)$06, $Q(s,a)$07, $Q(s,a)$08, and $Q(s,a)$09. On the device-fleet benchmark, it reports $Q(s,a)$10, $Q(s,a)$11, $Q(s,a)$12, $Q(s,a)$13, and $Q(s,a)$14. On the multi-condition C-MAPSS subsets, the reported QAQL-only RMSE values are $Q(s,a)$15 for FD002, $Q(s,a)$16 for FD003, and $Q(s,a)$17 for FD004. A paired Wilcoxon signed-rank test of RMSE across 30 seeds finds QAQL significantly better than Transformer and Quantum VQL on both main datasets, with $Q(s,a)$18 and Cliff’s $Q(s,a)$19 (Gandhudi et al., 16 Jun 2026).

The paper also provides operational ablations. Removing quantum annealing increases RMSE from $Q(s,a)$20 to $Q(s,a)$21 on FD001 and from $Q(s,a)$22 to $Q(s,a)$23 on device-fleet. Removing TD backup and using immediate reward only gives RMSE $Q(s,a)$24 on FD001 and $Q(s,a)$25 on device-fleet. Wall-clock training time per run is $Q(s,a)$26 s for FD001 and $Q(s,a)$27 s for device-fleet, versus $Q(s,a)$28 s and $Q(s,a)$29 s for the Transformer baseline. The measured average end-to-end QPU access time per update is approximately $Q(s,a)$30 ms, while inference is classical and takes approximately $Q(s,a)$31 ms per prediction on CPU (Gandhudi et al., 16 Jun 2026).

This formulation is notable because it does not treat the annealer as a post hoc optimizer. The annealer is woven into the RL loop at every action-selection step, and the sampled low-energy action distribution is itself the exploration mechanism.

6. Limitations, adjacent methods, and unresolved questions

Across the QBM-based branch, the dominant limitation is sampling capacity. The 2021 multi-agent paper states that MARL with QBMs is currently constrained by QPU size and therefore by the size of the input and Boltzmann machine. As grid size and agent count increase, visible dimensionality grows—for example, to 30 visible inputs in the 5×3 multi-agent case—and the corresponding QUBO/Ising instance requires more spins and couplings. Larger models make sampling harder, increase anneal time or the number of SQA sweeps and reads, and degrade solution quality if embedded naively on hardware with limited connectivity. The same paper recommends experience replay and separate policy/target QBMs, modest visible dimensions, tuning of $Q(s,a)$32, $Q(s,a)$33, $Q(s,a)$34, and $Q(s,a)$35, and, for QPU implementations, embeddings that minimize chain length together with gauge averaging and chain-break monitoring (Müller et al., 2021).

The CERN line adds further practical constraints. QBM size is bounded by hardware and embedding: the TT24–T4 task uses a 16-qubit QBM, while the 10D AWAKE task uses a 128-qubit QBM. Extensive hyperparameter search is required, including replay fraction, learning rate, batch size, discount $Q(s,a)$36, target-net soft update factor, inverse annealing temperature $Q(s,a)$37, and final transverse field $Q(s,a)$38. The actor update in the hybrid scheme uses finite differences of $Q(s,a)$39 with respect to $Q(s,a)$40, which introduces noise and computational overhead, and the entropy derivative is not tractable without strong assumptions (Schenk et al., 2022).

The predictive-maintenance QAQL formulation has a different failure mode. One-hot QUBOs with $Q(s,a)$41 actions are trivially embeddable, but scaling $Q(s,a)$42 upward increases chain length and noise susceptibility; extremely large action sets or richer combinatorial action structures would require decomposition or hybrid solvers. The penalty weight must satisfy $Q(s,a)$43, but too large a $Q(s,a)$44 inflates coefficient dynamic range and stresses embeddings. The paper also notes that its environment has exogenous transitions: actions do not affect future states, so the problem is a contextual prediction problem in MDP form rather than full control (Gandhudi et al., 16 Jun 2026).

A recurrent source of confusion is the boundary between annealing-based QAQL and other quantum accelerators for RL. “A hybrid classical-quantum approach to speed-up Q-learning” (Sannia et al., 2022) is related in purpose but not in mechanism: it is a gate-model algorithm that prepares action-selection distributions on a quantum register using amplitude amplification and quantum counting, with theoretical action-selection complexity improving from $Q(s,a)$45 classically to $Q(s,a)$46 quantumly for fixed numbers of classes. The paper itself distinguishes this from annealing: annealing samples from Boltzmann-like or low-energy distributions over Ising/QUBO landscapes, whereas the gate-model routine prepares a targeted amplitude distribution over actions. It is therefore adjacent to QAQL, not an instance of it (Sannia et al., 2022).

The literature nevertheless points toward a coherent engineering lesson. QAQL is most beneficial in small-to-moderate state-action spaces where energy landscapes are rugged, where sample quality materially affects TD updates or action selection, and where annealing-based low-energy sampling can replace either classical Gibbs sampling or hand-designed exploration schedules. It is least mature when scaling to large multi-agent domains, dense embeddings, or continuous-control settings that demand expensive finite-difference actor gradients. The combined evidence does not establish a universal advantage over classical RL, but it does show that quantum annealing can function as a usable internal component of a reinforcement-learning loop rather than merely as an external optimizer (Crawford et al., 2016, Müller et al., 2021, Schenk et al., 2022, Gandhudi et al., 16 Jun 2026).