---
title: Quantum Policy Evaluation (QPE)
url: https://www.emergentmind.com/topics/quantum-policy-evaluation-qpe
type: topic
---

# Quantum Policy Evaluation (QPE)

Quantum Policy Evaluation (QPE) denotes a family of quantum reinforcement-learning procedures for evaluating a fixed policy in a Markov decision process by encoding returns, value functions, or Bellman updates into quantum states, amplitudes, phases, or linear-system instances, and then extracting value information with quantum subroutines such as amplitude estimation, phase estimation, block-encoding plus quantum linear-system algorithms, or quantum annealing. Across the literature, QPE appears both as a standalone evaluation primitive and as the evaluation stage of broader quantum policy-iteration schemes; representative realizations include finite-horizon amplitude-estimation methods, infinite-horizon discounted policy evaluation via QLSA, hybrid amplitude-encoded Bellman backups, a D-Wave QUBO embedding for finite-episode games, and a phase-estimation-based evaluator inside the Grover Policy Agent (GPA) [2206.04741], [2203.01889], [2505.11862], [1708.09354], [2502.13755].

## 1. Formal problem setting

In the finite-MDP setting used throughout the literature, an MDP is specified as $(\mathcal S,\mathcal A,P,R,\gamma)$ or $(S,A,P,r,\gamma)$, with finite state and action spaces, transition–reward distribution $P(s',r\mid s,a)$, reward set or reward function, and discount factor $\gamma\in[0,1)$ [2509.07614], [2206.04741]. For a fixed policy $\pi(a\mid s)$, QPE targets either the state-value function
$$
V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]
$$
or the action-value function $Q^\pi$, depending on the construction [2509.07614], [2203.01889].

For finite-horizon formulations, the return of a trajectory $t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$ is
$$
G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,
$$
and the objective is to estimate $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$ under the policy and transition dynamics [2206.04741]. For infinite-horizon discounted problems, one may instead write policy evaluation as a Bellman linear system. In the action-value formulation of "Quantum Reinforcement Learning via Policy Iteration" [2203.01889], the matrix $P^\pi$ and reward vector $R$ satisfy
$$
(I-\gamma P^\pi)\,Q^\pi=R,
$$
while the state-value formulation obeys
$$
V^\pi=r^\pi+\gamma P^\pi V^\pi.
$$

These formulations identify the central computational burden that QPE seeks to alter: classical Monte Carlo rollout averaging, repeated Bellman backups, or linear-system solution. In amplitude-estimation-based work, the comparison baseline is explicitly Monte Carlo policy evaluation with $O(1/\varepsilon^2)$ samples for error $\varepsilon$ [2206.04741], [2509.07614]. In QLSA-based work, the emphasis shifts to preparing a quantum state proportional to the value vector with polylogarithmic dependence on the number of state–action pairs under oracle assumptions [2203.01889]. In hybrid amplitude-encoding work, the evaluation step is framed as simultaneous quantum Bellman backups over all state–action pairs [2505.11862].

## 2. Core algorithmic constructions

A dominant construction in finite-horizon QPE encodes the normalized return of a trajectory into an ancilla amplitude. Given known bounds $g^-\le G(t_H)\le g^+$, one defines
$$
\chi(G(t_H))=\frac{G(t_H)-g^-}{g^+-g^-}\in[0,1],
$$
prepares a qsample over trajectories with unitary $M$, computes $G(t_H)$ with a return unitary $G$, and applies a controlled rotation $U_\chi$ so that the ancilla-$|1\rangle$ amplitude satisfies
$$
\alpha^2=\sum_{t_H}p(t_H\mid s)\,\chi(G(t_H))
=\frac{V^\pi(s)-g^-}{g^+-g^-}.
$$
QPE then applies phase estimation to the Grover operator
$$
Q=-\,S_0A^\dagger S_1A
$$
to estimate $\theta$ with $\sin\theta=\alpha$, recovering $\hat V^\pi(s)=g^-+(g^+-g^-)\hat\alpha^2$ [2206.04741].

A distinct construction, used in GPA, encodes the policy value directly as a phase. For $v_H(s)\in[g,\overline g]$, GPA defines
$$
\phi=\frac{v_H(s)-g}{\overline g-g}\in[0,1),
$$
implements
$$
\Phi:\;|\psi_{\rm env}\rangle\mapsto e^{2\pi i\phi}|\psi_{\rm env}\rangle,
$$
and uses a $t$-qubit counting register, Hadamards, controlled-$\Phi^{2^j}$ operations, inverse QFT, and measurement to obtain a $t$-bit estimate $\widetilde\phi\approx \phi$, hence an approximate value $\widetilde v_H(s)$ [2502.13755]. In that setting, QPE is explicitly the policy-evaluation stage of a larger Grover-based policy-search loop.

The infinite-horizon discounted line of work based on QLSA does not estimate a scalar value via amplitude estimation. Instead, it block-encodes $P^\pi$, forms $A^\pi=I-\gamma P^\pi$, and uses a quantum linear-system solver to prepare the normalized value-function state
$$
|Q^\pi\rangle=\frac{1}{\|Q^\pi\|}\sum_{s,a}Q^\pi(s,a)\,|s,a\rangle.
$$
The resulting output is an $\varepsilon$-approximation to $|(I-\gamma P^\pi)^{-1}R\rangle$, after which tomography or sampling is required for classical recovery [2203.01889].

A fourth construction, introduced by Q-Policy, amplitude-encodes the entire action-value table,
$$
|Q^\pi\rangle
=
\frac{1}{\|Q^\pi\|_2}
\sum_{s\in\mathcal S}\sum_{a\in\mathcal A}
Q^\pi(s,a)\,|\mathrm{idx}(s,a)\rangle,
$$
and applies a quantum Bellman-update unitary
$$
\mathcal U_{\rm Bellman}:\;
|\mathrm{idx}(s,a)\rangle|0\rangle
\mapsto
|\mathrm{idx}(s,a)\rangle|r(s,a)+\gamma V^\pi(s')\rangle
$$
so that all $(s,a)$ pairs are updated in parallel by linearity. Classical approximation is then obtained through quantum amplitude estimation combined with a classical control variate [2505.11862].

## 3. Methodological branches in the literature

The term QPE spans several technically distinct branches rather than a single canonical circuit. The following comparison captures the main variants described in the literature.

| Branch | Representation | Distinctive focus |
|---|---|---|
| "Quantum-enhanced reinforcement learning for finite-episode games with discrete state spaces" [1708.09354] | QUBO embedding on a D-Wave 2000Q QPU | Monte Carlo policy iteration and embedding of sub-optimal state-value functions |
| "Quantum Reinforcement Learning via Policy Iteration" [2203.01889] | Block-encoding and QLSA | Prepare an approximate $|Q^\pi\rangle$ for infinite-horizon discounted problems |
| "Quantum Policy Iteration via Amplitude Estimation and Grover Search -- Towards Quantum Advantage for Reinforcement Learning" [2206.04741] | Trajectory qsampling, amplitude estimation, Grover operator | Finite-MDP QPE with quadratic improvement over classical Monte Carlo estimation |
| "GPA: Grover Policy Agent for Generating Optimal Quantum Sensor Circuits" [2502.13755] | Phase estimation on $\Phi$ | QPE as search-space generation for Grover-style policy improvement |
| "Q-Policy: Quantum-Enhanced Policy Evaluation for Scalable Reinforcement Learning" [2505.11862] | Amplitude encoding and superposed Bellman backups | Hybrid quantum-classical policy iteration with variance-reduced readout |
| "From Classical Data to Quantum Advantage -- Quantum Policy Evaluation on Quantum Hardware" [2509.07614] | Learned $U_{\rm env}$ plus amplitude estimation on hardware | End-to-end integration of QML-learned environments and QPE |

Historically, the 2017 D-Wave work addressed finite-episode games with discrete state spaces by partially embedding Monte Carlo policy iteration and sub-optimal state-value-function aggregation as QUBO instances, with the stated aim of explaining how to represent and solve parts of these problems on a QPU rather than proving supremacy over every existing classical policy evaluation algorithm [1708.09354]. Later work shifted toward gate-model formulations with explicit asymptotic guarantees: QLSA-based policy evaluation for infinite-horizon discounted MDPs [2203.01889], amplitude-estimation-based QPE and Grover policy improvement for finite MDPs [2206.04741], and amplitude-encoded Bellman backups in a hybrid quantum-classical policy-iteration framework [2505.11862].

The GPA formulation broadens the term still further. There, QPE is not primarily introduced for classical-control benchmarks; it evaluates candidate policies that generate quantum sensor circuits, with the estimated phases serving as the marked search space for subsequent Grover-amplitude amplification [2502.13755]. This suggests that the acronym names the policy-evaluation role within a quantum RL loop rather than a unique low-level primitive. A common source of ambiguity is therefore terminological: some QPE schemes literally employ phase estimation, whereas others use amplitude estimation, QLSA, or annealing-based optimization.

## 4. Complexity claims and resource scaling

The best-known asymptotic claim attached to amplitude-estimation-based QPE is a quadratic improvement over classical Monte Carlo evaluation. In the finite-MDP construction of [2206.04741], choosing
$$
t=\left\lceil\log_2\frac{1}{\varepsilon}\right\rceil
+\left\lceil\log_2\frac{2}{\delta}\right\rceil
$$
ancillas guarantees
$$
\Pr\!\left[|\hat V^\pi(s)-V^\pi(s)|\le\varepsilon\right]\ge 1-\delta,
$$
with
$$
O\!\left(\frac{1}{\varepsilon}\log\frac{1}{\delta}\right)
$$
calls to $A$ or $A^\dagger$, compared with classical Monte Carlo
$$
O\!\left(\varepsilon^{-2}\log\frac{1}{\delta}\right)
$$
samples. The hardware-oriented formulation of [2509.07614] states the same qualitative separation as $O(1/\varepsilon)$ queries to $O_{\rm agent}$ and $O_{\rm env}$ versus $O(1/\varepsilon^2)$ classical samples.

Q-Policy derives a different scaling because it evaluates all state–action pairs through amplitude encoding and a Bellman-update oracle. Under assumptions (A1) sparsity, (A2) spectral bound, and (A3) amplitude preparation, each Bellman update costs $\widetilde O(d\,\kappa)$ gates and amplitude estimation contributes $O(1/\varepsilon)$ queries, leading to overall evaluation calls
$$
\widetilde O(d\,\kappa/\varepsilon)
$$
for $\ell_2$-accurate evaluation, versus $\mathcal O(1/\varepsilon^2)$ classically. Combined with policy-iteration convergence, the paper gives
$$
\widetilde O\bigl(d\,\kappa\,\eta^{-1}(1-\gamma)^{-2}\bigr)
$$
quantum subroutine calls to reach an $\eta$-optimal policy, versus classical $\widetilde O(\eta^{-2})$ [2505.11862].

The QLSA-based framework offers a different kind of scaling advantage. Theorem 3.1 in [2203.01889] gives circuit-depth or gate-count cost
$$
O\!\left((\mu_{P^\pi}T_{P^\pi}+T_R)\cdot \Gamma\cdot \operatorname{polylog}(\Gamma/\varepsilon)\right),
$$
where $\Gamma=1/(1-\gamma)$ and $\mu_{P^\pi}=\sqrt{c_P}$. With typical quantum-RAM or sparse-access oracles, this becomes
$$
O\!\left(\mu_{P^\pi}\Gamma\,\operatorname{polylog}(SA\,\Gamma/\varepsilon)\right).
$$
However, the same work emphasizes that extracting a fully classical vector incurs tomography or sampling overhead, including $O(SA/\varepsilon^2)$ for recovering magnitudes from measurement statistics [2203.01889].

In GPA, resource accounting is expressed in terms of counting qubits and circuit depth rather than policy-evaluation sample complexity alone. The QPE subroutine uses $t$ Hadamards, $t$ controlled-$\Phi^{2^j}$ gates, one $t$-qubit inverse QFT requiring $O(t^2)$ two-qubit rotations, and $t$ measurements, for overall depth
$$
O(t+\mathrm{depth}(\Phi)\,t+t^2).
$$
The phase-estimation error scales as $O(1/2^t)$ in the absence of noise, while the median absolute error decays roughly as $O(1/\sqrt N)$ with the number of shots $N$ [2502.13755].

These heterogeneous results matter because they attach speedup claims to different access models and output models. A plausible implication is that “QPE is quadratically faster” is accurate for amplitude-estimation-based scalar evaluation, but it is not a universal summary of every QPE formulation.

## 5. Implementations and empirical studies

The earliest reported implementation direction in this corpus is the D-Wave annealing approach of 2017. "Quantum-enhanced reinforcement learning for finite-episode games with discrete state spaces" [1708.09354] shows how to express Monte Carlo policy iteration on random observations and the embedding of $(n)$ sub-optimal state-value functions as QUBO problems on a D-Wave 2000Q QPU, and reports that quantum-enhanced Monte Carlo policy evaluation allows finding equivalent or better state-value functions for a given policy with the same number episodes compared to a purely classical Monte Carlo algorithm.

The QLSA-based policy-iteration framework was validated numerically on OpenAI Gym environments rather than hardware. For FrozenLake on $4\times4$ and $8\times8$ grids, simulated quantum noise and sampling error with $M=36\log(4XY)/\varepsilon^2$ measurements still led the quantum policy-iteration loop to converge to the classical optimum within $\le 5$ iterations for $\varepsilon=10^{-2}$. For InvertedPendulum with continuous 2D state, three actions, Fourier-basis features with $k=4$ and $K=96$ features, and $M=100$ measurements per sample, convergence was reached within $\le 8$ iterations for $\varepsilon=10^{-2}$ [2203.01889].

The amplitude-estimation-plus-Grover line provided a full implementation and simulation for a two-armed bandit MDP, intended as a proof of concept for how quantum algorithms can solve reinforcement-learning problems given access to error-free, efficient quantum realizations of the agent and environment [2206.04741]. The paper’s emphasis is formal and asymptotic rather than hardware-realistic, and it explicitly frames the result as a detailed proof of concept for combining amplitude estimation and Grover search into policy evaluation and improvement.

Q-Policy reports classical emulations on small GridWorld tasks. In a $4\times4$ grid with stochastic transitions given by 80% intended move and 20% uniform random, goal state reward $+1$, and other rewards $0$, the log-scale Bellman error decays geometrically, matching theoretical $O(\gamma^t)$ contraction; Table 1 reports query complexity per iteration of $232$ for Q-Policy and $1\,000$ for Monte Carlo, with totals at $50$ iterations of $11\,600$ and $50\,000$, respectively. The ablation over amplitude-estimation precision $\varepsilon\in\{0.001,0.01,0.05\}$ and shot counts $\{128,\dots,4096\}$ further reports that even moderate precision $(\varepsilon=0.01,\ 512\ \text{shots})$ outperforms classical in query count [2505.11862].

The most explicit hardware realization is "From Classical Data to Quantum Advantage -- Quantum Policy Evaluation on Quantum Hardware" [2509.07614]. There, a 2-armed bandit environment is first learned from classical observational data through QML and then used for QPE on IonQ forte-1. The hardware description specifies 11 fully-connected qubits, single- and two-qubit gate fidelities of approximately 99.9% and 98%, coherence times $T_2\approx 30\,\mathrm s$, daily randomized benchmarking and crosstalk calibration, and FireOpal pulse-level detuning plus zero-noise extrapolation for error mitigation. The reported QPE circuits use 5 qubits and depth approximately 412 gates for $n=3$, and 6 qubits and approximately 883 gates for $n=4$. For the 70%/20% bandit case, the learned parameters satisfy $\theta^\leftarrow_{\rm learned}=1.96$ versus $\theta^\leftarrow_{\rm emp}=1.98$ and $\theta^\rightarrow_{\rm learned}=0.91$ versus $\theta^\rightarrow_{\rm emp}=0.93$, with MSE loss approximately $0.015$. For policy evaluation, true values are $v_{50}=0.45$ and $v_0=0.20$; RMSE on hardware is approximately $0.07$ for $\pi_{50}$ at $n=3$, approximately $0.05$ at $n=4$, and approximately $0.01$ for $\pi_0$ even on hardware [2509.07614].

GPA provides a different experimental target: quantum sensor circuits rather than classical control. In its reported QPE experiments, the authors used $t=4$ and obtained a value-function estimate of $0.99$ in $4096$ shots for a 12-qubit QPE instance, embedded in a loop that seeks circuits maximizing Quantum Fisher Information while minimizing the number of gates [2502.13755].

## 6. Assumptions, limitations, and recurrent points of interpretation

Nearly all QPE proposals derive their advantages from strong oracle and state-preparation assumptions. The finite-MDP amplitude-estimation framework assumes fault-tolerant, error-free implementation of the policy unitary $U_\pi$, environment unitary $U_E$, return arithmetic $G$, controlled powers of the Grover operator $Q$, and inverse QFT; it further notes that access to coherent sampling oracles is essential and that inefficient state preparation can erase the quadratic advantage [2206.04741]. The QLSA framework likewise depends on quantum row/column oracles for $P$, a state-preparation oracle for $R$, low-depth block-encoding constructions, and manageable condition number and normalization factors [2203.01889].

The hybrid amplitude-encoded framework makes its assumptions fully explicit. Q-Policy requires (A1) sparsity, namely that each $P(\cdot\mid s,a)$ has at most $d$ nonzero entries; (A2) a spectral bound $\|T_\pi\|\le\kappa<1/\gamma$; and (A3) QRAM-like amplitude preparation with $\widetilde O(1)$ depth per nonzero. The same paper states that these assumptions may not hold in dense or continuous-action MDPs without efficient sparse data structures, and that quantum readout still requires $O(|S||A|)$ measurements unless further compression or clever sampling is applied [2505.11862].

Hardware studies make the noise sensitivity concrete. In the IonQ bandit experiments, deeper circuits with larger $n$ suffer from gate errors and decoherence, yielding a trade-off between $n$ and fidelity; deterministic or low-branching policies are more robust; and FireOpal improved bandit fit but only partially recovered deep-circuit QPE accuracy. The paper therefore concludes that current hardware noise limits QPE to shallow $n$ and simple policies, even though noiseless regimes still exhibit the $O(1/\varepsilon)$ versus $O(1/\varepsilon^2)$ sample-complexity separation [2509.07614]. GPA states the same issue in circuit-design terms: more counting qubits increase precision but lengthen circuits, and deeper circuits are more prone to decoherence and gate errors on near-term hardware [2502.13755].

A further interpretive point concerns output format. Some QPE algorithms estimate a scalar $V^\pi(s)$ for a designated initial state, some prepare a normalized quantum state $|Q^\pi\rangle$, and some produce a search space over candidate-policy values for a Grover-style improvement stage. This suggests that “policy evaluation” in the quantum literature is task-level terminology. The concrete computational object being produced—an ancilla amplitude, a phase, an amplitude-encoded value table, or a QLSA output state—depends on the surrounding policy-iteration architecture.

Taken together, the literature presents QPE not as a single algorithm but as a structured design space. The common thread is the replacement of classical rollout averaging or classical Bellman evaluation by quantum-native encodings of policy value. The main divergence lies in the computational primitive chosen to expose that value—annealing, amplitude estimation, phase estimation, or linear-system solution—and in the assumptions under which the resulting complexity improvements are claimed.

Source: https://www.emergentmind.com/topics/quantum-policy-evaluation-qpe