Quantum Policy Evaluation (QPE)

• Quantum Policy Evaluation (QPE) is a quantum reinforcement learning approach that encodes returns, value functions, or Bellman updates into quantum states using techniques like amplitude and phase estimation.
• QPE methods cover finite-horizon and infinite-horizon Markov decision processes by replacing classical rollouts or Bellman backups with quantum subroutines such as QLSA and Grover-based search.
• Implementations range from D-Wave QUBO embeddings to gate-model quantum circuits, demonstrating promise in achieving quadratic improvements under specific oracle and state-preparation assumptions.

Quantum Policy Evaluation (QPE) denotes a family of quantum reinforcement-learning procedures for evaluating a fixed policy in a Markov decision process by encoding returns, value functions, or Bellman updates into quantum states, amplitudes, phases, or linear-system instances, and then extracting value information with quantum subroutines such as amplitude estimation, phase estimation, block-encoding plus quantum linear-system algorithms, or quantum annealing. Across the literature, QPE appears both as a standalone evaluation primitive and as the evaluation stage of broader quantum policy-iteration schemes; representative realizations include finite-horizon amplitude-estimation methods, infinite-horizon discounted policy evaluation via QLSA, hybrid amplitude-encoded Bellman backups, a D-Wave QUBO embedding for finite-episode games, and a phase-estimation-based evaluator inside the Grover Policy Agent (GPA) (Wiedemann et al., 2022, Cherrat et al., 2022, Cherukuri et al., 17 May 2025, Neukart et al., 2017, Alomari et al., 19 Feb 2025).

1. Formal problem setting

In the finite-MDP setting used throughout the literature, an MDP is specified as $(\mathcal S,\mathcal A,P,R,\gamma)$ or $(S,A,P,r,\gamma)$, with finite state and action spaces, transition–reward distribution $P(s',r\mid s,a)$, reward set or reward function, and discount factor $\gamma\in[0,1)$ (Hein et al., 9 Sep 2025, Wiedemann et al., 2022). For a fixed policy $\pi(a\mid s)$, QPE targets either the state-value function

$$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$

or the action-value function $Q^\pi$, depending on the construction (Hein et al., 9 Sep 2025, Cherrat et al., 2022).

For finite-horizon formulations, the return of a trajectory $t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$ is

$G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$

and the objective is to estimate $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$ under the policy and transition dynamics (Wiedemann et al., 2022). For infinite-horizon discounted problems, one may instead write policy evaluation as a Bellman linear system. In the action-value formulation of "Quantum Reinforcement Learning via Policy Iteration" (Cherrat et al., 2022), the matrix $(S,A,P,r,\gamma)$0 and reward vector $(S,A,P,r,\gamma)$1 satisfy

$(S,A,P,r,\gamma)$2

while the state-value formulation obeys

$(S,A,P,r,\gamma)$3

These formulations identify the central computational burden that QPE seeks to alter: classical Monte Carlo rollout averaging, repeated Bellman backups, or linear-system solution. In amplitude-estimation-based work, the comparison baseline is explicitly Monte Carlo policy evaluation with $(S,A,P,r,\gamma)$4 samples for error $(S,A,P,r,\gamma)$5 (Wiedemann et al., 2022, Hein et al., 9 Sep 2025). In QLSA-based work, the emphasis shifts to preparing a quantum state proportional to the value vector with polylogarithmic dependence on the number of state–action pairs under oracle assumptions (Cherrat et al., 2022). In hybrid amplitude-encoding work, the evaluation step is framed as simultaneous quantum Bellman backups over all state–action pairs (Cherukuri et al., 17 May 2025).

2. Core algorithmic constructions

A dominant construction in finite-horizon QPE encodes the normalized return of a trajectory into an ancilla amplitude. Given known bounds $(S,A,P,r,\gamma)$6, one defines

$(S,A,P,r,\gamma)$7

prepares a qsample over trajectories with unitary $(S,A,P,r,\gamma)$8, computes $(S,A,P,r,\gamma)$9 with a return unitary $P(s',r\mid s,a)$0, and applies a controlled rotation $P(s',r\mid s,a)$1 so that the ancilla-$P(s',r\mid s,a)$2 amplitude satisfies

$P(s',r\mid s,a)$3

QPE then applies phase estimation to the Grover operator

$P(s',r\mid s,a)$4

to estimate $P(s',r\mid s,a)$5 with $P(s',r\mid s,a)$6, recovering $P(s',r\mid s,a)$7 (Wiedemann et al., 2022).

A distinct construction, used in GPA, encodes the policy value directly as a phase. For $P(s',r\mid s,a)$8, GPA defines

$P(s',r\mid s,a)$9

implements

$\gamma\in[0,1)$0

and uses a $\gamma\in[0,1)$1-qubit counting register, Hadamards, controlled-$\gamma\in[0,1)$2 operations, inverse QFT, and measurement to obtain a $\gamma\in[0,1)$3-bit estimate $\gamma\in[0,1)$4, hence an approximate value $\gamma\in[0,1)$5 (Alomari et al., 19 Feb 2025). In that setting, QPE is explicitly the policy-evaluation stage of a larger Grover-based policy-search loop.

The infinite-horizon discounted line of work based on QLSA does not estimate a scalar value via amplitude estimation. Instead, it block-encodes $\gamma\in[0,1)$6, forms $\gamma\in[0,1)$7, and uses a quantum linear-system solver to prepare the normalized value-function state

$\gamma\in[0,1)$8

The resulting output is an $\gamma\in[0,1)$9-approximation to $\pi(a\mid s)$0, after which tomography or sampling is required for classical recovery (Cherrat et al., 2022).

A fourth construction, introduced by Q-Policy, amplitude-encodes the entire action-value table,

$\pi(a\mid s)$1

and applies a quantum Bellman-update unitary

$\pi(a\mid s)$2

so that all $\pi(a\mid s)$3 pairs are updated in parallel by linearity. Classical approximation is then obtained through quantum amplitude estimation combined with a classical control variate (Cherukuri et al., 17 May 2025).

3. Methodological branches in the literature

The term QPE spans several technically distinct branches rather than a single canonical circuit. The following comparison captures the main variants described in the literature.

Branch	Representation	Distinctive focus
"Quantum-enhanced reinforcement learning for finite-episode games with discrete state spaces" (Neukart et al., 2017)	QUBO embedding on a D-Wave 2000Q QPU	Monte Carlo policy iteration and embedding of sub-optimal state-value functions
"Quantum Reinforcement Learning via Policy Iteration" (Cherrat et al., 2022)	Block-encoding and QLSA	Prepare an approximate $\pi(a\mid s)$4 for infinite-horizon discounted problems
"Quantum Policy Iteration via Amplitude Estimation and Grover Search -- Towards Quantum Advantage for Reinforcement Learning" (Wiedemann et al., 2022)	Trajectory qsampling, amplitude estimation, Grover operator	Finite-MDP QPE with quadratic improvement over classical Monte Carlo estimation
"GPA: Grover Policy Agent for Generating Optimal Quantum Sensor Circuits" (Alomari et al., 19 Feb 2025)	Phase estimation on $\pi(a\mid s)$5	QPE as search-space generation for Grover-style policy improvement
"Q-Policy: Quantum-Enhanced Policy Evaluation for Scalable Reinforcement Learning" (Cherukuri et al., 17 May 2025)	Amplitude encoding and superposed Bellman backups	Hybrid quantum-classical policy iteration with variance-reduced readout
"From Classical Data to Quantum Advantage -- Quantum Policy Evaluation on Quantum Hardware" (Hein et al., 9 Sep 2025)	Learned $\pi(a\mid s)$6 plus amplitude estimation on hardware	End-to-end integration of QML-learned environments and QPE

Historically, the 2017 D-Wave work addressed finite-episode games with discrete state spaces by partially embedding Monte Carlo policy iteration and sub-optimal state-value-function aggregation as QUBO instances, with the stated aim of explaining how to represent and solve parts of these problems on a QPU rather than proving supremacy over every existing classical policy evaluation algorithm (Neukart et al., 2017). Later work shifted toward gate-model formulations with explicit asymptotic guarantees: QLSA-based policy evaluation for infinite-horizon discounted MDPs (Cherrat et al., 2022), amplitude-estimation-based QPE and Grover policy improvement for finite MDPs (Wiedemann et al., 2022), and amplitude-encoded Bellman backups in a hybrid quantum-classical policy-iteration framework (Cherukuri et al., 17 May 2025).

The GPA formulation broadens the term still further. There, QPE is not primarily introduced for classical-control benchmarks; it evaluates candidate policies that generate quantum sensor circuits, with the estimated phases serving as the marked search space for subsequent Grover-amplitude amplification (Alomari et al., 19 Feb 2025). This suggests that the acronym names the policy-evaluation role within a quantum RL loop rather than a unique low-level primitive. A common source of ambiguity is therefore terminological: some QPE schemes literally employ phase estimation, whereas others use amplitude estimation, QLSA, or annealing-based optimization.

4. Complexity claims and resource scaling

The best-known asymptotic claim attached to amplitude-estimation-based QPE is a quadratic improvement over classical Monte Carlo evaluation. In the finite-MDP construction of (Wiedemann et al., 2022), choosing

$\pi(a\mid s)$7

ancillas guarantees

$\pi(a\mid s)$8

with

$\pi(a\mid s)$9

calls to $$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$0 or $$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$1, compared with classical Monte Carlo

$$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$2

samples. The hardware-oriented formulation of (Hein et al., 9 Sep 2025) states the same qualitative separation as $$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$3 queries to $$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$4 and $$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$5 versus $$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$6 classical samples.

Q-Policy derives a different scaling because it evaluates all state–action pairs through amplitude encoding and a Bellman-update oracle. Under assumptions (A1) sparsity, (A2) spectral bound, and (A3) amplitude preparation, each Bellman update costs $$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$7 gates and amplitude estimation contributes $$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$8 queries, leading to overall evaluation calls

$$V^\pi(s)=\mathbb E_\pi\!\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$$9

for $Q^\pi$0-accurate evaluation, versus $Q^\pi$1 classically. Combined with policy-iteration convergence, the paper gives

$Q^\pi$2

quantum subroutine calls to reach an $Q^\pi$3-optimal policy, versus classical $Q^\pi$4 (Cherukuri et al., 17 May 2025).

The QLSA-based framework offers a different kind of scaling advantage. Theorem 3.1 in (Cherrat et al., 2022) gives circuit-depth or gate-count cost

$Q^\pi$5

where $Q^\pi$6 and $Q^\pi$7. With typical quantum-RAM or sparse-access oracles, this becomes

$Q^\pi$8

However, the same work emphasizes that extracting a fully classical vector incurs tomography or sampling overhead, including $Q^\pi$9 for recovering magnitudes from measurement statistics (Cherrat et al., 2022).

In GPA, resource accounting is expressed in terms of counting qubits and circuit depth rather than policy-evaluation sample complexity alone. The QPE subroutine uses $t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$0 Hadamards, $t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$1 controlled-$t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$2 gates, one $t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$3-qubit inverse QFT requiring $t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$4 two-qubit rotations, and $t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$5 measurements, for overall depth

$t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$6

The phase-estimation error scales as $t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$7 in the absence of noise, while the median absolute error decays roughly as $t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$8 with the number of shots $t_H=(s_0,a_0,r_0,\dots,s_H,r_H)$9 (Alomari et al., 19 Feb 2025).

These heterogeneous results matter because they attach speedup claims to different access models and output models. A plausible implication is that “QPE is quadratically faster” is accurate for amplitude-estimation-based scalar evaluation, but it is not a universal summary of every QPE formulation.

5. Implementations and empirical studies

The earliest reported implementation direction in this corpus is the D-Wave annealing approach of 2017. "Quantum-enhanced reinforcement learning for finite-episode games with discrete state spaces" (Neukart et al., 2017) shows how to express Monte Carlo policy iteration on random observations and the embedding of $G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$0 sub-optimal state-value functions as QUBO problems on a D-Wave 2000Q QPU, and reports that quantum-enhanced Monte Carlo policy evaluation allows finding equivalent or better state-value functions for a given policy with the same number episodes compared to a purely classical Monte Carlo algorithm.

The QLSA-based policy-iteration framework was validated numerically on OpenAI Gym environments rather than hardware. For FrozenLake on $G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$1 and $G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$2 grids, simulated quantum noise and sampling error with $G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$3 measurements still led the quantum policy-iteration loop to converge to the classical optimum within $G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$4 iterations for $G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$5. For InvertedPendulum with continuous 2D state, three actions, Fourier-basis features with $G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$6 and $G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$7 features, and $G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$8 measurements per sample, convergence was reached within $G(t_H)=\sum_{h=0}^{H}\gamma^h r_h,$9 iterations for $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$0 (Cherrat et al., 2022).

The amplitude-estimation-plus-Grover line provided a full implementation and simulation for a two-armed bandit MDP, intended as a proof of concept for how quantum algorithms can solve reinforcement-learning problems given access to error-free, efficient quantum realizations of the agent and environment (Wiedemann et al., 2022). The paper’s emphasis is formal and asymptotic rather than hardware-realistic, and it explicitly frames the result as a detailed proof of concept for combining amplitude estimation and Grover search into policy evaluation and improvement.

Q-Policy reports classical emulations on small GridWorld tasks. In a $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$1 grid with stochastic transitions given by 80% intended move and 20% uniform random, goal state reward $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$2, and other rewards $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$3, the log-scale Bellman error decays geometrically, matching theoretical $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$4 contraction; Table 1 reports query complexity per iteration of $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$5 for Q-Policy and $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$6 for Monte Carlo, with totals at $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$7 iterations of $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$8 and $V^\pi(s)=\mathbb E[G(t_H)\mid s_0=s]$9, respectively. The ablation over amplitude-estimation precision $(S,A,P,r,\gamma)$00 and shot counts $(S,A,P,r,\gamma)$01 further reports that even moderate precision $(S,A,P,r,\gamma)$02 outperforms classical in query count (Cherukuri et al., 17 May 2025).

The most explicit hardware realization is "From Classical Data to Quantum Advantage -- Quantum Policy Evaluation on Quantum Hardware" (Hein et al., 9 Sep 2025). There, a 2-armed bandit environment is first learned from classical observational data through QML and then used for QPE on IonQ forte-1. The hardware description specifies 11 fully-connected qubits, single- and two-qubit gate fidelities of approximately 99.9% and 98%, coherence times $(S,A,P,r,\gamma)$03, daily randomized benchmarking and crosstalk calibration, and FireOpal pulse-level detuning plus zero-noise extrapolation for error mitigation. The reported QPE circuits use 5 qubits and depth approximately 412 gates for $(S,A,P,r,\gamma)$04, and 6 qubits and approximately 883 gates for $(S,A,P,r,\gamma)$05. For the 70%/20% bandit case, the learned parameters satisfy $(S,A,P,r,\gamma)$06 versus $(S,A,P,r,\gamma)$07 and $(S,A,P,r,\gamma)$08 versus $(S,A,P,r,\gamma)$09, with MSE loss approximately $(S,A,P,r,\gamma)$10. For policy evaluation, true values are $(S,A,P,r,\gamma)$11 and $(S,A,P,r,\gamma)$12; RMSE on hardware is approximately $(S,A,P,r,\gamma)$13 for $(S,A,P,r,\gamma)$14 at $(S,A,P,r,\gamma)$15, approximately $(S,A,P,r,\gamma)$16 at $(S,A,P,r,\gamma)$17, and approximately $(S,A,P,r,\gamma)$18 for $(S,A,P,r,\gamma)$19 even on hardware (Hein et al., 9 Sep 2025).

GPA provides a different experimental target: quantum sensor circuits rather than classical control. In its reported QPE experiments, the authors used $(S,A,P,r,\gamma)$20 and obtained a value-function estimate of $(S,A,P,r,\gamma)$21 in $(S,A,P,r,\gamma)$22 shots for a 12-qubit QPE instance, embedded in a loop that seeks circuits maximizing Quantum Fisher Information while minimizing the number of gates (Alomari et al., 19 Feb 2025).

6. Assumptions, limitations, and recurrent points of interpretation

Nearly all QPE proposals derive their advantages from strong oracle and state-preparation assumptions. The finite-MDP amplitude-estimation framework assumes fault-tolerant, error-free implementation of the policy unitary $(S,A,P,r,\gamma)$23, environment unitary $(S,A,P,r,\gamma)$24, return arithmetic $(S,A,P,r,\gamma)$25, controlled powers of the Grover operator $(S,A,P,r,\gamma)$26, and inverse QFT; it further notes that access to coherent sampling oracles is essential and that inefficient state preparation can erase the quadratic advantage (Wiedemann et al., 2022). The QLSA framework likewise depends on quantum row/column oracles for $(S,A,P,r,\gamma)$27, a state-preparation oracle for $(S,A,P,r,\gamma)$28, low-depth block-encoding constructions, and manageable condition number and normalization factors (Cherrat et al., 2022).

The hybrid amplitude-encoded framework makes its assumptions fully explicit. Q-Policy requires (A1) sparsity, namely that each $(S,A,P,r,\gamma)$29 has at most $(S,A,P,r,\gamma)$30 nonzero entries; (A2) a spectral bound $(S,A,P,r,\gamma)$31; and (A3) QRAM-like amplitude preparation with $(S,A,P,r,\gamma)$32 depth per nonzero. The same paper states that these assumptions may not hold in dense or continuous-action MDPs without efficient sparse data structures, and that quantum readout still requires $(S,A,P,r,\gamma)$33 measurements unless further compression or clever sampling is applied (Cherukuri et al., 17 May 2025).

Hardware studies make the noise sensitivity concrete. In the IonQ bandit experiments, deeper circuits with larger $(S,A,P,r,\gamma)$34 suffer from gate errors and decoherence, yielding a trade-off between $(S,A,P,r,\gamma)$35 and fidelity; deterministic or low-branching policies are more robust; and FireOpal improved bandit fit but only partially recovered deep-circuit QPE accuracy. The paper therefore concludes that current hardware noise limits QPE to shallow $(S,A,P,r,\gamma)$36 and simple policies, even though noiseless regimes still exhibit the $(S,A,P,r,\gamma)$37 versus $(S,A,P,r,\gamma)$38 sample-complexity separation (Hein et al., 9 Sep 2025). GPA states the same issue in circuit-design terms: more counting qubits increase precision but lengthen circuits, and deeper circuits are more prone to decoherence and gate errors on near-term hardware (Alomari et al., 19 Feb 2025).

A further interpretive point concerns output format. Some QPE algorithms estimate a scalar $(S,A,P,r,\gamma)$39 for a designated initial state, some prepare a normalized quantum state $(S,A,P,r,\gamma)$40, and some produce a search space over candidate-policy values for a Grover-style improvement stage. This suggests that “policy evaluation” in the quantum literature is task-level terminology. The concrete computational object being produced—an ancilla amplitude, a phase, an amplitude-encoded value table, or a QLSA output state—depends on the surrounding policy-iteration architecture.

Taken together, the literature presents QPE not as a single algorithm but as a structured design space. The common thread is the replacement of classical rollout averaging or classical Bellman evaluation by quantum-native encodings of policy value. The main divergence lies in the computational primitive chosen to expose that value—annealing, amplitude estimation, phase estimation, or linear-system solution—and in the assumptions under which the resulting complexity improvements are claimed.