Quantum Value Iteration (QVI)

• Quantum Value Iteration (QVI) is a family of quantum dynamic programming algorithms that replace classical expectation and maximization with quantum primitives for finite-horizon MDPs.
• It employs quantum mean estimation and quantum maximum search to achieve quadratic speedups in action selection and improved sample complexity over classical methods.
• The framework spans multiple variants, including exact-dynamics and generative-model settings, providing provable quantum speedups and near-optimal complexity in planning and reinforcement learning.

Searching arXiv for the cited QVI paper and related quantum RL context. {"query":"arXiv (Luo et al., 7 Aug 2025) Quantum Algorithms for Finite-horizon Markov Decision Processes", "max_results": 5} {"query":"Quantum reinforcement learning via policy iteration (Cherrat et al., 2022)", "max_results": 5} {"query":"Wang et al 2021 quantum reinforcement learning infinite horizon MDP arXiv", "max_results": 10} Quantum Value Iteration (QVI) denotes a family of quantum dynamic-programming algorithms for episodic, finite-horizon, time-inhomogeneous Markov decision processes (MDPs) that retain the backward-induction structure of classical finite-horizon value iteration while replacing its two dominant subroutines—the expectation over next states and the maximization over actions—by quantum mean estimation and quantum maximum searching, respectively. In the formulation introduced in "Quantum Algorithms for Finite-horizon Markov Decision Processes" (Luo et al., 7 Aug 2025), QVI comprises four principal algorithms, QVI-1 through QVI-4, together with a sparse-transition extension QVI-5, spanning both the exact-dynamics and quantum generative-model settings. The framework is notable for proving quantum improvements over classical value-iteration baselines and, in the generative-model setting, establishing quantum lower bounds that render the resulting procedures asymptotically optimal up to logarithmic factors when the horizon is constant (Luo et al., 7 Aug 2025).

1. Formal problem setting and Bellman structure

The QVI framework is defined for finite-horizon, time-dependent MDPs

$$\mathcal M = (S, A, \{P_h\}_{h=0}^{H-1}, \{r_h\}_{h=0}^{H-1}, H),$$

with finite state space $S$, finite action space $A$, horizon $H$, reward functions $r_h(s,a)\in[0,1]$, and transition kernels $P_h(\cdot\mid s,a)$. A Markov policy is a map $\pi:S\times[H]\to A$. For any policy $\pi$, the value and action-value functions are

$$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$

and

$$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$

Optimal quantities satisfy

$S$0

and an optimal policy $S$1 obeys $S$2 (Luo et al., 7 Aug 2025).

Classical finite-horizon value iteration is the backward-induction recursion driven by the Bellman optimality operator

$S$3

Starting from $S$4, one computes, for $S$5,

$S$6

then

$S$7

With a classical model oracle $S$8 returning $S$9, the classical query complexity for exact optimal $A$0 and $A$1 is $A$2, and any classical algorithm that outputs an $A$3-approximation of $A$4 or $A$5 with constant success probability must use $A$6 queries in the exact-dynamics oracle model (Luo et al., 7 Aug 2025).

This Bellman structure is the invariant core of QVI. The quantum algorithms do not alter the dynamic-programming objective; they alter the cost of executing each Bellman update.

2. Quantum formulation of backward induction

QVI is the direct quantum analogue of classical backward induction. Its central design principle is to decompose each Bellman update into two tasks and replace both by standard quantum primitives: the maximization $A$7 is implemented by quantum maximum search, and the expectation $A$8 is implemented by quantum mean estimation (Luo et al., 7 Aug 2025).

Two oracle models are used. In the exact dynamics setting, the environment is known through a classical oracle

$A$9

and its coherent quantum analogue

$H$0

In the generative model setting, a classical generative model provides i.i.d. samples from $H$1, whereas the quantum generative model gives coherent superposition access,

$H$2

Query complexity to $H$3 measures quantum time in the exact-dynamics setting; sample complexity to $H$4 measures cost in the generative-model setting (Luo et al., 7 Aug 2025).

A compact summary of the algorithm family is as follows.

Algorithm	Setting and output	Complexity
QVI-1	Exact dynamics; exact $H$5 and $H$6	$H$7
QVI-2	Exact dynamics; $H$8-optimal policy and values	$H$9
QVI-3	Quantum generative model; $r_h(s,a)\in[0,1]$0-optimal $r_h(s,a)\in[0,1]$1	$r_h(s,a)\in[0,1]$2
QVI-4	Quantum generative model; $r_h(s,a)\in[0,1]$3-optimal $r_h(s,a)\in[0,1]$4	$r_h(s,a)\in[0,1]$5

Within this taxonomy, QVI-1 and QVI-2 are planning algorithms over a known model, whereas QVI-3 and QVI-4 are model-based reinforcement-learning algorithms under a quantum generative model. QVI-5 is an appendix extension for sparse transitions in the exact-dynamics setting (Luo et al., 7 Aug 2025).

3. Exact-dynamics QVI: QVI-1, QVI-2, and sparse extensions

QVI-1 addresses the exact-dynamics setting with exact output guarantees. Its input model oracle $r_h(s,a)\in[0,1]$6 encodes rewards and transition probabilities in binary, and value vectors are likewise encoded by binary oracles

$r_h(s,a)\in[0,1]$7

For a fixed $r_h(s,a)\in[0,1]$8, the Bellman right-hand side

$r_h(s,a)\in[0,1]$9

is computed exactly by querying $P_h(\cdot\mid s,a)$0 for all successor states, querying the oracle for $P_h(\cdot\mid s,a)$1, and applying quantum arithmetic such as QFT-based adders and multipliers. The resulting oracle $P_h(\cdot\mid s,a)$2 is then used inside the Dürr–Høyer quantum maximum searching routine to compute

$P_h(\cdot\mid s,a)$3

If all quantum maximum search calls succeed, the recursion coincides exactly with the classical Bellman recursion, and backward induction yields $P_h(\cdot\mid s,a)$4 and $P_h(\cdot\mid s,a)$5 for all $P_h(\cdot\mid s,a)$6. The total query complexity is

$P_h(\cdot\mid s,a)$7

which gives a quadratic speedup in $P_h(\cdot\mid s,a)$8 over classical $P_h(\cdot\mid s,a)$9 while matching the classical dependence on $\pi:S\times[H]\to A$0 (Luo et al., 7 Aug 2025).

QVI-2 relaxes exactness to obtain improved state-space dependence. The obstacle is that exact evaluation of

$\pi:S\times[H]\to A$1

costs $\pi:S\times[H]\to A$2 per state-action pair when probabilities are binary-encoded rather than amplitude-encoded. To overcome this, the paper introduces QMEBO (Quantum Mean Estimation with Binary Oracles), a subroutine that converts a binary probability oracle $\pi:S\times[H]\to A$3 into a probability-amplitude oracle and then applies amplitude-estimation-based quantum mean estimation. In its abstract form, QMEBO estimates

$\pi:S\times[H]\to A$4

for $\pi:S\times[H]\to A$5, and in the MDP application it reduces the cost of estimating $\pi:S\times[H]\to A$6 to $\pi:S\times[H]\to A$7 rather than $\pi:S\times[H]\to A$8 (Luo et al., 7 Aug 2025).

The algorithmic modification is not merely approximate evaluation. QVI-2 uses a monotone one-sided-error construction: after estimating $\pi:S\times[H]\to A$9, it rescales the result and subtracts a small bias,

$\pi$0

then defines

$\pi$1

This downward bias preserves monotonicity and prevents error blow-up over $\pi$2 stages. The resulting guarantee is

$\pi$3

with probability at least $\pi$4, so QVI-2 returns an $\pi$5-optimal policy and values. Its total query complexity is

$\pi$6

which improves the state dependence from $\pi$7 to $\pi$8 while retaining the quadratic improvement in $\pi$9 (Luo et al., 7 Aug 2025).

The sparse-transition extension QVI-5, presented in Appendix A.5, further improves the state dependence in sparse MDPs. By converting the binary oracle to an amplitude-style probability oracle and applying QME1 directly, it attains $$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$0 in sparse settings; the comparative summary later states the dependence as $$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$1, where $$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$2 encodes sparsity (Luo et al., 7 Aug 2025). This suggests that QVI is sensitive not only to cardinalities $$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$3 and $$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$4 but also to structural properties of the transition kernel.

4. Generative-model QVI: QVI-3 and QVI-4

QVI-3 and QVI-4 operate in the model-free, generative-model setting, where the agent lacks explicit access to transition probabilities but can sample from the environment. In the classical setting, finite-horizon model-based algorithms of Sidford et al. (2018) and Li et al. (2020) achieve sample complexities on the order $$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$5. The quantum generative model replaces classical sampling by coherent superposition access, enabling amplitude-estimation costs of $$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$6 instead of $$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$7 (Luo et al., 7 Aug 2025).

QVI-3 is a quantum version of RandomizedFiniteHorizonVI. At each time $$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$8, it encodes $$V_h^\pi(s) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],$$9, estimates $$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$0 by QME1 with error $$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$1, subtracts $$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$2 to bias the estimate downward, constructs

$$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$3

and applies quantum maximum search to obtain $$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$4. The same monotonicity argument used in QVI-2 yields

$$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$5

Its sample complexity is

$$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$6

improving the classical dependences from linear in $$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$7 to $$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$8, from $$Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].$$9 to $S$00, and from $S$01 to $S$02 (Luo et al., 7 Aug 2025).

QVI-4 is designed for stronger output guarantees: $S$03-optimal policy, values, and Q-values. It combines two classical ideas with quantum mean estimation. The first is variance reduction, implemented through a multi-epoch scheme with target accuracies $S$04. The second is the total variance technique, which uses local variance bounds $S$05 together with a global inequality

$S$06

This allows larger per-step errors than the naive $S$07 scaling would permit (Luo et al., 7 Aug 2025).

Operationally, QVI-4 estimates variance-like terms $S$08, baseline means $S$09, and increment terms $S$10, then forms

$S$11

It defines

$S$12

but uses a monotonicity update: if $S$13, the previous epoch’s value and policy are retained. This guarantees that values are non-decreasing across epochs and bounded by the value functions of the evolving policy. At the final epoch,

$S$14

The overall sample complexity is

$S$15

with leading term $S$16 in the regime $S$17 (Luo et al., 7 Aug 2025).

A recurrent misconception is that all QVI variants necessarily improve the action dependence. QVI-3 does, but QVI-4 does not improve $S$18 because it must estimate all Q-values rather than only locate a maximizer (Luo et al., 7 Aug 2025).

5. Complexity separations, lower bounds, and optimality

The significance of QVI lies not only in upper bounds but also in matching lower bounds under explicit oracle models. In the exact-dynamics setting, classical backward induction requires $S$19 operations, and the lower bound $S$20 shows that the classical dependence on $S$21 cannot be avoided in general. Against this baseline, QVI-1 achieves $S$22, and QVI-2 attains $S$23, thereby proving quantum improvements in the dependences on $S$24 and, for approximate algorithms, on $S$25 as well (Luo et al., 7 Aug 2025).

In the generative-model setting, the paper derives lower bounds by reducing infinite-horizon MDPs to finite-horizon ones with

$S$26

The resulting finite-horizon lower bounds state that any classical algorithm with a classical generative model that outputs $S$27-approximations to $S$28, $S$29, or $S$30 with probability at least $S$31 must use

$S$32

samples. Under a quantum generative model, the lower bounds become

$S$33

for Q-values and

$S$34

for V-values or policy (Luo et al., 7 Aug 2025).

These lower bounds clarify the status of QVI-3 and QVI-4. QVI-3 has complexity $S$35, while QVI-4 has complexity $S$36. Up to extra factors of $S$37 and logarithms, these match the lower bounds when $S$38 is constant. Consequently, for fixed horizon, QVI-3 matches the $S$39 dependence for policy and value estimation within polylogarithmic factors, and QVI-4 matches the $S$40 dependence for Q-values and policies. In that regime, the algorithms are asymptotically optimal in the quantum generative-model setting (Luo et al., 7 Aug 2025).

The main complexity picture is therefore asymmetric. QVI-1 and QVI-2 establish query-complexity separations from classical planning in a known-model oracle setting; QVI-3 and QVI-4 establish near-minimax quantum sample complexity in a generative-model setting. This suggests that “QVI” is not a single algorithm but a complexity-theoretic program for Bellman-style quantum dynamic programming.

6. Relation to broader quantum RL, assumptions, and limitations

QVI should be distinguished from earlier quantum reinforcement-learning approaches based on policy iteration rather than Bellman optimality updates. "Quantum Reinforcement Learning via Policy Iteration" (Cherrat et al., 2022) develops quantum policy evaluation for infinite-horizon discounted MDPs by solving

$S$41

as a quantum linear system and representing value functions as amplitude-encoded states

$S$42

That framework provides block-encoding-based access to transition and reward structure, statewise action-value states, and measurement-based policy improvement, but it does not implement a quantum analogue of the value-iteration recursion $S$43 or repeatedly apply a Bellman optimality operator (Cherrat et al., 2022).

The relationship is therefore foundational rather than identical. The policy-iteration framework supplies the quantum representations of value functions, the oracles for $S$44 and $S$45, and fixed-point methods for policy evaluation that a value-iteration-style quantum algorithm could reuse. QVI, by contrast, is explicitly organized around time-dependent, finite-horizon Bellman updates, with quantum maximum searching handling the nonlinearity of the $S$46 step and quantum mean estimation handling the expectation over successor states (Cherrat et al., 2022).

The principal assumptions behind QVI are strong and explicit. The exact-dynamics algorithms assume ideal coherent access to rewards and transition probabilities in binary through $S$47, sufficient fixed-point precision so that rounding is negligible relative to $S$48, constant-time oracle calls, and fault-tolerant implementations of quantum maximum search and quantum mean estimation. The generative-model algorithms assume coherent superposition sampling from the environment. Query complexity is taken as the primary cost measure, with gate complexity ignored up to polynomial and polylogarithmic factors (Luo et al., 7 Aug 2025).

These assumptions delimit the practical scope of the results. The paper states that dependence on the horizon can remain relatively high, with factors such as $S$49 and $S$50, so the clearest advantages arise for small or moderate episodic horizons. Hidden polylogarithmic factors in $S$51, $S$52, $S$53, and $S$54, as well as the constants associated with amplitude estimation and QFT-based arithmetic, may be substantial. The exact-dynamics oracle model is unrealistic for many reinforcement-learning applications, and implementation on near-term noisy devices is characterized as unrealistic; the algorithms are asymptotic and assume large-scale fault-tolerant quantum computers (Luo et al., 7 Aug 2025).

Within those assumptions, QVI marks a precise point of contact between quantum algorithms and finite-horizon dynamic programming. It shows that Bellman updates admit Grover-type and amplitude-estimation-type speedups, that these speedups can be stated in explicit $S$55-dependences under well-specified oracle models, and that in the generative-model setting the resulting algorithms are optimal up to logarithmic factors for constant horizon (Luo et al., 7 Aug 2025).