---
title: Quantum Value Iteration (QVI)
url: https://www.emergentmind.com/topics/quantum-value-iteration-qvi
type: topic
---

# Quantum Value Iteration (QVI)

Searching arXiv for the cited QVI paper and related quantum RL context.
{"query":"arXiv 2508.05712 Quantum Algorithms for Finite-horizon Markov Decision Processes", "max_results": 5}
{"query":"Quantum reinforcement learning via policy iteration 2203.01889", "max_results": 5}
{"query":"Wang et al 2021 quantum reinforcement learning infinite horizon MDP arXiv", "max_results": 10}
Quantum Value Iteration (QVI) denotes a family of quantum dynamic-programming algorithms for *episodic*, *finite-horizon*, *time-inhomogeneous* Markov decision processes (MDPs) that retain the backward-induction structure of classical finite-horizon value iteration while replacing its two dominant subroutines—the expectation over next states and the maximization over actions—by quantum mean estimation and quantum maximum searching, respectively. In the formulation introduced in "Quantum Algorithms for Finite-horizon Markov Decision Processes" [2508.05712], QVI comprises four principal algorithms, QVI-1 through QVI-4, together with a sparse-transition extension QVI-5, spanning both the exact-dynamics and quantum generative-model settings. The framework is notable for proving quantum improvements over classical value-iteration baselines and, in the generative-model setting, establishing quantum lower bounds that render the resulting procedures asymptotically optimal up to logarithmic factors when the horizon is constant [2508.05712].

## 1. Formal problem setting and Bellman structure

The QVI framework is defined for finite-horizon, time-dependent MDPs
\[
\mathcal M = (S, A, \{P_h\}_{h=0}^{H-1}, \{r_h\}_{h=0}^{H-1}, H),
\]
with finite state space \(S\), finite action space \(A\), horizon \(H\), reward functions \(r_h(s,a)\in[0,1]\), and transition kernels \(P_h(\cdot\mid s,a)\). A Markov policy is a map \(\pi:S\times[H]\to A\). For any policy \(\pi\), the value and action-value functions are
\[
V_h^\pi(s)  := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s\Big],
\]
and
\[
Q_h^\pi(s,a) := \mathbb E\Big[\sum_{t=h}^{H-1} r_t(s_t,a_t)\ \Big|\ \pi,\ s_h=s,\ a_h=a\Big].
\]
Optimal quantities satisfy
\[
V_h^*(s) := \max_{\pi\in\Pi} V_h^\pi(s),\qquad
Q_h^*(s,a) := \max_{\pi\in\Pi} Q_h^\pi(s,a),
\]
and an optimal policy \(\pi^*\) obeys \(V_0^{\pi^*}=V_0^*\) [2508.05712].

Classical finite-horizon value iteration is the backward-induction recursion driven by the Bellman optimality operator
\[
[\mathcal T^h V_{h+1}]_s = \max_{a\in A}\Bigl\{ r_h(s,a) + \sum_{s'} P_h(s'|s,a)\,V_{h+1}(s')\Bigr\}.
\]
Starting from \(V_H(s)=0\), one computes, for \(h=H-1,\dots,0\),
\[
Q_h(s,a)=r_h(s,a)+\sum_{s'}P_h(s'|s,a)V_{h+1}(s'),
\]
then
\[
\pi(s,h)=\arg\max_{a\in A}Q_h(s,a),\qquad V_h(s)=Q_h(s,\pi(s,h)).
\]
With a classical model oracle \(\mathcal O\) returning \((r_h(s,a),P_h(s'|s,a))\), the classical query complexity for exact optimal \(\pi^*\) and \(V_0^*\) is \(O(S^2AH)\), and any classical algorithm that outputs an \(\epsilon\)-approximation of \(\{V_h^*\}\) or \(\pi^*\) with constant success probability must use \(\Omega(S^2A)\) queries in the exact-dynamics oracle model [2508.05712].

This Bellman structure is the invariant core of QVI. The quantum algorithms do not alter the dynamic-programming objective; they alter the cost of executing each Bellman update.

## 2. Quantum formulation of backward induction

QVI is the direct quantum analogue of classical backward induction. Its central design principle is to decompose each Bellman update into two tasks and replace both by standard quantum primitives: the maximization \(\max_a\) is implemented by quantum maximum search, and the expectation \(\sum_{s'}P_h(s'|s,a)V_{h+1}(s')\) is implemented by quantum mean estimation [2508.05712].

Two oracle models are used. In the **exact dynamics setting**, the environment is known through a classical oracle
\[
\mathcal O:(s,a,h,s')\mapsto (r_h(s,a),P_h(s'|s,a)),
\]
and its coherent quantum analogue
\[
U_{\mathcal O}:
|s\rangle|a\rangle|h\rangle|s'\rangle|0\rangle|0\rangle
\mapsto
|s\rangle|a\rangle|h\rangle|s'\rangle|Bi(r_h(s,a))\rangle|Bi(P_h(s'|s,a))\rangle.
\]
In the **generative model setting**, a classical generative model provides i.i.d. samples from \(P_h(\cdot\mid s,a)\), whereas the quantum generative model gives coherent superposition access,
\[
G: |s\rangle|a\rangle|h\rangle|0\rangle|0\rangle
\mapsto |s\rangle|a\rangle|h\rangle\sum_{s'}\sqrt{P_h(s'|s,a)}\,|s'\rangle|j_{s'}\rangle.
\]
Query complexity to \(U_{\mathcal O}\) measures quantum time in the exact-dynamics setting; sample complexity to \(G\) measures cost in the generative-model setting [2508.05712].

A compact summary of the algorithm family is as follows.

| Algorithm | Setting and output | Complexity |
|---|---|---|
| QVI-1 | Exact dynamics; exact \(\pi^*\) and \(V_0^*\) | \(O\bigl(S^2\sqrt{A}H\log(SH/\delta)\bigr)\) |
| QVI-2 | Exact dynamics; \(\epsilon\)-optimal policy and values | \(O\left( \frac{S^{1.5}\sqrt{A}H^3\log(SA^{1.5}H/\delta)}{\epsilon} \right)\) |
| QVI-3 | Quantum generative model; \(\epsilon\)-optimal \(\pi,\{V_h\}\) | \(O\Bigl(\frac{S\sqrt{A}H^3\log(SA^{1.5}H/\delta)}{\epsilon}\Bigr)\) |
| QVI-4 | Quantum generative model; \(\epsilon\)-optimal \(\pi,\{V_h\},\{Q_h\}\) | \(O\Bigl( SA\bigl(\frac{H^{2.5}}{\epsilon} + H^3\bigr)\log^2(H^{1.5}/\epsilon)\log(\log(H/\epsilon)HSA/\delta) \Bigr)\) |

Within this taxonomy, QVI-1 and QVI-2 are planning algorithms over a known model, whereas QVI-3 and QVI-4 are model-based reinforcement-learning algorithms under a quantum generative model. QVI-5 is an appendix extension for sparse transitions in the exact-dynamics setting [2508.05712].

## 3. Exact-dynamics QVI: QVI-1, QVI-2, and sparse extensions

QVI-1 addresses the exact-dynamics setting with exact output guarantees. Its input model oracle \(U_{\mathcal O}\) encodes rewards and transition probabilities in binary, and value vectors are likewise encoded by binary oracles
\[
B_V: |s\rangle|0\rangle \mapsto |s\rangle|Bi(V(s))\rangle.
\]
For a fixed \((h,s,a)\), the Bellman right-hand side
\[
r_h(s,a)+\sum_{s'}P_h(s'|s,a)V_{h+1}(s')
\]
is computed exactly by querying \(U_{\mathcal O}\) for all successor states, querying the oracle for \(V_{h+1}\), and applying quantum arithmetic such as QFT-based adders and multipliers. The resulting oracle \(B_{\hat Q_{h,s}}\) is then used inside the Dürr–Høyer quantum maximum searching routine to compute
\[
\hat\pi(s,h)=\arg\max_{a\in A}\hat Q_{h,s}(a),\qquad
\hat V_h(s)=\hat Q_{h,s}(\hat\pi(s,h)).
\]
If all quantum maximum search calls succeed, the recursion coincides exactly with the classical Bellman recursion, and backward induction yields \(\hat V_h=V_h^*\) and \(\hat\pi=\pi^*\) for all \(h\). The total query complexity is
\[
O\bigl(S^2\sqrt{A}H\log(SH/\delta)\bigr),
\]
which gives a quadratic speedup in \(|A|\) over classical \(O(S^2AH)\) while matching the classical dependence on \(|S|\) [2508.05712].

QVI-2 relaxes exactness to obtain improved state-space dependence. The obstacle is that exact evaluation of
\[
\sum_{s'}P_h(s'|s,a)V_{h+1}(s')
\]
costs \(O(S)\) per state-action pair when probabilities are binary-encoded rather than amplitude-encoded. To overcome this, the paper introduces **QMEBO** (Quantum Mean Estimation with Binary Oracles), a subroutine that converts a binary probability oracle \(B_p\) into a probability-amplitude oracle and then applies amplitude-estimation-based quantum mean estimation. In its abstract form, QMEBO estimates
\[
\mu = p^T f = \mathbb E_{x\sim p}[f(x)]
\]
for \(f:\Omega\to[0,1]\), and in the MDP application it reduces the cost of estimating \(P_h(\cdot|s,a)^TV_{h+1}\) to \(\tilde O(\sqrt{S}/\epsilon)\) rather than \(O(S)\) [2508.05712].

The algorithmic modification is not merely approximate evaluation. QVI-2 uses a monotone one-sided-error construction: after estimating \(P_h(\cdot|s,a)^T\tilde V_{h+1}\), it rescales the result and subtracts a small bias,
\[
z_{h,s}(a)\gets H\cdot \mathrm{QMEBO}_\zeta\big(P_h(\cdot|s,a)^T\tilde V_{h+1}\big)-\epsilon/(2H),
\]
then defines
\[
\hat Q_{h,s}(a)=\max\{r_h(s,a)+z_{h,s}(a),0\}.
\]
This downward bias preserves monotonicity and prevents error blow-up over \(H\) stages. The resulting guarantee is
\[
V_h^*-\epsilon \le \hat V_h \le V_h^{\hat\pi}\le V_h^*,\qquad \forall h,
\]
with probability at least \(1-\delta\), so QVI-2 returns an \(\epsilon\)-optimal policy and values. Its total query complexity is
\[
O\left( \frac{S^{1.5}\sqrt{A}H^3\log(SA^{1.5}H/\delta)}{\epsilon} \right),
\]
which improves the state dependence from \(S^2\) to \(S^{1.5}\) while retaining the quadratic improvement in \(|A|\) [2508.05712].

The sparse-transition extension QVI-5, presented in Appendix A.5, further improves the state dependence in sparse MDPs. By converting the binary oracle to an amplitude-style probability oracle and applying QME1 directly, it attains \(\tilde O(S\sqrt{A}H^3/\epsilon)\) in sparse settings; the comparative summary later states the dependence as \(\tilde O(S\sqrt{A}H^3/(\epsilon\eta))\), where \(\eta\) encodes sparsity [2508.05712]. This suggests that QVI is sensitive not only to cardinalities \(S\) and \(A\) but also to structural properties of the transition kernel.

## 4. Generative-model QVI: QVI-3 and QVI-4

QVI-3 and QVI-4 operate in the model-free, generative-model setting, where the agent lacks explicit access to transition probabilities but can sample from the environment. In the classical setting, finite-horizon model-based algorithms of Sidford et al. (2018) and Li et al. (2020) achieve sample complexities on the order \(SAH^4/\epsilon^2\). The quantum generative model replaces classical sampling by coherent superposition access, enabling amplitude-estimation costs of \(\tilde O(1/\epsilon)\) instead of \(O(1/\epsilon^2)\) [2508.05712].

QVI-3 is a quantum version of RandomizedFiniteHorizonVI. At each time \(h\), it encodes \(\hat V_{h+1}\), estimates \(P_h(\cdot|s,a)^T\hat V_{h+1}\) by QME1 with error \(\epsilon/(2H)\), subtracts \(\epsilon/(2H)\) to bias the estimate downward, constructs
\[
\hat Q_{h,s}(a)=\max\{r_h(s,a)+z_{h,s}(a),0\},
\]
and applies quantum maximum search to obtain \(\hat\pi(s,h)\). The same monotonicity argument used in QVI-2 yields
\[
V_h^*-\epsilon \le \hat V_h\le V_h^{\hat\pi}\le V_h^*,\qquad \forall h.
\]
Its sample complexity is
\[
O\Bigl(\frac{S\sqrt{A}H^3\log(SA^{1.5}H/\delta)}{\epsilon}\Bigr),
\]
improving the classical dependences from linear in \(A\) to \(\sqrt{A}\), from \(1/\epsilon^2\) to \(1/\epsilon\), and from \(H^4\) to \(H^3\) [2508.05712].

QVI-4 is designed for stronger output guarantees: \(\epsilon\)-optimal policy, values, and Q-values. It combines two classical ideas with quantum mean estimation. The first is **variance reduction**, implemented through a multi-epoch scheme with target accuracies \(\epsilon_k=H/2^k\). The second is the **total variance technique**, which uses local variance bounds \(\sigma_h(V_{h+1})\) together with a global inequality
\[
\sum_{h'=h}^{H-1} \Bigl(\prod_{i=h+1}^{h'} P_i^\pi\Bigr)\sigma_{h'}(V_{h'+1}^\pi)
\le H^{3/2}.
\]
This allows larger per-step errors than the naive \(\epsilon_k/H\) scaling would permit [2508.05712].

Operationally, QVI-4 estimates variance-like terms \(y_{k,h}(s,a)\), baseline means \(x_{k,h}(s,a)\), and increment terms \(g_{k,h}(s,a)\), then forms
\[
Q_{k,h}(s,a)=\max\{r_h(s,a)+x_{k,h}(s,a)+g_{k,h}(s,a),0\}.
\]
It defines
\[
V_{k,h}(s)=\max_a Q_{k,h}(s,a),\qquad \pi_k(s,h)=\arg\max_a Q_{k,h}(s,a),
\]
but uses a monotonicity update: if \(V_{k,h}(s)\le V_{k,h}^{(0)}(s)\), the previous epoch’s value and policy are retained. This guarantees that values are non-decreasing across epochs and bounded by the value functions of the evolving policy. At the final epoch,
\[
V_h^*-\epsilon \le \hat V_h \le V_h^{\hat\pi}\le V_h^*,\qquad
Q_h^*-\epsilon \le \hat Q_h \le Q_h^{\hat\pi}\le Q_h^*.
\]
The overall sample complexity is
\[
O\Bigl( SA\Bigl(\frac{H^{2.5}}{\epsilon} + H^3\Bigr) \log^2(H^{1.5}/\epsilon)\log(\log(H/\epsilon)HSA/\delta) \Bigr),
\]
with leading term \(\tilde O(SAH^{2.5}/\epsilon)\) in the regime \(\epsilon\in O(1/\sqrt H)\) [2508.05712].

A recurrent misconception is that all QVI variants necessarily improve the action dependence. QVI-3 does, but QVI-4 does not improve \(|A|\) because it must estimate *all* Q-values rather than only locate a maximizer [2508.05712].

## 5. Complexity separations, lower bounds, and optimality

The significance of QVI lies not only in upper bounds but also in matching lower bounds under explicit oracle models. In the exact-dynamics setting, classical backward induction requires \(O(S^2AH)\) operations, and the lower bound \(\Omega(S^2A)\) shows that the classical dependence on \(S^2A\) cannot be avoided in general. Against this baseline, QVI-1 achieves \(\tilde O(S^2\sqrt{A}H)\), and QVI-2 attains \(\tilde O(S^{1.5}\sqrt{A}H^3/\epsilon)\), thereby proving quantum improvements in the dependences on \(|A|\) and, for approximate algorithms, on \(|S|\) as well [2508.05712].

In the generative-model setting, the paper derives lower bounds by reducing infinite-horizon MDPs to finite-horizon ones with
\[
H=\Theta\left(\Gamma\log(\Gamma/\epsilon)\right).
\]
The resulting finite-horizon lower bounds state that any classical algorithm with a classical generative model that outputs \(\epsilon\)-approximations to \(\{Q_h^*\}\), \(\{V_h^*\}\), or \(\pi^*\) with probability at least \(0.9\) must use
\[
\tilde\Omega\left(\frac{SAH^3}{\epsilon^2}\right)
\]
samples. Under a quantum generative model, the lower bounds become
\[
\tilde\Omega\left(\frac{SAH^{1.5}}{\epsilon}\right)
\]
for Q-values and
\[
\tilde\Omega\left(\frac{S\sqrt{A}H^{1.5}}{\epsilon}\right)
\]
for V-values or policy [2508.05712].

These lower bounds clarify the status of QVI-3 and QVI-4. QVI-3 has complexity \(\tilde O(S\sqrt{A}H^3/\epsilon)\), while QVI-4 has complexity \(\tilde O(SAH^{2.5}/\epsilon)\). Up to extra factors of \(H\) and logarithms, these match the lower bounds when \(H\) is constant. Consequently, for fixed horizon, QVI-3 matches the \(S\sqrt{A}/\epsilon\) dependence for policy and value estimation within polylogarithmic factors, and QVI-4 matches the \(SA/\epsilon\) dependence for Q-values and policies. In that regime, the algorithms are asymptotically optimal in the quantum generative-model setting [2508.05712].

The main complexity picture is therefore asymmetric. QVI-1 and QVI-2 establish query-complexity separations from classical planning in a known-model oracle setting; QVI-3 and QVI-4 establish near-minimax quantum sample complexity in a generative-model setting. This suggests that “QVI” is not a single algorithm but a complexity-theoretic program for Bellman-style quantum dynamic programming.

## 6. Relation to broader quantum RL, assumptions, and limitations

QVI should be distinguished from earlier quantum reinforcement-learning approaches based on policy iteration rather than Bellman optimality updates. "Quantum Reinforcement Learning via Policy Iteration" [2203.01889] develops quantum policy evaluation for infinite-horizon discounted MDPs by solving
\[
(I-\gamma P^\pi)Q^\pi = R
\]
as a quantum linear system and representing value functions as amplitude-encoded states
\[
\ket{Q^\pi}=\frac{1}{\|Q^\pi\|}\sum_{s,a}Q^\pi(s,a)\ket{s,a}.
\]
That framework provides block-encoding-based access to transition and reward structure, statewise action-value states, and measurement-based policy improvement, but it does **not** implement a quantum analogue of the value-iteration recursion \(Q_{k+1}=TQ_k\) or repeatedly apply a Bellman optimality operator [2203.01889].

The relationship is therefore foundational rather than identical. The policy-iteration framework supplies the quantum representations of value functions, the oracles for \(P\) and \(R\), and fixed-point methods for policy evaluation that a value-iteration-style quantum algorithm could reuse. QVI, by contrast, is explicitly organized around time-dependent, finite-horizon Bellman updates, with quantum maximum searching handling the nonlinearity of the \(\max_a\) step and quantum mean estimation handling the expectation over successor states [2203.01889].

The principal assumptions behind QVI are strong and explicit. The exact-dynamics algorithms assume ideal coherent access to rewards and transition probabilities in binary through \(U_{\mathcal O}\), sufficient fixed-point precision so that rounding is negligible relative to \(\epsilon\), constant-time oracle calls, and fault-tolerant implementations of quantum maximum search and quantum mean estimation. The generative-model algorithms assume coherent superposition sampling from the environment. Query complexity is taken as the primary cost measure, with gate complexity ignored up to polynomial and polylogarithmic factors [2508.05712].

These assumptions delimit the practical scope of the results. The paper states that dependence on the horizon can remain relatively high, with factors such as \(H^3\) and \(H^{2.5}\), so the clearest advantages arise for small or moderate episodic horizons. Hidden polylogarithmic factors in \(\epsilon^{-1}\), \(S\), \(A\), and \(H\), as well as the constants associated with amplitude estimation and QFT-based arithmetic, may be substantial. The exact-dynamics oracle model is unrealistic for many reinforcement-learning applications, and implementation on near-term noisy devices is characterized as unrealistic; the algorithms are asymptotic and assume large-scale fault-tolerant quantum computers [2508.05712].

Within those assumptions, QVI marks a precise point of contact between quantum algorithms and finite-horizon dynamic programming. It shows that Bellman updates admit Grover-type and amplitude-estimation-type speedups, that these speedups can be stated in explicit \((S,A,H,\epsilon)\)-dependences under well-specified oracle models, and that in the generative-model setting the resulting algorithms are optimal up to logarithmic factors for constant horizon [2508.05712].

Source: https://www.emergentmind.com/topics/quantum-value-iteration-qvi