QV-Learning: Dual Value Estimation

• QV-learning is a reinforcement learning method that jointly approximates state-value and action-value functions, offering enhanced stability over standard Q-learning.
• It employs a dual estimation approach where the learned V function serves as a bootstrap target for Q, reducing overestimation and accelerating convergence.
• QV-learning variants like DQV and DQV-Max demonstrate improved performance in tasks ranging from Atari games to complex MDPs, while addressing off-policy challenges.

QV-learning denotes a family of temporal-difference control methods that learn a state-value function $V(s)$ and an action-value function $Q(s,a)$ together, using the learned state value as an intermediate bootstrap target for the action-value estimate rather than relying exclusively on a single $Q$-function backup. In this formulation, the central departure from standard Q-learning is the joint approximation of two related value functions, with $V$ serving as a potentially more stable target for $Q$ while $Q$ retains the action-specific information needed for control (Sabatelli et al., 2019). In more recent terminology, this family is treated as one branch of “asymmetric two-value-function” methods, distinct from AV-learning, which decomposes $Q$ into value and advantage components rather than learning $V$ as an intermediate target (Daley et al., 13 Jul 2025).

1. Formal definition and Bellman structure

In standard notation, the state-value function is

$v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$

with

$v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$

QV-learning replaces the bootstrap term of single-function TD control with a learned state-value estimate. The canonical update equations are

$Q(s,a)$0

and

$Q(s,a)$1

The literature notes that $Q(s,a)$2 is updated before $Q(s,a)$3 at each step, and that the two updates are commonly assumed to share the same step size $Q(s,a)$4 (Daley et al., 13 Jul 2025).

The motivation is most transparent when contrasted with standard deep Q-learning. Classical Q-learning methods such as DQN and DDQN directly approximate only $Q(s,a)$5, typically with a target of the form $Q(s,a)$6. QV-learning instead asks why one should not approximate $Q(s,a)$7 as well, and then use that estimate to stabilize the learning of $Q(s,a)$8. The stated hypothesis is that this may reduce the tendency of $Q(s,a)$9-learning to bootstrap off itself and to overestimate action values (Sabatelli et al., 2019).

The same basic intuition is present in the deep version called Deep Quality-Value Learning. There, the method is described as learning both a state-value function $Q$0 and an action-value function $Q$1, while using the learned $Q$2-network to provide the bootstrap target for training $Q$3 (Sabatelli et al., 2018).

2. Canonical algorithms: tabular QV-learning and DQV

The tabular family is organized around the idea that both $Q$4 and $Q$5 can regress toward a common target generated by the state-value estimate. In the Deep Quality-Value Learning formulation, the tabular QV$Q$6 update for $Q$7 is

$Q$8

This is the defining substitution: the usual $Q$9 term is replaced by $V$0 (Sabatelli et al., 2018).

The canonical deep implementation, DQV, uses two distinct neural networks: one parameterized by $V$1 for $V$2, and one parameterized by $V$3 for $V$4. Both are trained from the same temporal-difference target

$V$5

where $V$6 is a target copy of the $V$7-network. The losses are

$V$8

and

$V$9

In this design, both networks regress toward the same bootstrapped target, but only the $Q$0-network uses a target network; the learning dynamic is therefore described as “$Q$1-driven” rather than “$Q$2-driven” (Sabatelli et al., 2019).

The deep version evaluated in Atari-style settings uses experience replay and a value target network. Algorithmically, the procedure observes $Q$3, chooses $Q$4, stores $Q$5 in replay, samples a minibatch, forms

$Q$6

then updates $Q$7 by minimizing $Q$8 and $Q$9 by minimizing $Q$0 (Sabatelli et al., 2018).

A recurrent interpretation in this literature is that $Q$1 may be easier and faster to learn than $Q$2, because it does not depend directly on action selection. This suggests why a more stable, faster-converging $Q$3 can accelerate learning of $Q$4, although the stronger causal explanation remains an object of analysis rather than a settled theorem (Sabatelli et al., 2018).

3. Off-policy variants and asymmetric extensions

A major line of development concerns whether the two-value-function idea remains useful in off-policy control. One early extension is DQV-Max, introduced specifically to test whether the benefit of jointly approximating $Q$5 and $Q$6 persists in an off-policy setting. In DQV-Max, the $Q$7-network is learned from a greedy $Q$8-based target,

$Q$9

while $Q$0 is still learned from the $Q$1-network: $Q$2 Because the $Q$3-update now depends on a greedy $Q$4 term, DQV-Max uses a target network for $Q$5, denoted $Q$6, rather than for $Q$7 (Sabatelli et al., 2019).

A later analysis systematizes the off-policy family in different notation. For the off-policy control variant, QVMAX uses

$Q$8

together with

$Q$9

A corrected off-policy version, BC-QVMAX, changes the $V$0-update to

$V$1

while leaving the $V$2-update driven by $V$3 (Daley et al., 13 Jul 2025).

These variants clarify that QV-learning can be either unidirectional or reciprocal. In vanilla QV-learning, the information flow is unidirectional from $V$4 to $V$5. In QVMAX and DQV-Max, reciprocal bootstrapping is reintroduced because $V$6 is itself tied back to a greedy $V$7-target (Daley et al., 13 Jul 2025). This shift is precisely where the benefits of off-policy control become entangled with the usual stability problems of bootstrapped value learning.

4. Theoretical properties, convergence, and bias

A central theoretical issue is whether learning two value functions yields a sound stochastic approximation scheme rather than a heuristic coupling. For on-policy prediction, recent analysis proves that the expected QV-learning update corresponds to a joint affine operator $V$8 whose unique fixed point is the concatenated vector of the true value functions. The operator is stated to be a contraction mapping with unique fixed point $V$9 in the on-policy case, under standard conditions including annealed step sizes, bounded conditional variances, and asynchronous updates (Daley et al., 13 Jul 2025).

This result matters because the common informal picture—$v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$0 converges first, then $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$1 benefits from it—is treated as insufficient. The explicit point made in the analysis is that one does not need to assume that $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$2 first converges exactly before $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$3 can benefit from it; the two functions can move jointly toward their fixed points (Daley et al., 13 Jul 2025).

The other persistent theoretical claim concerns overestimation. In standard DQN, the target contains $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$4, so $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$5 bootstraps from targets directly constructed from its own current estimates. This can induce the familiar upward bias in action values. DQV is said to go further than DDQN in a different way: it removes the $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$6 target entirely and bootstraps $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$7 from a separately learned $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$8-estimate. Since $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$9 is not the same function as $v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$0, the update is less circular and less susceptible to self-reinforcing overestimation (Sabatelli et al., 2019).

The empirical analysis attached to this claim reports that DQV’s estimated values remain more stable and bounded than DQN’s, and that DQV-Max, though less stable than DQV, still diverges less than DQN. The same study explicitly tests whether these methods are merely overestimating $v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$1 instead of $v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$2, and reports that this is not the case: typically $v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$3 is not larger than $v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$4 (Sabatelli et al., 2019).

At the same time, the literature is explicit about limitations. DQV-Max is described as generally more prone to divergence than DQV because it reintroduces the third element of the Deadly Triad. A plausible implication is that the gains of joint $v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$5-$v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$6 approximation depend not only on representation and bootstrapping structure, but also on whether the method preserves the reduced circularity that motivated QV-learning in the first place (Sabatelli et al., 2019).

5. Empirical behavior and scaling with action-set size

The empirical record associated with QV-learning is mixed across settings rather than uniformly favorable. In early deep RL experiments, DQV is reported to learn significantly faster and better than DQN and DDQN on Acrobot, Cartpole, and several Atari games. On Pong, DQV solves the game in less than 400 episodes, more than twice as fast as DQN and DDQN. On Boxing, it reaches near-maximal reward about 300 episodes earlier than the baselines. On Enduro, it achieves substantially higher cumulative reward, roughly twice the baselines’ performance, and on Ice-Hockey it is reported as the only method that improves the policy over the time horizon considered (Sabatelli et al., 2018).

Subsequent work aimed less at headline performance and more at family-level characterization. In a 4-state parametric MDP, QV-learning is compared to Expected Sarsa in the prediction setting and reported to be more sample efficient, especially as the number of actions grows. The argument is that Expected Sarsa must estimate

$v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$7

which becomes more expensive as $v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$8 increases, whereas QV-learning only learns $v_\pi(s) = \sum_{a \in \A} \pi(a|s) q_\pi(s,a).$9, which is described as roughly insensitive to action-set size (Daley et al., 13 Jul 2025).

The same analysis also emphasizes the cost of this compression. QV-learning’s updates are noisier, and if trained long enough, Expected Sarsa can eventually become more accurate. The resulting assessment is therefore conditional rather than absolute: QV-learning appears useful when “good performance quickly” is more important than best asymptotic precision, especially in on-policy prediction with large action spaces (Daley et al., 13 Jul 2025).

In control, the later verdict is more cautious. QVMAX is reported to perform poorly because it is biased, and BC-QVMAX, while correcting that bias, is still generally slower than Q-learning in the reported experiments. The conclusion drawn is explicit: QV-learning is helpful for prediction, but in control it does not provide a major benefit over standard Q-learning once the bias is fixed (Daley et al., 13 Jul 2025).

6. Relation to neighboring methods and terminological scope

QV-learning is best understood in contrast with three adjacent families: Q-learning, Expected Sarsa, and AV-learning. Q-learning learns only $Q(s,a)$00 and bootstraps through a greedy $Q(s,a)$01 operator. Expected Sarsa also learns only $Q(s,a)$02, but bootstraps through a policy expectation over next actions. QV-learning learns both $Q(s,a)$03 and $Q(s,a)$04, with $Q(s,a)$05 intended to approximate the actual state value and to serve as an intermediate target for $Q(s,a)$06 (Daley et al., 13 Jul 2025).

AV-learning differs more fundamentally. There, $Q(s,a)$07 is decomposed into $Q(s,a)$08 plus an advantage function $Q(s,a)$09, and both components are updated through the composite TD error on $Q(s,a)$10. In the cited classification, QV-learning is about learning state values as an intermediate step toward action values, whereas AV-learning is about factoring action values into state and advantage components (Daley et al., 13 Jul 2025).

The term also has a broader terminological perimeter outside reinforcement learning. In quantum machine learning, “QV” may denote “quantum variational,” as in QV-SVM and QVK-SVM, where the variational component is a trainable parametrized quantum circuit optimized from data by minimizing a loss function (Innan et al., 2023). In quantum-inspired representation learning for audio deepfake detection, “QV theory” denotes “Quantum Vision,” where a QV block converts spectrogram-like inputs into information waves before classification (Zaman et al., 9 Apr 2026). Those usages are conceptually unrelated to temporal-difference QV-learning in reinforcement learning.

Within reinforcement learning proper, however, the stable meaning is narrower: QV-learning names the family of methods that jointly estimate $Q(s,a)$11 and $Q(s,a)$12, with the state-value estimate functioning as an intermediate bootstrap target. The historical record suggests that this design can reduce self-referential bootstrapping and improve sample efficiency in prediction, while its advantage in off-policy control remains limited and contingent (Sabatelli et al., 2019).