Regularized Dueling Q-learning (RDQ) Overview

• RDQ is an advantage-based temporal difference method that decomposes Q-values into a state-value and an advantage component, with identifiability enforced by an explicit ℓ2 regularizer.
• It differs from Dueling DQN by removing the zero-mean advantage constraint and applying a penalty directly on value and advantage outputs, thereby improving stability.
• Empirical results in MinAtar and tabular settings show RDQ outperforms standard DQN and dueling variants, especially in environments with large or redundant action spaces.

Searching arXiv for the cited RDQ and dueling papers to ground the article. Regularized Dueling Q-learning (RDQ) is an advantage-based temporal-difference control method that learns an action-value function through a decomposition into a state-value component and an advantage component. It was introduced as a new AV-learning algorithm in “An Analysis of Action-Value Temporal-Difference Methods That Learn State Values” (Daley et al., 13 Jul 2025), and is most naturally understood relative to the dueling network architecture of “Dueling Network Architectures for Deep Reinforcement Learning” (Wang et al., 2015). Whereas Dueling DQN enforces identifiability by subtracting the mean advantage in the forward map, RDQ retains the unconstrained decomposition $Q(s,a)=V(s)+A(s,a)$ and restores identifiability through an explicit $\ell_2$ regularizer on the value and advantage outputs. In the formulation reported for deep control, RDQ uses a standard DQN target network, experience replay, and an output-space penalty that encourages a minimum-norm decomposition while preserving the dueling separation between global state value and per-action deviations.

1. Conceptual position within action-value TD methods

In single-action-value TD methods such as Q-learning, Sarsa, and Expected Sarsa, learning bootstraps exclusively from a single action-value function $Q(s,a)$. AV-learning methods instead explicitly leverage the decomposition

$Q(s,a)=V(s)+A(s,a),$

maintaining two asymmetric value functions: a state-value $V$ and an advantage $A$. In this formulation, both streams integrate only through the $Q$ aggregator and bootstrap from $Q$’s TD targets (Daley et al., 13 Jul 2025).

This places RDQ within the AV-learning family rather than the QV-learning family. QV-learning learns $Q$ and $V$ with explicit bootstrapping between them, and the analysis in the 2025 paper distinguishes sharply between these families: in control, AV-learning methods “work much more consistently” and “easily outperform” Q-learning, whereas the off-policy control variant of QV-learning, QVMAX, is biased, and its corrected variant BC-QVMAX behaves like a delayed Q-learning with worse sample efficiency (Daley et al., 13 Jul 2025).

The dueling architecture provides the immediate historical precursor. In that architecture, the value stream learns how good a state is regardless of action, while the advantage stream learns differences between actions when action selection matters. The original motivation was that many states have multiple actions with similar values, so learning a separate state-value estimate can improve generalization across actions and sample efficiency (Wang et al., 2015). RDQ preserves that decomposition but changes how uniqueness is enforced.

2. Formal specification and relation to dueling aggregation

The standard discounted return is

$\ell_2$0

Under a policy $\ell_2$1, the action-value, state-value, and advantage functions are

$\ell_2$2

$\ell_2$3

$\ell_2$4

with $\ell_2$5 (Wang et al., 2015).

In the dueling architecture, a shared feature extractor with parameters $\ell_2$6 feeds a value stream and an advantage stream. The value stream outputs a scalar $\ell_2$7 and the advantage stream outputs a vector $\ell_2$8. The original paper considered three aggregation operators:

1. naïve aggregation,

$\ell_2$9

which is unidentifiable;

1. max-normalized aggregation, which subtracts $Q(s,a)$0;
2. mean-normalized aggregation,

$Q(s,a)$1

which enforces zero-mean advantage per state and was found more stable in practice than max-normalization (Wang et al., 2015).

RDQ deliberately returns to the unconstrained decomposition

$Q(s,a)$2

and compensates for the resulting lack of identifiability with an explicit regularizer. In the control setting, its TD target is

$Q(s,a)$3

with TD error

$Q(s,a)$4

The deep RDQ loss is

$Q(s,a)$5

where $Q(s,a)$6 is the regularization strength (Daley et al., 13 Jul 2025).

The difference from Dueling DQN is therefore structural rather than cosmetic. Dueling DQN enforces uniqueness by constraining the advantage outputs to have zero mean in the forward map. RDQ imposes no zero-mean constraint and instead uses a statewise weight decay on $Q(s,a)$7 and on all per-action advantages. The 2025 analysis characterizes this as searching for the closest point on the hyperplane defined by two arbitrary functions that sum to $Q(s,a)$8, i.e. the minimum-norm decomposition (Daley et al., 13 Jul 2025).

3. Optimization, updates, and deep training procedure

For tabular soft RDQ, the paper gives explicit one-step updates. For all $Q(s,a)$9,

$Q(s,a)=V(s)+A(s,a),$0

and

$Q(s,a)=V(s)+A(s,a),$1

The shrinkage factor $Q(s,a)=V(s)+A(s,a),$2 acts as weight-decay-style contraction, and in deep RDQ the corresponding loss is optimized by backpropagation with Adam (Daley et al., 13 Jul 2025).

For tabular hard RDQ, obtained by setting $Q(s,a)=V(s)+A(s,a),$3, the updates become

$Q(s,a)=V(s)+A(s,a),$4

$Q(s,a)=V(s)+A(s,a),$5

In this case there is an invariant,

$Q(s,a)=V(s)+A(s,a),$6

up to a fixed translation determined at initialization. The stated reason is that each step increments exactly one $Q(s,a)=V(s)+A(s,a),$7 and $Q(s,a)=V(s)+A(s,a),$8 by the same amount $Q(s,a)=V(s)+A(s,a),$9 (Daley et al., 13 Jul 2025).

The reported deep RDQ training loop in MinAtar is specific. The online network is composed as $V$0, with target parameters initialized as $V$1. The replay buffer has capacity $V$2. Exploration uses $V$3 for the first $V$4 steps, then linearly anneals to $V$5 over the next $V$6 steps, and remains at $V$7 thereafter. The optimizer is Adam with learning rate $V$8 and $V$9. The regularization coefficient used in the paper is $A$0. Every $A$1 environment steps, a minibatch of size $A$2 is sampled uniformly from replay, and every $A$3 environment steps the target network is updated by setting $A$4 (Daley et al., 13 Jul 2025).

The dueling precursor paper provides the broader architectural and stabilization context. In Atari, the shared trunk used three convolutional layers with ReLU nonlinearities, followed by two fully connected streams with hidden layer size $A$5, one stream producing a scalar value and the other $A$6 advantages. That paper also reported target networks, experience replay, gradient clipping to norm $A$7, and rescaling the combined gradient entering the last convolutional layer by $A$8 because both streams backpropagate through it (Wang et al., 2015). A plausible implication is that these stabilization devices remain relevant when RDQ is instantiated in deeper dueling-style networks, although the MinAtar RDQ experiments themselves were reported with the simpler configuration above.

4. Identifiability, geometry, and learning dynamics

The central technical issue is identifiability. With the naïve decomposition $A$9, for any constant $Q$0 one can define

$Q$1

without changing $Q$2. In the original dueling formulation, this ambiguity was described as leading to poor training behavior, and normalization terms were introduced to remove it (Wang et al., 2015).

Dueling DQN resolves the ambiguity by centering the advantage outputs. Under mean-normalization, the forward map enforces

$Q$3

for each state. This preserves action rankings because subtracting the same state-dependent constant from all advantages does not change the order of $Q$4 across actions. The original paper also derived the local Jacobian of the mean-normalized aggregator:

$Q$5

Thus gradients to the advantage outputs are coupled across actions via the mean, which regularizes and stabilizes updates (Wang et al., 2015).

RDQ replaces that forward-map constraint with a regularized objective. Its $Q$6 penalty

$Q$7

selects the minimum-norm point on the solution line defined by $Q$8. The 2025 paper states that, given typical near-zero initialization of $Q$9 and $Q$0, this can require fewer updates and empirically improves learning stability and efficiency (Daley et al., 13 Jul 2025).

This change also alters how TD information reaches the advantage stream. In Dueling DQN, the mean term spreads the TD error across all actions, so the chosen action receives $Q$1 while all advantages receive $Q$2 in the tabular update. In RDQ, the TD signal is applied only to the chosen action’s advantage, while the regularizer shrinks all advantage outputs. The paper characterizes this as potentially focusing learning on the sampled action while still regularizing the full advantage vector (Daley et al., 13 Jul 2025). This suggests a different bias-variance trade-off from the mean-subtracted dueling update: Dueling DQN couples actions directly through the aggregator, whereas RDQ couples them indirectly through the shared value stream, shared trunk, and output penalty.

The tabular hard-RDQ invariant $Q$3 is especially notable in this regard. The paper states that it matches the minimum-norm direction on average without requiring an explicit penalty, but also emphasizes that this invariant does not transfer to general function approximation, because gradients for the two streams break the invariant. That observation is the stated motivation for soft $Q$4 regularization in deep RDQ (Daley et al., 13 Jul 2025).

5. Empirical evidence

The empirical case for RDQ is reported in MinAtar and in a tabular 4-state control problem, while the empirical case for the underlying dueling decomposition comes from a corridor policy-evaluation task and Atari 2600 benchmarks (Daley et al., 13 Jul 2025, Wang et al., 2015).

Setting	Reported result	Comparison
Corridor policy evaluation	With 5 actions, dueling and single-stream converge similarly; with 10 or 20 actions, dueling consistently achieves lower squared error and converges faster	Gap increases with number of actions
Atari 2600, 57 games, “30 no-ops”	Duel Clip: mean 373.1%, median 151.5%; Prior. Duel Clip: mean 591.9%, median 172.1%	Exceeds Single Clip, Single, Nature DQN, and prioritized single-stream baselines
Atari 2600, 57 games, “Human Starts”	Duel Clip: mean 343.8%, median 117.1%; Prior. Duel Clip: mean 567.0%, median 115.3%	Exceeds listed single-stream baselines
MinAtar	Across all five games, RDQ significantly outperforms both DQN and Dueling DQN given identical architectures and hyperparameters	Figures show consistently higher learning curves
4-state MDP with large $Q$5	Hard RDQ and Dueling Q-learning both dramatically outperform Q-learning; Hard RDQ slightly outperforms Dueling Q-learning in a small majority of instances	Instability observed near $Q$6

In the MinAtar benchmark, the environments were Asterix, Breakout, Freeway, Seaquest, and Space Invaders. Training ran for $Q$7 million steps, with $Q$8 independent runs per game. Evaluation occurred every $Q$9 million training steps for $Q$0 episodes with $Q$1, using mean undiscounted return and reporting mean $Q$2 confidence intervals. Under this protocol, RDQ significantly outperformed both DQN and Dueling DQN across all five games, with identical architectures and hyperparameters (Daley et al., 13 Jul 2025).

The earlier dueling results are important context rather than direct evidence for RDQ itself. They show that factoring $Q$3 into value and advantage can improve performance particularly when the action space contains many similar-valued or redundant actions. The corridor experiment explicitly enlarged the action set by adding no-ops and found that the advantage of the dueling architecture increased with the number of actions. The Atari experiments further reported that improvements were particularly strong in games with large action spaces, and that prioritized replay was complementary to dueling (Wang et al., 2015). A plausible implication is that RDQ’s improvements are most relevant in regimes where advantage structure is informative but the zero-mean constraint of Dueling DQN is not the best inductive bias.

6. Practical usage, limitations, and points of interpretation

The recommended defaults reported for MinAtar-like deep setups are: aggregator $Q$4, loss $Q$5 with $Q$6, a standard dueling split after a shared trunk, Adam with learning rate $Q$7, minibatch size $Q$8, an update every $Q$9 environment steps, target update every $V$0 steps, replay capacity $V$1, uniform sampling, and an $V$2-greedy schedule of $V$3 for $V$4k steps, annealed to $V$5 over $V$6k steps, then held constant (Daley et al., 13 Jul 2025).

The principal tuning parameter is $V$7, the regularization strength. The paper recommends starting at $V$8. If learning becomes too conservative, with values too small and progress slow, $V$9 should be reduced; if training is unstable or exhibits very large advantages or value drift, $\ell_2$00 should be increased. Because the penalty acts on both heads and sums over actions, its effective scale depends on $\ell_2$01 and on network activation magnitudes (Daley et al., 13 Jul 2025).

The same source also specifies several limitations. RDQ inherits off-policy issues from DQN, including overestimation bias of the max target, because the deep experiments used the standard DQN target rather than Double DQN. It further notes that one can combine RDQ with Double DQN, though this was not evaluated in that paper. Soft RDQ can be noisy in tabular form because the shrinkage terms make $\ell_2$02 and $\ell_2$03 “leaky” toward zero, so the recommendation is to use hard RDQ tabularly and soft RDQ with function approximation (Daley et al., 13 Jul 2025).

Several misconceptions can be resolved directly from the formulations. RDQ is not merely Dueling DQN with weight decay on network parameters; the reported regularizer acts on the output heads, penalizing $\ell_2$04 and all sampled-state advantages $\ell_2$05 rather than penalizing $\ell_2$06 directly. RDQ is also not equivalent to mean-normalized dueling: Dueling DQN imposes a zero-mean constraint in the forward map, whereas RDQ leaves the decomposition unconstrained and makes it well posed through the loss. Finally, RDQ does not come with a formal convergence proof beyond the fixed-point properties inherited from using the same Q-learning target on $\ell_2$07; with function approximation, it remains a practical algorithm with the same general caveats that apply to DQN-style methods (Daley et al., 13 Jul 2025).

From the perspective of the broader dueling literature, RDQ can therefore be read as a different answer to the same structural problem. Dueling DQN resolves ambiguity by normalization inside the architecture; RDQ resolves it by preferring a minimum-norm factorization through explicit regularization. The reported empirical and theoretical analyses present that choice as a principled alternative rather than a replacement of the underlying value-advantage decomposition (Wang et al., 2015, Daley et al., 13 Jul 2025).