Convergent DQN (C-DQN)

• Convergent DQN is a reinforcement learning algorithm that guarantees monotonic convergence by taking the maximum between DQN and MSBE losses.
• It enhances training stability and reproducibility in both discrete and continuous control tasks, even under high discount factors.
• C-DQN resolves divergence issues typical of standard DQN, enabling reliable performance in complex, high-dimensional environments.

Convergent Deep Q-Network (C-DQN) is a reinforcement learning algorithm designed to address and resolve the divergence and instability issues inherent in standard Deep Q-Networks (DQN), especially under function approximation and large discount factor ($\gamma$) regimes. The architecture and methodology of C-DQN guarantee monotonic convergence of the training loss, provide theoretical guarantees absent in standard DQN, and exhibit superior robustness and reproducibility in both discrete and continuous control domains (Wang et al., 2021, Wang, 2022).

1. Motivation and Problem Statement

Standard DQN, while empirically successful, lacks formal convergence guarantees in function-approximation settings. The root of the problem is that both the current $Q$-estimates and the targets in the Bellman update depend on the current parameters, inducing a coupled and potentially divergent update dynamic. Even gradient-based convergent methods, such as residual-gradient (RG) Q-learning, commonly fail due to ill-conditioning and “average-value” stalling.

Modern DQN variants typically deploy a target network ($\theta^-$) held fixed for $K$ steps, with the loss

$$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$

but this ad-hoc stabilization lacks a guarantee that training loss does not increase sharply upon target network synchronization. Divergence persists in hard exploration tasks, large-$\gamma$ domains, and settings with non-ideal sampling (Wang et al., 2021, Wang, 2022).

C-DQN directly targets this issue by designing a loss and update rule that ensures monotonic, non-increasing behavior of the minimal achievable training loss at every target-network update, thus enforcing global convergence.

2. Mathematical Formulation

C-DQN operates within the standard Markov Decision Process (MDP) framework: state space $S$, action space $A$, reward $r_t \in \mathbb{R}$, and discount factor $\gamma \in [0,1)$. The Q-function is approximated by a neural network $Q$0.

Loss Construction

C-DQN introduces two key per-sample squared errors:

• DQN-style Bellman fixed-target error:

$Q$1

• Mean Squared Bellman Error (MSBE), with both arguments evaluated at $Q$2:

$Q$3

C-DQN's batch loss is defined as:

$Q$4

This construction guarantees that, upon target network update ($Q$5), the minimal achievable $Q$6 can never increase:

$Q$7

3. Algorithmic Details

C-DQN's training protocol closely mirrors DQN, differing only in loss computation:

1. Initialize replay buffer $Q$8, online network parameters $Q$9, and target network $\theta^-$0.
2. At each time step:
   • Observe state $\theta^-$1, select $\theta^-$2 via $\theta^-$3-greedy policy.
   • Execute $\theta^-$4, observe $\theta^-$5, $\theta^-$6, and store $\theta^-$7 in $\theta^-$8.
3. Every fixed number of steps:
   • Sample minibatch $\theta^-$9
   • For each transition, compute $K$0 and $K$1
   • Compute $K$2
   • Minimize $K$3 with Adam.
4. Every $K$4 updates, synchronize target network: $K$5.

Practical Implementation

• Slightly increased per-sample computational cost ($K$6 that of DQN) due to the need to evaluate both $K$7 and $K$8 Q-values (Wang et al., 2021).
• No extra memory requirements beyond standard DQN; loss can optionally use a Huber function for outlier robustness.
• Target-network update period $K$9 can be shortened by up to $$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$0 without affecting stability.

Discount Factor Selection

C-DQN remains convergent for arbitrary $$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$1. For long-horizon tasks, $$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$2 can be set as large as $$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$3, with reward-interval-based heuristics to select $$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$4.

4. Convergence Properties and Theoretical Guarantees

Let $$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$5 denote the parameters after the $$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$6-th target network update. Define the sequence $$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$7. The following holds:

$$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$8

Thus, $$L_{\mathrm{DQN}}(\theta;\theta^-) = \mathbb{E} \left[(Q_\theta(s,a) - (r + \gamma \max_{a'} Q_{\theta^-}(s', a')))^2\right],$$9 is monotonic non-increasing and bounded below by 0, and so converges. The proof observes that the maximal loss construction upper-bounds both DQN and MSBE losses, and target network preservation ensures the non-increase property (Wang et al., 2021, Wang, 2022).

In stochastic environments, $\gamma$0 includes a variance correction, so C-DQN may converge to a fixed point between the DQN and MSBE minima. Empirically, this bias is negligible and does not impede learning.

5. Empirical Performance and Benchmarks

C-DQN has been benchmarked extensively on the Atari 2600 and on continuous quantum control tasks (Wang et al., 2021, Wang, 2022).

Atari Benchmarks

Game	Human	DQN	Rainbow	C-DQN	Agent57
Skiing	-4337	-29,751 ± 224	-12,958	-3,697 ±157	-4,203 ± 608
Tennis	-8.3	-2.6 ±1.4	0.0	+10.9 ±6.3	+23.8 ± 0.1
Private Eye	69,571	7,948 ± 749	4,234	14,730 ±37	79,716 ± 29,545
Venture	1,188	386 ± 85	5.5	893 ± 51	2,624 ± 442

Key findings:

• On stable games, C-DQN matches DQN; RG fails to produce competitive policies.
• For “prone-to-diverge” or long-horizon settings (large $\gamma$1, sparse rewards, buffer manipulations), standard DQN produces exploding losses or oscillatory learning, while C-DQN remains stable with only a marginal increase in computational time.
• C-DQN achieves reliable learning in notoriously difficult games without additional tuning and at a fraction of the computational budget of highly-engineered agents.

Quantum Control Tasks

C-DQN was applied to measurement-feedback cooling of quantum systems:

• Quartic oscillator: C-DQN converges monotonically to optimal control policies while DQN's loss and performance oscillate or collapse mid-training.
• Trapped quantum rigid body: C-DQN achieves robust cooling policies with small performance variance between seeds, whereas standard DQN and even LQG-type feedback exhibit higher failure rates or early instability under noise or nonlinearity (Wang, 2022).

In both domains, when DQN is stable, C-DQN matches its performance; when DQN diverges, C-DQN learns successfully.

6. Practical Implementation and Considerations

• Loss computation requires one additional forward pass per sample ($\gamma$2) compared to standard DQN.
• No changes to memory allocation, replay buffer, or optimizer are needed.
• The same sampling and optimizer schedules as DQN can be used; the only modification is the pointwise max construction in loss calculation.
• Huber loss can be optionally wrapped around both loss arguments.
• Annealing exploration ($\gamma$3) should be slower for large-$\gamma$4 tasks to enhance exploration coverage.
• Target update intervals can be substantially reduced, enhancing adaptability to fast-changing environments or online settings (Wang et al., 2021, Wang, 2022).

7. Significance and Related Work

C-DQN addresses the fundamental tension between stability and efficiency in deep RL with function approximation:

• Guarantees monotonic, non-increasing training loss at every target-network update, a property previously lacking in DQN variants.
• Retains practical efficiency and high performance even in regimes (large $\gamma$5, suboptimal replay) that destabilize standard algorithms.
• Outperforms both DQN and RG methods in practice, combining the rapid policy improvement of DQN with the theoretical guarantees of RG.
• Demonstrates generality by extending to high-dimensional continuous-control settings such as quantum feedback—domains where conventional deep RL exhibits unreliability and large variance (Wang et al., 2021, Wang, 2022).

The premise of taking the maximum between DQN and MSBE losses at every sample, rather than adding regularization or constraints, provides an operational upper bound that disciplines updates without averaging away progress. A plausible implication is that maximal-loss frameworks could serve more broadly as stabilizing mechanisms in off-policy RL algorithms under function approximation.

---

References:

• "Convergent and Efficient Deep Q Network Algorithm" (Wang et al., 2021)
• "Control of Continuous Quantum Systems with Many Degrees of Freedom based on Convergent Reinforcement Learning" (Wang, 2022)