Averaged-DQN: Variance Reduction in Deep RL

• Averaged-DQN is a reinforcement learning algorithm that averages recent Q-value estimates to reduce target approximation error and mitigate overestimation bias.
• The technique achieves at least a 1/K variance reduction compared to standard DQN, resulting in more stable and improved performance on benchmarks like Breakout and Seaquest.
• It integrates with existing DQN improvements with minimal overhead by using a sliding window of past network parameters to compute stable target values.

Averaged-DQN is a reinforcement learning algorithm designed as an extension to Deep Q-Networks (DQN), addressing instability and high variance inherent in standard DQN by averaging multiple past Q-value estimates to compute more stable target values. This technique provably reduces the variance of the target approximation error (TAE), controls overestimation bias, and empirically leads to improved learning stability and performance across a suite of deep reinforcement learning benchmarks (Anschel et al., 2016).

1. Background: Instability and Sources of Error in Deep Q-Learning

Standard DQN employs target networks to stabilize value estimation, updating network parameters $\theta$ by minimizing the squared error between the predicted Q-value and a temporally bootstrapped target,

$$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$

where $\theta^-$ denotes the target network's parameters, periodically synced with $\theta$.

However, practical implementations of DQN exhibit pronounced instability:

• Run-to-run variability: Identical hyperparameters and architectures can yield widely diverging performance due to sensitivity to random seed.
• Spiking and collapse in evaluation scores during training.

Two primary mechanisms underlie this instability:

• Target Approximation Error (TAE): Inexact minimization, restricted network capacity, and finite replay buffers inject noise into the value estimate targets.
• Overestimation Bias: The $\max_a$ in the Bellman target amplifies positive noise, further inflating Q-values.

An error decomposition (after Thrun & Schwartz, 1993) for the per-update deviation $\Delta_i(s, a)$ of the learned Q-value vs. $Q^*(s, a)$ yields: $$\Delta_i = \underbrace{Q(s,a;\theta_i) - y_i}_{\text{Target Approximation Error}} + \underbrace{y_i - (r + \gamma \max_a Q^*(s', a))}_{\text{Overestimation}} + \underbrace{(r + \gamma \max_a Q^*(s', a) - Q^*(s, a))}_{\text{Optimality Gap}}$$ This decomposition motivates explicit variance reduction in the target network update mechanism.

2. Algorithmic Definition and Implementation

Averaged-DQN maintains a sliding window of the $K$ most recent Q-network parameter snapshots, $\{\theta_{i-1}, ..., \theta_{i-K}\}$. The target for each Bellman update is computed as the mean value output by these $$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$0 historical Q-networks: $$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$1 At each learning iteration:

1. An $$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$2-greedy policy is used with respect to $$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$3, and the experienced transition $$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$4 is stored.
2. A mini-batch is sampled from the replay buffer.
3. For each transition, the update target is $$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$5 as above.
4. Parameters $$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$6 are updated by descending the stochastic gradient of the mean squared loss,

$$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$7

The final Q-function for policy extraction is the ensemble mean,

$$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$8

Averaged-DQN thus only replaces the target calculation step and requires circular buffer bookkeeping for historical networks.

3. Theoretical Analysis of Variance Reduction

A core analytical contribution is the characterization of how Averaged-DQN reduces the variance of the learned Q-values under TAE. In a stylized M-state chain MDP with zero rewards and deterministic dynamics, the standard DQN update accumulates noise as: $$y_i = r_i + \gamma \max_{a}\, Q(s_{i+1}, a; \theta^-),$$9 where $\theta^-$0 are i.i.d., zero-mean noise terms of variance $\theta^-$1. This yields

$\theta^-$2

For "Ensemble-DQN" (independently trained, non-interacting ensemble averaged at inference): $\theta^-$3 Averaged-DQN, by averaging across the historical parameter trajectory within a single training run, achieves strictly lower variance: $\theta^-$4 where $\theta^-$5 for $\theta^-$6, and thus $\theta^-$7. Therefore, Averaged-DQN attains at least a $\theta^-$8 reduction in TAE-driven variance compared to standard DQN, and often strictly better.

4. Empirical Evaluation and Results

The algorithm has been evaluated on the Arcade Learning Environment benchmarks (Breakout, Seaquest, Asterix), using a canonical deep convolutional architecture with three convolutional layers and a fully connected layer, following [Mnih et al., 2015]. Each method is run across multiple seeds for 120 million frames.

Key empirical results:

Game	DQN (K=1)	Avg-DQN (K=5)	Avg-DQN (K=10)
Breakout	245.1 (124.5)	381.5 (20.2)	381.8 (24.2)
Seaquest	3775.2 (1575.6)	5740.2 (664.8)	9961.7 (1946.9)
Asterix	195.6 (80.4)	6960.0 (999.2)	8008.3 (243.6)

Empirically:

• Increasing $\theta^-$9 nearly eliminates late-training performance collapse and shrinks run-to-run standard deviation by a factor of three or more.
• Final scores are substantially higher as $\theta$0 increases, with effect sizes sufficient to yield human-level performance where DQN fails.

Computational overhead arises only at target-calculation time (proportional to $\theta$1 forward passes per sample), with backpropagation cost unchanged. This is a modest cost on modern hardware.

5. Applications, Integration, and Best Practices

Averaged-DQN is particularly effective:

• In domains with sparse rewards or long horizons (where TAE accumulation is severe).
• When standard DQN exhibits noisy learning curves or persistent overestimation.

Typical practical choices:

• An averaging window of $\theta$2 to $\theta$3 captures most of the stabilizing effect with moderate additional cost.
• Tuning $\theta$4 via held-out validation or early-training stability metrics is advisable for new domains.

The algorithm is compatible with orthogonal DQN improvements (Double-DQN, prioritized replay, dueling networks). In modular DRL libraries (e.g., Stable Baselines, Dopamine), implementation requires adding a history buffer and modifying the target computation as above.

6. Comparative Perspective and Limitations

Averaged-DQN shares conceptual motivation with ensemble methods, but differs by exploiting the temporal correlation in the learning trajectory of a single agent rather than requiring independent learners. Compared to Full Gradient DQN and average-reward variants, Averaged-DQN targets reduction of variance in the discounted Bellman update setting and does not alter the policy evaluation or control protocol (Pagare et al., 2023). The computational expense, though moderate, is not negligible for very large $\theta$5 values or in resource-constrained settings.

Averaged-DQN does not explicitly address the bias induced by the max operator but controls the amplification of noise that leads to overestimation. In problems where other sources of bias or instability dominate, additional techniques may be needed.

7. Summary

Averaged-DQN is a principled, low-overhead extension of DQN, systematically reducing target estimation variance and suppressing learning instabilities arising from target approximation error. It offers provable and demonstrated empirical benefits, with adoption facilitated by its ease of integration into standard deep RL frameworks (Anschel et al., 2016).