Natural Gradient Deep Q-Learning

• NGDQN is a reinforcement learning algorithm that leverages natural-gradient optimization via the Fisher information matrix to improve stability and accelerate learning.
• It replaces the standard target network update strategy with a matrix-free Krylov solver for efficient natural gradient computation, leading to smoother and faster convergence.
• Empirical evaluations on classic control tasks demonstrate that NGDQN achieves competitive performance with reduced hyperparameter sensitivity compared to standard DQN.

Natural Gradient Deep Q-Learning (NGDQN) is a reinforcement learning algorithm that integrates natural-gradient optimization into the Deep Q-Network (DQN) framework. By embedding second-order information via the Fisher information matrix, NGDQN significantly stabilizes and accelerates value-function optimization, achieving competitive or superior results to DQN (with or without target networks) across multiple classic control domains. Furthermore, NGDQN demonstrates reduced sensitivity to hyperparameter choices, streamlining the training pipeline for deep value-based agents (Knight et al., 2018).

1. Foundations and Natural-Gradient Derivation

Let $\theta$ denote the parameters of a Q-network $Q(s,a;\theta)$. Standard DQN minimizes the one-step squared temporal difference (TD) loss,

$$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$

where the TD target is defined as $y = r + \gamma \max_{a'} Q(s',a';\theta)$ and is held constant with respect to $\theta$ during optimization.

The gradient of this loss is: $$\nabla_\theta \mathcal{L}(\theta) = \mathbb{E}_\mathcal{D} \left[ 2 (Q(s,a;\theta) - y) \nabla_\theta Q(s,a;\theta) \right].$$

To derive the natural gradient, the Q-network is interpreted as a probabilistic model: for any state $s$ and action $a$, the output is modeled as the mean of a Gaussian with fixed variance $\beta^2$, i.e.,

$$p_\theta(q \mid s) = \mathcal{N}\left(q \mid Q(s,a;\theta), \beta^2\right).$$

The Fisher information matrix (FIM) for this model is: $Q(s,a;\theta)$0 where $Q(s,a;\theta)$1 is the Jacobian of $Q(s,a;\theta)$2 with respect to $Q(s,a;\theta)$3.

The natural-gradient update at iteration $Q(s,a;\theta)$4 is given by: $Q(s,a;\theta)$5 where $Q(s,a;\theta)$6 is the learning rate. For computational tractability, the update is computed as the solution $Q(s,a;\theta)$7 to the linear system

$Q(s,a;\theta)$8

with $Q(s,a;\theta)$9 providing damping.

Due to the prohibitive cost of explicit matrix inversion, NGDQN employs matrix-free Krylov methods—either MinRes-QLP (robust under ill-conditioning) or linear conjugate gradient (LinCG, for symmetric positive definite systems)—to iteratively solve for $$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$0. Jacobian-vector products required in this procedure are efficiently computed via automatic differentiation.

2. Algorithm, Implementation, and Hyperparameters

An outline of the NGDQN algorithm is as follows:

• Experience is accumulated in a replay buffer with capacity $$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$1.
• The Q-network is initialized with random parameters $$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$2.
• At each timestep within an episode:
  • The agent acts $$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$3-greedily.
  • New experience tuples $$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$4 are added to the replay buffer.
  • A minibatch of size $$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$5 is sampled.
  • For each batch sample, the TD target $$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$6 is computed.
  • The loss gradient $$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$7 is computed over the batch.
  • The Fisher operator (matrix-vector product) $$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$8 is defined using auto-differentiation.
  • For input vector $$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[(y - Q(s,a;\theta))^2\right],$$9, the Krylov solver is used to solve $y = r + \gamma \max_{a'} Q(s',a';\theta)$0.
  • Parameters are updated: $y = r + \gamma \max_{a'} Q(s',a';\theta)$1.
  • Learning rate $y = r + \gamma \max_{a'} Q(s',a';\theta)$2 is decayed.

Significant differences from standard DQN:

• NGDQN does not use a separate target network.
• Updates use natural gradients rather than SGD or Adam.
• The damping constant $y = r + \gamma \max_{a'} Q(s',a';\theta)$3 and Krylov solver type are essential hyperparameters.

Empirical settings use $y = r + \gamma \max_{a'} Q(s',a';\theta)$4 (decayed by $y = r + \gamma \max_{a'} Q(s',a';\theta)$5), batch sizes $y = r + \gamma \max_{a'} Q(s',a';\theta)$6, replay sizes up to $y = r + \gamma \max_{a'} Q(s',a';\theta)$7, and $y = r + \gamma \max_{a'} Q(s',a';\theta)$8-greedy schedules decaying $y = r + \gamma \max_{a'} Q(s',a';\theta)$9 from $\theta$0. The damping parameter can be static or adaptively adjusted via the Levenberg–Marquardt heuristic.

NGDQN displays marked robustness to hyperparameter choices, outperforming DQN in sensitivity tests over wide grids.

3. Theoretical Properties and Algorithmic Consequences

Natural gradient methods precondition the loss gradient with the inverse Fisher matrix, which geometrically corresponds to taking steps with respect to the Riemannian metric defined on the model’s output distributions. This sidesteps plateaus and unstable regions in parameter space arising from ill-conditioning. NGDQN's order-invariance to sample sequencing further enhances the efficacy of experience replay.

Unlike standard DQN, which relies on a slowly updated target network to provide stability, NGDQN’s second-order updates from the Fisher information matrix obviate the need for target-network stabilization. This leads to both faster convergence—often requiring fewer episodes to reach optimal policy performance—and reduced training variance.

While global convergence in deep Q-learning with nonlinear function approximation remains unproven, natural-gradient approaches guarantee local convergence under smoothness assumptions, as established in foundational work (Amari, Pascanu). In practical experiments, NGDQN was never observed to diverge.

4. Empirical Evaluation and Benchmark Results

Empirical comparisons were performed on OpenAI Gym classic control tasks: CartPole-v0, CartPole-v1, Acrobot-v1, and LunarLander-v2. The evaluation compared:

• DQN (no target network, Adam optimizer)
• DQN with target network
• NGDQN (no target network)

Performance was measured as the best 100-episode average reward over training, summarized as follows:

Environment	DQN (no TN)	DQN (with TN)	NGDQN
CartPole-v0	180 (±20)	195 (±5)	195 (±3)
CartPole-v1	320 (±50)	475 (±10)	480 (±8)
Acrobot-v1	−100 (±20)	−80 (±5)	−75 (±4)
LunarLander-v2	150 (±30)	200 (±15)	210 (±10)

In all domains, NGDQN matches or outperforms DQN with a target network and exhibits dramatic gains over DQN without one. Learning curves for NGDQN are characteristically smoother and less variable. Statistical analysis (paired t-test, 10 trials) shows NGDQN's final performance is significantly superior ($\theta$1) to DQN on all tasks.

NGDQN’s performance is robust across approximately 100 hyperparameter configurations, as indicated by the relatively flat ranking of final rewards compared to DQN, which exhibits greater sensitivity.

5. Practical Guidelines and Limitations

Optimal practice with NGDQN involves:

• Krylov solvers (MinRes-QLP or LinCG) with mild damping (5–10% of the maximum eigenvalue of $\theta$2).
• Batch sizes in the 32–128 range and replay capacity of at least $\theta$3.
• Decaying $\theta$4-greedy policy and learning rate decay ($\theta$5).
• No requirement to maintain a target network, as training is inherently stabilized.

NGDQN’s reduced hyperparameter sensitivity offers operational simplicity over baseline DQN implementations that rely heavily on tuning Adam’s parameters and target network settings.

Open questions and future avenues include scaling to high-dimensional observation spaces (such as Atari environments), extensions to actor-critic architectures, and rigorous analysis of global convergence properties.

6. References and Related Work

Significant prior work on natural-gradient and second-order optimization in neural models include Amari (1998) [Neural Computation], Pascanu & Bengio [ICLR Workshop, 2014], Martens & Grosse [ICML 2015], and foundational policy-gradient methods (Kakade, Peters & Schaal). The original DQN was introduced by Mnih et al. [Nature 2015]. The complete account of NGDQN is provided by Knight & Lerner (Knight et al., 2018).