NoisyNet-DQN: Learnable Noise for Exploration

• NoisyNet-DQN is a deep Q-learning algorithm that integrates learnable, factorized Gaussian noise into network parameters to replace heuristic exploration methods.
• The algorithm replaces ε-greedy strategies with a noise-driven approach, enabling efficient exploration and faster learning, notably improving Atari benchmark scores.
• Extensions such as NROWAN-DQN, state-aware variants, and multi-agent adaptations further enhance policy stability and performance while keeping computational overhead minimal.

NoisyNet-DQN is a deep Q-learning algorithm that induces efficient exploration by injecting learnable, parameterized noise into the neural network weights, effectively replacing heuristic exploration schemes, such as ε-greedy, with stochasticity learned via gradient descent. The method, introduced by Fortunato et al., is designed to enable agents to discover superior policies by embedding noise directly into value function estimation, resulting in consistent empirical improvements in discrete action reinforcement learning benchmarks, most notably in Atari 2600 domains (Fortunato et al., 2017). NoisyNet-DQN variants have since been extended and analyzed both theoretically and in practical, multi-agent, and stability-critical settings (Aravindan et al., 2021, Han et al., 2020, He, 2023).

1. Core Architecture and Noise Parameterization

NoisyNet-DQN replaces fully connected (linear) layers’ weights and biases with learnable stochastic variables:

• For a linear layer with $y = W x + b$, the parameters become:

$$W = \mu_W + \sigma_W \odot \varepsilon_W, \quad b = \mu_b + \sigma_b \odot \varepsilon_b$$

where $\mu_W, \mu_b$ are trainable means, $\sigma_W, \sigma_b$ are trainable, non-negative noise scales, and $\varepsilon_W, \varepsilon_b$ are zero-mean random noise variables.

To reduce computational cost, NoisyNet-DQN employs factorized Gaussian noise: given input dimension $p$ and output dimension $q$, sample independent standard normal vectors $\varepsilon^{i} \in \mathbb{R}^p$, $\varepsilon^{o} \in \mathbb{R}^q$, compute $f(u) = \text{sgn}(u)\sqrt{|u|}$, and set:

$$\varepsilon_W[i, j] = f(\varepsilon^{i}_i)f(\varepsilon^{o}_j), \quad \varepsilon_b[j] = f(\varepsilon^{o}_j)$$

This reduces Gaussian samples per layer from $O(pq)$ to $O(p+q)$. Backpropagation is performed through the noisy parameters using standard automatic differentiation: for the loss $$\bar{L}(\theta) = \mathbb{E}_{\varepsilon}[L(\mu + \sigma \odot \varepsilon)]$$, gradients with respect to means and noise-scales are unbiased estimators.

2. Algorithmic Modifications to DQN

The NoisyNet-DQN variant follows the standard DQN algorithm but alters both the network structure and exploration strategy:

• At every step, sample a new $\varepsilon$ for the online network and use the perturbed $Q$-function for action selection via $$\arg\max_a Q_{\text{online}}(x_t,a; \theta, \varepsilon^{\text{online}})$$.
• Transitions are stored in the experience replay buffer as usual.
• During learning, for each minibatch, independently resample $\varepsilon$ for online and target networks and compute targets, loss, and gradient updates accordingly.
• The exploration policy is now inherently stochastic due to the perturbed $Q$-function, obviating the need for external ε-greedy policies.

Hyperparameter settings reported by Fortunato et al. include learning rate $=2.5 \times 10^{-4}$, RMSProp decay $=0.99$, gradient clipping to $[-1,1]$, target network update frequency $=10,000$ steps, replay buffer size $=1,000,000$, batch size $=32$, and default noise initialization (factorized) with $\mu_W[i,j] \sim U[-1/\sqrt{p}, 1/\sqrt{p}]$, $\sigma_W[i,j]=\sigma_0/\sqrt{p}$, $\sigma_0=0.5$ (Fortunato et al., 2017).

3. Theoretical Interpretation and Variational Connection

NoisyNet-DQN has a direct theoretical connection to variational Bayesian deep Q-learning and Thompson sampling. The injection of noise via learned Gaussian perturbations can be interpreted as defining a simple diagonal Gaussian variational posterior $$q(\theta) = \mathcal{N}(\mu, \operatorname{diag}(\sigma^2))$$. In this perspective, NoisyNet-DQN training corresponds to minimizing the expected TD-error (Bellman error) with respect to the current parameter posterior, discarding the explicit KL regularization used in the full evidence lower bound (ELBO) (Aravindan et al., 2021). The resultant policy realizes an implicit, parameter-space version of approximate Thompson sampling, where randomness in action selection emerges from stochasticity in the $Q$-function's parameterization rather than policy-level randomization.

4. Empirical Performance and Practical Utility

Empirical evaluation on the full 57-game Atari 2600 suite, using the "human-normalized" scoring protocol, demonstrates substantial improvements:

• Median human-normalized score improved from 83 for ε-greedy DQN to 123 for NoisyNet-DQN (a 48% improvement).
• Mean human-normalized score increased from 319 to 379.
• Out of 57 games, NoisyNet-DQN matched or exceeded vanilla DQN performance in ≈50 games.
• Learning curves exhibited both more rapid progress and higher final asymptotic scores (Fortunato et al., 2017).

NoisyNet-DQN also does not require manual tuning of ε-schedules or entropy terms, as exploration properties are learned end-to-end with the value function.

5. Algorithmic Extensions and Domain-Specific Variants

Several extensions and variants have been proposed:

• NROWAN-DQN: This variant introduces an explicit loss penalty driving the noise parameters in the output layer toward zero once high-scoring policies have emerged. The penalty term $D$ incentivizes low-variance (i.e., stable) actions in critical stages. The penalty weight $k$ is scheduled online as a function of cumulative reward, producing a smooth transition from broad exploration to exploitation. Across four Gym benchmarks (CartPole, MountainCar, Acrobot, Pong), NROWAN-DQN achieves consistently higher mean returns and significantly reduced variance compared to both standard DQN and vanilla NoisyNet-DQN, particularly in environments sensitive to action stochasticity (Han et al., 2020).
• State-Aware Noisy Exploration (SANE): SANE generalizes the noise-scaling mechanism by making the noise scale $\sigma$ state-dependent via an auxiliary, differentiable network. This enables the agent to modulate exploration based on state risk, resulting in higher scores in games where high state-dependent risk is present. SANE outperforms both ε-greedy and standard NoisyNet-DQN on selected Atari games, especially when injected noise is maintained at evaluation (Aravindan et al., 2021).
• NoisyNet-Multi-Agent DQN: NoisyNet parameter noise has been adopted in multi-agent DQN settings, such as autonomous vehicle platoon overtaking. In such decentralized, multi-agent domains, factorized noise enables scalable, coordinated exploration without $\epsilon$-greedy schedules. Empirical results in multi-agent traffic simulations demonstrated improvements in learning speed, collision avoidance (≈35% relative reduction), and overtaking success rate (≈88% success vs. 72% for standard MADQN) (He, 2023).

6. Computational Overhead and Implementation

The main computational impact of NoisyNet-DQN arises from the doubling of parameters in noisy layers (due to simultaneous learning of $\mu$ and $\sigma$) and the requirement of stochastic forward passes for each policy and learning update. However, the use of factorized Gaussian noise reduces random number generation cost from $O(pq)$ to $O(p+q)$ per layer, which is negligible in standard convolutional and fully connected architectures used in Atari and continuous control tasks. Modern autodiff frameworks (e.g., TensorFlow, PyTorch) handle the gradient computation through stochastic parameters seamlessly (Fortunato et al., 2017, He, 2023).

7. Limitations and Benchmarking Considerations

While NoisyNet-DQN improves convergence rate, final score, and reduces the necessity for exploration hyperparameter tuning, it introduces additional network parameters and stochasticity, which may not always be desirable in highly deterministic or safety-critical deployment scenarios. Results from NROWAN-DQN suggest that, in long-horizon or action-sensitive environments, persistent noise can be detrimental after policy convergence, motivating adaptive or state-aware noise attenuation. The original empirical benchmarks do not consider continuous action domains, and further adaptation may be required in such settings (Han et al., 2020).

---

Algorithm	Core Mechanism	Learning/Exploration Policy	Performance Gains
DQN	Deterministic network, ε-greedy	Fixed stochastic action policy	Baseline
NoisyNet-DQN	Learnable noise in weights/biases	Parameter noise, action greedy	+48% median Atari score, faster learning (Fortunato et al., 2017)
NROWAN-DQN	Output layer noise reduction	Online k-scheduling	Higher return, lower variance (Han et al., 2020)
SANE	State-dependent noise scaling	Auxiliary perturbation net	State-sensitive gains on risk-sensitive tasks (Aravindan et al., 2021)

The table summarizes the principal variations established in the literature, their exploration mechanisms, and corresponding empirical advantages as reported.