---
title: Recurrent Deep Q-Networks (DRQN) Overview
url: https://www.emergentmind.com/topics/recurrent-deep-q-networks-drqn
type: topic
---

# Recurrent Deep Q-Networks (DRQN) Overview

A Deep Recurrent Q-Network (DRQN) is a variant of the Deep Q-Network (DQN) architecture for reinforcement learning (RL) specifically designed to handle environments with partial observability. By integrating recurrent layers such as LSTMs or GRUs into Q-learning frameworks, DRQN allows agents to maintain a memory of past observations, aggregate information over time, and thus act effectively even when the underlying Markov state is hidden. DRQN and its derivatives have been widely studied across domains, including Atari games, autonomous driving, fog computing, adaptive systems, dialogue management, and multi-agent learning.

## 1. Foundations and Motivation

The canonical DQN framework assumes full observability and processes the current state (or a stack of recent frames) to estimate Q-values for all actions. In partially observable Markov decision processes (POMDPs), the agent receives incomplete or aliased observations $o_t$ rather than the true state $s_t$. The original DQN addresses this via stacking a fixed window of previous frames, but this is insufficient when critical information lies outside this temporal window or the structure of observation noise is irregular.

To address such scenarios, Hausknecht and Stone introduced DRQN, which augments the DQN with a recurrent layer (typically an LSTM or GRU) after the convolutional backbone. This recurrence enables the agent to construct an internal belief state $h_t$ by integrating arbitrary histories of observations, making DRQN particularly well-suited for environments with occlusion, missing input, or rich temporal dependencies [1507.06527].

DRQN’s advantage becomes more pronounced as the complexity and partial observability of the environment increase, enabling robust policies that adapt to varying degrees of observation fidelity. This property underpins its adoption across partially observable domains [2010.13407], [1704.05495], [1606.02560].

## 2. Network Architectures and Algorithmic Variants

### Core DRQN Structure

A prototypical DRQN consists of:
- **Input:** At each time-step $t$, the agent receives an observation $o_t$ (e.g., an image, feature vector, or sequence).
- **Feature Extraction:** A stack of convolutional or dense layers processes $o_t$. In vision domains, this is typically a 3-layer CNN (e.g., $84\times84$ input to $7\times7\times64$ activation) [1507.06527].
- **Recurrence:** The flattened feature map feeds into an LSTM (or GRU), yielding a hidden state $h_t = \mathrm{LSTM}(h_{t-1}, \phi(o_t))$.
- **Q-Head:** The recurrent output $h_t$ is projected to $|A|$ Q-values via a fully connected layer, where $|A|$ is the discrete action space.

This structure is extensible: stacked LSTMs have been used to improve capacity in urban autonomous driving [2010.13407], and GRU-based DRQN variants are common in resource allocation for fog networks [2007.10581].

### Algorithmic Enhancements

Modern DRQN deployments incorporate key advances for stability and performance:
- **Double Q-Learning:** Decouples action selection and evaluation in the Bellman target to mitigate value overestimation. The Double DRQN (“DRDQN”) update for transition $(o_t, h_{t-1}, a_t, r_t, o_{t+1}, h_t)$ is:
  $$
  y_t = r_t + \gamma Q(o_{t+1}, h_t, a^*; \theta^-), \quad a^* = \arg\max_a Q(o_{t+1}, h_t, a; \theta)
  $$
  [1908.06040], [2010.13407], [1801.01000]
- **Prioritized Experience Replay:** Samples sequences or transitions based on TD-error magnitude for efficient learning. Weighting is applied during backpropagation for bias correction [1801.01000].
- **Eligibility Traces:** Forward-view $\lambda$-returns are used to accelerate credit assignment; e.g., $R_t^\lambda = (1-\lambda)\sum_{n=1}^N \lambda^{n-1} R_t^{(n)}$ [1704.05495].
- **Attention Mechanisms:** Recent extensions (ARDDQN) apply an attention layer on RNN outputs before the Q-head to focus value estimation on salient temporal events [2405.11013].
- **Snapshot Ensembling:** Parameter snapshots from cyclical or annealed learning rates are averaged to form a more robust ensemble policy [1801.01000].

Table 1. Key Architectural Components

| Domain                         | Conv | Recurrence         | Output Head |
|------------------------------- |------|--------------------|-------------|
| Atari [1507.06527], [1908.06040]| Yes  | LSTM (512)         | FC to |A|    |
| Urban driving [2010.13407]      | Yes  | 2× LSTM (256+256)  | FC, 4 Q-values|
| Multi-fog [2007.10581]          | 1-D  | GRU (128)          | FC, |X| Q-values|
| UAV path planning [2405.11013]  | Yes  | LSTM/GRU/BiLSTM    | FC, per action|

## 3. Training Regimen, Optimization, and Experience Replay

### Sequence-based Experience Collection

Standard DQN's experience replay buffer is extended to accommodate sequences for truncated backpropagation through time (BPTT) in DRQN. Each replay batch typically contains $N$ sequences of fixed or variable length $L$, with the LSTM state zeroed at the start of each sampled sequence [1507.06527], [1704.05495].

- Example: In urban autonomous driving, full episodes (up to 1000 steps) are sampled, and training runs 8-step sequences per episode, resetting LSTM states between batches [2010.13407].
- In multi-fog scenarios, sliding-window histories of $T=10$ steps are concatenated and input to the GRU [2007.10581].

### Optimization

- **Optimizers:** Adam, RMSProp, and ADADELTA are commonly used. Adam, often used with a learning rate of $10^{-3}$, yields particularly rapid and stable convergence when combined with $\lambda$-returns [1704.05495], [2010.13407].
- **Gradient Clipping:** Applied to LSTM parameters for stability; e.g., max norm of 10 (or element-wise to $[-1,1]$) [1507.06527], [1908.06040].
- **Target Network Updates:** Employed to stabilize Q-learning; periodic hard updates (e.g., every 10,000 steps) or soft updates (Polyak averaging) [2010.13407], [2011.02243].
- **Exploration:** $\epsilon$-greedy, with schedules annealing $\epsilon$ from 1.0 to 0.1 (or 0.01), sometimes with restarts and decays per sub-task [2007.10581].

BPTT through unrolled sequences enables correct temporal credit assignment, and forward-view eligibility traces further speed up reward propagation [1704.05495].

## 4. Empirical Findings and Comparative Analyses

### Atari / ViZDoom (Games)

- **Performance:** DRQN matches DQN on fully observed Atari games but demonstrates increased robustness under partial observability (e.g., flickering screens) and in longer-horizon credit assignment [1507.06527]. On memory-dependent tasks (e.g., Beam Rider, Tennis with delayed rewards), DRQN and DRQN+$\lambda$ reach target performance faster and more stably [1704.05495].
- **Double DQN and PER:** Integration of double Q-learning yields additional stability and reduces overestimation. Prioritized replay accelerates early learning but may amplify value overestimation [1801.01000].
- **Attention Mechanisms:** In UAV coverage and data harvesting, the attention-augmented LSTM variant yields substantial coverage and landing gains compared to non-recurrent or non-attention models [2405.11013].

### Robotics and Control

- **Urban autonomous driving:** DRQN outperforms rule-based policies in collision avoidance and distance before task termination. Empirical metrics show collision-free rates of 70% versus 40% for baseline, and longer average distances traveled [2010.13407].
- **Fog computing:** DRQN (GRU-based) achieves higher task success rates and lower buffer overflows compared to DQN and DCQN. Under heavy load, improvements are 10–20% in main performance metrics [2007.10581].
- **Self-adaptive microservices:** DRQN enables faster convergence, higher cumulative reward, and lower adaptation time than DQN, DDQN, policy gradient (PGNN), and DDPG in microservice architecture adaptation [1901.04011].

### Dialogue Systems and Multi-Agent Scenarios

- **Dialogue management:** DRQN-based latent state encodings outperform flat RL and modular baselines in task completion rate and robustness under noisy slot recognition [2011.02243], [1606.02560].
- **Multi-agent communication:** Deep Distributed Recurrent Q-Networks (DDRQN) extend DRQN with weight sharing, agent IDs, last-action inputs, and on-policy updates, enabling agents to learn emergent communication protocols for complex riddles [1602.02672].

## 5. Limitations, Domain Considerations, and Design Choices

DRQN’s main strength—learned unbounded memory—incurs practical trade-offs:
- **Sample Efficiency and Overfitting:** DRQNs may require longer training and larger replay buffers; their over-parameterization can cause slower convergence in domains where task-relevant information is contained in a short observation window [1903.04311].
- **Sequence Length Sensitivity:** Truncation length for BPTT is a critical hyperparameter—too short fails to propagate credit; too long leads to instability and unmanageable memory cost [1507.06527], [2011.02243].
- **Stacked Frame vs. Recurrent Memory:** DRQN excels when temporal dependencies exceed the capability of frame stacking. If POMDP history is shallow, stacked-frame DQNs often train faster and more stably [1903.04311].

Authors recommend evaluating the temporal horizon of relevant information and adopting attention or eligibility traces if credit assignment remains inefficient [1704.05495], [2405.11013].

## 6. Advanced Applications and Hybridizations

DRQN’s modular architecture supports augmentation and domain specialization:
- **Hybrid Supervised-RL Models:** Supervised auxiliary losses on the RNN’s hidden state (predicting future user action/slots) accelerate convergence and improve robustness in dialogue systems [2011.02243], [1606.02560].
- **Dyna-style Synthetic Experience:** Mixes real and model-based synthetic transitions to cover rare slots and accelerate learning in dialogue tasks [1606.02560].
- **Attention-based Recurrence:** In ARDDQN, an attention mechanism over an RNN sequence allows the Q-head to selectively focus on time steps most relevant for UAV path planning and data harvesting metrics [2405.11013].
- **Multi-agent Coordination:** DDRQN demonstrates that weight sharing, agent identification, and last-action input are essential for emergence of communication protocols in decentralised settings [1602.02672].

Adaptation of DRQN to actor-critic, prioritized replay, dueling networks, and hierarchical or hybrid architectures is suggested for further performance gains [2010.13407], [2405.11013].

## 7. Outlook and Future Directions

Research directions identified in the literature include:
- Expanding training scales (episodes, replay buffer size) and mixing in more complex scenarios for robustness [2010.13407].
- Integration of advanced RL methods: prioritized experience replay, dueling architectures, and actor-critic schemes [2010.13407], [1801.01000].
- Exploration of domain-specific recurrence: attention, episodic memory, external memory augmentation, or memory hierarchies [2405.11013].
- Investigation of stabilizing techniques for longer BPTT, on-policy/off-policy hybridization, and scalability to large multi-agent systems [1602.02672].

The evidence to date positions DRQN and its augmented variants as foundational tools for deep RL in partially observed environments whenever temporal credit assignment, occlusion, or information asymmetry precludes fully Markovian solutions. The choice of architecture, training regime, and hybridization must be matched to domain requirements to optimize performance, sample efficiency, and generalization.

Source: https://www.emergentmind.com/topics/recurrent-deep-q-networks-drqn