---
title: Deep Attention Recurrent Q-Network (DARQN)
url: https://www.emergentmind.com/topics/deep-attention-recurrent-q-network-darqn
type: topic
---

# Deep Attention Recurrent Q-Network (DARQN)

Searching arXiv for DARQN and closely related attention-recurrent Q-network papers.
Deep Attention Recurrent Q-Network (DARQN) is a value-based deep reinforcement learning architecture that extends Deep Q-Networks by combining recurrent memory with neural visual attention. Introduced in "Deep Attention Recurrent Q-Network" [1512.01693], DARQN augments the DQN lineage with an LSTM and an attention mechanism over convolutional feature maps, so that the agent can integrate observations over time and selectively emphasize spatially informative regions of the input. In its canonical formulation, a single grayscale frame is encoded by a convolutional network, transformed into a set of spatial location vectors, filtered through either soft or hard attention to produce a context vector, and then processed by an LSTM whose hidden state is mapped to action values. DARQN was proposed both as a performance-oriented extension of DQN and as an interpretability-oriented architecture whose attention maps expose where the agent is focusing during decision making [1512.01693]. A closely related later formulation, described as a soft-attention recurrent Q-learning agent for Atari, follows the same core pattern of CNN feature maps, soft spatial attention, LSTM recurrence, and Q-value prediction, and is widely read as a DARQN-style architecture [1612.05753].

## 1. Historical position and conceptual motivation

DARQN emerged from the early sequence of Atari-based deep RL architectures built around Q-learning from raw visual input. DQN used a convolutional network over the last four frames and produced action values directly from that stacked representation. DRQN then replaced DQN’s fully connected layer with an LSTM and used only the most recent frame, delegating temporal integration to recurrence. DARQN extends this progression by inserting an attention mechanism between the convolutional encoder and the recurrent state update, making it an attention-augmented DRQN in both architecture and motivation [1512.01693].

The original DARQN paper identifies two practical limitations of DQN. The first is limited temporal memory: a fixed four-frame stack may be insufficient when relevant information extends farther back in time. The second is inefficient visual processing: the network processes the entire image uniformly, despite the fact that only a subset of visual regions may be relevant to action selection. DARQN addresses the first issue with an LSTM and the second with attention over spatial CNN features [1512.01693].

A further motivation is interpretability. Whereas DQN does not directly reveal which visual regions are responsible for a chosen action, DARQN exposes an explicit attention distribution over locations. This made it one of the early deep RL models in which internal perceptual focus could be monitored online [1512.01693].

Later work reinforced this conceptual framing. A 2016 Atari study proposed a soft attention mechanism combined with DQN to teach an RL agent both how to play and where to look, explicitly emphasizing task-relevant fixation prediction in interactive environments [1612.05753]. In a different domain, a multimodal robot-interaction system, MDARQN, adapted the same design logic—CNN feature grid, soft attention, LSTM, and Q-values—to grayscale and depth streams, making the attention not only interpretable but physically perceivable through robot orientation [1702.08626]. These later systems suggest that DARQN is best understood as an architectural family rather than a single isolated model.

## 2. Core architecture

The canonical DARQN pipeline in [1512.01693] is:

\[
s_t \rightarrow \text{CNN} \rightarrow v_t=\{v_t^1,\dots,v_t^L\} \rightarrow z_t \rightarrow \text{LSTM} \rightarrow h_t \rightarrow Q(s_t,\cdot)
\]

Here, the input is a single grayscale frame of size \(84 \times 84 \times 1\), rather than DQN’s conventional four-frame stack [1512.01693]. The CNN is described as similar to that of Mnih et al., except that the output of the third convolutional layer contains 256 feature maps of size \(7 \times 7\). This yields a spatial feature tensor of shape \(7 \times 7 \times 256\), which is then reorganized into \(L=49\) location vectors:

\[
v_t = \{v_t^1,\ldots,v_t^L\}, \qquad v_t^i \in \mathbb{R}^{256}.
\]

Each \(v_t^i\) summarizes one spatial region of the screen across channels. The attention mechanism operates over these 49 vectors, producing a context vector \(z_t \in \mathbb{R}^{256}\) that is then fed to an LSTM with 256 hidden units [1512.01693]. The LSTM hidden state is passed through a linear layer to produce one Q-value per legal action.

This organization is central to the DARQN idea. Instead of collapsing the full image into a monolithic global representation before temporal processing, the network preserves a structured set of candidate spatial descriptors and lets attention compute a task-conditioned summary. A later DARQN-like formulation for Atari makes this design explicit in the sequence

\[
\text{frame} \rightarrow \text{CNN feature maps} \rightarrow \text{soft spatial attention} \rightarrow \text{LSTM} \rightarrow \text{Q-values over actions}
\]

and interprets the hidden state as simultaneously supporting control and attentional allocation [1612.05753].

The same pattern recurs in MDARQN, although with two modalities. Each stream processes \(1 \times 198 \times 198\) input through four convolutional layers, yielding \(7 \times 7 \times 256\) features that are reshaped into \(L=49\) vectors of dimension \(D=256\). Soft attention computes an annotation vector \(z_t\), an LSTM with 256 units updates the recurrent state, and a Q-head outputs values for four social actions [1702.08626]. This multimodal adaptation illustrates that the DARQN template is not tied to Atari-specific preprocessing or a single modality.

## 3. Attention mechanisms

DARQN studies two attention variants: soft attention and hard attention [1512.01693].

In the soft-attention model, the attention network produces a normalized weight for each location vector \(v_t^i\), conditioned on that location and the previous recurrent hidden state \(h_{t-1}\). The paper writes the attention scoring function as

\[
g(v_t^i, h_{t-1}) = exp(Linear(Tanh(Linear(v_t^i) + W h_{t-1})))/Z,
\]

with \(Z\) as the normalizing constant. The context vector is then

\[
z_t = \sum_{i=1}^L g(v_t^i, h_{t-1})v_t^i.
\]

This produces a convex combination of all 49 location vectors, so the attention mechanism is deterministic and fully differentiable [1512.01693]. Because every operation is smooth, gradients from the Bellman error can propagate through the Q-head, the LSTM, the attention network, and the CNN.

The later Atari attention model in [1612.05753] uses the same essential structure. There, the final CNN tensor has shape \(7 \times 7 \times 64\), giving \(K^2=49\) regional descriptors \(C_{t,i}\in\mathbb{R}^{64}\). Attention scores are computed by a compatibility function between \(C_{t,i}\) and \(h_{t-1}\), normalized with a softmax over spatial positions, and used to form the attended context

\[
c_t = \sum_{i=1}^{K^2} \alpha_{t,i} C_{t,i}.
\]

That paper explicitly characterizes the mechanism as top-down, since the previous hidden state modulates where the model looks next [1612.05753]. This is a useful interpretive lens for DARQN more generally: attention is not a purely bottom-up saliency estimator, but a task-conditioned operator driven by recurrent state.

In hard attention, DARQN samples a single location \(i_t\) from a categorical distribution \(\pi_g(i_t \mid v_t,h_{t-1})\) parameterized by the same attention network. The selected location determines the context vector, but the discrete sampling step is non-differentiable, so ordinary backpropagation cannot train the attention policy directly [1512.01693]. The paper therefore uses a REINFORCE-style update:

\[
\Delta \theta_t^g \propto \nabla_{\theta_t^g} \log \pi_g(i_t \mid v_t,h_{t-1}) R_t,
\]

and then gives the specific update

\[
\theta_{t+1}^g = \theta_t^g + \alpha \nabla_{\theta_t^g} \log \pi_g(i_t \mid v_t,h_{t-1}) (G_t - Y_t),
\]

where \(G_t = Linear(h_t)\) serves as a learned baseline or return approximation [1512.01693]. The paper notes a sign convention that is somewhat unusual relative to standard advantage notation, but the equation should be preserved as written.

Hard attention introduces several stabilization details. CNN weights are initialized from a trained soft-attention model; with 50% probability the context is computed using the soft-attention formula even during hard-attention training; and the convolutional layers receive gradients from both Bellman-error optimization and the attention-policy update [1512.01693]. These details reflect the practical difficulty of training discrete perceptual selection in Q-learning systems.

## 4. Recurrent state, Q-learning objective, and optimization

DARQN remains a Q-learning algorithm. The LSTM integrates attended context vectors over time, producing hidden state \(h_t\) and cell state \(c_t\). Although [1512.01693] does not print the gate equations, its role is explicit: it summarizes motion, delayed dependencies, and latent task state when only one frame is observed per step. The same hidden state also conditions future attention, creating a feedback loop between memory and perceptual selection.

The Bellman target in the DARQN paper is

\[
Y_t = r_t + \gamma \max_{a_{t+1}} Q(s_{t+1},a_{t+1};\theta_{t-1}),
\]

and the semi-gradient update is

\[
\theta_{t+1} = \theta_t + \alpha(Y_t - Q(s_t,a_t;\theta_t))\nabla_{\theta_t}Q(s_t,a_t;\theta_t).
\]

The corresponding loss is a squared TD error expectation over behavior samples [1512.01693]. The paper’s notation uses \(\theta_{t-1}\) in the target, but it also clearly states that a target network is used and updated periodically.

The training protocol follows the DQN paradigm with replay memory and target networks. DARQN uses replay memory of 500,000 tuples, minibatch size 32, target network updates every 10,000 steps, and network updates every 4 steps. Training runs for 5M steps with RMSProp, momentum \(0.95\), and discount factor \(\gamma=0.99\). Exploration follows an \(\epsilon\)-greedy schedule with \(\epsilon\) linearly annealed from 1 to 0.1 over 1M steps. Learning rates differ by attention type: soft attention uses \(\alpha: 0.01 \rightarrow 0.00025\) over 1M steps, while hard attention uses \(\alpha: 0.001 \rightarrow 0.00025\) over the same period [1512.01693].

Recurrent training uses backpropagation through time with 4 unroll steps in the main experiments, and an additional 10-step unroll experiment on Breakout. For each new minibatch, the initial LSTM hidden and memory states are zeroed [1512.01693]. This implies that replayed segments are treated as short finite sequences rather than long recurrent trajectories with carried-over state.

A closely related 2016 soft-attention recurrent Q-network likewise uses end-to-end TD learning with experience replay, replay memory 500,000, minibatch size 32, RMSProp with learning rate \(0.00025\), momentum \(0.95\), \(\gamma=0.99\), and 2 million training steps. It specifies a two-layer LSTM with 64 hidden units per layer, truncated BPTT of sequence length 4, zero initialization of hidden and cell states at each training step, and gradient clipping on LSTM gradients to 10 [1612.05753]. This later configuration shows that DARQN-like designs rapidly settled into a recognizable recurrent Q-learning recipe.

## 5. Empirical results and evaluation

The original DARQN paper evaluates on five Atari 2600 games: Breakout, Seaquest, Space Invaders, Tutankham, and Gopher [1512.01693]. The reported best average reward per episode across 100 epochs is summarized below.

| Model | Breakout | Seaquest | Space Invaders |
|---|---:|---:|---:|
| DQN | 241 | 1,284 | 916 |
| DRQN | 72 | 1,421 | 571 |
| DARQN hard | 20 | 3,005 | 558 |
| DARQN soft | 11 | 7,263 | 650 |

| Model | Tutankham | Gopher |
|---|---:|---:|
| DQN | 197 | 1,976 |
| DRQN | 181 | 3,512 |
| DARQN hard | 128 | 2,510 |
| DARQN soft | 197 | 5,356 |

These results show that DARQN is not uniformly superior to DQN. Its strongest success case is Seaquest, where soft DARQN reaches 7,263 versus 1,284 for DQN and 1,421 for DRQN. Soft DARQN is also best on Gopher with 5,356. On Tutankham, soft DARQN matches DQN at 197. By contrast, DQN remains strongest on Space Invaders and especially Breakout, where DQN’s 241 exceeds all recurrent or attention-based variants by a wide margin [1512.01693].

The paper therefore supports a selective rather than universal interpretation: attention and recurrence can materially help on some games, but may underperform on tasks where short-horizon reactive control is already well handled by stacked-frame DQN. The authors hypothesize that Breakout’s poor DARQN results may be partly due to the short 4-step unroll; increasing the unroll to 10 improves performance somewhat, but still does not surpass DQN [1512.01693].

One architectural claim supported by the experiments is parameter efficiency. For Seaquest with 18 actions, DQN and DRQN have 1,693,362 parameters, while DARQN hard has 845,428 and DARQN soft has 845,171 [1512.01693]. This suggests that attention can reduce network size while still achieving strong performance on some tasks.

A later DARQN-like Atari paper shifts the evaluation emphasis from control score to fixation prediction. On Pong, Phoenix, Enduro, Breakout, and Seaquest, the soft-attention model is compared against Itti-Koch saliency and GBVS using NSS and ROC/AUC based on human click judgments. Reported scores include Breakout soft attention NSS \(1.326\), ROC \(0.787\); Pong NSS \(0.846\), ROC \(0.760\); and Seaquest NSS \(0.571\), ROC \(0.694\), generally outperforming bottom-up baselines [1612.05753]. The protocol uses three human subjects who watch agent gameplay videos at 5 fps and click where they believe they should look. This is not eye tracking, but it provides evidence that task-conditioned RL attention better matches human fixation behavior than bottom-up saliency in interactive settings [1612.05753].

## 6. Interpretability, extensions, and related architectures

One of DARQN’s most durable contributions is interpretability through attention visualization. The original paper reconstructs visual focus by creating 256 subsidiary \(7 \times 7\) feature maps from the attention outputs, then upsampling them into a spatial overlay. Qualitatively, soft DARQN in Breakout focuses on the ball trajectory, while in Seaquest it can emphasize the oxygen indicator during resurfacing and shift toward the submarine as the situation changes [1512.01693]. Hard DARQN also exhibits interpretable focus, such as immediate responses to short-term ball disappearance in Breakout or attention to an enemy until destruction in Seaquest [1512.01693].

This interpretability later became a design objective in its own right. In MDARQN, the attention output is used to orient the robot’s body or head, making its focus visible to nearby humans [1702.08626]. The system has two identical streams, grayscale and depth, each consisting of four convolutional layers, soft attention over 49 locations, an LSTM with 256 units, and a Q-head over four actions: wait, look towards human, wave hand, and handshake. The per-stream Q-values are normalized and averaged for late fusion [1702.08626]. The robot learns through a 14-day two-phase process, alternating between public interaction and offline learning. Reported results show that MDARQN(Aug) achieves a handshake ratio of 0.74 versus 0.48 for a multimodal DQN baseline, while offline accuracy remains similar, suggesting that attention contributed more to socially perceivable interaction than to raw classification-style action accuracy [1702.08626]. This suggests that DARQN-style attention can matter behaviorally by externalizing intent, not only computationally.

DARQN also serves as a useful contrast point for architectures that use recurrence but not visual attention. DRPIQN, for example, is recurrent and Q-learning based but introduces a policy-inference branch rather than an explicit attention mechanism. Its contribution is to infer collaborators’ or opponents’ behavior policies and fuse those policy features into the Q-value pathway. The paper explicitly states that it uses no DARQN-style visual attention; its use of the term “attention” refers only to adaptive loss weighting during training [1712.07893]. This contrast clarifies that DARQN’s defining addition is perceptual attention over visual or spatial features, not just any auxiliary mechanism inserted into a recurrent Q-network.

More recent domain-specific work such as ARDDQN for UAV coverage path planning and data harvesting remains DARQN-like in a broad sense—combining deep function approximation, recurrence, attention, and Q-learning—but differs materially from canonical DARQN. In ARDDQN, attention is described over LSTM hidden states rather than spatial CNN features, and the backbone is DDQN with global-local map engineering rather than Atari screen processing [2405.11013]. This suggests that the DARQN family has expanded into a wider class of attention-recurrent value-learning systems, though not all descendants preserve the original spatial-attention formulation.

## 7. Limitations, ambiguities, and enduring significance

DARQN has several limitations visible already in the original paper. First, performance gains are heterogeneous across games. The model excels on Seaquest and Gopher but performs substantially worse than DQN on Breakout and somewhat worse on Space Invaders [1512.01693]. Second, the hard-attention variant is difficult to optimize, plausibly because of high-variance policy-gradient updates and local optimum issues. Third, the number of recurrent unroll steps matters: short BPTT windows can handicap recurrent architectures in tasks requiring longer temporal credit assignment [1512.01693].

Reproducibility is also imperfect. The DARQN paper does not restate exact convolutional kernel sizes and strides, instead describing the CNN as similar to that of Mnih et al. [1512.01693]. The hard-attention context-selection equation is described operationally rather than fully formalized. The later Atari fixation-prediction paper is explicit about CNN layer sizes and attention equations, but does not clearly specify a separate DQN-style target network, and some replay details for recurrent training remain sparse [1612.05753]. MDARQN also leaves several details ambiguous, including the exact treatment of the stated 8 most recent frames relative to its single-image CNN description and the precise normalization used before Q-value fusion [1702.08626].

Despite these caveats, DARQN retains historical importance as an early synthesis of three strands that later became central across deep RL and sequence modeling: convolutional representation learning, recurrent state estimation, and attention-guided information selection. Its key conceptual move was to make action-value estimation depend on a selective, temporally contextualized representation rather than a fixed global image encoding. In practical terms, this meant that an agent could learn both what to do and where to look from the reinforcement signal itself [1512.01693]. The subsequent Atari fixation study strengthened the claim that such attention is task-driven rather than purely saliency-driven [1612.05753], while MDARQN showed that in embodied systems the same mechanism can support socially interpretable behavior [1702.08626].

DARQN is therefore best understood not as a uniformly dominant Atari agent, but as a foundational architecture in attention-based deep RL: an attention-augmented recurrent Q-network that established the feasibility of jointly learning spatial focus, temporal memory, and value-based control from end-to-end reinforcement learning [1512.01693].

Source: https://www.emergentmind.com/topics/deep-attention-recurrent-q-network-darqn