Episodic Memory Deep Q-Networks (EMDQN)

• Episodic Memory DQN (EMDQN) augments standard DQN with a memory module recording highest returns, enhancing learning in sparse-reward settings.
• Key results show EMDQN achieves human-comparable performance on Atari games faster than traditional DQN, reducing required frames significantly.
• EMDQN effectively stabilizes learning and improves sample efficiency using auxiliary memory-based supervision in reinforcement learning.

Episodic Memory Deep Q-Networks (EMDQN) is a biologically inspired reinforcement learning (RL) framework that augments the standard Deep Q-Network (DQN) algorithm with a tabular episodic memory of maximal empirical returns. EMDQN addresses the sample inefficiency prevalent in deep RL by supervising value learning with previously observed high-return trajectories, thereby accelerating reward propagation, particularly in sparse- or delayed-reward environments. This approach is structured around integrating a memory module that records the highest returns seen for each encountered state–action pair, and utilizing this information as an auxiliary signal during Q-learning updates (Lin et al., 2018).

1. Motivational Context and Background

Sample inefficiency represents a critical bottleneck in vanilla DQN, which relies on bootstrapping with one-step temporal difference targets: $Q(s, a) \leftarrow r + \gamma \max_{a'} Q(s', a')$. This form of learning can propagate information about delayed or sparse rewards slowly across the value function, often demanding tens or hundreds of millions of frames for human-level performance on benchmarks such as Atari games. Conventional experience replay partially ameliorates temporal correlations in the data but still fails to ensure rapid reward propagation, particularly for rare high-reward episodes that may be underrepresented in the replay buffer.

Neuroscientific studies of episodic memory highlight the benefit of recalling entire past episodes, rather than isolated transitions, for credit assignment in decision-making. If an agent can store and retrieve the highest observed return for each state–action pair, this information can serve as an instructive target during value learning, potentially enhancing convergence speed and robustness, especially in environments where high returns are infrequent.

2. Algorithmic Framework

2.1 Episodic Memory Storage and Querying

EMDQN maintains a table $\mathcal{H}$ indexed by state–action pairs $(s, a)$, initialized to $-\infty$. After each episode, the discounted return $G_t$ is computed for every time step $t$ as $G_t = \sum_{k=0}^{T-t} \gamma^k r_{t+k}$, where $T$ is the episode length. $\mathcal{H}(s_t, a_t)$ is then updated as $$\mathcal{H}(s_t, a_t) = \max\{\mathcal{H}(s_t, a_t), G_t\}$$, ensuring that only the maximal observed return for each $\mathcal{H}$0 is retained.

Querying $\mathcal{H}$1 for a given $\mathcal{H}$2 is an $\mathcal{H}$3 operation with appropriate hashing or tabular storage, enabling the agent to rapidly access the best empirical experience for supervision during learning.

2.2 Loss Augmentation and Target Interpolation

EMDQN modifies the DQN loss by introducing an episodic supervision term:

$\mathcal{H}$4

where

$\mathcal{H}$5

$\mathcal{H}$6

Here, $\mathcal{H}$7 weights the episodic supervision loss, and the indicator ensures the penalty is imposed only when $\mathcal{H}$8 has been initialized via actual experience. The target network parameters $\mathcal{H}$9 are periodically synchronized with the online network. Alternatively, the target can be interpolated as $(s, a)$0, but the two-term loss is equivalent when expanded.

3. Training Procedure and Implementation

A high-level outline of EMDQN training is as follows:

1. Initialization: Q-network parameters $(s, a)$1, target network $(s, a)$2, episodic memory $(s, a)$3 (all $(s, a)$4), and replay buffer $(s, a)$5.
2. Interaction: For each episode:
   • Record trajectory $(s, a)$6 at each step, appending to both the episode buffer and $(s, a)$7.
   • With probability $(s, a)$8, select a random action; otherwise, act greedily with respect to $(s, a)$9.
   • Every $-\infty$0 steps (or episode termination), sample minibatches from $-\infty$1 and update $-\infty$2 by gradient descent on $-\infty$3.
   • Periodically, synchronize $-\infty$4 with $-\infty$5.
3. Episodic Memory Update: At the end of each episode, for each timestep traversed backwards, compute and propagate $-\infty$6, updating $-\infty$7 to the maximal value ever observed.

In the Atari domain, inputs are $-\infty$8 preprocessed frames, processed by a convolutional architecture (three convolutional layers followed by a 512-unit fully connected layer). Typical hyperparameters: replay buffer size $-\infty$9, minibatch size 32, learning rate $G_t$0 (using Adam or RMSProp), discount $G_t$1, $G_t$2-greedy decay schedule, and episodic loss coefficient $G_t$3 in the range $G_t$4.

4. Empirical Evaluation and Results

4.1 Experimental Protocol

EMDQN was evaluated on the full suite of 49 Atari 2600 games using the Arcade Learning Environment, measuring average score at 2M, 10M, and 50M frames over 3 independent seeds. Human-normalized scores served as the principal metric. Performance was compared to DQN, Neural Episodic Control (NEC), Model-Free Episodic Control (MFEC), MERLIN, and other episodic memory-based baselines.

4.2 Quantitative Gains

• Sample Efficiency: On the median across all games, EMDQN reached DQN’s final (50M frame) performance within approximately 10M frames, yielding a 5-fold reduction in required environment interactions.
• Performance Metrics: At 50M frames, the median human-normalized score increased from 44% (DQN) to 65% (EMDQN).
• Variance Reduction: EMDQN exhibited a 5–10% lower standard deviation of final score across seeds, indicating improved learning stability.
• Benchmark Comparisons: At 10M frames, DQN attained a median of 16% of human performance, NEC reached 32%, and EMDQN achieved 48%, outperforming NEC on 35 of 49 games.

EMDQN notably excelled on sparse-reward environments such as Montezuma’s Revenge and Freeway, highlighting the advantage of episodic return recall. Learning curves for games such as Breakout, Frostbite, and Pong revealed faster initial improvements, higher plateaus, and reduced performance variance across seeds relative to standard DQN and episodic baselines.

5. Theoretical Perspectives and Analysis

The episodic supervision term $G_t$5 serves to anchor $G_t$6 from below by the empirical maximum return, substantially mitigating catastrophic forgetting of rare, high-reward experiences. As DQN’s bootstrapped estimates improve for frequently visited $G_t$7, the influence of $G_t$8 naturally diminishes, so it primarily accelerates early-stage learning without introducing long-term bias. Potential optimism bias may arise if $G_t$9 is dominated by outlier high returns from early episodes; empirical results suggest this is controlled effectively by tuning $t$0 and, if used, annealing the interpolation coefficient $t$1.

EMDQN remains an off-policy algorithm, inheriting the convergence characteristics of DQN augmented by an additional $t$2 penalty term. The principal limitation is the scalability of the tabular episodic memory as the set of unique $t$3 pairs grows, rendering exact indexing impractical in high-dimensional or continuous settings. Extensions proposed include utilizing approximate nearest-neighbor tables or locality-sensitive hashing to generalize episodic memory, and combining with prioritized replay to target both high temporal-difference error and high episodic return transitions.

6. Strengths, Limitations, and Future Directions

EMDQN is readily implemented atop any DQN framework, introducing minimal algorithmic complexity while delivering significantly improved sample efficiency, particularly in domains with delayed or sparse rewards. The memory requirement scales with the cardinality of the discrete $t$4 space, which poses challenges for scalability and generalization to continuous or high-dimensional regimes.

Potential future directions include exploring alternative episodic storage mechanisms to accommodate approximate matching across state–action equivalence classes, integrating prioritized sampling based on episodic return or TD-error, and theoretically characterizing the bias–variance regimes introduced by the interplay of bootstrapped and empirical episodic targets. A plausible implication is that enhanced episodic recall can further speed up RL in combinatorially complex or partially observable tasks, provided effective memory generalization strategies are deployed.

---

For further details, see "Episodic Memory Deep Q-Networks" (Lin et al., 2018).