---
title: Long-Term Episodic Memory Networks
url: https://www.emergentmind.com/topics/long-term-episodic-memory-networks-lemn
type: topic
---

# Long-Term Episodic Memory Networks

Long-term Episodic Memory Networks (LEMN) are memory-augmented neural architectures designed to address the scalability limitations of contemporary external-memory–based neural networks, particularly for lifelong learning with unbounded data streams in which informative content is sparse relative to memory capacity. LEMN introduces a learnable, RNN-based memory retention agent that dynamically identifies and retains memory entries of task-generic importance by leveraging both relative and historical information, supporting robust performance in navigation and question-answering domains [1812.04227].

## 1. Architectural Overview

LEMN is structured as an augmentation to any external-memory–based network (e.g., MemN2N, BiDAF, Memory Q-Networks) by integrating a memory retention (eviction) agent.

- **External Memory**: The memory comprises $N$ fixed-capacity slots $M_t = [m_{t,1},...,m_{t,N}]$ at timestep $t$, with each slot $m_{t,i} \in \mathbb{R}^d$.
- **Input Encoding**: Incoming datum $x_t$ (which may be a sentence, image, or other modality) is encoded via a learnable function $\psi$ into $c_t = \psi(x_t) \in \mathbb{R}^d$, forming the candidate for possible memory inclusion.
- **Memory Retention Agent**: Upon memory saturation, a policy $\pi$ selects a slot for eviction (or enacts a "no-op" to skip writing). LEMN provides three categories of retention agents, leveraging varying depths of spatial and temporal context.

## 2. Mathematical Formulation of Retention

LEMNs’ policy is formulated as a categorical distribution over memory slots, parameterized via task-specific retention scores.

### Memory Retention Pipeline
- **Slot Embeddings**: For each slot, per-slot embedding $e_{t,i} = \phi(m_{t,i})$ is computed.
- **Retention Scores**: Scores $g_{t,i}$ are computed per slot using one of the following agent types:
  - *Input-Matching LEMN (IM-LEMN)*: Computes dot-product similarity $z_{t,i} = e_{t,i}^\top c_t$, blended with an exponential moving average $v_{t,i}$ for recency (LRU tracking). A learned forgetting coefficient $\gamma_t$ modulates sensitivity.
  - *Spatial LEMN (S-LEMN)*: Utilizes a bidirectional GRU over spatial index $i$, generating context-aware features $f_{t,i}$ which are projected to a scalar score.
  - *Spatio-Temporal LEMN (ST-LEMN)*: Extends S-LEMN by incorporating a temporal GRU for each slot, enabling accumulation of historic slot importance.
- **Retention Policy**: The categorical policy is $\pi(m_i | M_t, c_t) = \mathrm{softmax}_i(\{g_{t,i}\})$. For interpretation as eviction, the retention probability is $p_{\mathrm{retain}}(m_i) = 1 - \pi(m_i | \cdots)$.

| Agent Variant        | Contextual Features Used         | Mechanism                                                             |
|----------------------|----------------------------------|-----------------------------------------------------------------------|
| IM-LEMN              | current input, slot similarity   | Dot-product + LRU-based EMA                                           |
| S-LEMN               | slot-to-slot (spatial)           | Bi-GRU over memory at timestep $t$                                    |
| ST-LEMN              | spatial + history (temporal)     | Bi-GRU (spatial) + GRU (temporal, per-slot)                           |

## 3. Training Methodology

Memory retention is cast as a policy-gradient RL task:

- **Action**: At each $t$, action $a_t = i$ corresponds to the memory slot chosen for eviction.
- **State**: The current memory augmented with the candidate embedding $s_t = [M_t; c_t]$.
- **Reward**: At a future step $t_f$, a downstream task (e.g., question answering, navigation) provides a reward signal $R_\mathcal{T}\in\{+1,-1\}$ or a dense RL reward.
- **Optimization**: The system (base network plus retention agent) is trained end-to-end using Asynchronous Advantage Actor-Critic (A3C) with Generalized Advantage Estimation (GAE). The loss is:
  \[
  L = -\mathbb{E}[A_t\log\pi(a_t|s_t)] + \text{value function and entropy regularization terms}
  \]
  where $A_t$ is the advantage estimate. Temporal hidden states in ST-LEMN, $h_{t,i}$, encode historical slot usage, inherently capturing historical importance.

## 4. Memory Update Procedure

Memory updating is based on the stochastic or deterministic selection of eviction candidates according to the learned policy. The operational pattern for memory management is as follows:

```python
# Pseudocode for LEMN memory update
Initialize memory M ← empty list (max size N)
Optional: Initialize hidden states {h_i} for ST-LEMN
for t in 1…T:
    c ← ψ(x_t)                                 # encode new input
    if |M| < N:
        append c to M                          # fill until full
    else:
        for i in 1…N:
            e_i ← φ(M[i])                      # per-slot embeddings
        g_i ← retention_scores(e_1…e_N, c)     # IM-LEMN, S-LEMN, or ST-LEMN
        π  ← softmax(g_1…g_N)
        i* ← sample_or_argmax(π)
        if i* ≠ NOP_index:
            M[i*] ← c                          # evict and replace
            if ST-LEMN: h_{i*} ← 0             # reset temporal state
```

The agent selectively evicts less relevant entries, as learned through experience and reward.

## 5. Empirical Evaluation

LEMN has demonstrated effectiveness in three task domains.

### 5.1 Maze Path-Finding (Memory Q-Networks)
- **Base Agents**: MQN (memory Q-network, no context RNN), FRMQN (with context RNN).
- **Tasks**: 
  - *I-Maze*: Long corridor where initial indicator color determines goal; corridor up to length 200.
  - *Random Maze Single-Goal*: Varied maze topologies.
- **Results**: With $N=5$ slots and ST-LEMN retention, MQN+ST-LEMN achieves ~100% success rate across all lengths, outperforming MQN+FIFO (≈0% for length > 40). FRMQN+ST-LEMN maintains ~100% while FRMQN+FIFO degrades with length. Visualization reveals that ST-LEMN learns to retain decision-relevant cues (indicator color) and discard repetitive or irrelevant frames.

### 5.2 Synthetic QA (bAbI Two-Supporting-Facts Task)
- **Base**: MemN2N with 3 hops, position encoding.
- **Datasets**: 
  - *Original*: 45 facts + 5 questions, fixed order.
  - *Noisy*: Inserted noise facts.
  - *Large*: Episode length extended (20–80).
- **Error Rates (memory=10)**:

| Baseline/Agent | Original | Noisy | Large |
|----------------|----------|-------|-------|
| FIFO           | 16.5%    | 44.1% | 32.4% |
| IM-LEMN        | 16.1%    | 18.9% |  9.0% |
| S-LEMN         | 5.0%     |  4.8% |  5.1% |
| ST-LEMN        | 4.6%     |  3.9% |  5.6% |

Shuffling spatial memory order causes performance drops, indicating that both absolute and relative ordering of memory contributes to effectiveness. Qualitative analysis confirms that ST-LEMN reliably preserves the minimal supporting facts needed for future queries, filtering out noise.

### 5.3 Real-World QA (TriviaQA)
- **Base**: BiDAF, modified to operate at sentence granularity.
- **Dataset**: TriviaQA, Wikipedia split (~2,900 words per doc, truncated to 800 = 40–50 sentences).
- **Metrics**: ExactMatch (EM), F1 (on “Distant Supervision” set).

| Baseline/Agent | EM    | F1    |
|----------------|-------|-------|
| FIFO           | 18.5% | 20.3% |
| IM-LEMN        | 34.9% | 38.7% |
| S-LEMN         | 43.0% | 46.6% |
| ST-LEMN        | 45.2% | 49.0% |

ST-LEMN achieves the highest performance, with qualitative findings indicating selective memory retention of question-relevant sentences.

## 6. Significance and Comparative Analysis

LEMN’s primary contribution is an adaptive, lightweight retention agent that outperforms both rule-based (FIFO, LRU) and simplistic learned scheduling baselines by combining:
- Short-term input-matching,
- Spatial context (relative importance among stored entries),
- Temporal aggregation (historical usage relevance).

A key insight is that memory retention policies benefiting from both spatial and temporal contextualization provide substantial advantages in both generalization (navigation, long contexts) and robustness to noisy or distractor-heavy environments. Sequential decision-theoretic framing for memory eviction, marshaled through A3C+GAE, yields end-to-end optimizable, task-sensitive memory schedulers without manual heuristics [1812.04227].

Source: https://www.emergentmind.com/topics/long-term-episodic-memory-networks-lemn