---
title: 'Memory Rewriting in RL: Concepts & Architectures'
url: https://www.emergentmind.com/topics/memory-rewriting-in-rl
type: topic
---

# Memory Rewriting in RL: Concepts & Architectures

Memory rewriting in reinforcement learning (RL) refers to the set of mechanisms by which agents selectively update, overwrite, or erase prior experiences or internal representations, balancing long-term retention against dynamic adaptation to new data, signals, or environmental shifts. Unlike pure retention—simply storing and retrieving prior information—memory rewriting is central to adaptive decision-making in partially observable or nonstationary domains, with concrete instantiations spanning experience replay buffers, external structured memories, recurrent and attention-based architectures, and agent-designed transition graphs. Modern approaches utilize explicit erasure, gating, merging, overwrite actions, or calibrated updates to regulate the persistence and transformation of learned traces, with significant impacts on sample efficiency, generalization, catastrophic forgetting, and memory interference. Recent research has shifted toward benchmarking and constructing mechanisms that support continual, context-driven update, recognizing that trainable forgetting and selective overwriting are as fundamental as stable retention.

## 1. Formalism and Core Principles of Memory Rewriting

At its foundation, memory rewriting in RL is defined within the POMDP framework as actions and update functions that modulate a latent memory state $m_t = f_\phi(h_t)$, where $h_t = (o_0, a_0, \dots, o_t)$ is the agent’s action-observation history. The general differentiable update is expressed as
\[
m_{t+1} = W_\phi(F_\phi(m_t), E_\phi(\eta_t)),
\]
where $F_\phi$ implements the retention/forgetting gate, and $E_\phi$ encodes new evidence $\eta_t = (a_t, o_{t+1})$ [2601.15086]. The crux of memory rewriting is that $F_\phi$ must actively erase outdated, irrelevant, or contradictory content in $m_t$ when $E_\phi(\eta_t)$ provides new salient signals, such that
\[
\frac{\partial m_{t'}}{\partial m_t} \approx 0 \quad \text{(erasure)}, \qquad \frac{\partial m_{t'}}{\partial \eta_t} > 0 \quad \text{(integration)}
\]
for specific sequences of inputs. This principle holds whether memory is implemented as buffer, recurrent state, attention-weighted cache, or external structured store.

Benchmarks such as Endless T-Maze and Color-Cubes explicitly separate regimes of pure retention ($N=1$ cues) from rewriting ($N>1$, sequential cue overwrite), exposing architectures incapable of adaptive erasure [2601.15086]. Empirical results indicate that trainable forgetting gates (as in LSTM/GRU) and explicit overwrite mechanisms are necessary conditions for robust, context-driven memory rewriting.

## 2. Architectures and Algorithms for Memory Rewriting

A full spectrum of RL approaches implement memory rewriting via discrete or continuous mechanisms:

- **Recurrent Networks (LSTM/GRU):** Employ input/forget/output gates, with the forget gate $f_t$ enabling selective erasure of latent cell features. LSTM’s gating supports high-dimensional, context-conditioned overwrite, yielding superior performance in rewriting benchmarks [2601.15086].
- **Transformer and Structured Memory Extensions:** Transformers maintain caches or global slots but lack explicit erasure, resulting in stale information persisting unless complemented by adaptive rewriting modules (e.g., ELMUR’s LRU-based slot overwrite and convex blending mechanism) [2510.07151].
- **External and Modular Memories:** Agents utilize buffer-based or structured memories, such as Stable Hadamard Memory (SHM), which update memory as
  \[
  M_t = M_{t-1} \odot C_\theta(x_t) + U_\varphi(x_t)
  \]
  where $C_\theta(x_t)$ calibrates (reinforces/erases) and $U_\varphi(x_t)$ writes new content [2410.10132]. Hadamard calibration bounds gradients and enables cell-wise selective rewriting.
- **Replay Buffers with Active Overwriting:** Experience replay control via buffer size and sample weighting directly implements rewriting. Adaptive algorithms (aER) adjust buffer capacity online in response to old samples’ TD-error, optimizing the trade-off between retaining and overwriting experiences [1710.06574].

Architectures such as GWR-R merge similar transitions into graph nodes and prune stale edges, dynamically rewriting stored state-action trajectories, while Forget-and-Grow (FoG) exploits explicit decay of sampling probabilities and critic expansion to combine continuous erasure and new feature encoding [2305.02054, 2507.02712].

## 3. Mechanisms: Write, Erase, Overwrite, and Blending

Memory rewriting mechanisms are instantiated as:

- **Explicit Overwrite Actions:** Agents decide on discrete write actions $w \in W$, directly choosing the new memory state $m' = w$ (binary or buffer-based external memory) [2010.01753].
- **Sliding Windows and Push/Skip Buffers:** Policies select when to push new observations/actions into a windowed buffer or skip (retain previous state), controlling the overwrite schedule [2010.01753].
- **Erasure/Calibration Gates:** Structured memories (SHM) generate calibration matrices $C_t$ whose element-wise product with old memory erases or reinforces, providing stability and selectivity [2410.10132].
- **Convex Blending with LRU:** ELMUR updates selected slots using replacement for empty and $\lambda$-weighted blending for least-recently used slots,
  \[
  m^{i+1}_{j^*} = \lambda\,\tilde{u}^{i+1}_{j^*} + (1-\lambda)\,m^{i}_{j^*}
  \]
  resulting in smooth decay of obsolete content and insertion of critical new cues [2510.07151].
- **Decay and Weighted Sampling:** FoG applies time-decayed weights $w_i(t) = \max\{T, (1-\epsilon)^{t-t_i}\}$ for buffer transitions, bounding replay counts and gradually erasing early samples [2507.02712].

- **Structural Merge and Pruning:** Map-based experience replay (GWR-R) dynamically merges similar nodes and prunes old edges, rewriting the graph-based memory representation [2305.02054].

## 4. Theoretical Bounds and Empirical Validation

The dynamics of memory rewriting have been characterized both analytically and empirically:

- **ODE Models of Replay Memory:** Continuous-time models reveal non-monotonic dependencies between buffer size $M$ and learning speed, with both small and large $M$ constraining convergence. Adaptive schemes (aER) auto-tune capacity for maximal efficiency [1710.06574].
- **Stability of Calibration Gates:** Randomized, context-driven calibration in SHM maintains bounded expectation of cumulative gating factors and decorrelates temporal updates, preventing gradient explosion/vanishing—contrary to fixed calibration [2410.10132].
- **Effective Memory Horizon:** In ELMUR, LRU rewrites combined with cross-attention decouple memory size from sequence length, sustaining retention and update over up to $10^6$ steps, exceeding standard transformer context by $>10^5\times$ [2510.07151].
- **Sample Diversity and Forgetting:** GWR-R’s rewrites increase the minimal pairwise distance in replay samples, decorrelating training batches, reducing catastrophic forgetting, but introducing a compression-performance trade-off controlled by activation threshold $a_T$ [2305.02054].

Benchmarks in Endless T-Maze and Color-Cubes, as well as meta-RL environments, consistently demonstrate that recurrent models (especially LSTM), structured calibration, and explicit overwrite actions outperform standard transformer and fixed-decay memories in rewriting regimes [2601.15086, 2410.10132].

## 5. Impact, Limitations, and Ongoing Challenges

Memory rewriting directly impacts sample efficiency, generalizability, and robustness to new or contradicting evidence. Major findings across recent works include:

- **Need for Trainable Forgetting:** Fixed schedules (e.g., exponential decay) and indiscriminate attention caches often fail under variable episode lengths or task nonstationarity. Architectures with trainable gating or context-driven overwrite generalize better [2601.15086].
- **Credit Assignment for Writes:** Scalable optimization of memory rewriting requires improved alignment of RL reward signals with specific erase/add or overwrite events, especially in differentiable or graph-structured memories [1904.06736].
- **Interference and Memory Capacity:** Merging or pruning in GWR-R and episodic buffers can introduce interference or under-represent critical transitions if overwriting is overly aggressive. Adaptive control of capacity and rewriting rate is necessary.
- **Benchmarks and Meta-Learning:** New diagnostic tasks (variable-length, multi-modal cues, sequential overwriting) and curriculum/meta-learned solutions to optimal rewrite scheduling are identified as priorities [2601.15086].

Limitations persist: context-independence assumptions in gating, recursive implementation overheads, and instability under sparse reward regimes for attention-based methods. There remains significant demand for designs that couple retention, selective erasure, and capacity management with scalable, sample-efficient RL under strict partial observability [2410.10132, 2601.15086].

## 6. Taxonomy and Comparative Table of Memory-Rewriting Approaches

Below is an explicit taxonomy assembling principal mechanisms and representative papers:

| Mechanism             | Core Update Formula                      | Representative Work (arXiv)        |
|-----------------------|------------------------------------------|------------------------------------|
| Explicit overwrite    | $m' = w$ (agent chooses new state)       | [2010.01753], [2507.02259]         |
| Erasure/calibration   | $M_t = M_{t-1} \odot C_t + U_t$          | [2410.10132], [2601.15086]         |
| Convex blending (LRU) | $m^{i+1}_j = \lambda u^{i+1}_j + (1-\lambda) m^i_j$ | [2510.07151]            |
| Time-decay buffer     | $w_i(t) = \max\{T, (1-\epsilon)^{t-t_i}\}$ | [2507.02712]                   |
| Graph merge/prune     | Node insertion, update, edge pruning     | [2305.02054]                       |
| Experience replay     | Overwrite oldest; adaptive buffer size   | [1710.06574], [1904.06736]         |

These strategies encapsulate the diverse trade-offs between capacity, stability, trainability, and retention-versus-update critical in contemporary RL. Ongoing research continues to refine mechanisms for alignment of credit assignment, interference suppression, and adaptive schedule learning.

## 7. Future Directions and Open Problems

Key areas for future advances include:

- **Meta-learned Forgetting Parameters:** Dynamic adaptation of rewrite schedules and gating factors to task statistics and environmental drift [2601.15086].
- **Integrating Planning with Memory-Rewriting:** Hybrid model-based and episodic recall architectures, facilitating joint adaptation of internal state and world model [1904.06736].
- **Unsupervised Memory Objectives:** Auxiliary objectives (e.g., reconstruction, contrastive losses) to shape more robust, interference-resistant memory rewriting [1904.06736].
- **Scalable Structural Memories:** Efficient graph, map, and matrix-based external memories with principled rewrite mechanisms balancing sample decorrelation and long-term retention [2305.02054, 2510.07151].

A plausible implication is that the act of forgetting—not merely remembering—is an essential ingredient for mastering RL in realistic, nonstationary, and partially observable domains, and architectures with explicit, context-sensitive rewrite dynamics are anticipated to dominate future agent design.

Source: https://www.emergentmind.com/topics/memory-rewriting-in-rl