---
title: Recurrent Memory Transformers
url: https://www.emergentmind.com/topics/recurrent-memory-transformers-rmt
type: topic
---

# Recurrent Memory Transformers

Recurrent Memory Transformers (RMT) are a class of neural sequence models that augment standard Transformer architectures with explicit segment-level memory recurrence, enabling efficient processing of extremely long sequences far beyond the quadratic limits imposed by global attention. RMTs partition input sequences into segments, process each with self-attention constrained to the segment, and propagate compact, trainable memory representations forward, thereby preserving global context with strictly linear scaling in both memory and computation. This paradigm, instantiated in diverse forms—including Associative Recurrent Memory Transformer (ARMT), Compact Recurrent Transformer (CRT), memory-augmented vision/language models, and biologically inspired variants—not only achieves state-of-the-art performance on long-context synthetic and real-world benchmarks but also enables scalable deployment on conventional hardware.

## 1. Architectural Foundations and Core Mechanism

At the heart of RMT lies segment-level recurrence implemented via explicit memory tokens. The input sequence of length $S$ is divided into $T$ non-overlapping segments $X_1, ..., X_T$, each of a fixed length $L$—with $S = T L$. For each segment $\tau$, a memory state $M_{\tau-1} \in \mathbb{R}^{m\times d}$ is prepended to the segment tokens. The augmented input for the Transformer at time $\tau$ is then:
$$
\tilde H^0_{\tau} = \begin{bmatrix} M_{\tau-1} \\ X_{\tau} \end{bmatrix} \in \mathbb{R}^{(m+L)\times d}
$$
After passing through $N$ Transformer layers, the output corresponding to the first $m$ positions is taken as the updated memory:
$$
[M_\tau; Y_\tau] \coloneqq H^{N}_{\tau}
$$
Memory read–write operations are thus learned end-to-end via the self-attention mechanism itself; no modifications to the underlying Transformer block or explicit gating are required in the baseline RMT formulation [2207.06881][2304.11062].

Variants may use memory concatenation at both the start and end of a segment, bidirectional memory attendance, or more complex per-layer or per-task memory organizations [2407.04841][2507.00453][2306.09459].

## 2. Mathematical Formulation and Memory Dynamics

Memory recurrence in RMT is defined by the update rule:
$$
M_{\tau} = f_{\mathrm{Trans}}(M_{\tau-1}, X_{\tau})
$$
where $f_{\mathrm{Trans}}$ denotes a multi-layer Transformer applied to the segment concatenated with its recurrent memory. During each step, all hidden states—both memory and segment tokens—participate in full self-attention within $[M_{\tau-1}; X_{\tau}]$.

More sophisticated memory mechanisms extend this formulation:
- **Associative updates** (ARMT): At each Transformer layer $l$ and segment $s$, an associative fast-weight store $A^l_s$ is updated by delta rules with normalization corrections involving key-value memory token pairs, supporting true $O(1)$ associative recall per-segment [2407.04841].
- **Gating or GRU-style updates**: Models such as HIPPO-CRAM, MART, or certain video/MT models integrate GRU- or LSTM-style gates into memory update equations, e.g.,
  $$
  M_{t+1} = (1 - z_t) \odot C_t + z_t \odot M_t
  $$
  for an update gate $z_t$ and candidate $C_t$ [2005.05402][2507.00453][2304.11062].
- **External episodic memory**: For discourse or commonsense tasks, episodic memory is managed as a dynamically expanding slot bank updated and accessed via global similarity projections rather than per-token attention [2010.01486].
- **Astrocyte-inspired retention**: In RMAAT, long-term memory compression and decay are adaptively managed by astrocyte-modeled retention factors $\alpha_t$ applied pointwise to segment memory updates, yielding normalization $\sum_{t=1}^T \alpha_t = 1$ and enforcing bounded memory growth [2601.00426].

## 3. Computational Complexity and Scaling

RMTs achieve linear or subquadratic complexity by restricting attention to small segments and propagating constant-size memory:
- **Standard Transformers** require $O(S^2d)$ per layer due to global attention over length-$S$ sequences.
- **RMTs** reduce segment cost to $O(L^2d)$ for attention over $(m+L)$ tokens and $O(mLd)$ for memory-related updates, with totals scaling as $O(T L^2 d) = O(S L d)$, i.e., linear in sequence length for fixed $L$ and $m$ [2207.06881][2304.11062].
- **ARMT and associative variants** maintain $O(1)$ per-segment time for memory operations; overall, this yields true constant time per segment as $S$ grows [2407.04841].
- **Diagonal Batching** further exploits the recurrent computation DAG structure, scheduling by layer+segment anti-diagonals to parallelize across segments and layers, reducing GPU kernel launch overhead from $O(LT)$ to $O(L+T)$ and enabling up to 3.3x speedup at 131,072 tokens [2506.05229].

## 4. Empirical Performance Across Domains

RMTs and their relatives consistently set new performance records on long-context benchmarks:
- **Synthetic and QA Tasks**: ARMT achieves 79.9% accuracy in the BABILong single-fact QA task at 50M tokens, outperforming Mamba and conventional RMT [2407.04841]. On associative retrieval, ARMT stores double the factual pairs of standard RMT.
- **Language Modeling**: On WikiText-103, RMT and CRT yield perplexity reductions compared to Transformer and Transformer-XL baselines—e.g., CRT achieves PPL 31.8 vs. 32.6 for Transformer-XL in a 3-layer, $n=150$ configuration [2505.00929]. RMT matches or improves on Transformer-XL in both PPL and computational cost [2207.06881][2304.11062].
- **Reinforcement Learning**: RATE outperforms Decision Transformer by large margins in ViZDoom and T-Maze, achieving stable high reward even as memory requirements increase with horizon [2306.09459].
- **Vision and Video**: RMT-augmented ViTs yield higher video quality assessment scores and outperform GRU-based baselines in blind video quality evaluation and paragraph captioning tasks [2405.08621][2005.05402].
- **Machine Translation**: Sentence-level RMT augmentation in document-level machine translation delivers +0.91 s-BLEU and +1.49 d-BLEU over baselines with minimal computational increase [2205.01546].

A summary table of long-context QA results for single-fact retrieval (QA1) from [2407.04841]:

| Context Length | GPT-4 | RMT | Mamba | ARMT |
|:--------------:|:-----:|:---:|:-----:|:----:|
| 64k            | 30.0% | 99.6% | 100.0% | 100.0% |
| 128k           | 24.0% | 99.1% | 99.5% ± 0.2 | 99.9% ± 0.2 |
| 1M             | –     | 94.2% | 92.3% ± 1.1 | 98.5% ± 1.0 |
| 10M            | –     | 76.4% | –          | 89.4% ± 8.1 |
| 50M            | –     | –     | –          | 79.9% |

## 5. Design Variants and Domain-Specific Extensions

RMT has been widely adapted for distinct tasks and modalities:
- **Associative/fast-weight memory**: ARMT uses a per-layer associative store updated by a delta rule with normalization, achieving high capacity for fact storage and supporting true key-value associative recall [2407.04841].
- **GRU/LSTM-inspired memory compression**: CRT and paragraph captioning models incorporate memory updates governed by gated RNNs or LSTM/GRU-style equations, facilitating gradient stability and informative compression [2505.00929][2005.05402].
- **Unification of local/global/chunked attention**: Hybrid architectures fuse global in-chunk attention, sliding-window local attention, and cross-chunk memory with learnable fusion weights for robust multi-scale modeling [2507.00453].
- **Biological inspiration**: RMAAT incorporates astrocyte-derived retention factors and Hebbian/astrocyte interaction rules for memory compression and capacity normalization [2601.00426].
- **Application-layer specialization**: Recurrent memory modules have been used to track paragraph-level commonsense inferences [2010.01486], maintain document-level translation context [2205.01546], and compress video sequences into global embeddings for VQA [2405.08621].

## 6. Implementation, Training, and Computational Considerations

Implementing RMT requires:
- Minor architectural modifications: only an additional memory-tokens input/output buffer and code for carrying forward these tokens across segments. The core Transformer block is unchanged in most variants [2207.06881][2304.11062].
- Curriculum learning is frequently essential: models are initially trained for short unroll lengths and gradually exposed to longer sequences, stabilizing memory usage and preventing divergence [2304.11062].
- Backpropagation through time (BPTT): Gradients are propagated through sequence-unrolled segments, typically for $K$ steps (truncated BPTT), though memory replay schemes such as AMRB minimize storage requirements by recomputation [2601.00426].

Key recommendations include careful choice of memory size $m$ (typically 1–32), judicious scheduling of segment lengths, and Pre-LayerNorm around blocks for stable deep stacks [2507.00453][2205.01546].

## 7. Limitations, Comparative Analysis, and Future Directions

While RMTs provide transformative gains in long-context efficiency and compactness, some limitations persist:
- **Parallelism**: Sequential processing of segments incurs latency for short contexts; Diagonal Batching partially mitigates this constraint [2506.05229].
- **Compression capacity**: Small fixed-size memory must distill exponentially growing context; poor memory or curriculum design leads to information bottlenecks and degraded performance [2304.11062].
- **Task suitability**: RMT may not improve over vanilla Transformers when the full context fits in a single segment, or when memory tokens are insufficient to store all relevant information [2306.09459][2505.00929].
- **Training complexity**: BPTT or replay is necessary for gradient propagation, with nontrivial memory and compute trade-offs [2304.11062][2601.00426].
- **Theoretical analysis of memory capacity**: Open questions include optimal memory sizing, hybridization with retrieval systems, and scaling to billion-parameter models [2407.04841].

Ongoing lines of inquiry target hybrid retrieval architectures, adaptive memory budgeting, further kernel/hardware optimization, and biological circuit–inspired compression [2506.05229][2601.00426]. The RMT paradigm provides a unifying abstraction for long-context modeling and is likely foundational for efficient next-generation sequence models across domains.

Source: https://www.emergentmind.com/topics/recurrent-memory-transformers-rmt