Recurrent Latent Robotic Memory

Updated 4 July 2026

Recurrent latent robotic memory comprises architectures that encode task-relevant history into compact internal states for robust control and planning.
These systems utilize recurrent hidden states, token slots, prototype memories, or hybrid external-plus-latent modules to integrate multi-modal sensory inputs over time.
Learning strategies such as replay, predictive modeling, and distillation enhance memory retention and improve performance in long-horizon robotic tasks.

Recurrent latent robotic memory denotes a family of robotic architectures that preserve task-relevant history in compact internal states rather than relying solely on the current observation. Across the literature, those states appear as recurrent hidden activations, belief vectors, learned memory tokens, prototype memories, or compact event summaries, but the shared objective is stable control, recognition, or planning when the decisive information is no longer visible. The surveyed work places this problem in partially observable control, continual perception, long-horizon manipulation, navigation, and map-free localization, while also showing that the term has nontrivial boundary cases: some systems maintain persistent memory across environment time, whereas others perform recurrent latent computation only within a single decision (Heess et al., 2015, Liang et al., 14 Sep 2025, Cherepanov et al., 10 Jun 2026).

1. Historical development and conceptual scope

Early work framed memory as an internal recurrent state learned directly for control in partially observed continuous-action domains. “Memory-based control with recurrent neural networks” showed that recurrent deterministic and stochastic policies could solve noisy sensing, hidden-parameter identification, long-term cue retention, Morris-water-maze-style exploration, and pixel-based control by conditioning actor and critic on history rather than instantaneous observation (Heess et al., 2015). A distinct line then demonstrated that a simulated robot in a triple T-maze could acquire navigation skills, spatial memory, and working memory through neuroevolution of a recurrent controller whose dynamics encoded location and future route information, with performance depending more on evolved recurrent dynamics than on any single sensory modality (Zou et al., 2021).

Subsequent work broadened the notion of latent robotic memory beyond controller hidden state. “Robust Robotic Control from Pixels using Contrastive Recurrent State-Space Models” introduced a recurrent latent dynamics model with prediction-correction structure and contrastive future prediction, explicitly treating the latent state as a belief-like internal memory robust to camera, background, and color distractions (Srivastava et al., 2021). “Initialization of Latent Space Coordinates via Random Linear Projections for Learning Robotic Sensory-Motor Sequences” instead treated memory at the sequence level: a whole sensory-motor trajectory was compressed into a latent vector that conditioned a recurrent decoder, so the latent variable functioned as a memory trace for a motion primitive rather than a per-step belief state (Nikulin et al., 2022).

A later wave of work expanded the design space in three directions. First, recurrent self-organizing lifelong-memory systems such as the growing dual-memory architecture used recurrent prototype graphs, temporal context, and replay for continual robotic perception (Parisi et al., 2018). Second, manipulation and navigation papers made persistent latent state a first-class design target under partial observability, as in MEMBOT’s belief encoder, SRU-based navigation memory, Kinaema’s recurrent transformer memory bank, and minimal recurrent VLA backbones such as $\mu$ VLA (Liang et al., 14 Sep 2025, Yang et al., 6 Jun 2025, Sariyildiz et al., 23 Oct 2025, Cherepanov et al., 10 Jun 2026). Third, several papers argued for broader or hybrid interpretations: RD-VLA introduced recurrent latent reasoning within a single control decision rather than persistent trajectory memory, while RoboMemory, RoboMemArena/PrediMem, and HiMem-WAM combined latent processing with explicit keyframes, graphs, or external memory banks (Tur et al., 8 Feb 2026, Lei et al., 2 Aug 2025, Lei et al., 11 May 2026, sun et al., 9 Jun 2026).

2. Major architectural families

The surveyed literature suggests several recurring architectural families. They differ mainly in what the memory object is, how it is updated, and whether persistence is achieved through hidden-state recurrence, token recurrence, prototype dynamics, or explicit external storage.

Family	Core memory object	Representative systems
Recurrent hidden or belief state	Hidden activations, belief vectors, RSSM state	RDPG/RSVG(0), CoRe, MEMBOT, SRU
Recurrent token or slot memory	Learned tokens or slots carried across time	$\mu$ VLA, ReMem-VLA, Kinaema, Chimera
Recurrent prototype memory	Gamma-GWR neurons with temporal context	Growing Dual-Memory
Hybrid external-plus-latent memory	Keyframes, graphs, memory banks with latent controllers	RoboMemory, PrediMem, HiMem-WAM
Depth-wise latent recurrence	Iterative scratchpad within one action decision	RD-VLA

The first family treats memory as a persistent latent state that is recurrently updated from observations and sometimes actions. In this category, recurrent actor-critic control stores history in RNN or LSTM state (Heess et al., 2015), CoRe maintains prior and posterior latent states inside an RSSM (Srivastava et al., 2021), MEMBOT exports an explicit belief vector $b_t$ from an LSTM or SSM observer (Liang et al., 14 Sep 2025), and SRU-based navigation policies use recurrent hidden state as an implicit local map-like representation (Yang et al., 6 Jun 2025).

The second family keeps memory in a distributed set of tokens or slots. $\mu$ VLA inserts learnable memory tokens into the OpenVLA-OFT transformer and carries them across timesteps (Cherepanov et al., 10 Jun 2026). ReMem-VLA maintains frame-level and chunk-level recurrent query sets $Q^f$ and $Q^c$ to provide short-term and long-term latent memory (Li et al., 13 Mar 2026). Kinaema uses a fixed-size bank of recurrent memory embeddings for localization and navigation (Sariyildiz et al., 23 Oct 2025), and Chimera supervises such a memory bank by distilling a full-history teacher bottleneck into a recurrent student (Weinzaepfel et al., 19 Jun 2026).

The third family is prototype-based rather than vector-state-based. The growing dual-memory architecture consists of a growing episodic memory and a growing semantic memory, both implemented as extended Gamma-GWR recurrent self-organizing networks; memory is distributed over prototype weights, temporal context descriptors, temporal synapses, and replay trajectories rather than over a dense hidden vector (Parisi et al., 2018).

The hybrid family combines persistent latent processing with explicit externalized storage. RoboMemory uses temporal summaries, a dynamic spatial knowledge graph, and episodic/semantic retrieval stores (Lei et al., 2 Aug 2025). PrediMem couples a high-level planner to a recent-frame buffer and a keyframe bank, with predictive coding used to improve keyframe sensitivity (Lei et al., 11 May 2026). HiMem-WAM stores compact continuous memory tokens in an external bank updated at predicted skill boundaries (sun et al., 9 Jun 2026). These systems are strongly relevant to robotic memory, but the persistent substrate is not purely a recurrent latent state.

A final boundary case is RD-VLA. Its latent scratchpad $S_k$ is recurrent and weight-tied, but it is reinitialized for each inference call; the recurrence is across depth within one control decision rather than across environment time (Tur et al., 8 Feb 2026). This distinction has become central in later discussions of what should count as recurrent latent robotic memory.

3. State updates and memory semantics

A defining property of recurrent latent robotic memory is that current matching or action selection depends on a compressed state summarizing prior experience. In the growing dual-memory architecture, each episodic or semantic neuron contains both a weight vector $w_j$ and temporal context descriptors $c_{j,k}$ . Best-matching-unit selection uses both sensory mismatch and contextual mismatch, and global context evolves recursively as

$C_k(t)=\beta \cdot w_b^{t-1} + (1-\beta)\cdot c_{b,k-1}^{t-1},$

so matching depends on recent neural history rather than the current frame alone (Parisi et al., 2018). This makes each neuron a prototype in a joint sensory-temporal state space rather than a static point in observation space.

Belief-filter formulations make the same principle explicit in state-estimation form. MEMBOT defines an observation encoder $\mu$ 0 and a recurrent observer

$\mu$ 1

with an SSM realization

$\mu$ 2

so that belief persists when observations are masked or absent (Liang et al., 14 Sep 2025). CoRe similarly combines a GRU-based deterministic latent state with stochastic prior and posterior state components, giving a prediction-correction latent state that is trained to be both control-relevant and predictive of future observation embeddings (Srivastava et al., 2021).

Token-memory systems generalize recurrence from vectors to slot sets. Kinaema maintains $\mu$ 3 memory embeddings of dimension $\mu$ 4, updates each slot by combining previous slot state with current encoded observation, contextualizes slots through transformer self-attention, and then merges candidate and prior memory through a shared GRU gate (Sariyildiz et al., 23 Oct 2025). $\mu$ 5VLA injects a bank of learnable memory tokens $\mu$ 6 into the transformer input and reads out the next memory tokens from the corresponding hidden positions, so the same forward pass both reads and writes memory (Cherepanov et al., 10 Jun 2026). ReMem-VLA uses dual-timescale latent queries with fixed EMA recurrence; frame-level memory updates every step,

$\mu$ 7

while chunk-level memory updates every $\mu$ 8 frames,

$\mu$ 9

This separates rapidly updated short-term context from slowly updated long-term episode context (Li et al., 13 Mar 2026).

Hybrid systems make memory semantics more explicit. HiMem-WAM stores a bank of latent memory tokens $b_t$ 0, reads them by attention, and writes a new compact task token only when a boundary-aware write gate fires (sun et al., 9 Jun 2026). RoboMemory treats persistent state as temporal summaries, a dynamic spatial graph $b_t$ 1, and episodic/semantic entities updated recurrently after each interaction step (Lei et al., 2 Aug 2025). PrediMem defines memory as the union of a recent sliding window and a keyframe bank,

$b_t$ 2

so the planner reasons over selected historical observations rather than a learned hidden state alone (Lei et al., 11 May 2026).

4. Learning mechanisms: replay, prediction, distillation, and truncated recurrence

Because long-horizon credit assignment is difficult, recurrent latent robotic memory is often trained with auxiliary mechanisms that shape what should be remembered. In the growing dual-memory architecture, consolidation relies on recurrent neural activation trajectories (RNATs): after each learning episode, episodic memory replays internal prototype sequences to itself and to semantic memory, which the paper reports as mitigating catastrophic forgetting (Parisi et al., 2018). This is a replay-based memory system without raw-sample storage and without pixel-level generative modeling.

Predictive and reconstructive objectives are common in belief-style memory models. CoRe trains its RSSM by contrastively predicting the next observation representation from the prior latent state, combined with KL regularization, reward prediction, and inverse dynamics; the core claim is that recurrence provides temporal smoothness and useful hard negatives for contrastive future prediction (Srivastava et al., 2021). MEMBOT first pretrains a task-agnostic belief encoder with behavior cloning and reconstruction,

$b_t$ 3

then fine-tunes with SAC while keeping a reduced reconstruction term, so the belief remains informative under observation dropout (Liang et al., 14 Sep 2025). ReMem-VLA adds Past Observation Prediction (POP),

$b_t$ 4

precisely because action supervision alone does not preserve enough fine-grained visual detail in the latent recurrent state (Li et al., 13 Mar 2026).

Distillation provides a second route to train latent memory. Chimera first trains a Latent Bottleneck History Transformer teacher whose fixed-size bottleneck $b_t$ 5 summarizes the full observation history, then distills that bottleneck into a recurrent transformer student memory $b_t$ 6 using

$b_t$ 7

The central claim is that recurrent models are not mainly limited by architecture but by the difficulty of learning compression online; direct bottleneck supervision narrows the gap to full-history transformers (Weinzaepfel et al., 19 Jun 2026).

TBPTT is the dominant optimization regime in modern VLA-style recurrent memory, but the papers disagree on how much recurrence should itself be learned. RD-VLA uses TBPTT to train a weight-tied recurrent action head over latent iterative depth and stops inference when successive action predictions converge (Tur et al., 8 Feb 2026). $b_t$ 8VLA varies memory width $b_t$ 9, TBPTT horizon $\mu$ 0, and update rule, explicitly showing that cross-step gradients matter and that detached EMA writes are weaker (Cherepanov et al., 10 Jun 2026). ReMem-VLA takes the opposite position: because its BPTT truncation horizon is 1, it keeps the recurrence path fixed and gradient-free and reports that replacing fixed EMA recurrence with trainable recurrent dynamics almost completely eliminates memory capability on the critical simulation task (Li et al., 13 Mar 2026). HiMem-WAM again differs: it discovers skill boundaries and latent actions first, then fine-tunes gated memory jointly with action prediction, using write-gate supervision against boundary labels rather than relying on unconstrained recurrent self-organization (sun et al., 9 Jun 2026).

5. Benchmarks and empirical behavior

The empirical record shows that recurrent latent robotic memory is not a single capability but a family of task-dependent advantages. In continual visual perception, the growing dual-memory model achieved 79.43% instance accuracy and 93.92% category accuracy in batch mode, compared to 69.08% and 80.23% for the VGG fine-tuning baseline, and replay improved incremental performance from $\mu$ 1 to $\mu$ 2 at instance level and from $\mu$ 3 to $\mu$ 4 at category level (Parisi et al., 2018). On CORe50 it reported 87.94% in NI, 86.14% in NC, and 87.06% in NIC.

For pixel control and intermittent sensing, CoRe reached $\mu$ 5 mean score on dynamic-medium DCS at 500K steps and $\mu$ 6 at 1M steps, outperforming reconstruction, CURL, bisimulation-style, and recurrent SAC baselines (Srivastava et al., 2021). MEMBOT reported that it “maintains up to 80% of peak performance under 50% observation availability,” with the conclusion restating maintenance of roughly 65–80% under $\mu$ 7 and detailed task-dependent gains over memoryless and naive recurrent baselines (Liang et al., 14 Sep 2025). In long-range navigation, SRU-Ours reached overall success rate 78.9 versus 63.5 for LSTM and 78.3 versus 60.4 for the explicit-mapping/history RL baseline, which the paper summarized as 23.5% and 29.6% improvements respectively (Yang et al., 6 Jun 2025).

Localization and navigation memory have produced some of the clearest fixed-size latent-memory results. Kinaema reached $\mu$ 8 on Mem-RPE at sequence length 200 and $\mu$ 9 at length 800 under the three reported pose thresholds, outperforming GRU, xLSTM, MooG, EMA, LRU, and truncated-history baselines (Sariyildiz et al., 23 Oct 2025). Chimera, using bottleneck distillation into the same recurrent student architecture, improved those numbers to $Q^f$ 0 at length 200 and $Q^f$ 1 at length 800, sharply narrowing the gap to the full-history teacher (Weinzaepfel et al., 19 Jun 2026).

Manipulation and VLA studies have exposed both the promise and the limits of recurrent latent memory. On MIKASA-Robo, $Q^f$ 2VLA improved average success on five training tasks from 0.42 to 0.84 at the strongest setting and reached 0.23 on matched-semantics held-out tasks versus 0.07 for the memoryless baseline, while still achieving 96.2% average success on LIBERO without regression under full observability (Cherepanov et al., 10 Jun 2026). ReMem-VLA reached 93, 99, 100, and 86 success on the four MemoryBench tasks, for 94.5 average, and 82.5% average success on four real-world tasks versus 8% for MemoryVLA and 11% for $Q^f$ 3 (Li et al., 13 Mar 2026). PrediMem, evaluated on RoboMemArena’s 26 tasks with average trajectory length 1,076 and 68.9% memory-dependent subtasks, obtained 38.5 TSR and 55.2 CSR versus 27.3 and 49.1 for MemER and 21.5 and 38.7 for $Q^f$ 4, and reached 52% average success on five real-world tasks (Lei et al., 11 May 2026). HiMem-WAM reported 97.7 average on LIBERO and 26.3 on RMBench, with the authors specifically attributing the latter to gains on memory-dependent long-horizon manipulation (sun et al., 9 Jun 2026).

At the same time, the literature shows that not every form of latent recurrence is trajectory memory. RD-VLA demonstrated that recurrent depth within a single decision can be decisive for manipulation difficulty scaling: on LIBERO, fixed recurrence rose from 8.4% at one iteration to 84.1% at four, 92.6% at eight, and 93.0% at twelve, while adaptive stopping reached 92.5% with mean 7.93 iterations and up to 80× inference speedup over prior reasoning-based VLAs (Tur et al., 8 Feb 2026). This is a substantial latent-memory result for robotics, but it is depth-wise rather than persistent across environment time.

6. Limits, boundary cases, and open problems

One recurring controversy is definitional. RD-VLA is explicitly an “adjacent or partial-match paper” because its latent scratchpad is recurrent across depth within a single control decision, not across the robot’s trajectory (Tur et al., 8 Feb 2026). RoboMemory is a strong example of recurrent robotic memory but only a partial example of recurrent latent robotic memory because its persistent state is mostly explicit: temporal summaries, a dynamic spatial knowledge graph, episodic entities, and semantic entities (Lei et al., 2 Aug 2025). PrediMem and HiMem-WAM are likewise hybrid cases in which explicit keyframes or memory-bank entries coexist with latent planners and executors (Lei et al., 11 May 2026, sun et al., 9 Jun 2026). These papers suggest that the field still lacks a universally accepted boundary between latent recurrent memory, explicit external memory, and recurrent reasoning.

A second limitation concerns what present-day recurrent latent states actually retain. The growing dual-memory model is prototype-based, uses a pre-trained VGG frontend, and is evaluated on continual object recognition rather than control or world modeling (Parisi et al., 2018). MEMBOT’s narrative says belief integrates observations and actions, but its published recurrent equations make only observation-driven recurrence explicit, and its training pipeline alternates between claims about behavior-cloning fine-tuning and SAC fine-tuning (Liang et al., 14 Sep 2025). $Q^f$ 5VLA shows that minimal in-backbone recurrence can be sufficient when the required memory structure is represented in training, yet it remains weak on held-out tasks with novel memory semantics and loses much of its recall as delay increases (Cherepanov et al., 10 Jun 2026). ReMem-VLA resets memory at episode boundaries and requires POP because latent action supervision alone underpreserves visual details (Li et al., 13 Mar 2026).

A third limitation is scalability and transfer. Chimera is trained only on Mem-RPE, relies on a high-quality history teacher, uses fixed camera intrinsics and simulator-generated alternative frames, and leaves generalization beyond map-free pose estimation unresolved (Weinzaepfel et al., 19 Jun 2026). HiMem-WAM improves memory-dependent manipulation but still underperforms stronger memory baselines on the hardest $Q^f$ 6-style tasks described in the paper, indicating that sparse event-aligned memory is not yet sufficient for repeated multi-update memory reasoning (sun et al., 9 Jun 2026). RoboMemArena’s results further suggest that explicit event-centric memory formation and memory-selection quality may matter as much as recurrence itself, since a hybrid keyframe-based system outperformed many alternatives on its benchmark (Lei et al., 11 May 2026).

Taken together, these works suggest two broad research directions. One is deeper integration of recurrent latent memory into backbone pretraining, rather than shallow attachment of recurrent state to a largely memoryless policy; $Q^f$ 7VLA makes this argument directly by calibrating the capability envelope of minimal recurrence (Cherepanov et al., 10 Jun 2026). The other is hybridization: recurrent latent state may need to coexist with explicit event memories, predictive objectives, or hierarchical abstraction, as argued separately by PrediMem’s keyframe-plus-predictive-coding design, HiMem-WAM’s boundary-triggered skill memory, and Chimera’s teacher-supervised compression view of long-horizon recurrence (Lei et al., 11 May 2026, sun et al., 9 Jun 2026, Weinzaepfel et al., 19 Jun 2026).