Dynamic Latent Vision Memories

Updated 18 November 2025

Dynamic latent vision memories are recurrent, learnable systems operating in latent space, enabling continual update and retrieval of visual representations.
They integrate biologically inspired mechanisms, transformer-based architectures, and reinforcement learning to manage temporal coherence and object permanence.
Empirical results demonstrate improvements in sample efficiency, dynamic scene reconstruction, and vision–language reasoning across various applications.

A dynamic latent vision memory is a recurrent, learnable, and temporally structured memory system operating in the latent space of visual models. It enables the continual update, storage, and retrieval of visual representations, supporting robust perception, reasoning, planning, and action in temporally extended tasks. This construct unifies biologically inspired mechanisms (short-term attentive and Hebbian memories), transformer-based memory architectures, and off-policy reinforcement learning with temporally coherent latent spaces. Recent advances show its centrality in reinforcement learning, world models, dynamic scene reconstruction, video tracking, and vision–language reasoning.

1. Mathematical Foundations and Canonical Structures

Dynamic latent vision memories are instantiated as temporally evolving variables or matrices in the latent (representation) space, distinct from raw pixel or token space. Several canonical mathematical forms have emerged:

Latent Markov transitions (PALM): Given a latent state $z_t$ and action $a_t$ , new states are sampled as

$z_{t+1} = \beta z_t + (1-\beta)(\alpha a_t + (1-\alpha)z_{\mathrm{rnd}})$

with exponential moving average (EMA) for persistence and a conditional generator $G(z_{t+1},c)$ for image decoding (Liu et al., 2022).

Hebb–Rosenblatt plasticity (STAWM): Memory as a learnable weight matrix $W$ updated by

$W(t+1) = W(t) + \eta[e(t)\otimes \phi(W(t)e(t)+\theta e(t))] - \delta W(t)$

with $e(t)$ a learned glimpse embedding and $\phi$ a bounded activation. This acts as a dynamic “latent scratchpad” for sequential visual evidence accumulation (Harris et al., 2019).

Slot-based autoregression and imagination gating (Loci-Looped): Each “slot” maintains a hidden state and fuses prior latent imagination with current observation by gating:

$G^t_k = \alpha^{t,G}_k\,\tilde G^t_k + (1-\alpha^{t,G}_k)\hat G^t_k$

enabling robust tracking and object permanence (Traub et al., 2023).

Dual memory stacks (Mem4D): Dynamic content is decoupled:
- Transient Dynamics Memory (TDM): short-term, high-frequency motion cues recomputed per frame.
- Persistent Structure Memory (PSM): long-term spatial structure in a queue, updated from global scene predictions and compressed via temporal convolutions (Cai et al., 11 Aug 2025).
Recurrent state-space models (HRSSM): Latent memory $s^\mathrm{m}_t = [h^\mathrm{m}_t; z^\mathrm{m}_t]$ is updated recurrently with spatio-temporal masking, bisimulation regularization, and an EMA-stabilized raw branch (Sun et al., 10 May 2024).
Token-injection in VLMs (VisMem): Short-term and long-term latent vision memories are injected as special token spans during text generation, generated on-the-fly from multimodal context by memory formers and query builders, supporting both fine perceptual retention and semantic consolidation (Yu et al., 14 Nov 2025).
Discrete multi-slot future memory (Farsighted-LAM, SSM-VLA): A sequence of discrete codebook-token slots $z_{t+k}$ encode predicted or observed future scene dynamics via spatio-temporal transformers, supporting reasoning and planning (Cai et al., 30 Sep 2025).

All designs share an emphasis on temporally coherent, task-relevant, and efficiently queryable latent memory, often using alternation, EMA, gating, or slot attention for controlled persistence and renewal.

2. Memory Formation, Update, and Access Mechanisms

Dynamic latent vision memories differ in formation and update logic:

Recurrent, action-influenced transitions: In PALM, the agent’s action and noise are linearly mixed and stabilized via EMA, yielding a continuous perceptual flow in latent space (Liu et al., 2022).
Sequential plasticity and snapshot querying: The Hebbian memory in STAWM is updated after each glimpse, allowing arbitrary querying after any number of steps for either classification or “drawing” (Harris et al., 2019).
Slot autoregression and imagination blending: Each slot in Loci-Looped is updated via a transition network, and the blending coefficient (percept gate) modulates reliance on imagination versus observation, important for tracking under occlusion (Traub et al., 2023).
Alternating short/long-term stacks: Mem4D readout alternates between motion-rich TDM and structure-rich PSM at each transformer decoding pass, providing separation of dynamic scene elements and background structure (Cai et al., 11 Aug 2025).
Masked synchronous recurrences: HRSSM applies spatio-temporal masking before encoding, updates both a masked and an EMA raw branch of the latent recurrent model, and aligns them via bisimulation loss and latent reconstruction loss, ensuring that the dynamic latent memory is robust to exogenous distractors and temporal noise (Sun et al., 10 May 2024).
Token-based memory injection: In VisMem, token-level cues (“invocations”) cause a dedicated transformer (query builder) to extract a summary from the current context, which then produces memory tokens via LoRA-adapted blocks, inserted into the model’s input stream for continued generation (Yu et al., 14 Nov 2025).
Multi-frame, multi-slot discrete updates: In Farsighted-LAM, a frozen vision encoder processes a window of frames jointly with slot queries, yielding a set of discrete latent tokens. These tokens are explicitly modeled as a dynamic latent memory, read and written by a downstream VLA policy (Cai et al., 30 Sep 2025).

The update schedule and control signals (e.g., actions, invocations, masking, occlusion, slot recruitment) are crucial for task-adaptive persistence and memory renewal.

3. Representation, Supervision, and Inductive Biases

Dynamic latent vision memories support a variety of training objectives and inductive biases:

Intrinsic reward for representation expansion: PALM employs nearest-neighbor entropy maximization in latent space, producing high-diversity rollouts and “free” data augmentation by temporal pairing (Liu et al., 2022).
Natural data augmentations without hand-design: Variations produced by dynamic latent transitions serve as positive pairs for self-supervised learning, eliminating the need for artificial augmentations (e.g., in SimSiam loss) (Liu et al., 2022).
Decoupling structure and dynamics: Mem4D explicitly bifurcates dynamic and static content, alternating transformer reads without explicit loss terms for separation; the design leverages multi-scale convolutional compression to compact PSM, and pyramid pooling for TDM (Cai et al., 11 Aug 2025).
Bisimulation principles and masking: HRSSM uses spatio-temporal masking in the observation stream, with latents aligned through unit-norm projection and bisimulation loss to focus on reward- and dynamics-relevant information (Sun et al., 10 May 2024).
Temporal continuity and object permanence: Slot-based architectures enable robust object-centric memory, tracking objects through occlusion, handling blackouts, and producing surprise signals upon unexpected reappearances (Traub et al., 2023).
RL-based optimization of invocation and formation: VisMem uses RL to optimize both the formation of memory tokens and the invocation policy, incorporating penalties for over- or mis-invocation, thus aligning the memory usage with task demands (Yu et al., 14 Nov 2025).
Multi-modal, multi-scale reconstruction targets: Farsighted-LAM and SSM-VLA reconstruct both RGB and depth for future keyframes, biasing the latent memory to be geometry- and dynamics-aware, then use the memory for causal planning (Cai et al., 30 Sep 2025).

The choice of memory representation, update, and training regime tightly controls the expressivity and stability of the dynamic latent vision memory.

4. Applications and Empirical Performance

Dynamic latent vision memories drive advances across several domains:

Model/framework	Primary domain(s)	Notable results
PALM	Unsupervised RL, vision	CIFAR-10 linear probe ≈92.3%, OOD detection
STAWM	Vision, attention, interpretability	MNIST cls error 0.35%, self-sup. draw 0.77%
Mem4D	Dynamic scene reconstruction	+20% AbsRel on Sintel, online 16 FPS
HRSSM	World-model RL, robustness	Maniskill, Matterport: state-of-the-art MBRL
VisMem	Vision–LLMs	+11.8% over vanilla VLM, superior continual learning
Loci-Looped	Video object tracking, object permanence	SOTA on occlusion, interpretable activity
SSM-VLA	Vision-Language-Action	SOTA on VLA tasks, strong generalizability

PALM enables sample-efficient offline RL and vision pretraining without explicit simulators. STAWM produces interpretable visual sketchpads and unifies attention with Hebbian learning. Mem4D’s dual-memory approach resolves drift and blur trade-offs in dynamic scene 3D modeling. HRSSM’s masking and dual-branch recurrence enhance robustness to distractors in RL. VisMem provides VLMs with cognitive-aligned memory, mitigating visual bottlenecks, preserving grounding, and improving OOD stability. Loci-Looped demonstrates model-based object permanence and surprise via latent continuity. SSM-VLA’s visual Chain-of-Thought integrates discrete latent plans for robust vision–language–action reasoning.

5. Biological and Cognitive Inspirations

Multiple architectures explicitly draw inspiration from neuroscientific and cognitive theory:

Short-term working memory: STAWM’s Hebb–Rosenblatt matrix mirrors recurrent, plastic cortical dynamics underlying visual primate memory, supporting sequential glimpse integration (Harris et al., 2019).
Slot-based object memory: Loci-Looped parallels findings in object-centric visual cortex, using independently evolving “slots” to model object continuity and surprise (Traub et al., 2023).
Short-term vs. long-term memory dichotomy: VisMem’s dual-memory mechanism is directly motivated by the Dennis–Norris cognitive model, distinguishing visually dominant short-term slots for immediate perception and semantically dominant long-term slots for higher-order reasoning (Yu et al., 14 Nov 2025).
EMA for “world persistence”: PALM’s EMA-based latent evolution simulates physical continuity and gradual scene changes, emulating animal perception (Liu et al., 2022).
Attention–replay alternation: Mem4D’s architectural alternation between dynamic TDM and structural PSM echoes hippocampal–neocortical interaction hypotheses in memory consolidation (Cai et al., 11 Aug 2025).

Such connections provide both justification and further research avenues for dynamic latent vision memory mechanisms.

6. Limitations, Open Problems, and Empirical Ablations

Despite documented progress, several open areas and ablations elucidate both the power and failure modes of dynamic latent vision memories:

Memory decay and trade-off control: Setting EMA coefficients (e.g., $\beta$ in PALM) affects the memory’s timescale—too low yields scene “jumps”; too high loses novelty and diversity. Empirically, $\beta\approx 0.5$ –$0.75$ is optimal (Liu et al., 2022).
Slot drop/add events: Slot-based models require careful slot recruitment strategies to maintain object-localization without “object collisions” or slot flushing (Traub et al., 2023).
Drift and catastrophic forgetting: Mem4D shows that omitting PSM yields severe static drift, while omitting TDM blurs dynamic content (Cai et al., 11 Aug 2025); VisMem demonstrates superior continual learning retention compared to direct fine-tuning (Yu et al., 14 Nov 2025).
Policy–representation instability: HRSSM avoids joint training collapse by using pure latent prediction objectives and stabilizing with EMA, bisimulation, and free-bits KL; this isolates policy gradients from destabilizing early signals (Sun et al., 10 May 2024).
Quantization vs. expressivity: Farsighted-LAM’s discrete memory trade-off enables chain-of-thought planning, but quantization can limit fine structural detail; a plausible implication is further gains with hierarchical continuous–discrete hybrids (Cai et al., 30 Sep 2025).
Memory formation triggers: In token-injected models (VisMem), inappropriate invocation frequencies degrade performance (“random invocation at P%” is sub-optimal), indicating the necessity of adaptive, learned control (Yu et al., 14 Nov 2025).

Extensive ablations and benchmarks guide hyperparameter selection and reveal the need for domain-adaptive control over memory staleness, capacity, and role specialization.

7. Outlook and Synthesis

Dynamic latent vision memories formalize and operationalize the concept of temporally evolving, queryable, and action-or-policy modifiable visual memory in deep neural architectures. Their centrality spans reinforcement learning, scene reconstruction, vision-language action, and biologically inspired sequential perception. Contemporary results establish their role in improving sample efficiency, robustness, OOD generalization, and continual learning.

Future directions can be anticipated along axes of: