PermaVid: Persistent Visual Memory

• PermaVid is a persistent visual-video framework offering consistent memory retrieval and edit-aware generation across extended timeframes.
• It employs a parallel visual memory branch and disentangled RGB-depth context to decouple visual retrieval from text-induced dilution.
• The architecture ensures long-term structural and semantic consistency via memory-guided edits and spatially-aware retrieval integrated with diffusion-Transformer models.

PermaVid refers to a class of frameworks and architectural principles for achieving persistent, consistent visual memory and coherent video generation across long temporal horizons and complex editing operations. The term is used both to describe the future direction of persistent visual–video memory in large multimodal models (Huang et al., 1 May 2026) and as the title of a specific memory-guided video generation architecture that enables long-term structural and semantic consistency under user-driven scene edits (Yang et al., 15 Jun 2026). Below is a comprehensive technical overview of PermaVid, focusing on both theoretical underpinnings and practical implementations, with reference to the state of the art in memory-augmented visual generation and edit-aware video synthesis.

1. Motivation and Theoretical Foundations

Long-term visual generation in multimodal systems, particularly for video synthesis or extended multimodal dialogue, is fundamentally limited by the “Visual Signal Dilution” phenomenon in autoregressive frameworks. As the contextual textual history $\mathcal{T}_t$ grows, the partition function for attention-based models causes the effective attention weight on a fixed set of visual tokens $\mathcal{V}$ to decrease as $\mathcal{O}(t^{-1})$ (see Eqns. (1)-(2), (Huang et al., 1 May 2026)). This results in the systematic decay of visual perception and ultimately in loss of scene consistency and visual recall capabilities.

The extension of persistent memory architectures to the video domain—coined "PermaVid"—aims to address these challenges by structurally decoupling the retrieval and integration of visual (and spatiotemporal) information from the noisy and growing textual/contextual background. This enables compositionality, precision, and long-term consistency, both in the presence of arbitrary edits and along open-ended camera trajectories.

2. Architectures for Persistent Visual-Video Memory

Two major PermaVid frameworks have been developed:

A. Parallel Persistent Visual Memory (PVM) for LVLMs

PVM is integrated as a parallel cross-attention branch alongside the backbone feed-forward path in transformer decoders. Its primary feature is a distance-agnostic retrieval mechanism that fuses visual context into the model's reasoning stream without the dilution effect caused by increasing text length (Huang et al., 1 May 2026). The architecture comprises:

• Three-Stage Bottleneck Adapter: Input state and visual embeddings are projected to a low-dimensional latent space, cross-attended (exclusively over $\mathcal{V}$), and then up-projected.
• Gated Fusion: Trainable scalar $\lambda$ and “silencing masks” restrict PVM output to textual positions, yielding

$$\mathbf{y} = \mathbf{x} + \mathbf{h}_{\mathrm{ffn}} + (\lambda \cdot \mathbf{h}_{\mathrm{pvm}})\odot \mathcal{M}_{\rm txt}$$

• Structural Guarantee: PVM attention partition is independent of text length, guaranteeing persistent visual recall.

B. PermaVid for Edit-Aware Video Generation

The PermaVid architecture of (Yang et al., 15 Jun 2026) operationalizes persistent visual memory for video generation under scene edits:

• Disentangled Multi-Modal Context Memory: Maintains parallel "banks" for RGB (appearance) and depth (geometry)—$\mathcal{M}^{\mathrm{rgb}}$ and $\mathcal{M}^{\mathrm{dep}}$—each storing past frames with metadata.
• Edit-Aware Memory Update:
  • Global edits: reset all RGB memory and increment semantic version.
  • Local edits: invalidate only RGB (and, if needed, depth) memory slots spatially overlapping with edited regions, using frustum-based intersection.
• Spatially-Aware Retrieval: For each generation step, references are drawn via maximal trajectory-overlap scoring and merged to compose a diverse, valid reference set $\mathcal{R}^{\mathrm{mem}}$.

3. Memory-Guided Video Generation and Multi-Modal Feature Fusion

The generation backbone in PermaVid uses a diffusion-Transformer (DiT) model. Key elements include:

• Conditional Generation Inputs:
  • Text encoding of prompts/captions.
  • Camera pose/trajectory encoding.
  • Sequence of retrieved memory slots (both modalities) encoded into latent tokens via a 3D-VAE.
• Memory Context Branch: A set of cascaded “Context Blocks” is inserted at select DiT layers. These blocks perform learned cross-attention over concatenated RGB and depth tokens, supplying appearance and geometry guidance.

During inference, the conditional denoising process at each timestep $t$ is as follows:

$\mathcal{V}$0

ensuring that generated frames are consistent with prescribed semantics, scene geometry, and editing history.

4. Edit-Aware Memory Dynamics and Selective Memory Invalidation

PermaVid's design centers around robust, edit-aware memory management for edit propagation and long-term consistency:

• Global Edits: Semantic style changes (e.g., day$\mathcal{V}$1night) trigger a reset of all RGB memory, with the depth bank retained for geometric stability.
• Local Edits: Targeted object insertions or region-specific modifications cause only spatially overlapping memory slots to be invalidated.
• Reference Selection: Only memory slots with sufficient view-frustum overlap and, for RGB, matching semantics are selected. These are then pruned for spatial diversity before use in generation.

This disentanglement enables precise propagation of appearance changes without corrupting stable geometric information, and vice versa.

5. Training Paradigms and Objective Functions

Training employs a two-stage protocol:

• Stage I: Conditional diffusion training on short videos (SpatialVid) with pose-plus-text conditioning, leaving memory references blank. This scaffolds the base denoising capability.
• Stage II: Full memory-guided training on long revisiting trajectories (UE-Mem), using both RGB/depth memory for context-aware conditioning.
• Core Loss:

$\mathcal{V}$2

No adversarial or explicit memory-consistency losses are employed; consistency emerges via implicit conditioning on the curated memory context.

6. Empirical Performance, Robustness, and Limitations

Quantitative evaluation on challenging datasets—such as UE-Mem (complex, long trajectories with revisits, dynamic edits)—demonstrate significant improvements:

• Structure Consistency: After global edits, PermaVid achieves depth-PSNR 22.84 (vs. next best 22.34); CLIP-Vid similarity 27.87 (vs. 26.17).
• Semantic Consistency and Recall: Under local edits, RGB-PSNR 22.33 (next best 21.92); robust re-rendering at revisit points with minimal blurring or artifact accumulation.
• Ablation Findings: Removing disentangled context, i.e., using only RGB without a separate depth memory, leads to persistent errors—demonstrating the necessity of modalities separation.
• Efficiency: Memory read and depth estimation add only a few milliseconds per frame; the overall system scales effectively to thousands of frames per trajectory.

Limitations include dependence on accurate depth estimation (monocular errors cause artifacts), computational cost (∼10 s/frame for diffusion), and challenges with highly dynamic multi-object scenes or scaling to extreme sequence lengths (Yang et al., 15 Jun 2026).

7. Relation to Broader Perpetual Video Generation and Future Directions

The persistent video memory approach embodied in PermaVid is also fundamental to the perpetual view generation task, as formalized in two-stage pipelines such as DreamJourney (Pan et al., 21 Jun 2025). These systems:

• Alternate spatial blending (camera pose-guided synthesis) with semantic/temporal animation (LLM-guided object motion).
• Incorporate stabilization strategies (early stopping, view padding) to improve cross-view coherence and visual quality.
• Expose open challenges in depth-to-geometry lifting, efficient diffusion solvers, and unified training that merges geometry, vision, language, and video priors.

A general "Persistent Visual–Video Memory" subsystem—structurally decoupling appearance and geometry, and equipped for hierarchical cache management and adaptive, RL-driven retrieval—emerges as the architectural foundation for long-range, edit-consistent multimodal intelligence (Huang et al., 1 May 2026, Yang et al., 15 Jun 2026, Pan et al., 21 Jun 2025).