Memory-Augmented Video Storytelling

• Memory-augmented video storytelling architectures are systems that integrate memory modules to maintain historical context, ensuring coherent multi-sentence narratives.
• They utilize techniques like recurrent transformer memory, keyframe selection, and adaptive retrieval to resolve coreferences, reduce redundancy, and maintain visual consistency.
• Empirical results demonstrate improved coherence and scalability on benchmarks such as ActivityNet and YouCookII, highlighting their practical effectiveness in video synthesis.

Memory-augmented video storytelling architectures define a research area focused on generating temporally coherent, visually relevant, and discourse-consistent multi-sentence narratives or multi-shot video sequences using explicit or implicit memory mechanisms. These approaches augment standard Transformers, diffusion models, or vision-language backbones with modules that store, retrieve, and summarize historical context throughout the video, yielding improved coreference resolution, reduced redundancy, and globally consistent storylines across long temporal horizons.

1. Core Principles and System Architectures

Memory-augmented storytelling systems typically operate by maintaining a memory state summarizing the visual and/or textual context up to the present, which is consulted to condition the generation of subsequent sentences or shots. Architectures vary:

• Recurrent Transformer Memory: MART (Lei et al., 2020) replaces the usual encoder–decoder with a stack of shared Transformer layers, appending a compact recurrent memory cell ($M^l_{t-1}$) alongside each layer. This cell is gated and updated via attention, ensuring the model remembers discourse history—resolving coreferences and suppressing repetition—while generating multi-sentence captions.
• Keyframe and Frame Selection Memories: In multi-shot video paradigms (e.g., StoryMem (Zhang et al., 22 Dec 2025), OneStory (An et al., 8 Dec 2025)), semantic keyframes extracted from previous shots are pooled into a memory bank. These memories are injected into the next-shot generation model via concatenation and positional encoding shifts, enabling cross-shot narrative consistency.
• Adaptive Retrieval and Compression: Long-form architectures address GPU and context window limitations by storing frame-level feature memories, applying hierarchical compression (e.g., cosine-similarity-based merging in MA-LMM (He et al., 2024)), and retrieving salient chunks (MemFlow (Ji et al., 16 Dec 2025), Context-as-Memory (Yu et al., 3 Jun 2025)) based on semantic or geometric relevance.
• Geometry-grounded Memories: For interactive scene generation, memory modules index views by 3D surfels (Li et al., 23 Jun 2025) or explicit volumetric grids (TSDF) (Wu et al., 5 Jun 2025), enabling robust retrieval of the most relevant spatial context for maintaining physical and visual consistency during revisits.

2. Memory Update, Retrieval, and Conditioning Mechanisms

Memory states in these architectures are updated and queried using heterogeneous strategies:

• Gated Recurrent Updates: In MART (Lei et al., 2020), the memory cell is updated via multi-head attention over new inputs and a tanh-sigmoid gate, ensuring that repetitive or noisy content is filtered out while high-level discourse information is retained.

  $$C^l_t = \tanh(W^l_{mc}M^l_{t-1} + W^l_{sc}S^l_t + b^l_c), \qquad Z^l_t = \sigma(W^l_{mz}M^l_{t-1} + W^l_{sz}S^l_t + b^l_z)$$

  $$M^l_t = (1 - Z^l_t) \odot C^l_t + Z^l_t \odot M^l_{t-1}$$

• Keyframe Selection & Filtering: StoryMem (Zhang et al., 22 Dec 2025) curates keyframes via CLIP-embedding similarity, adding only those passing a semantic threshold; aesthetic filtering ensures memory quality. Memory banks are maintained using a sliding window plus a "memory sink" of oldest persistent frames.
• Sparse Semantic Retrieval: MemFlow (Ji et al., 16 Dec 2025) and MA-LMM (He et al., 2024) use cross-attention between the current text/query and stored frame features. Only top-scoring candidates (based on pooled similarities or direct dot-products) are activated in the memory, controlling computational cost and focus.
• Geometric and Camera Overlap-Based Selection: Context-as-Memory (Yu et al., 3 Jun 2025) uses FOV overlap algorithms to select context frames with maximal co-visible content; VMem (Li et al., 23 Jun 2025) employs surfel rendering to tally which past frames most directly observed surfaces visible from the current camera, scoring each based on coverage for selection.
• Adaptive Importance-Guided Patchification: OneStory (An et al., 8 Dec 2025) splits selected frames into context tokens using layers with differing receptive fields, allocating finer patchification to frames with higher semantic importance.

3. Integration into Captioning and Generation Pipelines

Generated captions or video shots are conditioned on the current and historical context via these memory modules:

• Unified Input Sequences: In MART, video and text tokens for each segment are concatenated and attended jointly with memory slots at every Transformer layer.
• Memory Token Injection: In multi-shot models such as StoryMem, memory latents are concatenated with those of the to-be-generated clip, with negative RoPE indices ensuring temporal precedence; LoRA fine-tuning adapts the pretrained single-shot generator to operate under memory conditioning.
• Direct Context Prepending: OneStory performs conditioning by prepending context tokens derived from selected frames directly to diffusion noise tokens, leveraging pretrained I2V backbones.
• Hierarchical and Structured Memory Use: Advanced agents (e.g., VideoAgent (Fan et al., 2024)) maintain separate temporal and object-centric structured memories accessed via textual or feature-based keys, allowing LLM-driven tool-use workflows for queries, segment localization, and reasoning.

4. Efficiency and Scalability Considerations

Memory-based augmentation strategies focus on maximizing temporal support and coherence while minimizing computational overhead:

• Memory Compression: MeMViT (Wu et al., 2022) compresses memory per layer via learned pooling, avoiding quadratic scaling of attention with context length. MA-LMM merges adjacent similar entries to cap memory growth.
• Sparse Activation: MemFlow (Ji et al., 16 Dec 2025) restricts attention to only the top-$k$ relevant memory tokens, reducing inference speed degradation to 7.9% compared to a memory-free baseline.
• Memory Bank Sizing and Pruning: Both MA-LMM and StoryMem use explicit limits and compression mechanisms to maintain bounded GPU footprint while preserving critical historical information.
• Linear Scalability: VideoLLaMB (Wang et al., 2024) demonstrates bridge-layer based memory scaling linearly up to 320 frames on a single GPU, supporting long-form inference without excessive resource requirements.

5. Quantitative Results and Empirical Validation

Memory-augmented storytelling architectures consistently outperform standard baselines in metrics of coherence, relevance, and consistency:

Architecture	Key Metric (Coherence/Consistency)	Baseline	Memory-Augmented	Dataset/Task
MART	R@4 (para repetition); Human coherence	7.45%	5.44% (+16.5%)	ActivityNet, YouCookII
StoryMem	Cross-shot Consistency (ViCLIP sim)	0.3937	0.5065	ST-Bench
OneStory	Character Consistency (DINOv2)	0.515	0.587	Multi-shot generation
MemFlow	Consistency Score	96.60%	Highest	60s streaming gen.
MA-LMM	Long-video captioning (CIDEr, METEOR)	175.3	179.1	MSVD, YouCookII
VideoLLaMB	Egocentric Planning	30.26%	32.32%	MVBench
VMem	LPIPS (cycle revisit)	>0.5	~0.25	RealEstate10K

In all cases, memory enables the models to correctly resolve references, suppress n-gram and scene repetition, recall objects/subjects over long horizons, and produce narratives more consistent with human storytelling standards.

6. Extensions, Limitations, and Theoretical Context

Memory-augmentation has evolved from early attention-based video description models (Fakoor et al., 2016) through external memory bank networks (PFMN (Lee et al., 2018)) to complex architectures integrating both spatial/semantic and hierarchical narrative memories.

Limitations documented in recent works include:

• Memory saturation in extremely long narratives (>10 shots) requiring increased context budget or adaptive expansion (An et al., 8 Dec 2025).
• Challenges with dynamic scenes and rapid viewpoint shifts, which strain patchification and retrieval (An et al., 8 Dec 2025, Li et al., 23 Jun 2025).
• Geometric memory approaches demonstrating less robustness to occlusions and untuned domains (e.g., outdoor scenes) (Li et al., 23 Jun 2025).
• Computational cost for per-frame memory retrieval and compressive operations, mitigated but not eliminated by sparse activation/importance selection (Ji et al., 16 Dec 2025, He et al., 2024).

Recent systems propose integration of hierarchical plot and event memories, object-centric SQL databases, and reinforcement of global narrative vectors for enhanced long-horizon coherence (Fan et al., 2024, Wu et al., 5 Jun 2025). These developments position memory-augmented architectures as central to both practical long-form video generation and advanced story reasoning.

7. Historical Perspective and Future Directions

The area originated with PFMN's dual memory network approach for story-based video summarization (Lee et al., 2018), demonstrating that separate past and future external memories more effectively recover latent storylines than RNN/LSTM approaches. This paradigm has expanded to include explicit multi-modal and geometric memories, adaptive retrieval, transformer-based bridge layers (Wang et al., 2024), and recurrent fused memory slots.

Advances are being made in:

• Scene-consistent memory retrieval via spatial overlap and surfel-indexing for interactive environment generation (Yu et al., 3 Jun 2025, Li et al., 23 Jun 2025).
• Efficient long-term video–language modeling integrated with LLMs and multimodal tool-use workflows (He et al., 2024, Fan et al., 2024).
• Cross-shot multi-shot autoregressive video synthesis with referential captions and shot-level keyframe memory (Zhang et al., 22 Dec 2025, An et al., 8 Dec 2025).
• Importance-weighted conditioning modules guiding compact memory injection into diffusion backbones.

A plausible implication is that further progress will depend on scaling both the memory retrieval and compression engines and integrating more abstract event and character state memories for full cinematic and plot-level video storytelling. The trend toward hybrid architectures leveraging geometric, semantic, and hierarchical memory banks is likely to continue and expand.