Memento: Reconstruct to Remember for Consistent Long Video Generation

Abstract: Long-form video generation requires recurring subjects to remain consistent across various shots, viewpoints, motions, and scene transitions. Existing temporal decomposition methods improve scalability by generating videos shot by shot. However, they mainly focus on optimizing plausible next-shot continuations without verifying whether the historical memory preserves identity-critical subject evidence. Consequently, as generation proceeds, recurring subjects may be diluted, overwritten, or forgotten. In this paper, we propose Memento, a subject-reconstruction-guided framework that treats subject preservation as an explicit identity grounding problem, based on the premise that a memory bank faithfully preserving a subject should support reconstructing that subject from memory alone. Specifically, Memento jointly trains autoregressive next-shot generation with memory-based subject reconstruction, recovering target appearances using historical memory and global story captions. To disentangle long-range subject evidence from short-range cues, Memento introduces a dual-query memory mechanism, where one query retrieves identity-relevant memory and the other selects short-context keyframes for coherent continuation. Additionally, a subject-aware cinematic data pipeline provides precise reconstruction supervision via consistent, pronoun-free subject descriptions. Experiments demonstrate that Memento achieves state-of-the-art performance in long-term subject consistency, cross-shot coherence, and visual quality.

• The paper introduces explicit subject reconstruction to preserve long-term identity in video generation using a dual-query memory system.
• It employs a memory-conditioned autoregressive model with shot-level generation and subject-aware data curation to maintain narrative coherence.
• Empirical results show significant improvements in inter-shot and inter-scene subject consistency, validated by both quantitative metrics and human evaluations.

Memento: Reconstruct to Remember for Consistent Long Video Generation

Problem Statement and Motivation

Long-form video generation, particularly with recurring subjects and complex narratives, demands high-fidelity synthesis of subject identities and coherence across shots and scene transitions. Existing video generative paradigms—storyboard-based, joint multi-shot, and memory-conditioned autoregressive generation—encounter severe limitations: they fail to guarantee identity preservation over long temporal horizons due to undirected or entangled memory representations, leading to identity drift, weak narrative linkage, and subject inconsistency at scale.

Memento addresses these deficiencies by explicitly formulating long video subject consistency as an identity grounding problem. The core hypothesis is that for a generative model to faithfully preserve subjects, its memory mechanism must enable the reconstruction of subject appearances from memory alone, thus ensuring the persistence of fine-grained identity cues throughout the generative process.

Framework Architecture and Methodology

Memento is built upon a memory-conditioned, autoregressive shot-level generation backbone with explicit subject reconstruction supervision and a disentangled dual-query memory retrieval mechanism.

Figure 1: Architecture illustrating autoregressive generation, split self-attention, dual cross-attention for global/local captions, and the memory bank update process.

Subject-Aware Data Curation

To minimize subject ambiguity and enforce identity traceability, the authors introduce a three-stage data pipeline:

• Story Captioning: A global caption for each video sequence, establishing an inventory of recurring subjects with unique, pronoun-free identifiers.
• Shot Captioning: Each shot is annotated using fixed subject descriptors aligned with the global inventory, reducing reference ambiguity.
• Reconstruction Captioning: Selected, subject-salient frames are paired with explicit, identity-preserving captions to serve as targets for memory-based reconstruction.

This disciplined data design enables unambiguous subject annotation, robust to subtle intra- and inter-scene transformations.

Figure 2: The subject-aware captioning pipeline, demonstrating story, shot, and reconstruction captions for identity-consistent supervision.

Dual-Query Disentangled Memory Bank

Conventional single-channel memory selection entangles short-range scene context with long-term identity information, resulting in suboptimal retention of subject evidence. Memento overcomes this via:

• Dual-Query Memory: Separate learnable queries are independently conditioned on the global story caption (for long-term subject evidence) and per-shot caption (for short-term contextual cues).
• Adaptive Candidate Pooling: The memory bank updates by combining previous memory tokens and new visual features from the latest shot, from which subject- and shot-relevant top-$K$ tokens are independently retrieved and concatenated.
• Reduced Interference: This architecture mitigates memory competition between persistent identity traces and transient scene context, facilitating robust cross-shot and cross-scene consistency during generation.

Subject-Anchored Multi-Task Training

Memento’s total loss consists of two parallel objectives:

• Next-Shot Generation (TM2V): Standard conditional denoising for the next shot, leveraging memory, global, and local text conditions.
• Memory-Based Subject Reconstruction (TM2I): Reconstruction of target subject frames using memory and global caption alone, in the absence of any direct image reference to the target—imposing a strict pressure for the memory to encode all identity-critical details. The loss for each is weighted by a hyperparameter to balance scene progression and identity fidelity.

Joint optimization ensures that the model not only continues plausible video progressions but also preserves identify-critical cues retrievable for reconstruction at any time step.

Experimental Results

Quantitative Evaluation

Memento is benchmarked against leading approaches—StoryDiffusion+Wan2.2-I2V (storyboard-based), StoryMem (memory-conditioned), and HoloCine (joint multi-shot)—across metrics for aesthetic quality, semantic consistency, background consistency, and subject consistency at intra-shot, inter-shot, and inter-scene levels.

Strong numerical results: Memento registers highest values on inter-shot (0.7338), intra-shot (0.8578), and inter-scene subject consistency (0.7268), as well as shot-level (0.2893) and story-level (0.3063) semantic consistency. Notably, these improvements are achieved while maintaining competitive aesthetic quality, refuting the notion that visual fidelity must be sacrificed for long-term subject stability.

Qualitative Analysis

Qualitative comparisons exhibit visible superiority in preserving subject consistency—especially under challenging conditions involving complex motions, occlusions, or scene/camera transitions—where other methods exhibit rapid identity drift or semantic incoherence.

Figure 3: Representative multi-shot generations demonstrating robust subject identity retention and coherent storyline progression in challenging scenarios.

Figure 4: Key story moment comparison, highlighting subject drift in baselines; Memento prevents identity loss across scenes and shot boundaries.

Human Evaluation

A user study corroborates the automatic metrics: pairwise human assessments reveal Memento is strongly preferred over StoryMem and HoloCine in cross-shot consistency, prompt adherence, and aesthetic quality, with win rates ranging up to 69% for consistency. This further supports the theoretical and empirical advantages of explicit reconstruction-guided memory supervision.

Figure 5: User study win rates confirming consistent human preference for Memento in critical video quality dimensions.

Ablation Studies

Ablative experiments validate each core module. Incorporating explicit subject reconstruction significantly improves inter-shot consistency. The adoption of disentangled memory—while marginally lowering immediate inter-shot similarity due to the introduction of other subjects’ evidence—materially boosts global, cross-scene, and intra-shot consistency metrics. This confirms that separating memory paths is advantageous for overall long-term identity preservation.

Advanced Capabilities

Memento demonstrates the ability to sustain complex narrative arcs, such as aging a subject while maintaining consistent identity, and scales to minutes-long coherent video synthesis without decline in subject fidelity or story coherence.

Figure 6: Top: Age-consistent identity preservation across life stages. Bottom: Multi-minute generation with robust subject and story integrity.

Implications and Future Directions

The results imply that explicit, memory-based reconstruction supervision is essential for persistent subject consistency in autoregressive long video generation. The dual-query disentangled memory paradigm may generalize to other domains requiring temporally persistent entity tracking, such as long-context dialog, multi-agent simulations, or reinforcement learning environments.

Practical limitations include error propagation from autoregressive memory updates and limited physical realism in generative backbones lacking explicit world modeling. Future research should address these through robust memory gating/filtering mechanisms and integration of physics priors or external world models to further enhance controllability and realism in generative video modeling.

Conclusion

Memento sets a new standard for long-form, consistent video generation by transforming identity consistency into a direct reconstruction problem and employing disentangled, dual-query memory mechanisms. These innovations lead to substantial improvements over prior methods along both automatic and human-evaluated axes of subject consistency, semantic coherence, and temporal scalability, with broad implications for temporally consistent generative modeling.