SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation (2410.23277v2)

Abstract: Human beings are endowed with a complementary learning system, which bridges the slow learning of general world dynamics with fast storage of episodic memory from a new experience. Previous video generation models, however, primarily focus on slow learning by pre-training on vast amounts of data, overlooking the fast learning phase crucial for episodic memory storage. This oversight leads to inconsistencies across temporally distant frames when generating longer videos, as these frames fall beyond the model's context window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning system for action-driven long video generation. Our approach incorporates a masked conditional video diffusion model for the slow learning of world dynamics, alongside an inference-time fast learning strategy based on a temporal LoRA module. Specifically, the fast learning process updates its temporal LoRA parameters based on local inputs and outputs, thereby efficiently storing episodic memory in its parameters. We further propose a slow-fast learning loop algorithm that seamlessly integrates the inner fast learning loop into the outer slow learning loop, enabling the recall of prior multi-episode experiences for context-aware skill learning. To facilitate the slow learning of an approximate world model, we collect a large-scale dataset of 200k videos with language action annotations, covering a wide range of scenarios. Extensive experiments show that SlowFast-VGen outperforms baselines across various metrics for action-driven video generation, achieving an FVD score of 514 compared to 782, and maintaining consistency in longer videos, with an average of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm significantly enhances performances on long-horizon planning tasks as well. Project Website: https://slowfast-vgen.github.io

An Analysis of SlowFast-VGen for Action-Driven Long Video Generation

The paper entitled "SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation" introduces a sophisticated framework that emulates the dual-learning processes observed in biological systems, specifically targeting the task of generating coherent and consistent long-duration videos. The authors' primary contribution is the integration of slow and fast learning phases, designed to mimic the complementary learning systems found in human cognition.

Key Contributions

The paper proposes a novel architecture combining slow learning, for general dynamics capture across scenarios, with fast learning, aimed at episodic memory storage. The following key elements are synthesized into the model's design:

1. Masked Conditional Video Diffusion Model: This model serves the slow learning phase, pre-training on a vast set of diverse data. It effectively captures general world dynamics through action-conditioned video generation.
2. Temporal LoRA Module for Fast Learning: During inference, this module adapts and stores episodic memory, enhancing long-term consistency across video segments. The Temp-LoRA module is inspired by analogous techniques in text generation, focusing here on video memory.
3. Slow-Fast Learning Loop: The dual-speed system encapsulates an innovative looping mechanism where fast-learning outputs are integrated into the slow-learning structure, enabling the model to leverage multi-episode data. This loop facilitates context-aware skill learning from the accumulated prior experiences.
4. Extensive Dataset Collection: The research introduces a large-scale dataset consisting of 200,000 videos annotated with language actions. This dataset is integral to training the model, ensuring a broad coverage of scenarios such as games, simulations, driving sequences, and more.

Experimental Performance

The experimental evaluations underline the significant improvements brought by SlowFast-VGen over existing models. The system exhibits superior performance in generating longer, coherent video sequences, achieving an FVD score of 514, notably outperforming other benchmarks like 782 achieved by competitors. This result is accompanied by a reduction in scene cuts—demonstrating consistency—and achieving high scene revisit consistency, crucial for tasks where trajectory memory is important.

The model also excels in specific long-horizon planning tasks, demonstrating its dual-speed system's ability to store and utilize episodic memory efficiently. The innovative three-phase loop enhances the model's capacity to perform context-sensitive actions within extended videos.

Implications and Future Directions

The integration of fast learning into a traditionally slow-learning domain like video generation introduces a new frontier in video-LM synthesis models. This dual approach could redefine frameworks beyond video generation, potentially impacting robotics, autonomous navigation, and real-time simulation environments where consistent recall of previous experiences is critical.

Future research could explore:

• Optimization of Temp-LoRA: Refining the memory and computational efficiency of the fast-learning modules.
• Diverse Scenario Applications: Extending the architecture's applicability to even more complex, real-world datasets.
• Adaptive Learning Mechanisms: Incorporating on-the-fly learning adjustments during active inferences to handle unseen scenarios dynamically.

In conclusion, SlowFast-VGen stands as a substantial advancement in the field of long video generation, providing a robust and adaptable framework that harmonizes slow and fast learning processes effectively. The model’s architecture and its successful application across diverse domains reveal a promising advancement towards more intelligent and adaptive video generation systems.