AnimateDiff-Lightning: Cross-Model Diffusion Distillation (2403.12706v1)

Abstract: We present AnimateDiff-Lightning for lightning-fast video generation. Our model uses progressive adversarial diffusion distillation to achieve new state-of-the-art in few-step video generation. We discuss our modifications to adapt it for the video modality. Furthermore, we propose to simultaneously distill the probability flow of multiple base diffusion models, resulting in a single distilled motion module with broader style compatibility. We are pleased to release our distilled AnimateDiff-Lightning model for the community's use.

AnimateDiff-Lightning: A Leap in Few-Step Video Generation Through Cross-Model Diffusion Distillation

Introduction to the Paper

In the constantly evolving landscape of generative models for video, the efficiency of video generation models has been a bottleneck, limiting their wider application. Among the various approaches, AnimateDiff has emerged as a popular choice due to its innovative use of learnable temporal motion modules integrated with frozen image generation models. This allows for the leveraging of image priors to generate temporally coherent frames efficiently. However, the iterative nature of the diffusion process intrinsic to these models entails a significant computational cost, particularly apparent in the context of video generation. Addressing this challenge, the paper introduces AnimateDiff-Lightning, a model that significantly accelerates the video generation process while maintaining, and in some aspects enhancing, the quality of the generated videos.

Methodology

The core innovation of AnimateDiff-Lightning lies in its application of progressive adversarial diffusion distillation specifically tailored for video models. This approach has shown promising results in few-step image generation and is now extended to videos for the first time. The model simultaneously distills the probability flow of multiple base diffusion models, resulting in a distilled motion module that exhibits broader style compatibility.

Key aspects:

• Cross-Model Distillation: The proposed distillation methodology effectively combines the probability flow of various base models to distill into a single, shared motion module. This approach not only improves quality across several pre-selected base models but also enhances compatibility with unseen base models.
• Model and Data Preparation: Using popular base models for both realistic and anime styles, the authors generated an extensive dataset to facilitate the distillation process. This dataset addresses the challenge of out-of-distribution samples when applying anime-style models.
• Flow-Conditional Video Discriminator: The discriminator is extended to accommodate multiple flows of different base models, enabling it to critique separate flow trajectories accordingly. This is a crucial component of the progressive adversarial diffusion distillation technique adapted for video.

Evaluation & Results

Quantitative and qualitative evaluations demonstrate that AnimateDiff-Lightning sets a new benchmark in few-step video generation quality. Compared to its predecessor, AnimateLCM, it achieves better quality with reduced inference steps across various styles, including unseen base models. Notably, this model shows compatibility with existing modules for motion control and aspect ratio adjustments, underscoring its versatility and practical utility.

Implications & Future Directions

AnimateDiff-Lightning represents a significant step forward in the development of fast and efficient generative models for video content. By effectively addressing the speed-quality trade-off, it opens new possibilities for real-time applications and complex video generation tasks. The success of cross-model diffusion distillation in this context also suggests a promising avenue for further research, potentially leading to even more versatile and universally applicable distillation modules for various modalities.

Conclusion

In summary, AnimateDiff-Lightning achieves its goal of accelerating video generation without compromising quality through the innovative use of cross-model diffusion distillation. The release of this model to the community is expected to catalyze further advancements in generative AI, particularly in applications requiring high-quality video content generated efficiently.