From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2412.07772v2)

Abstract: Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. The generation of a single frame requires the model to process the entire sequence, including the future. We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly. To further reduce latency, we extend distribution matching distillation (DMD) to videos, distilling 50-step diffusion model into a 4-step generator. To enable stable and high-quality distillation, we introduce a student initialization scheme based on teacher's ODE trajectories, as well as an asymmetric distillation strategy that supervises a causal student model with a bidirectional teacher. This approach effectively mitigates error accumulation in autoregressive generation, allowing long-duration video synthesis despite training on short clips. Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models. It enables fast streaming generation of high-quality videos at 9.4 FPS on a single GPU thanks to KV caching. Our approach also enables streaming video-to-video translation, image-to-video, and dynamic prompting in a zero-shot manner. We will release the code based on an open-source model in the future.

• The paper introduces a causal transformer-based model, CausVid, that generates video frames on-the-fly at approximately 9.4 FPS after a 1.3-second startup delay.
• It employs asymmetric distribution matching distillation to transfer multi-step bidirectional knowledge into a four-step causal student model, ensuring competitive output quality.
• The approach enables fast, interactive video synthesis for real-time applications like video editing and dynamic content generation while improving temporal and frame consistency.

An Evaluation of "From Slow Bidirectional to Fast Causal Video Generators"

The paper "From Slow Bidirectional to Fast Causal Video Generators" presents a novel approach to improving the efficiency of video diffusion models through the introduction of a fast, streaming video generation model named CausVid. The motivation behind this work stems from the inherent limitations of existing video diffusion models that operate in a bidirectional manner, resulting in substantial latency and computational overhead due to their requirement to process entire video sequences for the generation of individual frames.

Key Contributions

1. Causal Video Generation Architecture: The authors propose a conversion from bidirectional to causal video generation by adapting a pretrained bidirectional diffusion transformer into a causal transformer that enables on-the-fly frame generation. This transformation significantly reduces the initial latency required for video generation by facilitating continuous frame streaming, offering approximately 9.4 FPS after a brief 1.3 seconds of initial latency.
2. Asymmetric Distribution Matching Distillation (DMD): To maintain high-quality output and stability despite reduced inference steps, the paper introduces an asymmetric distillation technique where knowledge from a rigorous, multi-step, bidirectional teacher model is distilled into a four-step causal student model. This approach leverages ODE initialization techniques and demonstrates the capability of autoregressive models in achieving competitive video quality while enhancing interactivity and speed.
3. Implications for Interactive Applications: The reduced latency and continuous streaming capabilities of CausVid open new possibilities for interactive applications, such as dynamic video-to-video translation and image-to-video generation. The system supports responsive workflows by adapting to user input changes more effectively than previous methods.

Experimental Evaluation

The authors conducted extensive experiments and benchmarked CausVid on various datasets, including the VBench, to evaluate its performance in terms of temporal quality, frame quality, and text alignment. Compared to existing state-of-the-art models like CogVideoX, OpenSORA, and MovieGen, CausVid demonstrated superior temporal consistency and frame quality, validating its effectiveness.

Moreover, the human perceptual paper further highlighted CausVid's capability by showing a preference for its outputs over those produced by both its teacher model and other contemporary video generation methods. The introduction of block-wise causal attention and KV caching mechanisms support efficient inference, greatly enhancing practical usability.

Implications and Future Directions

The development of CausVid represents an important step towards practical video generation using diffusion models. Its architecture facilitates rapid and interactive video synthesis, enabling applications beyond static content creation, such as real-time video editing and interactive game rendering.

Despite these advancements, the paper acknowledges areas for future exploration, such as long-range video consistency and reducing inter-chunk temporal inconsistencies. Addressing these challenges could further improve the robustness of causal generators for even more demanding applications.

Additionally, the paper's approach to reducing output diversity, an artifact of its reliance on KL-divergence-based distribution matching, suggests potential areas for refining the model's ability to handle diverse and complex generation tasks. Future research might explore alternative distillation objectives that maintain diversity without compromising quality.

In summary, "From Slow Bidirectional to Fast Causal Video Generators" offers a significant contribution to the field of video generation by introducing a method that balances efficiency and output quality. The implications for interactive and real-time applications make it a notable advancement in the scalability and applicability of diffusion-based video synthesis techniques.