ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation (2410.20502v1)

Abstract: Text-to-video models have recently undergone rapid and substantial advancements. Nevertheless, due to limitations in data and computational resources, achieving efficient generation of long videos with rich motion dynamics remains a significant challenge. To generate high-quality, dynamic, and temporally consistent long videos, this paper presents ARLON, a novel framework that boosts diffusion Transformers with autoregressive models for long video generation, by integrating the coarse spatial and long-range temporal information provided by the AR model to guide the DiT model. Specifically, ARLON incorporates several key innovations: 1) A latent Vector Quantized Variational Autoencoder (VQ-VAE) compresses the input latent space of the DiT model into compact visual tokens, bridging the AR and DiT models and balancing the learning complexity and information density; 2) An adaptive norm-based semantic injection module integrates the coarse discrete visual units from the AR model into the DiT model, ensuring effective guidance during video generation; 3) To enhance the tolerance capability of noise introduced from the AR inference, the DiT model is trained with coarser visual latent tokens incorporated with an uncertainty sampling module. Experimental results demonstrate that ARLON significantly outperforms the baseline OpenSora-V1.2 on eight out of eleven metrics selected from VBench, with notable improvements in dynamic degree and aesthetic quality, while delivering competitive results on the remaining three and simultaneously accelerating the generation process. In addition, ARLON achieves state-of-the-art performance in long video generation. Detailed analyses of the improvements in inference efficiency are presented, alongside a practical application that demonstrates the generation of long videos using progressive text prompts. See demos of ARLON at \url{http://aka.ms/arlon}.

Overview of ARLON: Enhancing Diffusion Transformers for Long Video Generation

The research article titled "ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation" introduces a novel framework designed to enhance the efficiency and quality of text-to-video (T2V) generation. The proposed methodology, ARLON, combines autoregressive (AR) models with diffusion transformers (DiT) to address the challenges associated with generating long videos, particularly those involving rich motion dynamics and temporal consistency.

Key Innovations

ARLON introduces several notable innovations to enable efficient long video generation:

1. Latent VQ-VAE Compression: The framework employs a Vector Quantized Variational Autoencoder (VQ-VAE) to compress the latent input space of the DiT model. This compression results in compact, quantized visual tokens that bridge the AR and DiT models, optimizing learning complexity and information density.
2. Semantic Injection Module: An adaptive norm-based semantic injection module is used to integrate coarse discrete visual units from the AR model into the DiT model. This integration ensures effective guidance during the video generation process.
3. Noise Tolerance Strategy: The DiT model is trained with coarser visual latent tokens using an uncertainty sampling module, enhancing its tolerance to noise introduced during AR inference.

Experimental Results

The paper reports that ARLON significantly outperforms the baseline model, OpenSora-V1.2, achieving superior results on eight out of eleven metrics from the VBench benchmark. Particularly, ARLON excels in dynamic degree and aesthetic quality, demonstrating its prowess in generating high-quality, dynamic, and temporally coherent long videos.

Theoretical and Practical Implications

The integration of AR models provides a richer dynamic range for the generation of long videos, a domain where traditional diffusion models struggle, especially concerning temporal coherence and detail richness. The paper's results imply that ARLON not only elevates the quality of generated long-form content but also accelerates the generation process, thus showcasing a balanced trade-off between efficiency and quality.

Future Prospects

The innovative approach of leveraging AR models for initializing and guiding the DiT process suggests several future research directions:

• Expanded Use Cases: The ARLON framework could be adapted for various applications beyond traditional T2V, such as interactive media generation and virtual reality content creation.
• Enhanced Model Architectures: Future work could explore more advanced semantic injection methods and refined compression techniques to further improve model robustness and output fidelity.
• Scalability with Larger Datasets: Given the emergence of larger and more complex datasets, ARLON's methods may evolve to handle increased data volumes and diversity, thus broadening its applicability in real-world scenarios.

Conclusion

The paper presents a well-defined strategy for combining the strengths of diffusion transformers and autoregressive models to produce long videos that are both aesthetically pleasing and temporally consistent. ARLON's methodological contributions mark a significant step forward in the development of efficient and high-quality T2V generation techniques, setting a new benchmark for long video synthesis.