LTX-Video: Realtime Video Latent Diffusion (2501.00103v1)

Abstract: We introduce LTX-Video, a transformer-based latent diffusion model that adopts a holistic approach to video generation by seamlessly integrating the responsibilities of the Video-VAE and the denoising transformer. Unlike existing methods, which treat these components as independent, LTX-Video aims to optimize their interaction for improved efficiency and quality. At its core is a carefully designed Video-VAE that achieves a high compression ratio of 1:192, with spatiotemporal downscaling of 32 x 32 x 8 pixels per token, enabled by relocating the patchifying operation from the transformer's input to the VAE's input. Operating in this highly compressed latent space enables the transformer to efficiently perform full spatiotemporal self-attention, which is essential for generating high-resolution videos with temporal consistency. However, the high compression inherently limits the representation of fine details. To address this, our VAE decoder is tasked with both latent-to-pixel conversion and the final denoising step, producing the clean result directly in pixel space. This approach preserves the ability to generate fine details without incurring the runtime cost of a separate upsampling module. Our model supports diverse use cases, including text-to-video and image-to-video generation, with both capabilities trained simultaneously. It achieves faster-than-real-time generation, producing 5 seconds of 24 fps video at 768x512 resolution in just 2 seconds on an Nvidia H100 GPU, outperforming all existing models of similar scale. The source code and pre-trained models are publicly available, setting a new benchmark for accessible and scalable video generation.

• The paper's main contribution is LTX-Video, which shifts patchifying to the VAE input, achieving a 1:192 compression ratio for efficient video processing.
• The framework supports both text-to-video and image-to-video generation in a single model that emphasizes prompt adherence, visual quality, and motion fidelity.
• The model generates 5 seconds of 768x512 video in roughly 2 seconds on an Nvidia H100 GPU, outperforming comparable models in speed and quality.

An Overview of LTX-Video: Realtime Video Latent Diffusion

The paper presents LTX-Video, a novel framework in the domain of video generation utilizing transformer-based latent diffusion models. LTX-Video distinguishes itself by integrating the Video-VAE and denoising transformer components more cohesively compared to existing methodologies, optimizing their interaction for enhanced video generation efficiency and quality.

Key Contributions and Methodology

1. Holistic Approach to Latent Diffusion: LTX-Video innovates by relocating the patchifying operation to the VAE’s input rather than the transformer's. This structural adjustment enables significant compression (1:192 ratio) and reduced data redundancy, thereby facilitating efficient processing in a compressed latent space. The VAE decoder uniquely combines latent-to-pixel conversion with the final step of denoising, circumventing the need for separate upsampling processes and enhancing detail generation in the pixel space.
2. Model Capabilities: The research highlights LTX-Video's ability to support both text-to-video and image-to-video generation. The integrated model approach ensures both functionalities are trained simultaneously, emphasizing prompt adherence, visual quality, and motion fidelity.
3. Performance Metrics: LTX-Video exhibits a remarkable performance profile, generating 5 seconds of video at a resolution of 768x512 pixels in approximately 2 seconds on an Nvidia H100 GPU, surpassing existing models of similar scale regarding speed and output quality. This demonstration underlines the model's synthesis capabilities in real-time applications.
4. Architecture Design: The model builds upon previous frameworks, incorporating improvements such as Rotary Positional Embeddings, QK-normalization, and a restructured diffusion strategy. These enhancements focus on optimizing spatial and temporal coherence in the generated videos, contributing to more stable attention calculations.

Experimental Validation

In a series of experiments, LTX-Video's efficiency and capacity for high-quality video generation were validated. It exhibited a distinct advantage in generating coherent and aesthetically aligned video outputs compared to models like Open-Sora Plan and CogVideoX within similar computational budgets.

Implications and Future Directions

From a practical perspective, LTX-Video represents a significant advancement in scalable and accessible video generation. Its public availability encourages applications across diverse sectors, potentially revolutionizing content creation processes. Theoretically, the approach paves the way for more efficient video modeling techniques that balance compression with detail retention.

Future research could explore extending the architecture for longer video sequences and enhancing domain-specific adaptability. In doing so, LTX-Video could further its impact on AI-driven video synthesis, offering broader applicability and refinement in visual content generation.

The presented framework of LTX-Video sets a new benchmark by delivering a robust and efficient model for video generation, backed by empirical evidence and designed with practical considerations for the wider research community.