T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design (2410.05677v2)

Abstract: In this paper, we focus on enhancing a diffusion-based text-to-video (T2V) model during the post-training phase by distilling a highly capable consistency model from a pretrained T2V model. Our proposed method, T2V-Turbo-v2, introduces a significant advancement by integrating various supervision signals, including high-quality training data, reward model feedback, and conditional guidance, into the consistency distillation process. Through comprehensive ablation studies, we highlight the crucial importance of tailoring datasets to specific learning objectives and the effectiveness of learning from diverse reward models for enhancing both the visual quality and text-video alignment. Additionally, we highlight the vast design space of conditional guidance strategies, which centers on designing an effective energy function to augment the teacher ODE solver. We demonstrate the potential of this approach by extracting motion guidance from the training datasets and incorporating it into the ODE solver, showcasing its effectiveness in improving the motion quality of the generated videos with the improved motion-related metrics from VBench and T2V-CompBench. Empirically, our T2V-Turbo-v2 establishes a new state-of-the-art result on VBench, with a Total score of 85.13, surpassing proprietary systems such as Gen-3 and Kling.

• The paper introduces a novel consistency distillation framework that leverages high-quality datasets and diverse reward models to improve text-to-video generation.
• It rigorously demonstrates through ablation studies that tailored dataset selection and conditional guidance significantly enhance visual quality and semantic alignment.
• The enhanced model achieves state-of-the-art performance on VBench with a Total score of 85.13, underscoring its potential for practical video synthesis applications.

Overview of "T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design"

The paper presents T2V-Turbo-v2, a sophisticated approach to improving text-to-video (T2V) models in the post-training phase. By employing a diffusion-based model, the authors aim to refine video generation through a comprehensive framework that integrates high-quality data, diverse reward feedback, and conditional guidance.

Key Contributions

1. Consistency Distillation (CD) Enhancement: T2V-Turbo-v2 advances the post-training of a diffusion-based T2V model by distilling from a pretrained model using a revised consistency model (CM). This involves integrating supervision signals from new datasets and various reward models (RMs), resulting in videos with superior visual quality and semantic alignment.
2. Innovative Use of Datasets and RMs: By conducting rigorous ablation studies, the authors underline the critical importance of dataset selection tailored to specific learning objectives. Utilizing datasets like VidGen-1M and WebVid-10M, T2V-Turbo-v2 achieves significant gains in performance by selectively optimizing reward models, including HPSv2.1 and InternV2.
3. Conditional Guidance Design: The paper explores extensive design space in conditional guidance strategies, introducing an energy function that augments the ODE solver. Motion guidance is extracted from training videos, significantly enhancing motion features and quality.
4. Results and Performance: The model establishes a new state-of-the-art on VBench, with a Total score of 85.13, exceeding proprietary systems such as Gen-3 and Kling, demonstrating substantial improvements across multiple video dimensions.

Methodology

The core methodological innovation lies in the integration of diverse supervision signals during the CD process. The approach conditions the consistency function on guidance parameters, providing a direct mapping in ODE solver space. The use of high-quality video datasets serves dual purposes: providing a base for CD loss minimization and optimizing reward feedback for text-video alignment.

Implications and Future Directions

The implications of T2V-Turbo-v2 are multifaceted:

• Theoretical Advancements: The methodology offers a robust framework for enhancing video generation models, paving the way for future research that could explore alternative energy functions or novel reward models.
• Practical Applications: In real-world implementations, the improved text-video alignment and motion quality could significantly impact industries reliant on video content generation, such as entertainment, advertising, and e-learning.

Future developments could focus on creating long-context RMs to exploit high-quality datasets further and improve alignment in detailed prompts. Additionally, replacing or enhancing the text encoder to handle dense captions could enhance model capacity even more.

Conclusion

T2V-Turbo-v2 represents a pivotal advancement in T2V model post-training. By strategically selecting high-quality data and employing a diverse set of RMs, combined with innovative conditional guidance, the authors have laid a foundation for subsequent innovations in video synthesis technology, both theoretically and in applied settings.