DreamFactory: Pioneering Multi-Scene Long Video Generation with a Multi-Agent Framework (2408.11788v1)

Abstract: Current video generation models excel at creating short, realistic clips, but struggle with longer, multi-scene videos. We introduce \texttt{DreamFactory}, an LLM-based framework that tackles this challenge. \texttt{DreamFactory} leverages multi-agent collaboration principles and a Key Frames Iteration Design Method to ensure consistency and style across long videos. It utilizes Chain of Thought (COT) to address uncertainties inherent in LLMs. \texttt{DreamFactory} generates long, stylistically coherent, and complex videos. Evaluating these long-form videos presents a challenge. We propose novel metrics such as Cross-Scene Face Distance Score and Cross-Scene Style Consistency Score. To further research in this area, we contribute the Multi-Scene Videos Dataset containing over 150 human-rated videos.

• The paper introduces a multi-agent framework where LLMs fulfill roles in film production to generate long, multi-scene videos.
• It employs a Keyframe Iteration Design Method to maintain style and character consistency across scenes by transforming long-term memory into iterative tasks.
• It presents novel evaluation metrics such as CSFD, CSSC, and CLIP scores, demonstrating significant improvements over existing video generation models.

DreamFactory: Pioneering Multi-Scene Long Video Generation with a Multi-Agent Framework

The paper DreamFactory: Pioneering Multi-Scene Long Video Generation with a Multi-Agent Framework by Zhifei Xie et al. introduces an innovative framework that addresses the challenges of generating long, multi-scene videos. Current video generation models typically excel in creating short, realistic clips but falter with longer content that requires consistency in style and character portrayal across scenes. DreamFactory leverages multi-agent collaboration principles, inspired by human cooperative behaviors, to address this issue.

Overview

DreamFactory is developed as a comprehensive framework that facilitates the generation of long, consistent videos by simulating a virtual film production environment. Key to this framework is the introduction of LLMs acting as various roles within a production team, such as directors, art directors, and screenwriters. The framework utilizes a Key Frames Iteration Design Method to maintain consistency and style across different video scenes, ensuring a seamless visual narrative.

Methodology

DreamFactory's architecture involves several critical phases that mirror the traditional video production workflow:

1. Task Definition
2. Style Decision
3. Story Prompting
4. Script Design
5. Key-frame Design

Each phase involves collaborative interactions among LLM-based agents, ensuring detailed and coherent progression from scriptwriting to video synthesis. A notable innovation is the Keyframe Iteration Design Method, which ensures long-term memory retention by transforming long-term memory needs into short-term iterative tasks. This method involves generating a foundational keyframe (Base Frame) and iterating upon this frame while maintaining style and character consistency.

Evaluations and Metrics

The paper proposes novel metrics to evaluate the quality and consistency of the generated videos:

1. Cross-Scene Face Distance Score (CSFD Score): Measures the consistency of characters' facial features across different scenes.
2. Cross-Scene Style Consistency Score (CSSC Score): Ensures stylistic consistency throughout the video.
3. Average Key-Frames CLIP Score: Evaluates the alignment of each scene's keyframes with the corresponding textual description.

These metrics address the lack of robust evaluation mechanisms in the domain of long video generation and provide quantitative measures for assessing the framework's effectiveness.

Experimental Results

The framework's performance was validated using state-of-the-art video generation models and various datasets such as UTF-101 and HMDB51. The authors compared DreamFactory's results with those from existing video generation tools, demonstrating substantial improvements in video quality and consistency. Notably, the following results were highlighted:

• FID, IS, and CLIP Scores: Showed significant enhancement in image quality and alignment with textual descriptions.
• FVD and KVD Scores: Indicated marked improvements in long video quality metrics.
• Human Evaluations: Reinforced the superior preference for videos generated by DreamFactory in terms of role and scene consistency, plot quality, and overall fluidity.

Practical and Theoretical Implications

Practical Implications: DreamFactory has significant potential in content creation industries, where long, consistent videos are critical. Its ability to automate extensive portions of the video production process can save substantial time and resources.

Theoretical Implications: The introduction of a multi-agent collaborative framework to the video generation domain opens new avenues for research. The paper's methodological innovations, such as the Keyframe Iteration Design, contribute to the broader understanding of maintaining consistency in AI-generated content.

Future Developments

The research sets the groundwork for future advancements in AI-driven video generation. Potential developments might include:

• Enhancing the creative capabilities of agents to improve plot and artistic elements.
• Increasing the framework's efficiency to handle even longer video compositions.
• Expanding the dataset and fine-tuning models to further refine the quality of generated content.

Conclusion

DreamFactory represents a significant step forward in the domain of AI-generated video content, addressing long-standing challenges of consistency and quality in multi-scene video generation. By leveraging multi-agent collaboration and innovative design methods, the framework paves the way for more sophisticated video production capabilities in artificial intelligence. As the field continues to evolve, the principles and metrics introduced by DreamFactory will undoubtedly contribute to future research and application in video generation technologies.