Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation (2408.09787v1)

Abstract: Traditional animation generation methods depend on training generative models with human-labelled data, entailing a sophisticated multi-stage pipeline that demands substantial human effort and incurs high training costs. Due to limited prompting plans, these methods typically produce brief, information-poor, and context-incoherent animations. To overcome these limitations and automate the animation process, we pioneer the introduction of large multimodal models (LMMs) as the core processor to build an autonomous animation-making agent, named Anim-Director. This agent mainly harnesses the advanced understanding and reasoning capabilities of LMMs and generative AI tools to create animated videos from concise narratives or simple instructions. Specifically, it operates in three main stages: Firstly, the Anim-Director generates a coherent storyline from user inputs, followed by a detailed director's script that encompasses settings of character profiles and interior/exterior descriptions, and context-coherent scene descriptions that include appearing characters, interiors or exteriors, and scene events. Secondly, we employ LMMs with the image generation tool to produce visual images of settings and scenes. These images are designed to maintain visual consistency across different scenes using a visual-language prompting method that combines scene descriptions and images of the appearing character and setting. Thirdly, scene images serve as the foundation for producing animated videos, with LMMs generating prompts to guide this process. The whole process is notably autonomous without manual intervention, as the LMMs interact seamlessly with generative tools to generate prompts, evaluate visual quality, and select the best one to optimize the final output.

• The paper introduces an autonomous animation generation framework powered by LMMs that enhances text-to-image and video quality.
• It employs a six-stage process integrating GPT-4 with generative tools such as Midjourney and Pika to transform simple narratives into detailed animated scripts.
• Quantitative evaluations using CLIP and VBench metrics demonstrate superior character consistency, scene coherence, and video quality compared to baseline models.

Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation

The paper "Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation" introduces a novel framework aiming at automating the animation video creation process using Large Multimodal Models (LMMs). Traditional methods for animation generation entailed developing generative models trained with human-labeled datasets, leading to a complexity that required substantial human intervention and incurred high costs. This paper proposes leveraging LMMs' advanced understanding and reasoning capabilities to design an autonomous animation-making agent.

Framework and Methodology

Anim-Director employs LMMs, particularly GPT-4, integrated with generative tools such as Midjourney for images and Pika for video creation. The model operates through a six-stage process:

1. Story Refinement: LMMs refine a brief narrative into a detailed and coherent storyline, expanding character dialogues and enhancing plot details.
2. Script Generation: The system generates a detailed director-like script from the refined narrative, outlining character profiles, scene settings, and context-coherent descriptions to structure the animation workflow.
3. Scene Image Generation: Using Midjourney, the model creates high-quality visual representations for each scene, ensuring characters and settings are vividly depicted to maintain visual consistency.
4. Scene Image Improvement: LMMs evaluate and refine generated images for content accuracy and visual consistency using a self-reflection mechanism and image segmentation techniques.
5. Video Production: The framework uses scene images combined with descriptive text to generate animations through Pika, optimizing the generative process by predicting the best hyperparameters.
6. Video Quality Enhancement: The final stage involves evaluating the generated video quality using distortion detection and consistency metrics, ensuring the best possible output by selecting the optimal candidate video.

Quantitative and Qualitative Evaluation

The effectiveness of Anim-Director is validated using a dataset of concise narratives from TinyStories and evaluated against several state-of-the-art models in image and video generation domains.

Text-to-Image Evaluation:

The contextual coherence, image-text, and image-image similarities were assessed using CLIP feature space. The results demonstrated that Anim-Director achieves superior performance, particularly in maintaining character and background consistency across scenes.

Video Quality Assessment:

The paper adopts VBench metrics to quantify video quality, assessing distortion, subject, and background consistency, as well as text-video alignment. Anim-Director exhibited the highest performance, generating longer and contextually richer videos compared to baseline models.

Implications and Future Directions

The introduction of Anim-Director signifies a step forward in democratizing and enhancing the animation production process. By automating intricate aspects of animation creation, this approach reduces the reliance on large studios and extensive human resources, making high-quality animation accessible to smaller teams and individual creators.

The implications of this research extend to various industry applications, including entertainment, education, and marketing, where animation plays a critical role. The integration of LMMs and generative tools into autonomous agents opens avenues for refined content generation, broader creative freedom, and more efficient production workflows.

Future research could focus on further improving the visual quality and contextual coherence of generated videos, particularly for longer animations. Enhancing control over the generative process to manage scene transitions smoothly will be crucial. Owing to the flexibility of training-free approaches, the underlying methodology could be adapted for other creative content generation domains, potentially combining audio and interactive elements.

In summary, Anim-Director demonstrates significant potential in streamlining and advancing animation generation, leveraging the capabilities of LMMs to foster innovation and accessibility in multimedia content creation.