MagicVideo-V2: Multi-Stage High-Aesthetic Video Generation (2401.04468v1)

Published 9 Jan 2024 in cs.CV and cs.AI

Abstract: The growing demand for high-fidelity video generation from textual descriptions has catalyzed significant research in this field. In this work, we introduce MagicVideo-V2 that integrates the text-to-image model, video motion generator, reference image embedding module and frame interpolation module into an end-to-end video generation pipeline. Benefiting from these architecture designs, MagicVideo-V2 can generate an aesthetically pleasing, high-resolution video with remarkable fidelity and smoothness. It demonstrates superior performance over leading Text-to-Video systems such as Runway, Pika 1.0, Morph, Moon Valley and Stable Video Diffusion model via user evaluation at large scale.

View on arXiv

References (15)

Authors (12)

Weimin Wang (52 papers)
Jiawei Liu (156 papers)
Zhijie Lin (30 papers)
Jiangqiao Yan (3 papers)
Shuo Chen (127 papers)
Chetwin Low (2 papers)
Tuyen Hoang (3 papers)
Jie Wu (230 papers)
Jun Hao Liew (29 papers)
Hanshu Yan (28 papers)
Daquan Zhou (47 papers)
Jiashi Feng (295 papers)

Citations (24)

View on Semantic Scholar

Summary

Overview of MagicVideo-V2

The domain of video generation from textual prompts has taken a leap forward with the introduction of MagicVideo-V2. This sophisticated framework encapsulates various components of video generation into a seamless, end-to-end pipeline, markedly improving the quality and aesthetics of generated videos. It encompasses several independent modules, each designed to perform specific tasks in the creation of videos that are not only high-resolution but also display an impressive fidelity to the initial text prompts.

Key Components of the System

At the heart of MagicVideo-V2 lie four critical modules that work in concert to transform text descriptions into visual narratives:

Text-to-Image Module: The first step involves generating an initial high-fidelity image based on a given text prompt. This image serves as a reference for the video contents and aesthetic style.
Image-to-Video Module: Using the initial image along with the prompt, this module generates keyframes for the video, infusing movement while maintaining the scene's visual quality and content consistency.
Video-to-Video Module: This component refines the keyframes produced by the previous module, enhancing their resolution and detail to yield a high-resolution video.
Video Frame Interpolation: To achieve motion smoothness across frames, this module interpolates additional frames between the existing keyframes, resulting in a fluid and cohesive video sequence.

Evaluation and Performance

MagicVideo-V2 was evaluated through human judgment against several state-of-the-art text-to-video systems. In a large user paper with 61 evaluators, MagicVideo-V2 consistently outperformed other methods across various benchmarks, including visual appeal, temporal consistency, and incidence of structural errors. These comparisons attest to the advanced capabilities of MagicVideo-V2 in generating videos that meet human visual standards for quality and aesthetic appeal.

Conclusion and Implications

Concluding, MagicVideo-V2 establishes a new benchmark in the text-to-video generation landscape with its innovative multi-stage approach. Its modular architecture allows for the generation of videos that are both visually stunning and temporally coherent. With human evaluators favoring MagicVideo-V2 over other methods, it signifies a notable stride in video synthesis technology, promising advancements in areas such as entertainment, content creation, and more. MagicVideo-V2 indeed marks a significant milestone in the interplay between artificial intelligence and creative video production.