EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions (2402.17485v3)

Abstract: In this work, we tackle the challenge of enhancing the realism and expressiveness in talking head video generation by focusing on the dynamic and nuanced relationship between audio cues and facial movements. We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose EMO, a novel framework that utilizes a direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks. Our method ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations. Experimental results demonsrate that EMO is able to produce not only convincing speaking videos but also singing videos in various styles, significantly outperforming existing state-of-the-art methodologies in terms of expressiveness and realism.

• The paper introduces a diffusion-based framework that directly synthesizes expressive talking head videos from audio inputs.
• It incorporates speed and face region controllers to stabilize generation and improve visual-temporal consistency.
• It leverages a large-scale, diverse dataset and identity-preserving techniques to achieve state-of-the-art performance.

Enhancing Realism in Talking Head Video Generation with the EMO Framework under Weak Conditions

Introduction to the EMO Framework

The emergence of Diffusion Models has marked a significant milestone in the field of generative models, particularly in image and video synthesis. These models have demonstrated exceptional capabilities in generating images and videos with high levels of detail and realism. An area of keen interest and notable challenge within video generation research is the synthesis of human-centric videos, such as talking heads, where the goal is to generate facial expressions from audio inputs. Traditional methodologies often employ 3D models or extract sequences of head movements, which, while simplifying the task, tend to compromise the naturalness of the resulting facial expressions.

Addressing these limitations, the EMO framework introduces a novel talking head generation approach that leverages the generative prowess of Diffusion Models to synthesize character head videos directly from an image and an audio clip. This method circumvents the need for intermediate representations or intricate preprocessing, thereby facilitating the creation of talking head videos that closely align with the audio input in terms of visual and emotional fidelity.

Addressing the Challenges of Audio-to-Video Synthesis

One of the primary challenges in synthesizing expressive portrait videos from audio cues lies in the ambiguity inherent to the mapping between audio signals and facial expressions. This ambiguity can lead to instability in the generated videos, manifesting as facial distortions or inconsistencies across frames. To mitigate these issues, the EMO framework integrates stable control mechanisms, specifically a speed controller and a face region controller. These mechanisms act as hyperparameters that provide subtle control signals, ensuring the stability of video generation without sacrificing expressiveness or diversity.

Furthermore, maintaining the character's identity consistency across the video is paramount for the realism of the output. To achieve this, EMO employs a component similar to ReferenceNet, termed FrameEncoding, which is designed to preserve the character’s identity throughout the video generation process.

Training and Performance

To train the EMO model, a vast and diverse audio-video dataset was compiled, featuring over 250 hours of footage and more than 150 million images. This dataset encompasses a wide range of content types and linguistic varieties, providing a comprehensive foundation for the development of the EMO framework. In comparative evaluations on the HDTF dataset, the EMO approach surpassed current state-of-the-art methods across several metrics, including FID, SyncNet, F-SIM, and FVD. Both quantitative assessments and user studies confirmed EMO's ability to generate highly realistic and expressive talking and singing videos.

Related Work and Theoretical Underpinnings

The paper situates the EMO framework within the broader context of advancements in Diffusion Models and their applications in video generation. It draws a distinction between video-based and single-image-based approaches to audio-driven talking head generation, highlighting the limitations of existing methodologies. Moreover, it elaborates on the principles of Diffusion Models, emphasizing their utility in generating high-quality visuals and their adaptability to video synthesis tasks.

Future Directions and Limitations

Despite its notable achievements, the EMO framework is not without limitations. The generation process is relatively time-consuming compared to non-diffusion model-based methods, and the absence of explicit control signals for non-facial body parts can lead to the generation of artifacts. Addressing these challenges presents an avenue for future research, potentially involving the integration of control signals for finer manipulation of body parts.

Conclusion

The EMO framework represents a significant advancement in the generation of expressive portrait videos from audio inputs. By leveraging the capabilities of Diffusion Models and introducing innovative control mechanisms, it achieves a high degree of realism and expressiveness, setting a new benchmark for future developments in the field of talking head video generation.