V-Express: Conditional Dropout for Progressive Training of Portrait Video Generation (2406.02511v1)

Abstract: In the field of portrait video generation, the use of single images to generate portrait videos has become increasingly prevalent. A common approach involves leveraging generative models to enhance adapters for controlled generation. However, control signals (e.g., text, audio, reference image, pose, depth map, etc.) can vary in strength. Among these, weaker conditions often struggle to be effective due to interference from stronger conditions, posing a challenge in balancing these conditions. In our work on portrait video generation, we identified audio signals as particularly weak, often overshadowed by stronger signals such as facial pose and reference image. However, direct training with weak signals often leads to difficulties in convergence. To address this, we propose V-Express, a simple method that balances different control signals through the progressive training and the conditional dropout operation. Our method gradually enables effective control by weak conditions, thereby achieving generation capabilities that simultaneously take into account the facial pose, reference image, and audio. The experimental results demonstrate that our method can effectively generate portrait videos controlled by audio. Furthermore, a potential solution is provided for the simultaneous and effective use of conditions of varying strengths.

• The paper presents V-Express, a method that integrates conditional dropout in progressive training to balance multiple control signals for portrait video generation.
• It uses a three-stage training strategy with modules like ReferenceNet and V-Kps Guider to enhance video quality and synchronization.
• Experimental results on TalkingHead-1KH and AVSpeech datasets show superior FID and FVD scores compared to methods such as Wav2Lip and DiffusedHeads.

Conditional Dropout for Progressive Training of Portrait Video Generation

In the paper titled "V-Express: Conditional Dropout for Progressive Training of Portrait Video Generation," the authors present a comprehensive methodology to address the challenge of generating high-quality portrait videos from single images using multiple control signals of varying strengths. The method, termed V-Express, balances control signals through progressive training and a conditional dropout operation.

Introduction

Utilizing single images to generate portrait videos is a growing trend in areas such as virtual avatars and personalized video content. Due to the advancements in diffusion models, control signals like text, audio, reference images, keypoints, and depth maps can be employed to enhance the quality and accuracy of the generated videos. However, control signals differ in strength, creating a significant challenge in achieving a balanced and effective control mechanism. The weaker conditions, such as audio signals, often get overshadowed by stronger ones like facial pose and reference images. V-Express aims to address these issues.

Method

The V-Express methodology employs a Latent Diffusion Model (LDM) to generate video frames and incorporates three key modules: ReferenceNet, V-Kps Guider, and Audio Projection. These modules effectively handle various control inputs to generate synchronized portrait videos. The architecture of V-Express consists of a denoising U-Net with four specific attention layers that capture spatial and temporal relationships. The training strategy is progressive, ensuring the model initially focuses on easier tasks and gradually takes on more complex tasks, thereby allowing effective control by weaker conditions.

Progressive Training Strategy

The training is divided into three stages:

1. Stage I: Focuses on single-frame generation, training the ReferenceNet, V-Kps Guider, and the denoising U-Net while freezing the Audio and Motion Attention Layers.
2. Stage II: Involves multi-frame generation, training the Audio Projection, Audio Attention Layers, and Motion Attention Layers while keeping other parameters fixed.
3. Stage III: Global fine-tuning, updating all parameters to refine the overall performance.

During Stages II and III, conditional dropout is employed to disrupt shortcut patterns, ensuring balanced control from various signals.

Experimental Results

The experiment section presents a quantitative comparison against Wav2Lip and DiffusedHeads on subsets of TalkingHead-1KH and AVSpeech datasets. The metrics used are FID, FVD, $\Delta$FaceSim, KpsDis, and SyncNet. V-Express excels in video quality and signal alignment, although it does not achieve the best lip synchronization score. Notably, V-Express shows superiority in video quality with FID and FVD scores significantly lower than those of the competing methods.

Results and Implications

The qualitative results illustrate V-Express's capability in generating high-quality portrait videos controlled by both audio and V-Kps signals. The attention weights of the cross-attention layers can modulate the influence of each control signal, which is pivotal for achieving desired video generation outcomes.

Conclusion and Future Work

V-Express presents a robust solution for balancing control signals in portrait video generation, effectively addressing the challenges posed by varying signal strengths. The method enhances video quality and synchronization, providing a balanced approach to integrating multiple control signals. Future work proposes addressing multilingual support, reducing computational burden, and enabling explicit face attribute control.

V-Express introduces significant advancements in generating high-quality, synchronized portrait videos from single images, paving the way for more advanced and balanced systems in video generation. The implications are profound for various applications in digital entertainment, virtual avatars, and personalized content creation.

The references provide a detailed background for the methodologies and datasets used, contributing to a well-rounded understanding of the advancements in portrait video generation.