HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characters (2505.20156v2)

Abstract: Recent years have witnessed significant progress in audio-driven human animation. However, critical challenges remain in (i) generating highly dynamic videos while preserving character consistency, (ii) achieving precise emotion alignment between characters and audio, and (iii) enabling multi-character audio-driven animation. To address these challenges, we propose HunyuanVideo-Avatar, a multimodal diffusion transformer (MM-DiT)-based model capable of simultaneously generating dynamic, emotion-controllable, and multi-character dialogue videos. Concretely, HunyuanVideo-Avatar introduces three key innovations: (i) A character image injection module is designed to replace the conventional addition-based character conditioning scheme, eliminating the inherent condition mismatch between training and inference. This ensures the dynamic motion and strong character consistency; (ii) An Audio Emotion Module (AEM) is introduced to extract and transfer the emotional cues from an emotion reference image to the target generated video, enabling fine-grained and accurate emotion style control; (iii) A Face-Aware Audio Adapter (FAA) is proposed to isolate the audio-driven character with latent-level face mask, enabling independent audio injection via cross-attention for multi-character scenarios. These innovations empower HunyuanVideo-Avatar to surpass state-of-the-art methods on benchmark datasets and a newly proposed wild dataset, generating realistic avatars in dynamic, immersive scenarios.

• The paper introduces HunyuanVideo-Avatar, a model using an MM-DiT architecture to generate high-fidelity, audio-driven human animations for multiple characters, tackling consistency and emotion alignment challenges.
• It features innovative modules like the Face-Aware Audio Adapter (FAA) enabling independent audio control for individual characters in multi-character scenes.
• Experimental results show HunyuanVideo-Avatar outperforms state-of-the-art models in video quality, lip sync, and emotional expression alignment across various datasets.

HunyuanVideo-Avatar: An Advanced Model for Audio-Driven Human Animation

The paper presents HunyuanVideo-Avatar, a sophisticated model for generating high-fidelity, audio-driven human animations, incorporating multiple characters. At the core of this approach is the MM-DiT (Multimodal Diffusion Transformer)-based architecture which aims to address prevailing challenges in the field, such as maintaining character consistency, precise emotion alignment with audio, and enabling multi-character dialogues driven by audio inputs.

Key Innovations

HunyuanVideo-Avatar introduces several innovative modules, each designed to enhance different aspects of audio-driven animation:

1. Character Image Injection Module: This module replaces conventional addition-based conditioning methods that often result in condition mismatches between training and inference phases. It ensures dynamic motion coupled with robust character consistency, thereby solving the dynamism-consistency trade-off.
2. Audio Emotion Module (AEM): AEM extracts emotional cues from reference images and infuses them into target videos. This allows for fine-grained control over emotions in generated animations, ensuring that facial expressions align closely with the audio's emotional tone.
3. Face-Aware Audio Adapter (FAA): By applying latent-level face masks, FAA isolates audio-driven animation effects to specific characters, enabling them to receive independent audio inputs in multi-character scenarios. This facilitates realistic multi-character dialogues without cross-interference.

Experimental Results and Baseline Comparison

The efficacy of HunyuanVideo-Avatar is demonstrated through extensive qualitative and quantitative assessments against state-of-the-art methods across different datasets, including CelebV-HQ and HDTF for portrait animations and a custom wild dataset for full-body scenarios. The model consistently outperforms others concerning video quality metrics such as IQA, ASE, FID, and FVD, as well as lip sync accuracy and emotional expression alignment.

In user studies, HunyuanVideo-Avatar received superior scores in Identity Preservation and Lip Synchronization compared to previous methods, further validating the effectiveness of the modules introduced. However, it noted some limitations inherent in the foundational HunyuanVideo model, particularly regarding motion naturalness compared to Omnihuman-1's enhanced features.

Implications and Future Directions

Practically, HunyuanVideo-Avatar promises significant improvements in creating realistic avatars for dynamic, multi-character animations, potentially enhancing applications in virtual entertainment, game design, and digital avatars synthesis. The paper suggests future work should focus on directly inferring emotions from audio, improving inference speed to meet real-time requirements, and exploring interactive human animations capable of real-time feedback.

Theoretically, HunyuanVideo-Avatar contributes to advancing understanding in audio-driven animation, offering a more nuanced approach to emotion representation and multi-character interaction. The insights gained from its novel modules could inform future developments in generative models, particularly those employing diffusion networks for complex scenario synthesis.

In conclusion, while addressing limitations such as dependence on reference images and computational demand, HunyuanVideo-Avatar establishes itself as a pivotal development in audio-driven human animation, setting a new benchmark for both single and multi-character scenarios. The promising results open the door to further research and practical deployments, bolstering the capabilities of AI-driven character animation technologies.