MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement (2104.08223v2)

Abstract: This paper presents a generic method for generating full facial 3D animation from speech. Existing approaches to audio-driven facial animation exhibit uncanny or static upper face animation, fail to produce accurate and plausible co-articulation or rely on person-specific models that limit their scalability. To improve upon existing models, we propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face. At the core of our approach is a categorical latent space for facial animation that disentangles audio-correlated and audio-uncorrelated information based on a novel cross-modality loss. Our approach ensures highly accurate lip motion, while also synthesizing plausible animation of the parts of the face that are uncorrelated to the audio signal, such as eye blinks and eye brow motion. We demonstrate that our approach outperforms several baselines and obtains state-of-the-art quality both qualitatively and quantitatively. A perceptual user study demonstrates that our approach is deemed more realistic than the current state-of-the-art in over 75% of cases. We recommend watching the supplemental video before reading the paper: https://github.com/facebookresearch/meshtalk

• The paper introduces a novel cross-modality loss to disentangle audio-correlated lip movements from independent facial expressions.
• The method employs an autoregressive sampling strategy conditioned on speech, achieving superior lip synchronization with over 75% user preference.
• The approach offers scalable, person-independent facial animation applicable to virtual reality, telepresence, and other interactive domains.

MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement

The paper "MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement" presents a sophisticated method for generating full 3D facial animations driven by speech input. Unlike previous approaches, this method successfully addresses the challenge of achieving realistic upper face animations alongside accurate lip synchronization, without being constrained by person-specific models.

Methodology Overview

The core innovation lies in the introduction of a categorical latent space for facial animation, which disentangles audio-correlated components, such as lip movements, from audio-uncorrelated facial features, like eye blinks and eyebrow movements. This separation is achieved through a novel cross-modality loss, ensuring the model accurately reflects speech-driven dynamics while integrating realistic facial expressions that are not directly tied to audio cues.

The authors employ an autoregressive sampling strategy over this latent space, conditioned on audio input. This design enables the synthesis of facial animations that are both highly accurate and natural, even when applied to identities unseen during training.

Experimental Evaluation

The proposed method demonstrates a significant improvement over previous baselines. In quantitative terms, the approach yields superior lip synchronization compared to existing models like VOCA. The perceptual evaluations conducted in the user paper highlight that MeshTalk obtains greater user preference over alternative methods, achieving favorable rankings in over 75% of cases for both full-face animations and lip sync accuracy.

Implications and Future Directions

The successful disentanglement of audio-correlated and uncorrelated information in the latent space paves the way for more adaptable and scalable facial animation solutions across different applications. Practically, this could enhance facial animation in various domains such as virtual reality, telepresence, and entertainment, providing more lifelike and believable character interactions.

Theoretically, this advancement contributes to an improved understanding of cross-modality in machine learning, where audio-visual synchronization is paramount. Future research could explore reducing computational demands for real-time applications and improving robustness across varied facial features and conditions.

Overall, MeshTalk represents a significant step forward in the field of audio-driven 3D facial animation, offering both theoretical insights and practical improvements applicable to a wide range of digital environments.