CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior (2301.02379v2)

Abstract: Speech-driven 3D facial animation has been widely studied, yet there is still a gap to achieving realism and vividness due to the highly ill-posed nature and scarcity of audio-visual data. Existing works typically formulate the cross-modal mapping into a regression task, which suffers from the regression-to-mean problem leading to over-smoothed facial motions. In this paper, we propose to cast speech-driven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty. The codebook is learned by self-reconstruction over real facial motions and thus embedded with realistic facial motion priors. Over the discrete motion space, a temporal autoregressive model is employed to sequentially synthesize facial motions from the input speech signal, which guarantees lip-sync as well as plausible facial expressions. We demonstrate that our approach outperforms current state-of-the-art methods both qualitatively and quantitatively. Also, a user study further justifies our superiority in perceptual quality.

• The paper introduces CodeTalker as a novel framework that uses a discrete motion codebook derived via VQ-VAE to synthesize 3D facial animations.
• It employs an autoregressive model to translate speech signals into motion codes, ensuring precise lip synchronization and natural facial expressions.
• The method outperforms state-of-the-art techniques on BIWI and VOCASET datasets, achieving lower lip vertex error and enhanced expression realism.

Overview of "CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior"

This paper addresses the challenging task of generating speech-driven 3D facial animations, where the goal is to produce realistic and vivid facial movements synchronized with an audio signal. Traditional approaches often suffer from over-smoothed outputs due to the regression-to-mean problem, and the complexities involved in audio-to-visual mappings result in ambiguity.

Key Contributions and Method

The authors propose "CodeTalker," a novel method that reframes the problem as a code query task within a discrete proxy space. The introduction of a learned discrete motion codebook, derived through self-reconstruction from real motion data, serves as the pivotal innovation in this work. This methodology is grounded in the use of a vector-quantized autoencoder (VQ-VAE), ensuring that the facial animations possess embedded realistic motion priors.

The CodeTalker architecture employs an autoregressive model to synthesize facial movements by processing the speech signal into a sequence of motion codes. By doing so, it reduces the uncertainty associated with traditional cross-modal mapping techniques. The temporal autoregressive nature of the model ensures accurate lip synchronization and the generation of natural facial expressions.

Experimental Results and Evaluation

The paper rigorously evaluates the performance of CodeTalker against state-of-the-art methods using the BIWI and VOCASET datasets. The method demonstrates superior quantitative results, particularly in terms of lip synchronization and motion realism. Specific metrics, such as the lip vertex error and upper-face dynamics deviation (FDD), are used to show that CodeTalker consistently outperforms its counterparts in achieving lower errors.

Qualitatively, the paper provides evidence of the enhanced expressiveness and accurate synchronization of the facial animations generated by CodeTalker. It also introduces the concept of style interpolation, allowing for the synthesis of novel speaking styles by combining learned style vectors.

Implications and Future Directions

The proposed framework has significant implications for applications in virtual reality, gaming, and film production, where high fidelity and nuanced facial animations are crucial. The discrete representation of motion priors offers robustness against the cross-modal ambiguity that plagues conventional methods, making CodeTalker a potentially valuable tool for industry practitioners.

Future research could explore the extension of this method to incorporate larger and more diverse datasets, which would further enhance the generalizability and realism of the synthesized animations. Moreover, integrating additional contextual information, such as emotional states or environmental factors, might further refine the animation quality and applicability to real-world scenarios.

Conclusion

CodeTalker represents a meaningful advancement in the domain of speech-driven 3D facial animation by leveraging discrete motion priors within an autoregressive framework. It successfully addresses the limitations of prior approaches, offering a more reliable and expressive solution for generating synchronized and visually appealing facial animations.