Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis (2404.01647v1)

Published 2 Apr 2024 in cs.CV

Abstract: Achieving disentangled control over multiple facial motions and accommodating diverse input modalities greatly enhances the application and entertainment of the talking head generation. This necessitates a deep exploration of the decoupling space for facial features, ensuring that they a) operate independently without mutual interference and b) can be preserved to share with different modal input, both aspects often neglected in existing methods. To address this gap, this paper proposes a novel Efficient Disentanglement framework for Talking head generation (EDTalk). Our framework enables individual manipulation of mouth shape, head pose, and emotional expression, conditioned on video or audio inputs. Specifically, we employ three lightweight modules to decompose the facial dynamics into three distinct latent spaces representing mouth, pose, and expression, respectively. Each space is characterized by a set of learnable bases whose linear combinations define specific motions. To ensure independence and accelerate training, we enforce orthogonality among bases and devise an efficient training strategy to allocate motion responsibilities to each space without relying on external knowledge. The learned bases are then stored in corresponding banks, enabling shared visual priors with audio input. Furthermore, considering the properties of each space, we propose an Audio-to-Motion module for audio-driven talking head synthesis. Experiments are conducted to demonstrate the effectiveness of EDTalk. We recommend watching the project website: https://tanshuai0219.github.io/EDTalk/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (80)
  1. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 44(12):8717–8727, 2018.
  2. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pages 609–617, 2017.
  3. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187–194, 1999.
  4. Video rewrite: Driving visual speech with audio. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 715–722. 2023.
  5. What makes fake images detectable? understanding properties that generalize. Lecture Notes in Computer Science, 2020.
  6. Talking-head generation with rhythmic head motion. In European Conference on Computer Vision, pages 35–51. Springer, 2020.
  7. Lip movements generation at a glance. In Proceedings of the European Conference on Computer Vision (ECCV), pages 520–535, 2018.
  8. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7832–7841, 2019.
  9. Vast: Vivify your talking avatar via zero-shot expressive facial style transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2977–2987, 2023.
  10. Voxceleb2: Deep speaker recognition. arXiv preprint arXiv:1806.05622, 2018.
  11. Lip reading in the wild. In Computer Vision–ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 87–103. Springer, 2017.
  12. Out of time: automated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017.
  13. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20311–20322, 2022.
  14. Speech-driven facial animation using cascaded gans for learning of motion and texture. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, pages 408–424. Springer, 2020.
  15. Facial action coding system. Environmental Psychology & Nonverbal Behavior, 1978.
  16. Efficient emotional adaptation for audio-driven talking-head generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22634–22645, 2023.
  17. Emotionally enhanced talking face generation. In Proceedings of the 1st International Workshop on Multimedia Content Generation and Evaluation: New Methods and Practice, pages 81–90, 2023.
  18. Deepfake video detection using recurrent neural networks. Advanced Video and Signal Based Surveillance, 2018.
  19. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  20. Implicit identity representation conditioned memory compensation network for talking head video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23062–23072, 2023.
  21. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3397–3406, 2022.
  22. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
  23. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501–1510, 2017.
  24. Eamm: One-shot emotional talking face via audio-based emotion-aware motion model. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022.
  25. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14080–14089, 2021.
  26. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 694–711. Springer, 2016.
  27. Realistic one-shot mesh-based head avatars. In European Conference on Computer Vision, pages 345–362. Springer, 2022.
  28. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673, 2020.
  29. Emoberta: Speaker-aware emotion recognition in conversation with roberta. arXiv preprint arXiv:2108.12009, 2021.
  30. Ae-nerf: Audio enhanced neural radiance field for few shot talking head synthesis. arXiv preprint arXiv:2312.10921, 2023.
  31. Expressive talking head generation with granular audio-visual control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3387–3396, 2022.
  32. Semantic-aware implicit neural audio-driven video portrait generation. In European Conference on Computer Vision, pages 106–125. Springer, 2022.
  33. Talkclip: Talking head generation with text-guided expressive speaking styles. arXiv preprint arXiv:2304.00334, 2023.
  34. Styletalk: One-shot talking head generation with controllable speaking styles. arXiv preprint arXiv:2301.01081, 2023.
  35. Frame attention networks for facial expression recognition in videos. In 2019 IEEE international conference on image processing (ICIP), pages 3866–3870. IEEE, 2019.
  36. Expressive talking avatars. IEEE Transactions on Visualization and Computer Graphics, 2024.
  37. Emotional voice puppetry. IEEE Transactions on Visualization and Computer Graphics, 29(5):2527–2535, 2023.
  38. Dpe: Disentanglement of pose and expression for general video portrait editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 427–436, 2023.
  39. Synctalkface: Talking face generation with precise lip-syncing via audio-lip memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2062–2070, 2022.
  40. Ai-generated characters for supporting personalized learning and well-being. Nature Machine Intelligence, 3(12):1013–1022, 2021.
  41. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, pages 484–492, 2020.
  42. Pirenderer: Controllable portrait image generation via semantic neural rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13759–13768, 2021.
  43. Variational inference with normalizing flows. In International conference on machine learning, pages 1530–1538. PMLR, 2015.
  44. High-resolution image synthesis with latent diffusion models, 2021.
  45. Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https://github.com/mseitzer/pytorch-fid, August 2020. Version 0.3.0.
  46. Learning dynamic facial radiance fields for few-shot talking head synthesis. In European Conference on Computer Vision, pages 666–682. Springer, 2022.
  47. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1982–1991, 2023.
  48. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
  49. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  50. Emotion-controllable generalized talking face generation.
  51. Emotion-controllable generalized talking face generation. In International Joint Conference on Artificial Intelligence. IJCAI, 2021.
  52. Talking face generation by conditional recurrent adversarial network. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Aug 2019.
  53. Say anything with any style. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5088–5096, 2024.
  54. Emmn: Emotional motion memory network for audio-driven emotional talking face generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22146–22156, 2023.
  55. Flowvqtalker: High-quality emotional talking face generation through normalizing flow and quantization. arXiv preprint arXiv:2403.06375, 2024.
  56. Style2talker: High-resolution talking head generation with emotion style and art style. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5079–5087, 2024.
  57. Neural voice puppetry: Audio-driven facial reenactment. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pages 716–731. Springer, 2020.
  58. Progressive disentangled representation learning for fine-grained controllable talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17979–17989, 2023.
  59. Seeing what you said: Talking face generation guided by a lip reading expert. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14653–14662, 2023.
  60. Lipformer: High-fidelity and generalizable talking face generation with a pre-learned facial codebook. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13844–13853, 2023.
  61. Mead: A large-scale audio-visual dataset for emotional talking-face generation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI, pages 700–717. Springer, 2020.
  62. Efficient video portrait reenactment via grid-based codebook. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–9, 2023.
  63. Audio2head: Audio-driven one-shot talking-head generation with natural head motion. In International Joint Conference on Artificial Intelligence. IJCAI, 2021.
  64. One-shot talking face generation from single-speaker audio-visual correlation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2531–2539, 2022.
  65. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10039–10049, 2021.
  66. A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding. arXiv preprint arXiv:2111.02735, 2021.
  67. Latent image animator: Learning to animate images via latent space navigation. In International Conference on Learning Representations, 2021.
  68. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004.
  69. Face2face ρ𝜌\rhoitalic_ρ: Real-time high-resolution one-shot face reenactment. In European conference on computer vision, pages 55–71. Springer, 2022.
  70. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In European conference on computer vision, pages 85–101. Springer, 2022.
  71. Talking head generation with probabilistic audio-to-visual diffusion priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7645–7655, 2023.
  72. Few-shot adversarial learning of realistic neural talking head models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9459–9468, 2019.
  73. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  74. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8652–8661, 2023.
  75. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021.
  76. Identity-preserving talking face generation with landmark and appearance priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2023.
  77. Talking face generation by adversarially disentangled audio-visual representation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9299–9306, 2019.
  78. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4176–4186, 2021.
  79. Makelttalk: speaker-aware talking-head animation. ACM Transactions On Graphics (TOG), 39(6):1–15, 2020.
  80. Joint audio-visual deepfake detection. International Conference on Computer Vision, 2021.
Citations (10)
View on Semantic Scholar

Summary

  • The paper introduces EDTalk, an efficient framework for emotional talking head synthesis that disentangles facial components using novel orthogonal bases and latent spaces.
  • EDTalk achieves superior performance compared to state-of-the-art methods on standard datasets, showing high quality and synchronization while being computationally efficient.
  • The disentangled control and efficiency of EDTalk have significant implications for realistic emotional avatars in entertainment, VR, and digital creativity applications.

An Analysis of EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis

This paper showcases a novel approach to the synthesis of talking head animations, with a particular focus on emotional expression. The authors present EDTalk, a framework designed to facilitate efficient disentanglement of facial components such as mouth shape, head pose, and emotional expression. The proposed solution adeptly handles both video-driven and audio-driven inputs, allowing for nuanced facial animation that is consistent with conditioned emotions and motions.

Methodology Overview

At the core of EDTalk is an innovative framework that efficiently disentangles facial components into separate latent spaces using an autoencoder architecture. The system's key contribution lies in its use of orthogonal bases within component-aware latent spaces to avoid interference among different motions. Specifically, the method relies on component-aware latent navigation modules to decompose the global facial motion into discrete spaces and support both video and audio inputs.

The authors detail a two-fold efficient disentanglement strategy: the mouth-pose decoupling process and the expression decoupling process. The mouth-pose decoupling leverages cross-reconstruction and self-reconstruction to separate mouth and head dynamics effectively. This is augmented by an orthogonal constraint across bases, ensuring clear distinctions between components. Expression decoupling, on the other hand, incorporates complementary learning via a latent navigation module dedicated to emotional expression, facilitating precise control over facial expressions derived from context in audio inputs.

Furthermore, the authors exhibit EDTalk's prowess in audio-driven talking head synthesis, introducing modules for predicting weights for lip, pose, and expression spaces from audio inputs. This effort leverages audio encoding, normalizing flows for head pose, and semantic-driven expression generation based on textual and acoustic cues.

Evaluation and Results

The authors conducted extensive experiments, assessing EDTalk against several state-of-the-art methods. Through evaluations on datasets such as MEAD and HDTF, EDTalk demonstrates superior performance, particularly in metrics like PSNR, SSIM, and FID, while ensuring high synchronization confidence. The model's capability for probabilistic pose generation is especially notable, illustrating the framework's aptitude for recreating naturally diverse head motions.

Moreover, the paper presents comparative analyses with existing face reenactment methods, highlighting EDTalk's capacity to maintain identity integrity and realistic motion representation. Empirical validations further demonstrate the framework's efficiency in terms of computational resources, training time, and data requirements.

Implications and Future Directions

EDTalk presents significant implications for facial animation fields, offering a means to achieve realistic emotional expressions in virtual avatars while minimizing resource demands. The disentangled motion spaces allow for granular control over facial animations, with potential applications in entertainment, virtual reality, and digital creativity realms.

The approach is poised to influence future advancements in AI-driven animation techniques, encouraging efficient resource utilization while maintaining high-quality outputs. Future work could further explore the integration of higher resolution data, addressing limitations in output fidelity, and extend the framework's scope by modeling the interplay of head poses through emotive cues.

In conclusion, the paper presents a significant advancement in the field of talking head synthesis by explicitly addressing the intricacies of disentangling facial components and proposing an efficient, modular solution adaptable to multiple modalities.