Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SAiD: Speech-driven Blendshape Facial Animation with Diffusion (2401.08655v2)

Published 25 Dec 2023 in cs.CV, cs.AI, cs.GR, cs.LG, and cs.MM

Abstract: Speech-driven 3D facial animation is challenging due to the scarcity of large-scale visual-audio datasets despite extensive research. Most prior works, typically focused on learning regression models on a small dataset using the method of least squares, encounter difficulties generating diverse lip movements from speech and require substantial effort in refining the generated outputs. To address these issues, we propose a speech-driven 3D facial animation with a diffusion model (SAiD), a lightweight Transformer-based U-Net with a cross-modality alignment bias between audio and visual to enhance lip synchronization. Moreover, we introduce BlendVOCA, a benchmark dataset of pairs of speech audio and parameters of a blendshape facial model, to address the scarcity of public resources. Our experimental results demonstrate that the proposed approach achieves comparable or superior performance in lip synchronization to baselines, ensures more diverse lip movements, and streamlines the animation editing process.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Adobe. Animated lip-syncing powered by adobe ai. 2020.
  2. CVXOPT: A python package for convex optimization. 2013.
  3. Apple. Apple developer documentation - ARSCNFaceGeometry. 2017a.
  4. Apple. Apple developer documentation - ARKit. 2017b.
  5. Apple. Apple developer documentation - ARFaceAnchor.BlendShapeLocation. 2017c.
  6. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460, 2020.
  7. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013.
  8. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713, 2020.
  9. Out of time: automated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017.
  10. Animated speech: Research progress and applications. In AVSP 2001-International Conference on Auditory-Visual Speech Processing, 2001.
  11. Capture, learning, and synthesis of 3d speaking styles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10101–10111, 2019.
  12. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  13. Wind: Wasserstein inception distance for evaluating generative adversarial network performance. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3182–3186. IEEE, 2020.
  14. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450–455, 1982.
  15. Jali: an animator-centric viseme model for expressive lip synchronization. ACM Transactions on graphics (TOG), 35(4):1–11, 2016.
  16. Jali-driven expressive facial animation and multilingual speech in cyberpunk 2077. In ACM SIGGRAPH 2020 Talks, New York, NY, USA, 2020. Association for Computing Machinery.
  17. Faceformer: Speech-driven 3d facial animation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770–18780, 2022.
  18. Cletus G Fisher. Confusions among visually perceived consonants. Journal of speech and hearing research, 11(4):796–804, 1968.
  19. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. In North American Chapter of the Association for Computational Linguistics, 2019.
  20. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021–2029, 2020.
  21. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  22. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
  23. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  24. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  25. Diff-tts: A denoising diffusion model for text-to-speech. arXiv preprint arXiv:2104.01409, 2021.
  26. Flame: Free-form language-based motion synthesis & editing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8255–8263, 2023.
  27. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  28. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761, 2020.
  29. Practice and theory of blendshape facial models. Eurographics (State of the Art Reports), 1(8):2, 2014.
  30. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI conference on artificial intelligence, pages 11020–11028, 2022.
  31. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  32. Repaint: Inpainting using denoising diffusion probabilistic models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11451–11461, 2022.
  33. Meta. Tech note: Enhancing oculus lipsync with deep learning. 2018.
  34. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830, 2011.
  35. Emotalk: Speech-driven emotional disentanglement for 3d face animation. arXiv preprint arXiv:2303.11089, 2023.
  36. Speech-driven 3d facial animation with implicit emotional awareness: A deep learning approach. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 80–88, 2017.
  37. End-to-end learning for 3d facial animation from speech. In Proceedings of the 20th ACM International Conference on Multimodal Interaction, pages 361–365, 2018.
  38. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
  39. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  40. Audio-and gaze-driven facial animation of codec avatars. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 41–50, 2021a.
  41. Meshtalk: 3d face animation from speech using cross-modality disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1173–1182, 2021b.
  42. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  43. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In Proceedings of the IEEE international conference on computer vision workshops, pages 397–403, 2013.
  44. Palette: Image-to-image diffusion models. ACM SIGGRAPH 2022 Conference Proceedings, 2021.
  45. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022a.
  46. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713–4726, 2022b.
  47. Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01418, 2023.
  48. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  49. Denoising diffusion implicit models. ArXiv, abs/2010.02502, 2020.
  50. Facediffuser: Speech-driven 3d facial animation synthesis using diffusion. arXiv preprint arXiv:2309.11306, 2023.
  51. Deformation transfer for triangle meshes. ACM Transactions on graphics (TOG), 23(3):399–405, 2004.
  52. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
  53. A deep learning approach for generalized speech animation. ACM Transactions On Graphics (TOG), 36(4):1–11, 2017.
  54. Dynamic units of visual speech. In Proceedings of the 11th ACM SIGGRAPH/Eurographics conference on Computer Animation, pages 275–284, 2012.
  55. Human motion diffusion model. In The Eleventh International Conference on Learning Representations, 2023.
  56. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 448–458, 2023.
  57. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  58. Codetalker: Speech-driven 3d facial animation with discrete motion prior. arXiv preprint arXiv:2301.02379, 2023.
  59. A practical and configurable lip sync method for games. In Proceedings of Motion on Games, pages 131–140. 2013.
  60. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39(6):1–16, 2020.
  61. Physdiff: Physics-guided human motion diffusion model. arXiv preprint arXiv:2212.02500, 2022.
  62. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022.
  63. Visemenet: Audio-driven animator-centric speech animation. ACM Transactions on Graphics (TOG), 37(4):1–10, 2018.
  64. Taming diffusion models for audio-driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10544–10553, 2023.
Citations (4)
View on Semantic Scholar

Summary

  • The paper introduces a diffusion-based approach that overcomes regression limitations by generating diverse, synchronized 3D facial animations from speech.
  • It presents BlendVOCA, a novel benchmark dataset with high-quality speech-blendshape pairs for rigorous evaluation of animation models.
  • Extensive experiments show that SAiD outperforms traditional methods in lip synchronization and animation editing, benefiting VR, gaming, and film.

Insights into "SAiD: Speech-driven Blendshape Facial Animation with Diffusion"

The paper "SAiD: Speech-driven Blendshape Facial Animation with Diffusion" presents a novel approach to generating 3D facial animations from speech. The suggested method, SAiD, integrates diffusion models to overcome limitations plaguing conventional regression-based methods, such as capturing the many-to-one nature of speech to lip synchronization and ensuring diverse, continuous lip movements. Here, the paper provides both a theoretical foundation along with a practical implementation that addresses the scarcity of datasets through the introduction of a novel benchmark dataset, BlendVOCA.

Key Contributions and Methods

  1. BlendVOCA Dataset: The authors introduce BlendVOCA, a benchmark composed of high-quality speech-blendshape pairs. This dataset allows for a direct evaluation of blendshape and vertex-based facial animation models. BlendVOCA was carefully constructed using deformation transfer techniques to obtain blendshapes and coefficients for various speakers, thereby addressing dataset scarcity.
  2. Diffusion Model Utilization: SAiD employs a diffusion-based method, representing a departure from traditional least squares regression models. Diffusion models, known for generating high-quality samples, allow for the subsequent generation and editing of facial animations in a consistent manner. The model leverages a lightweight Transformer-based U-Net architecture, designed to predict blendshape coefficients conditioned on audio input.
  3. Alignment Bias for Lip Syncing: To achieve tight synchronization between audio and visual outputs, an alignment bias is implemented in the cross-modal attention architecture. This mechanism biases attention towards temporally adjacent audio frames, enhancing synchronization.
  4. Performance Evaluation: Extensive experiments demonstrate that SAiD achieves superior results in synchronizing lip movements with speech while offering diverse outputs. In terms of objective metrics like AV offset/confidence and FD, SAiD often outperforms existing frameworks.
  5. Facilitating Animation Editing: A significant contribution of this work is its ability to facilitate animation editing and interpolation efficiently. Using SAiD, users can edit portions of facial animation without detracting from the overall temporal coherence, further underscoring the flexibility of diffusion models over regression-based approaches.

Implications and Future Directions

The development of SAiD opens up several new possibilities in the field of speech-driven facial animation. The diffusion model paradigm allows for greater flexibility in generating and editing animations, which could be beneficial for applications in virtual reality, video game development, and film production. Furthermore, SAiD's advantages in producing realistic and well-synchronized animations suggest potential in enhancing human-virtual character interaction.

Looking ahead, the integration of global attention mechanisms could further add to the model's ability to synthesize contextual and coherent animations. There is also potential to explore transfer learning approaches to extend SAiD's capabilities across different languages and dialects, further refining the animation's expressive abilities to match diverse spoken inputs.

Overall, the contribution of this work is significant in not only advancing the technical capability of facial animation but also in providing a valuable dataset that can spur further research in the domain. The combination of advanced neural techniques and comprehensive evaluation underscores this paper's role in progressing the state-of-the-art in AI-driven animation.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - yunik1004/SAiD: SAiD: Blendshape-based Audio-Driven Speech Animation with Diffusion (108 stars)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets