Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models (2404.00569v1)

Published 31 Mar 2024 in cs.SD, cs.CL, and eess.AS

Abstract: Neural Text-to-Speech (TTS) systems find broad applications in voice assistants, e-learning, and audiobook creation. The pursuit of modern models, like Diffusion Models (DMs), holds promise for achieving high-fidelity, real-time speech synthesis. Yet, the efficiency of multi-step sampling in Diffusion Models presents challenges. Efforts have been made to integrate GANs with DMs, speeding up inference by approximating denoising distributions, but this introduces issues with model convergence due to adversarial training. To overcome this, we introduce CM-TTS, a novel architecture grounded in consistency models (CMs). Drawing inspiration from continuous-time diffusion models, CM-TTS achieves top-quality speech synthesis in fewer steps without adversarial training or pre-trained model dependencies. We further design weighted samplers to incorporate different sampling positions into model training with dynamic probabilities, ensuring unbiased learning throughout the entire training process. We present a real-time mel-spectrogram generation consistency model, validated through comprehensive evaluations. Experimental results underscore CM-TTS's superiority over existing single-step speech synthesis systems, representing a significant advancement in the field.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Deep speech 2 : End-to-end speech recognition in english and mandarin. In Proceedings of ICML.
  2. Objective measure for estimating mean opinion score of synthesized speech.
  3. Wei Chu and Abeer Alwan. 2009. Reducing f0 frame error of f0 tracking algorithms under noisy conditions with an unvoiced/voiced classification frontend. In Proceedings of ICASSP.
  4. Glow-wavegan: Learning speech representations from gan-based variational auto-encoder for high fidelity flow-based speech synthesis. In Proceedings of Interspeech.
  5. Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. In Proceedings of NeurIPS.
  6. End-to-end adversarial text-to-speech. In Proceedings of ICLR.
  7. Automatic chemical design using A data-driven continuous representation of molecules. ACS central science.
  8. Bootstrap your own latent - A new approach to self-supervised learning. In Proceedings of NeurIPS.
  9. Denoising diffusion probabilistic models. In Proceedings of NeurIPS.
  10. Keith Ito and Linda Johnson. 2017. The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/.
  11. Diff-tts: A denoising diffusion model for text-to-speech. In Proceedings of Interspeech.
  12. Elucidating the design space of diffusion-based generative models. In Proceedings of NeurIPS.
  13. Guided-TTS: A diffusion model for text-to-speech via classifier guidance. In Proceedings of the ICML.
  14. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In Processing of NeurIPS.
  15. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proceedings of ICML.
  16. Guided-tts 2: A diffusion model for high-quality adaptive text-to-speech with untranscribed data. arXiv preprint arXiv:2205.15370.
  17. Flowavenet : A generative flow for raw audio. In Proceedings of ICML.
  18. Diederik P. Kingma and Max Welling. 2019. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning.
  19. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Proceedings of NeurIPS.
  20. Melgan: Generative adversarial networks for conditional waveform synthesis. In Proceedings of NeurIPS.
  21. Improved precision and recall metric for assessing generative models. In Processing of NeurIPS.
  22. Deep speaker: An end-to-end neural speaker embedding system. arXiv preprint arXiv:1705.02304.
  23. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of AAAI.
  24. Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans. arXiv preprint arXiv:2201.11972.
  25. A review of deep learning techniques for speech processing. Information Fusion.
  26. Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In Proceedings of ICML.
  27. Librispeech: An ASR corpus based on public domain audio books. In Proceedings of ICASSP.
  28. Deep voice 3: 2000-speaker neural text-to-speech. In Proceedings of ICLR.
  29. Grad-tts: A diffusion probabilistic model for text-to-speech. In Proceedings of ICML.
  30. Fastspeech 2: Fast and high-quality end-to-end text to speech. In Proceedings of ICLR.
  31. Fastspeech: Fast, robust and controllable text to speech. In Proceedings of NeurIPS.
  32. High-resolution image synthesis with latent diffusion models. In Proceedings of CVPR.
  33. Natural TTS synthesis by conditioning wavenet on MEL spectrogram predictions. In Proceedings of ICASSP.
  34. Rad-tts: Parallel flow-based tts with robust alignment learning and diverse synthesis. In Proceedings of ICML(Workshop).
  35. Consistency models. In Proceedings of ICML.
  36. Score-based generative modeling through stochastic differential equations. In Proceedings of ICLR.
  37. Integrating structure-based approaches in generative molecular design. Current Opinion in Structural Biology.
  38. Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis. In Proceedings of ICLR.
  39. Wavenet: A generative model for raw audio. In Proceedings of SSW.
  40. Neural discrete representation learning. In Proceedings of NeurIPS.
  41. Attention is all you need. In Processing of NeurIPS.
  42. The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. In Proceedings of COCOSDA.
  43. Tacotron: Towards end-to-end speech synthesis. In Proceedings of Interspeech.
  44. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
  45. Comospeech: One-step speech and singing voice synthesis via consistency model. In Proceedings of MM.
  46. Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation. In Proceedings of NeurIPS.
  47. Libritts: A corpus derived from librispeech for text-to-speech. In Proceedings of Interspeech.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Xiang Li (1003 papers)
  2. Fan Bu (23 papers)
  3. Ambuj Mehrish (15 papers)
  4. Yingting Li (8 papers)
  5. Jiale Han (14 papers)
  6. Bo Cheng (51 papers)
  7. Soujanya Poria (138 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now