Papers
Topics
Authors
Recent
Search
2000 character limit reached

Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units

Published 19 Dec 2022 in cs.SD, cs.CL, cs.LG, and eess.AS | (2212.09730v2)

Abstract: We introduce DISSC, a novel, lightweight method that converts the rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner. Unlike DISSC, most voice conversion (VC) methods focus primarily on timbre, and ignore people's unique speaking style (prosody). The proposed approach uses a pretrained, self-supervised model for encoding speech to discrete units, which makes it simple, effective, and fast to train. All conversion modules are only trained on reconstruction like tasks, thus suitable for any-to-many VC with no paired data. We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate that DISSC significantly outperforms the evaluated baselines. Code and samples are available at https://pages.cs.huji.ac.il/adiyoss-lab/dissc/.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Audiolm: a language modeling approach to audio generation. arXiv preprint arXiv:2209.03143.
  2. A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units. arXiv preprint arXiv:2211.06535.
  3. Meiying Chen and Zhiyao Duan. 2022. Controlvc: Zero-shot voice conversion with time-varying controls on pitch and rhythm. arXiv preprint arXiv:2209.11866.
  4. Again-vc: A one-shot voice conversion using activation guidance and adaptive instance normalization. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5954–5958. IEEE.
  5. One-shot voice conversion by separating speaker and content representations with instance normalization. arXiv preprint arXiv:1904.05742.
  6. ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In Interspeech 2020, pages 3830–3834. ISCA.
  7. On the robustness of self-supervised representations for spoken language modeling. arXiv preprint arXiv:2209.15483.
  8. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460.
  9. Any-to-one sequence-to-sequence voice conversion using self-supervised discrete speech representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5944–5948. IEEE.
  10. S3prl-vc: Open-source voice conversion framework with self-supervised speech representations. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6552–6556. IEEE.
  11. Alexander Kain and Michael W Macon. 1998. Spectral voice conversion for text-to-speech synthesis. In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP’98 (Cat. No. 98CH36181), volume 1, pages 285–288. IEEE.
  12. Stargan-vc: Non-parallel many-to-many voice conversion using star generative adversarial networks. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 266–273. IEEE.
  13. Takuhiro Kaneko and Hirokazu Kameoka. 2017. Parallel-data-free voice conversion using cycle-consistent adversarial networks. arXiv preprint arXiv:1711.11293.
  14. Kavita Kasi and Stephen A Zahorian. 2002. Yet another algorithm for pitch tracking. In 2002 ieee international conference on acoustics, speech, and signal processing, volume 1, pages I–361. IEEE.
  15. textless-lib: a library for textless spoken language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations, pages 1–9, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
  16. Text-free prosody-aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8666–8681, Dublin, Ireland. Association for Computational Linguistics.
  17. Assem-vc: Realistic voice conversion by assembling modern speech synthesis techniques. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6997–7001. IEEE.
  18. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022–17033.
  19. Textless speech emotion conversion using decomposed and discrete representations. arXiv preprint arXiv:2111.07402.
  20. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.15352.
  21. Audio language modeling using perceptually-guided discrete representations. arXiv preprint arXiv:2211.01223.
  22. Investigation into target speaking rate adaptation for voice conversion. arXiv preprint arXiv:2209.01978.
  23. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336–1354.
  24. Direct speech-to-speech translation with discrete units. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3327–3339, Dublin, Ireland. Association for Computational Linguistics.
  25. Textless speech-to-speech translation on real data. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 860–872, Seattle, United States. Association for Computational Linguistics.
  26. Voicemixer: Adversarial voice style mixup. In Advances in Neural Information Processing Systems, volume 34, pages 294–308. Curran Associates, Inc.
  27. Duration controllable voice conversion via phoneme-based information bottleneck. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:1173–1183.
  28. S2vc: a framework for any-to-any voice conversion with self-supervised pretrained representations. arXiv preprint arXiv:2104.02901.
  29. Fragmentvc: Any-to-any voice conversion by end-to-end extracting and fusing fine-grained voice fragments with attention. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5939–5943. IEEE.
  30. Any-to-many voice conversion with location-relative sequence-to-sequence modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1717–1728.
  31. Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. In Proc. Interspeech 2017, pages 498–502.
  32. Voice conversion in high-order eigen space using deep belief nets. In Interspeech, pages 369–372.
  33. Generative spoken dialogue language modeling. arXiv preprint arXiv:2203.16502.
  34. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE.
  35. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355.
  36. Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation. arXiv preprint arXiv:2204.02967.
  37. Scaling speech technology to 1,000+ languages. arXiv preprint arXiv:2305.13516.
  38. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pages 7836–7846. PMLR.
  39. Global prosody style transfer without text transcriptions. In International Conference on Machine Learning, pages 8650–8660. PMLR.
  40. Autovc: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pages 5210–5219. PMLR.
  41. Contentvec: An improved self-supervised speech representation by disentangling speakers. In International Conference on Machine Learning, pages 18003–18017. PMLR.
  42. Disentangling prosody representations with unsupervised speech reconstruction. arXiv preprint arXiv:2212.06972.
  43. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356.
  44. SpeechBrain: A general-purpose speech toolkit. ArXiv:2106.04624.
  45. A metric for distributions with applications to image databases. In Sixth international conference on computer vision (IEEE Cat. No. 98CH36271), pages 59–66. IEEE.
  46. Continuous probabilistic transform for voice conversion. IEEE Transactions on speech and audio processing, 6(2):131–142.
  47. Superseded-cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit.
  48. fairseq s^2: A scalable and integrable speech synthesis toolkit. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 143–152, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  49. Carl Earl Williams. 1965. The effects of selected factors on the aural identification of speakers. University of Illinois at Urbana-Champaign.
  50. Improving zero-shot voice style transfer via disentangled representation learning. In International Conference on Learning Representations.
  51. Emotional voice conversion: Theory, databases and esd. Speech Communication, 137:1–18.
Citations (9)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up