Papers
Topics
Authors
Recent
Search
2000 character limit reached

EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models

Published 21 Dec 2023 in cs.CL, cs.SD, and eess.AS | (2312.14069v2)

Abstract: We introduce EmphAssess, a prosodic benchmark designed to evaluate the capability of speech-to-speech models to encode and reproduce prosodic emphasis. We apply this to two tasks: speech resynthesis and speech-to-speech translation. In both cases, the benchmark evaluates the ability of the model to encode emphasis in the speech input and accurately reproduce it in the output, potentially across a change of speaker and language. As part of the evaluation pipeline, we introduce EmphaClass, a new model that classifies emphasis at the frame or word level.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Jonathan E Avila and Nigel G Ward. 2023. Towards cross-language prosody transfer for dialog. arXiv preprint arXiv:2307.04123.
  2. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296.
  3. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  4. Whisperx: Time-accurate speech transcription of long-form audio. arXiv preprint arXiv:2303.00747.
  5. Seamlessm4t-massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596.
  6. Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72.
  7. Audiolm: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
  8. The detection of emphatic words using acoustic and lexical features. In Ninth European Conference on Speech Communication and Technology.
  9. Prosody in the comprehension of spoken language: A literature review. Language and speech, 40(2):141–201.
  10. Delphine Dahan. 2015. Prosody and language comprehension. Wiley Interdisciplinary Reviews: Cognitive Science, 6(5):441–452.
  11. Probing phoneme, language and speaker information in unsupervised speech representations. In Interspeech 2022.
  12. Prosaudit, a prosodic benchmark for self-supervised speech models. In Interspeech 2023.
  13. Sequence-to-sequence models for emphasis speech translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(10):1873–1883.
  14. Preserving word-level emphasis in speech-to-speech translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(3):544–556.
  15. Enhancing expressivity transfer in textless speech-to-speech translation. arXiv preprint arXiv:2310.07279.
  16. Iqdubbing: Prosody modeling based on discrete self-supervised speech representation for expressive voice conversion. arXiv preprint arXiv:2201.00269.
  17. Lexical emphasis detection in spoken french using f-banks and neural networks. In International Conference on Statistical Language and Speech Processing, pages 241–249. Springer.
  18. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460.
  19. A holistic cascade system, benchmark, and human evaluation protocol for expressive speech-to-speech translation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
  20. Translatotron 2: High-quality direct speech-to-speech translation with voice preservation. In International Conference on Machine Learning, pages 10120–10134. PMLR.
  21. Direct speech-to-speech translation with a sequence-to-sequence model. arXiv preprint arXiv:1904.06037.
  22. Text-free prosody-aware generative spoken language modeling. arXiv preprint arXiv:2109.03264.
  23. D Robert Ladd and Amalia Arvaniti. 2023. Prosodic prominence across languages. Annual Review of Linguistics, 9:171–193.
  24. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336–1354.
  25. Textless speech-to-speech translation on real data. arXiv preprint arXiv:2112.08352.
  26. On the utility of self-supervised models for prosody-related tasks. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 1104–1111. IEEE.
  27. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pages 498–502.
  28. Yoonsook Mo. 2008. Acoustic correlates of prosodic prominence for naiïve listeners of american english. In Annual Meeting of the Berkeley Linguistics Society, volume 34, pages 257–267.
  29. Self-supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal Processing.
  30. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. arXiv preprint arXiv:2308.05725.
  31. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250–266.
  32. Learning cross-lingual knowledge with multilingual blstm for emphasis detection with limited training data. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5615–5619. IEEE.
  33. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR.
  34. Improving language understanding by generative pre-training.
  35. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925.
  36. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728.
  37. Prosodic prominence and boundaries in sequence-to-sequence speech synthesis. arXiv preprint arXiv:2006.15967.
  38. Jacques Terken and Dik Hermes. 2000. The perception of prosodic prominence. In Prosody: Theory and experiment: Studies presented to Gösta Bruce, pages 89–127. Springer.
  39. A study on the effect of prosodic emphasis transfer on overall speech translation quality. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8396–8400. IEEE.
  40. Deep learning for prominence detection in children’s read speech. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8157–8161. IEEE.
  41. Ann Wennerstrom. 2001. The music of everyday speech: Prosody and discourse analysis. Oxford University Press.
  42. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  43. Emphasis detection for voice dialogue applications using multi-channel convolutional bidirectional long short-term memory network. In 2018 11th International Symposium on Chinese Spoken Language Processing (ISCSLP), pages 210–214. IEEE.
  44. Inferring emphasis for real voice data: an attentive multimodal neural network approach. In MultiMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5–8, 2020, Proceedings, Part II 26, pages 52–62. Springer.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 47 likes about this paper.

Sign Up for Free