2000 character limit reached
Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initialization (2402.01692v1)
Published 23 Jan 2024 in cs.CL and cs.LG
Abstract: This paper presents an effective transfer learning framework for language adaptation in text-to-speech systems, with a focus on achieving language adaptation using minimal labeled and unlabeled data. While many works focus on reducing the usage of labeled data, very few consider minimizing the usage of unlabeled data. By utilizing self-supervised features in the pretraining stage, replacing the noisy portion of pseudo labels with these features during fine-tuning, and incorporating an embedding initialization trick, our method leverages more information from unlabeled data compared to conventional approaches. Experimental results show that our framework is able to synthesize intelligible speech in unseen languages with only 4 utterances of labeled data and 15 minutes of unlabeled data. Our methodology continues to surpass conventional techniques, even when a greater volume of data is accessible. These findings highlight the potential of our data-efficient language adaptation framework.
- “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” 2021.
- “Portaspeech: Portable and high-quality generative text-to-speech,” 2022.
- “Tacotron: Towards End-to-End Speech Synthesis,” Proc. Interspeech 2017, pp. 4006–4010, 2017.
- “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 4779–4783.
- “Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning,” in International Conference on Learning Representations, 2018.
- “Char2Wav: End-to-End Speech Synthesis,” in ICLR (Workshop), 2017.
- “Towards Universal Text-to-Speech.,” in INTERSPEECH, 2020, pp. 3171–3175.
- “Bytes are all you need: End-to-end multilingual speech recognition and synthesis with bytes,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5621–5625.
- “Almost unsupervised text to speech and automatic speech recognition,” in International Conference on Machine Learning. PMLR, 2019, pp. 5410–5419.
- “Unsupervised text-to-speech synthesis by unsupervised automatic speech recognition,” 2022.
- “Lrspeech: Extremely low-resource speech synthesis and recognition,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 2802–2812.
- “Bag of tricks for unsupervised text-to-speech,” in The Eleventh International Conference on Learning Representations, 2023.
- “Few Shot Cross-Lingual TTS Using Transferable Phoneme Embedding,” in Proc. Interspeech 2022, 2022, pp. 4566–4570.
- “Semi-supervised training for improving data efficiency in end-to-end speech synthesis,” 2018.
- “Unsupervised learning for sequence-to-sequence text-to-speech for low-resource languages,” 2020.
- “Transfer learning framework for low-resource text-to-speech using a large-scale unlabeled speech corpus,” in Interspeech 2022. sep 2022, ISCA.
- “End-to-End Text-to-Speech for Low-Resource Languages by Cross-Lingual Transfer Learning,” Proc. Interspeech 2019, pp. 2075–2079, 2019.
- “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 11, pp. 4037–4058, 2021.
- “Audio self-supervised learning: A survey,” Patterns, vol. 3, no. 12, pp. 100616, 2022.
- “Virtuoso: Massive multilingual speech-text joint semi-supervised learning for text-to-speech,” 2023.
- “Learning to speak from text: Zero-shot multilingual text-to-speech with unsupervised text pretraining,” 2023.
- “Unsupervised speech recognition,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, Eds. 2021, vol. 34, pp. 27826–27839, Curran Associates, Inc.
- “Towards end-to-end unsupervised speech recognition,” 2022.
- Alexander Gutkin, “Uniform Multilingual Multi-Speaker Acoustic Model for Statistical Parametric Speech Synthesis of Low-Resourced Languages,” Proc. Interspeech 2017, pp. 2183–2187, 2017.
- “Areal and Phylogenetic Features for Multilingual Speech Synthesis,” Proc. Interspeech 2017, pp. 2078–2082, 2017.
- “A Unified Phonological Representation of South Asian Languages for Multilingual Text-to-Speech,” in Proc. The 6th Intl. Workshop on Spoken Language Technologies for Under-Resourced Languages, pp. 80–84.
- “Cross-lingual Low Resource Speaker Adaptation Using Phonological Features,” arXiv preprint arXiv:2111.09075, 2021.
- Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet, Cambridge University Press, 1999.
- “Multilingual byte2speech models for scalable low-resource speech synthesis,” arXiv preprint arXiv:2103.03541, 2021.
- “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020.
- “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” 2021.
- “LibriTTS: A corpus derived from LibriSpeech for text-to-speech,” arXiv preprint arXiv:1904.02882, 2019.
- “Aishell-3: A multi-speaker mandarin tts corpus and the baselines,” arXiv preprint arXiv:2010.11567, 2020.
- “Css10: A collection of single speaker speech datasets for 10 languages,” arXiv preprint arXiv:1903.11269, 2019.
- “JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis,” arXiv preprint arXiv:1711.00354, 2017.
- “Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi.,” in Interspeech, 2017, vol. 2017, pp. 498–502.
- “Fastspeech 2: Fast and high-quality end-to-end text to speech,” arXiv preprint arXiv:2006.04558, 2020.
- “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
- “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” 2020.
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.