End to end Hindi to English speech conversion using Bark, mBART and a finetuned XLSR Wav2Vec2
Abstract: Speech has long been a barrier to effective communication and connection, persisting as a challenge in our increasingly interconnected world. This research paper introduces a transformative solution to this persistent obstacle an end-to-end speech conversion framework tailored for Hindi-to-English translation, culminating in the synthesis of English audio. By integrating cutting-edge technologies such as XLSR Wav2Vec2 for automatic speech recognition (ASR), mBART for neural machine translation (NMT), and a Text-to-Speech (TTS) synthesis component, this framework offers a unified and seamless approach to cross-lingual communication. We delve into the intricate details of each component, elucidating their individual contributions and exploring the synergies that enable a fluid transition from spoken Hindi to synthesized English audio.
- “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale” In Proc. Interspeech 2022, 2022, pp. 2278–2282 DOI: 10.21437/Interspeech.2022-143
- “Robust Speech Recognition via Large-Scale Weak Supervision”, 2022 arXiv:2212.04356 [eess.AS]
- , https://github.com/kaldi-asr/kaldi
- “Multilingual Denoising Pre-training for Neural Machine Translation” In Transactions of the Association for Computational Linguistics 8, 2020, pp. 726–742 DOI: 10.1162/tacl˙a˙00343
- , https://huggingface.co/docs/transformers/model_doc/marian
- “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, 2019
- James Betker “TorToiSe text-to-speech”, 2022 URL: https://github.com/neonbjb/tortoise-tts
- , https://github.com/suno-ai/bark
- “Tacotron: Towards End-to-End Speech Synthesis”, 2017 arXiv:1703.10135 [cs.CL]
- “Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions” In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 4779–4783 DOI: 10.1109/ICASSP.2018.8461368
- “FastSpeech: Fast, Robust and Controllable Text to Speech” In Advances in Neural Information Processing Systems 32 Curran Associates, Inc., 2019 URL: https://proceedings.neurips.cc/paper_files/paper/2019/file/f63f65b503e22cb970527f23c9ad7db1-Paper.pdf
- “FastSpeech 2: Fast and High-Quality End-to-End Text to Speech”, 2022 arXiv:2006.04558 [eess.AS]
- “Common Voice: A Massively-Multilingual Speech Corpus”, 2020 arXiv:1912.06670 [cs.CL]
- , https://github.com/serp-ai/bark-with-voice-clone
- “Low Resource Multi-ASR Speech Command Recognition” In 2022 Moratuwa Engineering Research Conference (MERCon), 2022, pp. 1–6 DOI: 10.1109/MERCon55799.2022.9906230
- “Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages”, 2021 arXiv:2012.12121 [cs.CL]
- “Domain and Language Adaptation Using Heterogeneous Datasets for Wav2vec2.0-Based Speech Recognition of Low-Resource Language” In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5 DOI: 10.1109/ICASSP49357.2023.10095644
- H.A.Z.Sameen Shahgir, Khondker Salman Sayeed and Tanjeem Azwad Zaman “Applying wav2vec2 for Speech Recognition on Bengali Common Voices Dataset”, 2022 arXiv:2209.06581 [eess.AS]
- “A Study on Fine-Tuning wav2vec2.0 Model for the Task of Mispronunciation Detection and Diagnosis” In Proc. Interspeech 2021, 2021, pp. 4448–4452 DOI: 10.21437/Interspeech.2021-1344
- “Exploring Wav2vec 2.0 Fine Tuning for Improved Speech Emotion Recognition” In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5 DOI: 10.1109/ICASSP49357.2023.10095036
- T. Gopalakrishnan, Syed Ayaz Imam and Archit Aggarwal “Fine Tuning and Comparing Tacotron 2, Deep Voice 3, and FastSpeech 2 TTS Models in a Low Resource Environment” In 2022 IEEE International Conference on Data Science and Information System (ICDSIS), 2022, pp. 1–6 DOI: 10.1109/ICDSIS55133.2022.9915932
- Leslie N. Smith “A disciplined approach to neural network hyper-parameters: Part 1 – learning rate, batch size, momentum, and weight decay”, 2018 arXiv:1803.09820 [cs.LG]
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
