Transcription and translation of videos using fine-tuned XLSR Wav2Vec2 on custom dataset and mBART
Abstract: This research addresses the challenge of training an ASR model for personalized voices with minimal data. Utilizing just 14 minutes of custom audio from a YouTube video, we employ Retrieval-Based Voice Conversion (RVC) to create a custom Common Voice 16.0 corpus. Subsequently, a Cross-lingual Self-supervised Representations (XLSR) Wav2Vec2 model is fine-tuned on this dataset. The developed web-based GUI efficiently transcribes and translates input Hindi videos. By integrating XLSR Wav2Vec2 and mBART, the system aligns the translated text with the video timeline, delivering an accessible solution for multilingual video content transcription and translation for personalized voice.
- , https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI
- “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale” In Proc. Interspeech 2022, 2022, pp. 2278–2282 DOI: 10.21437/Interspeech.2022-143
- “Multilingual Denoising Pre-training for Neural Machine Translation”, 2020 arXiv:2001.08210 [cs.CL]
- “Common Voice: A Massively-Multilingual Speech Corpus”, 2020 arXiv:1912.06670 [cs.CL]
- , https://github.com/devilismyfriend/ozen-toolkit
- “End-to-end speaker segmentation for overlap-aware resegmentation” In Proc. Interspeech 2021, 2021
- “wav2vec: Unsupervised Pre-training for Speech Recognition”, 2019 arXiv:1904.05862 [cs.CL]
- “Librispeech: An ASR corpus based on public domain audio books” In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210 DOI: 10.1109/ICASSP.2015.7178964
- “Low Resource Multi-ASR Speech Command Recognition” In 2022 Moratuwa Engineering Research Conference (MERCon), 2022, pp. 1–6 DOI: 10.1109/MERCon55799.2022.9906230
- “Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages”, 2021 arXiv:2012.12121 [cs.CL]
- “Domain and Language Adaptation Using Heterogeneous Datasets for Wav2vec2.0-Based Speech Recognition of Low-Resource Language” In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5 DOI: 10.1109/ICASSP49357.2023.10095644
- H.A.Z.Sameen Shahgir, Khondker Salman Sayeed and Tanjeem Azwad Zaman “Applying wav2vec2 for Speech Recognition on Bengali Common Voices Dataset”, 2022 arXiv:2209.06581 [eess.AS]
- “A Study on Fine-Tuning wav2vec2.0 Model for the Task of Mispronunciation Detection and Diagnosis” In Proc. Interspeech 2021, 2021, pp. 4448–4452 DOI: 10.21437/Interspeech.2021-1344
- “Exploring Wav2vec 2.0 Fine Tuning for Improved Speech Emotion Recognition” In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5 DOI: 10.1109/ICASSP49357.2023.10095036
- Leslie N. Smith “A disciplined approach to neural network hyper-parameters: Part 1 – learning rate, batch size, momentum, and weight decay”, 2018 arXiv:1803.09820 [cs.LG]
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
