Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

End-to-End Speech-to-Text Translation: A Survey (2312.01053v2)

Published 2 Dec 2023 in cs.CL, cs.SD, and eess.AS

Abstract: Speech-to-text translation pertains to the task of converting speech signals in a language to text in another language. It finds its application in various domains, such as hands-free communication, dictation, video lecture transcription, and translation, to name a few. Automatic Speech Recognition (ASR), as well as Machine Translation(MT) models, play crucial roles in traditional ST translation, enabling the conversion of spoken language in its original form to written text and facilitating seamless cross-lingual communication. ASR recognizes spoken words, while MT translates the transcribed text into the target language. Such disintegrated models suffer from cascaded error propagation and high resource and training costs. As a result, researchers have been exploring end-to-end (E2E) models for ST translation. However, to our knowledge, there is no comprehensive review of existing works on E2E ST. The present survey, therefore, discusses the work in this direction. Our attempt has been to provide a comprehensive review of models employed, metrics, and datasets used for ST tasks, providing challenges and future research direction with new insights. We believe this review will be helpful to researchers working on various applications of ST models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (107)
  1. Lapicque’s introduction of the integrate-and-fire model neuron (1907). Brain Research Bulletin 50, 303–304.
  2. FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN, in: Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), Association for Computational Linguistics, Toronto, Canada (in-person and online). pp. 1–61.
  3. On the locality of attention in direct speech translation. arXiv preprint arXiv:2204.09028 .
  4. Tied multitask learning for neural speech translation. arXiv preprint arXiv:1802.06655 .
  5. An unsupervised probability model for speech-to-translation alignment of low-resource languages. arXiv preprint arXiv:1609.08139 .
  6. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv:2006.11477.
  7. A comparative study on end-to-end speech to text translation, in: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE. pp. 792–799.
  8. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments, in: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72.
  9. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation. arXiv preprint arXiv:1809.01431 .
  10. Towards speech-to-text translation without speech recognition. arXiv preprint arXiv:1702.03856 .
  11. Cascade versus direct speech translation: Do the differences still make a difference?, in: Annual Meeting of the Association for Computational Linguistics.
  12. End-to-end automatic speech translation of audiobooks, in: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 6224–6228.
  13. Listen and translate: A proof of concept for end-to-end speech-to-text translation. arXiv preprint arXiv:1612.01744 .
  14. Exploring continuous integrate-and-fire for adaptive simultaneous speech translation. ArXiv abs/2204.09595.
  15. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909 .
  16. Direct simultaneous speech-to-text translation assisted by synchronized streaming asr, in: Findings.
  17. Thinking slow about latency evaluation for simultaneous machine translation. ArXiv abs/1906.00048.
  18. Kosp2e: Korean speech to english translation corpus. arXiv preprint arXiv:2107.02875 .
  19. Stylekqc: A style-variant paraphrase corpus for korean questions and commands. arXiv preprint arXiv:2103.13439 .
  20. Learning a similarity metric discriminatively, with application to face verification, in: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), IEEE. pp. 539–546.
  21. The chime corpus: a resource and a challenge for computational hearing in multisource environments, in: Eleventh annual conference of the international speech communication association.
  22. The fisher corpus: A resource for the next generations of speech-to-text., in: LREC, pp. 69–71.
  23. The miami corpus: Documentation file. Bangortalk, bangortalk. org. uk/docs/Miami_doc. pdf .
  24. Must-c: a multilingual speech translation corpus, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics. pp. 2012–2017.
  25. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics, in: Proceedings of the second international conference on Human Language Technology Research, pp. 138–145.
  26. Cif: Continuous integrate-and-fire for end-to-end speech recognition. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 6079–6083.
  27. Listen, understand and translate: Triple supervision decouples end-to-end speech-to-text translation, in: AAAI Conference on Artificial Intelligence.
  28. Learning when to translate for streaming speech. arXiv preprint arXiv:2109.07368 .
  29. An attentional model for speech translation without transcription, in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 949–959.
  30. Stemm: Self-learning with speech-text manifold mixup for speech translation. arXiv preprint arXiv:2203.10426 .
  31. Microsoft speech language translation (mslt) corpus: The iwslt 2016 release for english, french and german, in: Proceedings of the 13th International Conference on Spoken Language Translation.
  32. Simultaneous translation of lectures and speeches. Machine Translation 21, 209–252.
  33. On knowledge distillation for direct speech translation. ArXiv abs/2012.04964.
  34. End-to-end speech translation with pre-trained models and adapters: Upc at iwslt 2021. ArXiv abs/2105.04512.
  35. One-to-many multilingual end-to-end speech translation. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , 585–592.
  36. A very low resource language speech corpus for computational language documentation experiments. arXiv preprint arXiv:1710.03501 .
  37. Segmentation of input in simultaneous translation. Journal of Psycholinguistic Research 1, 127–140.
  38. Sequence transduction with recurrent neural networks. ArXiv abs/1211.3711.
  39. Don’t until the final verb wait: Reinforcement learning for simultaneous machine translation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Doha, Qatar. pp. 1342–1352. doi:10.3115/v1/D14-1140.
  40. Learning shared semantic space for speech-to-text translation, in: Findings.
  41. Distilling the knowledge in a neural network. arXiv:1503.02531.
  42. Parameter-efficient transfer learning for nlp, in: International Conference on Machine Learning, PMLR. pp. 2790–2799.
  43. An analysis of semantically-aligned speech-text embeddings, in: 2022 IEEE Spoken Language Technology Workshop (SLT), IEEE. pp. 747–754.
  44. Multilingual end-to-end speech translation. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , 570–577.
  45. Espnet-st: All-in-one speech translation toolkit. arXiv preprint arXiv:2004.10234 .
  46. Europarl-st: A multilingual corpus for speech translation of parliamentary debates, in: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 8229–8233.
  47. Leveraging weakly supervised data to improve end-to-end speech-to-text translation, in: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 7180–7184.
  48. Libri-light: A benchmark for asr with limited or no supervision. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 7669–7673.
  49. Cstnet: Contrastive speech translation network for self-supervised speech representation learning. arXiv preprint arXiv:2006.02814 .
  50. Joint ctc-attention based end-to-end speech recognition using multi-task learning, in: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE. pp. 4835–4839.
  51. Augmenting librispeech with french translations: A multimodal corpus for direct speech translation evaluation. arXiv preprint arXiv:1802.03142 .
  52. Pre-training for speech translation: Ctc meets optimal transport. ArXiv abs/2301.11716.
  53. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. CoRR abs/2011.00747. URL: https://arxiv.org/abs/2011.00747, arXiv:2011.00747.
  54. Lightweight adapter tuning for multilingual speech translation. ArXiv abs/2106.01463.
  55. Building a psychological ground truth dataset with empathy and theory-of-mind during the covid-19 pandemic, in: Proceedings of the Annual Meeting of the Cognitive Science Society.
  56. Multilingual speech translation from efficient finetuning of pretrained models, in: Annual Meeting of the Association for Computational Linguistics.
  57. Cross attention augmented transducer networks for simultaneous translation, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 39–55.
  58. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics 8, 726–742.
  59. End-to-end speech translation with knowledge distillation. arXiv preprint arXiv:1904.08075 .
  60. Bridging the modality gap for speech-to-text translation. arXiv preprint arXiv:2010.14920 .
  61. Stacl: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework, in: Annual Meeting of the Association for Computational Linguistics.
  62. Simuleval: An evaluation toolkit for simultaneous translation, in: Conference on Empirical Methods in Natural Language Processing.
  63. Gentle forced aligner. github. com/lowerquality/gentle .
  64. Optimizing segmentation strategies for simultaneous speech translation, in: Annual Meeting of the Association for Computational Linguistics.
  65. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038 .
  66. Waco: Word-aligned contrastive learning for speech translation. arXiv preprint arXiv:2212.09359 .
  67. Librispeech: an asr corpus based on public domain audio books, in: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE. pp. 5206–5210.
  68. Attention as a guide for simultaneous speech translation, in: Annual Meeting of the Association for Computational Linguistics.
  69. Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318.
  70. Kss dataset: Korean single speaker speech dataset.
  71. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning 11, 355–607.
  72. chrf: character n-gram f-score for automatic mt evaluation, in: Proceedings of the tenth workshop on statistical machine translation, pp. 392–395.
  73. Simulspeech: End-to-end simultaneous speech to text translation, in: Annual Meeting of the Association for Computational Linguistics.
  74. The multilingual tedx corpus for speech recognition and translation. arXiv preprint arXiv:2102.01757 .
  75. How2: a large-scale dataset for multimodal language understanding. arXiv preprint arXiv:1811.00347 .
  76. Prabhupadavani: A code-mixed speech translation data for 25 languages. arXiv preprint arXiv:2201.11391 .
  77. Evaluating speech translation systems: Applying score to transtac technologies, in: Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems, pp. 223–230.
  78. A study of translation edit rate with targeted human annotation, in: Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pp. 223–231.
  79. Attention-passing models for robust and data-efficient end-to-end speech translation. Transactions of the Association for Computational Linguistics 7, 313–325.
  80. Whitening sentence representations for better semantics and faster retrieval. arXiv:2103.15316.
  81. Unified speech-text pre-training for speech translation and recognition. arXiv preprint arXiv:2204.05409 .
  82. Cross-modal transfer learning for multilingual speech-to-text translation. ArXiv abs/2010.12829.
  83. Attention is all you need, in: NIPS.
  84. Simple and effective unsupervised speech translation. arXiv preprint arXiv:2210.10191 .
  85. Covost: A diverse multilingual speech-to-text translation corpus, in: International Conference on Language Resources and Evaluation.
  86. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation, in: Annual Meeting of the Association for Computational Linguistics.
  87. Fairseq s2t: Fast speech-to-text modeling with fairseq. arXiv preprint arXiv:2010.05171 .
  88. Covost 2 and massively multilingual speech-to-text translation. arXiv: Computation and Language .
  89. Large-scale self- and semi-supervised learning for speech translation, in: Interspeech.
  90. Curriculum pre-training for end-to-end speech translation. arXiv preprint arXiv:2004.10093 .
  91. Lamassu: Streaming language-agnostic multilingual speech recognition and translation using neural transducers. arXiv preprint arXiv:2211.02809 .
  92. Sequence-to-sequence models can directly translate foreign speech. arXiv preprint arXiv:1703.08581 .
  93. End-to-end speech translation for code switched speech. arXiv preprint arXiv:2204.05076 .
  94. Recent advances in direct speech-to-text translation. ArXiv abs/2306.11646.
  95. Large-scale streaming end-to-end speech translation with neural transducers. arXiv preprint arXiv:2204.05352 .
  96. Cmu’s iwslt 2023 simultaneous speech translation system, in: International Workshop on Spoken Language Translation.
  97. End-to-end speech translation via cross-modal progressive training. arXiv preprint arXiv:2104.10380 .
  98. Cross-modal contrastive learning for speech translation. arXiv preprint arXiv:2205.02444 .
  99. Gigast: A 10,000-hour pseudo speech translation corpus. arXiv preprint arXiv:2204.03939 .
  100. Realtrans: End-to-end simultaneous speech translation with convolutional weighted-shrinking transformer. ArXiv abs/2106.04833.
  101. Adatrans: Adapting with boundary-based shrinking for end-to-end speech translation. ArXiv abs/2212.08911.
  102. Open source toolkit for speech to text translation. Prague Bull. Math. Linguistics 111, 125–135.
  103. Revisiting end-to-end speech-to-text translation from scratch, in: International Conference on Machine Learning, PMLR. pp. 26193–26205.
  104. Improving speech translation by cross-modal multi-grained contrastive learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, 1075–1086.
  105. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 .
  106. Neurst: Neural speech translation toolkit. arXiv preprint arXiv:2012.10018 .
  107. M-adapter: Modality adaptation for end-to-end speech-to-text translation. arXiv preprint arXiv:2207.00952 .
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets