Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning

Abstract: Multimodal LLMs (MLLMs) have achieved significant success in Speech-to-Text Translation (S2TT) tasks. While most existing research has focused on English-centric translation directions, the exploration of many-to-many translation is still limited by the scarcity of parallel data. To address this, we propose a three-stage curriculum learning strategy that leverages the machine translation capabilities of LLMs and adapts them to S2TT tasks, enabling effective learning in low-resource settings. We trained MLLMs with varying parameter sizes (3B, 7B, and 32B) and evaluated the proposed strategy using the FLEURS and CoVoST-2 datasets. Experimental results show that the proposed strategy achieves state-of-the-art average performance in $15\times14$ language pairs, requiring fewer than 10 hours of speech data per language to achieve competitive results. The source code and models are released at https://github.com/yxduir/LLM-SRT.

• The paper presents CoT-ST, a novel model that integrates multimodal chain-of-thought reasoning to improve speech translation accuracy.
• It employs a three-stage curriculum learning strategy (ASR, MMT, SRT) to systematically activate LLM reasoning and reduce error propagation.
• Experimental results on CoVoST-2 and MuST-C datasets demonstrate significant BLEU score improvements, especially for Japanese, Chinese, and en-zh translations.

Enhancing LLM-based Speech Translation with Multimodal Chain-of-Thought: A Review of CoT-ST

The paper "CoT-ST: Enhancing LLM-based Speech Translation with Multimodal Chain-of-Thought" by Du et al. presents a speech translation model leveraging LLMs and introducing a novel multimodal chain-of-thought (CoT) approach. The method is validated on the CoVoST-2 and MuST-C datasets, demonstrating superior performance over previous state-of-the-art (SOTA) methods.

Introduction

Speech translation (ST) aims to convert speech from a source language into text in a target language. Traditional methods predominantly employ a cascade system integrating Automatic Speech Recognition (ASR) followed by Machine Translation (MT). While effective, these methods are susceptible to error propagation. Recently, end-to-end ST methods have shown to be advantageous by reducing such error propagation. This paper posits that existing Speech LLMs (SLMs) have not fully utilized their inherent reasoning capabilities, thus proposing a multimodal CoT framework specifically designed to enhance these capabilities.

Methodology

The core proposition of this paper is the CoT-ST model, which integrates multimodal CoT reasoning to decompose the speech translation task into sequential steps of speech recognition and translation. This framework is trained through a novel three-stage curriculum learning process, consisting of ASR, Multimodal Machine Translation (MMT), and speech recognition and translation (SRT) tasks. This structured training approach aims to progressively activate the CoT reasoning capabilities of the model.

Model Architecture

The CoT-ST model architecture comprises three main components: a frozen speech encoder, a Q-Former projection module, and a frozen LLM. The speech encoder processes the raw audio input into high-dimensional features, which are compressed and adjusted in dimension by the Q-Former before being fed into the LLM for final textual output generation.

Training Framework

The curriculum learning approach enhances the CoT-ST by sequentially training the model on tasks of increasing complexity:

1. ASR Task: Establishes the multimodal alignment and forms the basis for subsequent tasks by training the model to transcribe speech accurately.
2. MMT Task: Strengthens cross-lingual capabilities by training the model to generate both the transcription and the target language translation from the same input.
3. SRT Task: Activates full CoT reasoning by training the model to output both transcription and translation solely from the audio input.

Experimental Results

The performance of CoT-ST was evaluated against established baselines and recent models on the CoVoST-2 and MuST-C datasets. Key findings include:

• CoVoST-2: CoT-ST achieved superior BLEU scores compared to previous SOTA methods. Notable improvements were seen in translation tasks for Japanese and Chinese languages.
• MuST-C Zero-shot: Even in zero-shot settings, CoT-ST demonstrated robust performance, notably in the en-zh translation task, also setting new benchmarks in the field.

Discussion

The CoT-ST model's effectiveness is attributed to its novel multimodal CoT approach and its structured training paradigm. By breaking down complex translation tasks into manageable steps, CoT-ST leverages the intrinsic reasoning capabilities of LLMs more effectively than traditional or end-to-end methods without CoT.

Implications and Future Work

The promising results of CoT-ST suggest significant practical and theoretical implications. Practically, this approach has potential applications in multilingual and multimodal contexts, enhancing translation accuracy and contextual appropriateness across diverse languages and domains. Theoretically, it underscores the importance of reasoning in LLMs and opens up new avenues for research in multimodal CoT techniques.

Future research could focus on optimizing the CoT-ST model's architecture for better performance, exploring larger-scale LLMs to extend the capabilities of the approach, and further investigating the advantages of MMT tasks in various contexts. The ongoing exploration of multimodal CoT methodologies may significantly advance the field of speech translation and broaden the applicability of such models.

In summary, CoT-ST represents a substantial step forward in leveraging CoT reasoning within SLMs for speech translation tasks, setting new performance benchmarks and offering a robust framework for future research and applications in the field.