Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?

Abstract: Connecting a pre-trained speech encoder to a LLM is the standard architecture for building Speech LLMs. However, a structural misalignment exists between the encoder and the LLM. Unlike encoders based on automatic speech recognition, which often produce representations in separate language-specific spaces, LLMs operate within a unified language-agnostic space. A mechanism is required to align the encoder's language-specific representations with the LLM's shared space. We argue that speech translation provides a principled way to achieve this. Unlike monolingual transcription, translation requires the model to bridge different languages and learn language-agnostic representations. We experimentally evaluate the impact of incorporating translation objectives into speech encoder pre-training. Our results demonstrate that translation-enhanced pre-training improves cross-modal integration and leads to superior performance across downstream Speech LLM tasks.

• The paper demonstrates that bidirectional translation pre-training yields significant ASR improvements, reducing error rates across languages.
• The study compares different encoder configurations, evidencing that symmetric pre-training enhances semantic abstraction and cross-lingual transfer.
• Empirical results confirm that enhanced encoder representations boost classification and translation metrics without compromising acoustic sensitivity.

Translation-Enhanced Pre-training for Speech LLMs: Empirical Evaluation and Implications

Motivation and Problem Statement

Integrating a pre-trained speech encoder with a frozen LLM is now the dominant technique for direct audio input processing in Speech LLMs. However, a core technical challenge persists: the representational spaces of speech encoders trained with automatic speech recognition (ASR) or self-supervised learning (SSL) are disjoint from the high-level semantic embedding space of LLMs. ASR-pretrained encoders yield language-specific acoustic representations, while text-based LLMs operate in a language-agnostic conceptual space. Bridging this structural mismatch typically relies on a lightweight adaptor, whose effectiveness depends fundamentally on the encoder’s ability to extract semantics beyond language boundaries.

This paper posits that cross-lingual speech translation objectives, especially bidirectional translation (i.e., both $X \rightarrow \text{en}$ and $\text{en} \rightarrow X$), are uniquely suited to enforce language-agnostic semantic abstraction in the encoder, thus enhancing cross-modal integration with LLMs. The hypothesis further questions the adequacy of standard unidirectional Whisper-style training, which omits $\text{en} \rightarrow X$ translation, for extracting semantically unified representations from English speech.

Methodology

The authors design a controlled experimental protocol to isolate the contribution of speech encoder pre-training objectives. Three configurations are compared:

• ASR-Only: Multilingual transcription without translation, serving as the baseline.
• ASR + Unidirectional ST ($X \rightarrow \text{en}$): Standard Whisper-style, with transcription for all languages and translation from non-English to English.
• ASR + Bidirectional ST ($X \leftrightarrow \text{en}$): Extended paradigm including both $X \rightarrow \text{en}$ and $\text{en} \rightarrow X$ translation for all languages.

Pre-training employs a sequence-to-sequence architecture identical to Whisper, after which only the encoder is retained. Experiments are conducted on four languages (English, Japanese, Chinese, German), using a curated 130k-hour speech dataset augmented with synthetic parallel data generated by an LLM. The integrated Speech LLM architecture couples this frozen speech encoder with a Llama 3.2-1B or 3B LLM via a compact CNN+linear adaptor. All components except the adaptor are strictly frozen during downstream training, ensuring attribution of performance improvements solely to encoder representations.

Evaluation spans ASR, speech translation (ST) in both directions, intent classification, and emotion recognition, utilizing established benchmarks (FLEURS, CoVoST2, SLURP, MELD, Speech-MASSIVE). Metrics include WER/CER for ASR, BLEU for ST, and accuracy for classification.

Empirical Findings

Translation-enhanced pre-training configurations deliver consistent and substantial gains over ASR-only baselines across all downstream modalities. The following points summarize the key results:

• ASR Performance: Bidirectional translation pre-training reduces CER/WER for all languages. For example, Japanese CER drops from 29.2 (ASR-only) to 19.7 (bidirectional) in the 1B LLM, and similar gains are observed for other languages.
• Speech Translation: Inclusion of $\text{en} \rightarrow X$ direction in pre-training yields marked BLEU score increases for both seen and unseen languages, demonstrating robust cross-lingual transfer and superior exploitation of the LLM's language capabilities.
• Classification Tasks: Intent classification accuracy correlates with the pre-training directions used. English intent accuracy increases from 57.3 (ASR-only) to 64.5 (bidirectional); German from 57.9 to 66.3. Emotion recognition, depending predominantly on acoustic cues, remains unchanged, indicating semantic abstraction does not compromise acoustic fidelity.
• Unfreezing Encoder: Joint training further improves translation metrics, but the advantages of symmetric translation pre-training persist, demonstrating its impact is not attributable merely to downstream adaptation.
• Generalization: Performance improvements extend to language pairs unseen during encoder pre-training, confirming the encoder's ability to facilitate compositional cross-lingual semantic alignment when coupled to a frozen LLM.

Claims and Contrasts

The study makes empirically substantiated claims that contradict prevailing practice in Speech LLMs, where Whisper-based encoders are often constrained to unidirectional translation objectives. The paper asserts that:

• Bidirectional translation pre-training is uniquely effective in enforcing semantic abstraction and alignment between the encoder output and LLM input space.
• Omitting $\text{en} \rightarrow X$ translation undermines the semantic utility of the encoder for English speech, limiting cross-modal integration.
• The translation objective provides a direct and reliable path to language-agnostic representations, while mere scale or downstream adaptation does not induce equivalent abstraction.

Implications for Model Design and Future Directions

The practical consequence is clear: symmetric, bidirectional translation should be adopted as standard practice when pre-training speech encoders for Speech LLMs. This paradigm yields significant improvements in generative, cross-lingual, and spoken language understanding tasks with no degradation to acoustic-sensitive capabilities.

Theoretically, the results validate the necessity of explicit language-agnostic objectives during representation learning to overcome modality and language boundary mismatches. The findings motivate future investigations in several directions:

• Extending bidirectional translation objectives to broader language sets and modalities (e.g., code-switching, non-text modalities).
• Quantifying semantic abstraction via probing studies or zero-shot transfer beyond the tested scope.
• Exploring architectural innovations that directly integrate translation-based objectives into multimodal pre-training frameworks.

Conclusion

This work demonstrates that bidirectional, translation-enhanced pre-training is essential for speech encoders powering Speech LLMs. The empirical results confirm that such training directly improves downstream generative and classification performance, especially for English speech and cross-lingual tasks. The approach preserves acoustic sensitivity while deepening semantic abstraction, providing a robust foundation for future Speech LLM architectures and challenging the adequacy of standard Whisper-style unidirectional paradigms.