Multilingual Conversational Speech Models

• MLC-SLMs are advanced neural architectures designed to perform multilingual ASR, diarization, and language understanding in dynamic conversational scenarios.
• They integrate parameter-efficient adaptation techniques like LoRA and MoE with context-aware LLM decoders to optimize transcription accuracy and reasoning across diverse languages.
• Robust training strategies using multilingual corpora and data augmentation reduce error rates and enhance performance in real-world, code-switching environments.

Multilingual Conversational Speech LLMs (MLC-SLMs) are advanced neural architectures designed to perform automatic speech recognition (ASR), spoken language understanding, and diarization across a broad range of languages and dialects in unconstrained, real-world conversational scenarios. These models emerged in response to challenges such as code-switching, multilingual speaker interaction, non-standard orthographies, and the scarcity of aligned data spanning multiple conversational domains. MLC-SLMs integrate large pretrained speech encoders with powerful LLMs, and employ parameter-efficient adaptation techniques and curriculum-based training to achieve robust transcription and reasoning performance in multilingual dialogues, as exemplified by leading submissions to the INTERSPEECH 2025 MLC-SLM Challenge (Mu et al., 17 Sep 2025).

1. Model Architectures and Core Components

Current state-of-the-art MLC-SLMs follow the “speech encoder → modality projector → LLM decoder” pattern. The most effective systems employ large pretrained encoders, such as Whisper-large-v3 and self-supervised models like MMS-1B or mHuBERT-147, to convert raw audio into high-dimensional representations (Mei et al., 4 Jul 2025, Xue et al., 24 Jul 2025, Li et al., 15 Aug 2025). These are either concatenated, fused using cross-attention (e.g., gated bidirectional fusion (Mei et al., 4 Jan 2026)), or merged adaptively by language (e.g., MoE-adapted connectors (Xue et al., 24 Jul 2025)).

Projection modules, typically lightweight MLPs or temporal pooling layers (linear–ReLU–linear, SwiGLU), compress and align speech features to the input space of a decoder-only LLM (Qwen3-8B, Gemma-3-12B, Babel-9B, Llama-3). Rather than full joint fine-tuning, most models employ Low-Rank Adaptation (LoRA) (Meng et al., 11 Jul 2025) on select matrices in both the encoder and decoder, realizing domain-specific adaptation with minimal trainable parameters. Table 1 summarizes key architectural elements:

Speech Encoder(s)	Projector	Decoder LLMs
Whisper-large-v3	2-layer MLP, Conv	Qwen2.5-7B, Gemma-3-12B, Babel-9B
mHuBERT/MMS-1B	Downsample, MoE	Qwen3-8B
Dual encoders	Cross-attn, MoE	Llama-3, EuroLLM 1.7B

Language-specific adaptation is realized through explicit prompts (“Please transcribe in {lang}”), per-language LoRA modules (MoE), or learned embeddings, which significantly decrease error rates (Mu et al., 17 Sep 2025, Peng et al., 16 Jun 2025, Mei et al., 4 Jul 2025).

2. Training Methodologies and Data Augmentation

MLC-SLMs rely on large-scale, multilingual conversational corpora (MLC-SLM challenge: ≈1,604 h in 14 varieties) complemented by external datasets (GigaSpeech, CommonVoice, LibriSpeech, TEDx, MSR-86K, YODAS2) to cover low-resource languages and accents (Mu et al., 17 Sep 2025, Xue et al., 24 Jul 2025). Training typically follows a modular, staged approach:

• Projector Pretraining: Freeze all but the projector to stabilize modality alignment (Mei et al., 4 Jul 2025, Nguyen et al., 16 Jun 2025).
• Encoder/Adapter Fine-tuning: LoRA or MoE applied to speech encoders for domain adaptation; optionally CTC loss combined with autoregressive objectives (Xue et al., 24 Jul 2025, Mei et al., 4 Jul 2025).
• Decoder Adaptation: LoRA modules enable efficient specialization of LLMs; curriculum learning and chain-of-thought SFT are utilized to inject reasoning and self-correction capabilities (Li et al., 16 Jun 2025).
• Iterative/Pseudo-labeling Strategies: Robustness is increased via iterative LoRA stages and pseudo-label augmentation, especially in low-resource regimes (Meng et al., 11 Jul 2025).

Data augmentation strategies such as SpecAugment, speed perturbation, and volume manipulation are universally reported to improve conversational robustness, especially for speaker and accent diversity (Mu et al., 17 Sep 2025, Polok et al., 16 Jun 2025).

3. Integration of Conversational Context and Reasoning

Recent submissions have emphasized the importance of conversational context for robust ASR. Bi-directional context integration (preceding and subsequent utterances), character-level contextual masking, and prompt-based conditioning allow LLM decoders to leverage both history and local future, yielding up to 18% relative error reductions (Peng et al., 16 Jun 2025, Li et al., 16 Jun 2025). Approaches include:

• Contrastive Learning: Embedding-level contrastive losses integrate dialogue context, leading to improved semantic grounding (Concina et al., 25 Jul 2025).
• Chain-of-Thought (CoT) Reasoning: Explicit > … tags encourage models to self-reflect and self-correct intermediate hypotheses, further refined with RLVR and verifiable reward signals (Li et al., 16 Jun 2025).
• Context-Sensitive Decoding: Two-stage pipelines (context-agnostic, then context-aware) allow for re-decoding with neighboring hypotheses, demonstrating consistent performance gains across languages (Peng et al., 16 Jun 2025).

Qualitative ablations demonstrate that context-aware systems outperform those trained on five times more data but without context (Peng et al., 16 Jun 2025).

4. Diarization and Multi-Speaker ASR

End-to-end diarization+ASR models and pipeline approaches have both been explored. Pipeline strategies based on Pyannote (DiariZen), S2SND, and WavLM feature extraction enable accurate segmentation and speaker identification (Polok et al., 16 Jun 2025, Lin et al., 13 Jul 2025). Diarization-conditioned Whisper variants (DiCoW) and gated fusion between speaker embeddings (ResNet-34) and semantic features (Whisper) facilitate time-constrained, multi-speaker transcription (Lin et al., 13 Jul 2025, Saengthong et al., 26 Jun 2025).

Advanced diarization-aware frameworks inject VAD and clustering signals directly into encoder layers, and leverage time-bound “triplets” to force speaker alignment during decoding (Lin et al., 13 Jul 2025).

Diarization Technique	ASR Integration	tcpWER/CER (Best)
DiariZen (EEND-based)	DiCoW-conditioned	16.75%
S2SND + gated fusion	Qwen2.5 w/ adapters	18.08%
CoT-aware LLM	Unified end-to-end	27.25%

5. Evaluation Metrics and Benchmarks

MLC-SLM performance is assessed via word error rate (WER), character error rate (CER), mix error rate (MER), and time-constrained permutation WER (tcpWER/tcpCER) for multi-speaker scenarios (Mu et al., 17 Sep 2025).

Top-ranking models on the INTERSPEECH 2025 challenge have achieved MER below 10% on 14 varieties (NTU Speechlab: 10.58%; TEA-ASLP: 9.60%; Triple X: 9.67%; Seewo: 11.57%; Transsion: 9.83%) (Mu et al., 17 Sep 2025, Xue et al., 24 Jul 2025, Gao et al., 23 Jul 2025, Li et al., 16 Jun 2025, Li et al., 15 Aug 2025). Diarization+ASR pipelines yield tcpWER ~16–18%, while unified Speech-LLM models remain at ~27% (Saengthong et al., 26 Jun 2025).

Robust multilingual performance necessitates explicit language prompts or adapters; ablation studies confirm 0.3–0.7% MER reductions per adaptation mechanism (Peng et al., 16 Jun 2025, Mu et al., 17 Sep 2025). Data augmentation and balancing—especially for low-resource languages—are decisive for top performance (Xue et al., 24 Jul 2025, Meng et al., 11 Jul 2025).

6. Multilingual and Code-Switching Extensions

MLC-SLMs are increasingly equipped to handle code-switching and textless scenarios. Synthetic code-switched data construction (sampling and concatenation of constituent language units) enables joint ASR and TTS in mixed-language contexts, enhancing LLM code-switched generation and recognition abilities (Xu et al., 2024). Cross-lingual interleaving of speech tokens (without textual supervision) creates shared hidden-state alignment and robust cross-lingual continuation, as reflected by improved semantic cloze-task accuracy in EN–FR benchmarks (Moumen et al., 1 Dec 2025).

The integration of multilingual mixture-of-experts (MoE) LoRA structures and per-language fusion modules further reduce language confusion in both recognition and generation (Xue et al., 24 Jul 2025).

7. Challenges, Limitations, and Future Directions

Despite significant progress, several open challenges remain:

• End-to-end Diarization: Unified speech-LLM models must match the error rates of pipeline systems; advances in context modeling and speaker-turn boundary detection are required (Mu et al., 17 Sep 2025, Saengthong et al., 26 Jun 2025).
• Scaling and Low-Resource Generalization: Training larger LLM backbones (Gemma3, DeepSeek) or extending capacity via cross-lingual interleaving and pseudo-labeling strategies remains ongoing (Meng et al., 11 Jul 2025, Moumen et al., 1 Dec 2025).
• Code-Switching and Real-Time Streaming: Real-world dialogue phenomena—rapid language changes, multi-speaker overlap, non-causal inference—necessitate robust, latency-adaptive architectures (Xu et al., 2024).
• Reward and Reasoning Optimization: Designing dense, informative RL reward signals for error detail and self-correction, as well as efficient CoT pipelines, is an open research area (Li et al., 16 Jun 2025).
• Multi-modal Integration: Leveraging visual cues, text retrieval, and external embeddings for improved diarization and transcription is a prospective direction (Mu et al., 17 Sep 2025).

In summary, MLC-SLMs leveraging large-scale pretrained speech encoders, efficient adaptation pipelines, and context-aware LLM decoding have set new standards for multilingual, conversational ASR. Continued innovation in context modeling, speaker attribution, cross-lingual learning, and reward-based reasoning will further advance this vital area of speech and language technology.