Multilingual Conversational Speech Language Model

• Multilingual Conversational Speech Language Model is defined as a unified neural system that integrates large-scale speech encoding with language model decoding to perform both ASR and diarization.
• It employs modular components such as pretrained speech encoders, lightweight adapters, and LoRA-tuned LLM decoders for efficient cross-lingual transcription and speaker-aware segmentation.
• Evaluations report reduced error rates (WER and tcpWER) in open-domain, multi-speaker scenarios, demonstrating its potential for scalable and robust spoken language understanding.

A Multilingual Conversational Speech LLM (MLC-SLM) is a neural architecture integrating large-scale speech encoding and large language modeling to perform automatic speech recognition (ASR) and spoken language understanding across numerous languages and conversational domains. MLC-SLM systems are evaluated in open-domain, multichannel, and multi-speaker contexts, where they must transcribe real-world conversations, resolve speaker turns and boundaries, and robustly model linguistic and cross-lingual variability. This paradigm is motivated by the need for unified solutions capable of leveraging rich pretrained LLMs, massive multilingual audio corpora, and highly scalable adaptation methods, as exemplified in the INTERSPEECH 2025 MLC-SLM Challenge (Mu et al., 17 Sep 2025).

1. Task Formulation, Datasets, and Evaluation

MLC-SLM research centers on two principal tasks:

1. Conversational ASR with Oracle Segmentation
   • Input: Audio, segmented utterance boundaries, speaker labels.
   • Output: Per-utterance transcripts.
   • Metric: Word Error Rate (WER) for alphabetic languages, Character Error Rate (CER) for non-segmented scripts; Mixed Error Rate (MER) averages these across languages.
2. End-to-End Diarization + ASR
   • Input: Raw audio with no oracle segmentation.
   • Output: Time-stamped, speaker-attributed transcriptions.
   • Metric: Time-constrained permutation WER (tcpWER) that jointly penalizes content and temporal misalignment; analogous tcpCER and an aggregate tcpMER.

The primary benchmarking resource is the MLC-SLM corpus (1,500+ hours), covering 11–16 languages (English with 5 regional variants, French, German, Italian, Portuguese, Spanish, Japanese, Korean, Russian, Thai, Vietnamese), consisting of two-speaker, natural conversations with extensive annotation (Mu et al., 17 Sep 2025). Public corpora such as CommonVoice, GigaSpeech/GigaSpeech2, MLS, Multilingual LibriSpeech, TEDx, and others are used for augmentation, although strict per-language balancing is observed to avoid resource bias (Mei et al., 4 Jan 2026).

2. Model Architectures

The canonical MLC-SLM architecture adopts a modular composition, typically as follows:

• Speech Encoder: Large pretrained models (e.g., Whisper-large-v3, mHuBERT-147, MMS-1B), which convert log-mel or waveform inputs into dense frame-level representations. Some systems concatenate multiple encoders (e.g., Whisper + mHuBERT) to leverage both supervised and self-supervised representations (Mei et al., 4 Jul 2025, Mei et al., 4 Jan 2026).
• Projector/Adapter: Lightweight, often two-layer MLPs or conv-linear adapters, which subsample and map encoder features to the LLM embedding space. Adapter designs include Linear-ReLU-Linear (Transsion (Li et al., 15 Aug 2025)), SwiGLU-MLP (Qwen vs Gemma (Nguyen et al., 16 Jun 2025)), or stacking with LoRA (Meng et al., 11 Jul 2025, Lin et al., 13 Jul 2025).
• LLM Decoder: Large, general LLMs (Qwen2.5-7B, Gemma-2B/12B, Llama-3.x, Babel-9B, EuroLLM 1.7B) often left mostly frozen with task-specific adaptation (LoRA, adapters, or full-tuning).
• Speaker/Context Modules: For diarization, speaker-embedding extraction (e.g., ResNet34, ERes2Net, S2S neural diarizers), triplet-based gating, or explicit speaker tokens in the target sequence (Lin et al., 13 Jul 2025, Saengthong et al., 26 Jun 2025).
• Prompting: Language ID and context are often provided via explicit language-aware prompts, context tokens, or CTC-token prefixing (Mei et al., 4 Jul 2025, Xue et al., 24 Jul 2025).

Architectural innovations span cross-attention fusion (Res-Uni-CAF, bi-directional gating (Mei et al., 4 Jan 2026)), dual-encoder residual fusion (Mei et al., 4 Jul 2025, Xue et al., 24 Jul 2025), or end-to-end output formatting using special speaker/timestamp tokens for unified diarization and transcription (Saengthong et al., 26 Jun 2025). Table 1 summarizes leading Track 1 architecture patterns.

System/Team	Encoders/Adapters	LLM Decoder	Key Adaptation
Transsion (Li et al., 15 Aug 2025)	Whisper-v3 + 2L-Adapter	Qwen2.5-7B-LoRA	Adaptor+LoRA
SHNU-mASR (Mei et al., 4 Jul 2025)	Whisper-v3 + mHuBERT-cat, projector	Qwen2.5-7B-LoRA	Parallel LoRA
NTU Speechlab (Peng et al., 16 Jun 2025)	Whisper-v3 + Adapter	Gemma2-2B (FT)	Full-tune
BUT (Polok et al., 16 Jun 2025)	DiCoW (Whisper-v3, FDDT mask)	-	Frame-level Diar.
Triple X (Gao et al., 23 Jul 2025)	Whisper-v3 + Adapter	Qwen-3B (LoRA)	3-stage training
TEA-ASLP (Xue et al., 24 Jul 2025)	Whisper-v3 + MMS-1B + Gated	Qwen-3-8B + mLoRA	MoE, CTC Prompt

FT: full fine-tuning; mLoRA: Mixture-of-Experts LoRA adapters.

3. Training Strategies and Adaptation

Parameter-efficient adaptation is a central focus due to the size of modern LLMs and encoders. The dominant strategies include:

• Fine-tune Only Lightweight Modules: Freeze speech encoders and LLM backbone; adapt only projectors, adapters, and LoRA modules (Li et al., 15 Aug 2025, Lin et al., 13 Jul 2025).
• Iterative LoRA Training (ILT): Multi-phase adaptation (Focus, Feedback, Fix) with knowledge-task cycling, pseudo-labels, and composite loss (CE, CTC) (Meng et al., 11 Jul 2025). ILT demonstrably closes the gap between LoRA-only and full fine-tuning, mitigating overfitting and domain drift.
• Language-Specific LoRA/Adapters: Use per-language LoRA “experts” and route via explicit LID; MoE-style fusion or gating (sigmoid/softmax) to blend encoder or decoder updates by language (Xue et al., 24 Jul 2025).
• Full-Parameter Fine-Tuning: Complete LLM tuning offers further gains when combined with a frozen encoder, though requiring more compute (Peng et al., 16 Jun 2025).
• Curriculum and Multi-Stage: Curricular training (projector → adapter/LoRA → joint) stabilizes learning and enables scaling to large multilingual or stylistically heterogeneous corpora (Li et al., 15 Aug 2025, Gao et al., 23 Jul 2025).
• Chain-of-Thought (CoT) and RLVR: CoT data augmentation (hypothesis–error–correction triplets) and reinforcement learning with verifiable, automatic rewards can yield further error reductions by enforcing structured self-correction (Li et al., 16 Jun 2025).

Data augmentation (SpecAugment, noise/reverb, speed, synthetic TTS) is routine, but reliance on in-domain conversational data remains critical for transferability, as out-of-domain corpora show diminishing returns once core domains are covered (Peng et al., 16 Jun 2025, Mu et al., 17 Sep 2025).

4. Multilingual and Contextual Adaptation

Effective multilingual modeling employs explicit mechanisms:

• Language-Aware Prompting: Prepending language prompts in the target language reduces cross-language confusion and code-switch errors; even short, task-matched prompts significantly reduce MER (–10.4% rel.) (Peng et al., 16 Jun 2025, Peng et al., 16 Jun 2025).
• Mixture-of-Experts and LID Routing: Adapter-based MoE or per-language LoRA handling enables robust handling of both high- and low-resource languages and is synergized by language conditioning during both training and inference (Xue et al., 24 Jul 2025, Mu et al., 17 Sep 2025).
• Data Balancing: Uniform per-language sampling and augmentation prevent performance collapse in rare languages and ensure downstream metrics reflect true multilingual generality (Mu et al., 17 Sep 2025, Mei et al., 4 Jan 2026).
• Conversational Context: Conditioning on historical (or bi-directional) context via prompt templates and character-level context masking enhances disfluency and repair modeling, resolves local ambiguities, and improves error rates by up to 2% absolute over non-contextual decoding (Peng et al., 16 Jun 2025, Concina et al., 25 Jul 2025).

Contrastive alignment of speech–context pairs and context-aware re-decoding are especially effective in late-turn or low-resource dialog settings (Concina et al., 25 Jul 2025).

5. Diarization, Unified Modeling, and Joint Tasks

Emerging approaches integrate diarization and transcription rather than relying on cascaded pipelines:

• Speaker-Aware Embedding & Triplet Decoding: DKU’s diarization-aware decoding supplies (speaker embedding, time window) triplets, instructing the LLM to confine decoding to the relevant speaker/time interval (Lin et al., 13 Jul 2025).
• Unified Sequence Generation: Systems serialize the entire diarization + ASR task as a mixed speaker/timestamp/text token sequence, enabling single-pass decoding and supervision (Saengthong et al., 26 Jun 2025).
• Frame-Level Diarization Conditioning: DiCoW augments each encoder layer with learned affine transformations, weighted by time-varying probabilities of silence, target, non-target, or overlap resulting in improved zero-shot and adapted tcpWER even under OOD conditions (Polok et al., 16 Jun 2025).
• Global Alignment and RTTM Integration: Sliding-window inference combined with RTTM-based global merging of segments resolves speaker drift and boundary errors (Saengthong et al., 26 Jun 2025).

tcpWER emerges as the most informative metric for the joint task, as it jointly penalizes segmentation, assignment, and content errors across all languages.

6. Results, Ablations, and Lessons Learned

State-of-the-art MLC-SLM systems, as developed in the INTERSPEECH 2025 Challenge and follow-up works, achieve:

• Best published evaluation WER/CER: 9.60% (TEA-ASLP) using multilingual dual-encoder fusion, per-language MoE LoRA, and CTC-prompted LLM (Xue et al., 24 Jul 2025).
• Best published tcpWER/tcpCER: 16.75% (BUT) and 17.49% (TEA-ASLP) for combined diarization + ASR, with direct end-to-end or advanced diarization pipelines (Polok et al., 16 Jun 2025, Xue et al., 24 Jul 2025).
• Relative improvement: Up to 54.87% reduction in tcpWER over the official baseline when adopting unified or diarization-aware architectures (Saengthong et al., 26 Jun 2025, Lin et al., 13 Jul 2025).

Key ablation findings:

• Adapters/LoRA: Adapter alignment provides ~2–3% absolute WER gain; LoRA fine-tuning in the LLM decoder adds ~0.6% (Gao et al., 23 Jul 2025).
• Context modeling: Bi-directional context-aware re-decoding outweighs simple data scaling, with context masking during training a necessary regularizer (Peng et al., 16 Jun 2025).
• Data scale: In-domain, conversational, and balanced data are more valuable than sheer scale of generic speech for sustained improvement (Peng et al., 16 Jun 2025, Mu et al., 17 Sep 2025).
• MoE routing: Known-LID LoRA routing outperforms shared or softmax gating, especially when combined with language-adapted connector modules (Xue et al., 24 Jul 2025).
• Projector/compression: Simple linear or two-layer adapters generally outperform more complex attention-based modules for speech-to-LLM mapping (Mei et al., 4 Jan 2026, Concina et al., 25 Jul 2025).

7. Open Challenges and Future Directions

Despite strong advances, several unsolved challenges persist:

• Modality Alignment: The gap between fine-tuned end-to-end (Whisper) and Speech-LLM pipelines persists, likely due to lossy projection of temporal features and suboptimal joint pretraining (Mei et al., 4 Jan 2026).
• Scaling and Transfer: Further multilingual scaling (beyond 16 languages), including robust code-switching and rare-language handling, remains an open topic.
• Diarization Integration: Robust, end-to-end diarization, especially for overlapping speech, multi-party (>2) conversations, and dynamic speaker tracking, is still underexplored (Saengthong et al., 26 Jun 2025, Polok et al., 16 Jun 2025).
• Textless Modeling: Cross-lingual interleaving with discrete speech-only LMs offers a path for low-/zero-resource language adaptation and direct semantic transfer without text tokens (Moumen et al., 1 Dec 2025).
• Inference Efficiency and Self-Correction: Chain-of-Thought data augmentation, explicit error reasoning, and reward-driven RLVR show promise for reducing hallucinations but introduce inference and tuning complexity (Li et al., 16 Jun 2025).
• Automatic Speech–Language Pretraining: Tighter joint pretraining of speech encoders and LLMs, modality-shared latent spaces, and adaptive fusion remain key foci (Mei et al., 4 Jan 2026, Li et al., 15 Aug 2025).

The MLC-SLM paradigm, grounded in scalable transfer, modular adaptation, and contextually aware modeling, underpins the current frontier of multilingual, conversational speech–language systems. Recent results indicate that further progress will arise from joint optimization of all pipeline stages, richer pretraining across modalities and domains, and structurally robust integration of context, speaker, and diarization cues.