MLC-SLM Challenge: Multilingual Speech Modeling

Updated 12 November 2025

MLC-SLM Challenge is a benchmark suite designed to evaluate multilingual ASR, diarization, and conversation modeling under real-world, noisy conditions.
It integrates encoder–adapter–LLM and hybrid encoder architectures to enhance transcription accuracy and speaker segmentation in diverse languages.
The challenge leverages a 1,604-hour dataset across 11 languages, employing techniques like low-rank adaptation and iterative fine-tuning for robust performance.

Multilingual Conversational Speech Language Modeling (MLC-SLM) Challenge designates a benchmark suite, dataset, and community challenge focused on advancing robust, context-sensitive, LLM-based speech recognition and diarization across multiple languages and conversational settings. MLC-SLM synthesizes techniques at the intersection of automatic speech recognition (ASR), speaker diarization, and instruction-tuned LLMs, emphasizing real-world, multi-speaker, multilingual dialogues with naturalistic conversational phenomena.

1. Challenge Scope and Dataset

The MLC-SLM Challenge (Mu et al., 17 Sep 2025) is structured around two principal tasks:

Task 1: Multilingual Conversational ASR. Given oracle segmentation and speaker attribution, systems transcribe speech in 11 typologically diverse languages (15 English-including-accent varieties) with evaluation by Word Error Rate (WER), Character Error Rate (CER, for Japanese/Korean/Thai), and a composite Mixed Error Rate (MER).
Task 2: Multilingual Diarization + ASR. Systems must solve both the underlying diarization (segmentation and speaker labeling) and transcript generation from raw, unsegmented audio, with evaluation by time-constrained minimum-permutation WER/CER (tcpWER/tcpCER/tcpMER). This task mandates robustness to uncontrolled conversational dynamics (overlaps, interruptions, code-switching).

The provided dataset is 1,604 hours of real-world conversational audio distributed over 11 languages, each collected in quiet, indoor conditions from a wide speaker pool recorded on mobile devices. Datasets are partitioned into training (1,507 h), development (32 h), and evaluation (64 h across two test sets) subsets. Each session is structured as a two-speaker dialogue (~20 min, 2–10 s turns), with manual transcripts and speaker annotations for training and development sets.

Subset	Total Duration (h)	Speaker/Segmentation Transcripts
Training	1,507	Oracle segmentation, speaker labels, reference text
Dev	32	Oracle segmentation, speaker labels, reference text
Eval-1/2	64	Eval-1: no references; Eval-2 released post-challenge

The speech displays typical phenomena found in spontaneous dialogue: short turns, overlaps, rapid speaker changes, and frequent code-switching—contextual features often underrepresented in legacy ASR corpora.

2. Modeling Paradigms

2.1 Encoder-Adapter-LLM Architectures

A prevailing paradigm in MLC-SLM is the Encoder–Adapter–LLM stack (Li et al., 15 Aug 2025, Gao et al., 23 Jul 2025). An acoustic encoder (typically Whisper-large-v3) extracts frame-level audio features, a trainable adapter projects the features into the LLM token-embedding space, and an instruction-tuned LLM (e.g., Qwen2.5-7B, Gemma-2-2B, Babel-9B-Chat) performs autoregressive decoding. Adapter modules are typically compact two-layer feed-forward networks, often with frame-splicing (downsampling) to control sequence length. Whisper encoders are generally frozen—a design choice justified by their extensive multilingual pretraining and to prevent catastrophic forgetting.

2.2 Parallel/Hybrid Encoders

Systems such as SHNU-mASR (Mei et al., 4 Jul 2025) and TEA-ASLP (Xue et al., 24 Jul 2025) deploy dual parallel encoders (supervised + self-supervised, e.g., Whisper-large-v3 and mHuBERT-147 or MMS-1B) to fuse semantic and phonetic feature representations. Fusion strategies are often conditioned on language identification, enabling language-specific weighting of encoder outputs.

2.3 LLM Adaptation

Multiple groups employ Low-Rank Adaptation (LoRA) modules within LLMs for parameter-efficient adaptation, commonly with per-task or per-language specialization (Li et al., 15 Aug 2025, Lin et al., 13 Jul 2025, Xue et al., 24 Jul 2025). Mixture-of-Experts (MoE) LoRA structures specialize separate adapters for each language, routed by explicit language-ID tokens (Xue et al., 24 Jul 2025). Most systems freeze the main LLM weights, only updating LoRA and/or adapter parameters. Some, such as NTU Speechlab (Peng et al., 16 Jun 2025), pursue full-parameter fine-tuning for maximal adaptation.

2.4 Chain-of-Thought and Reasoning-Augmented Training

Curriculum learning and chain-of-thought (CoT) augmentation incorporate explicit intermediate reasoning into ASR (Li et al., 16 Jun 2025). For example, systems require the LLM to generate an initial hypothesis, a self-reflection with token-level error attribution, and finally a corrected transcript, supported by weighted losses focusing on the corrected segment.

2.5 Contextual Modeling and Prompting

Leading approaches leverage explicit context windows, prepending system prompts and/or preceding/following utterance hypotheses to the input, with context masking during training to promote robustness against imperfect previous/future hypotheses (Peng et al., 16 Jun 2025). Language-aware "prompts" or instructions are critical to prevent cross-lingual transfer errors and to enforce language-appropriate decoding (Peng et al., 16 Jun 2025, Mei et al., 4 Jul 2025, Peng et al., 16 Jun 2025).

3. Training Regimes and Data Utilization

State-of-the-art systems emphasize:

Curriculum or multi-stage fine-tuning: Progressive unfreezing (adapter → encoder → LLM) and staged parameter updating are widely applied (Li et al., 15 Aug 2025, Gao et al., 23 Jul 2025, Mei et al., 4 Jul 2025).
Large-scale augmentation: Incorporation of external corpora (CommonVoice, GigaSpeech, MLS, TEDx, ReazonSpeech, etc.) increases data volume and language coverage. Data balancing and quality filtering (via CTC-based methods) are vital for preventing performance collapse on underrepresented languages (Xue et al., 24 Jul 2025).
Iterative and semi-supervised learning: Iterative LoRA Training (ILT) with pseudo-labeling expands coverage and stabilizes adaptation, particularly in low-resource or new-language bootstrapping (Meng et al., 11 Jul 2025).
Parameter-efficient techniques: Fine-tuning only adapter and LoRA parameters keeps model storage manageable across N language variants and enables tractable experimentation (Lin et al., 13 Jul 2025, Li et al., 15 Aug 2025).

4. Diarization and Joint Models

Task 2 requires joint diarization and transcription. Pipeline systems first perform voice activity detection (VAD) and speaker embedding extraction, then cluster segments, and finally transcribe with SLLMs adapted for speaker context (e.g., TEA-ASLP uses ERes2Net-large, followed by a Qwen-3-8B ASR module (Xue et al., 24 Jul 2025); DKU employs S2SND diarization, triplet-embedding prompts, and language-specific LLM adapters (Lin et al., 13 Jul 2025)). BUT's DiariZen + DiCoW (Polok et al., 16 Jun 2025) demonstrates the integration of diarization-conditioned affine transforms at every Whisper encoder layer, conditioned on four-class speaker masks, enabling Whisper to distinguish silence, target, non-target, and overlap regions.

End-to-end Speech LLMs (e.g., (Saengthong et al., 26 Jun 2025)) jointly model speaker turns, text, and temporal alignment, learning to emit diarization markers and transcribe tokens in a unified autoregressive process.

5. Evaluation Metrics and Empirical Results

Task 1 performance is measured by WER (Latin-script languages), CER (East Asian languages), and averaged as Mixed Error Rate (MER). Task 2 is evaluated using tcpWER/tcpCER, enforcing correct alignment of speech and diarization under permutation constraints. Leading systems halve MER from baseline values (~20% to ~9.6% for ASR; ~60% to ~16–18% tcpMER for diarization+ASR) (Mu et al., 17 Sep 2025).

Sample performance table for Task 1 (selected systems) (Mu et al., 17 Sep 2025, Li et al., 15 Aug 2025, Gao et al., 23 Jul 2025, Xue et al., 24 Jul 2025):

Team/Approach	WER/CER (%)	System Features
TEA-ASLP	9.60	Dual encoder, MoE LoRA, CTC prompting, 180k h data
Triple X	9.67	Encoder–Adapter–LLM, 3B Qwen-LLM, 30k h ext. data, LoRA
Transsion	9.83	Frozen Whisper+Qwen2.5-7B, 2-layer adaptor, LoRA
NTU Speechlab	10.58	Gemma-2B, language prompts, model averaging
SHNU-mASR	11.76	Parallel Whisper+mHuBERT encoders, LoRA, language prompt
Seewo	11.57	Curriculum, multilingual prompt, CoT, RLVR, LoRA

For Task 2, state-of-the-art pipeline and joint models reach tcpWER ≈ 16.5%–18% (Xue et al., 24 Jul 2025, Lin et al., 13 Jul 2025, Polok et al., 16 Jun 2025).

6. Innovations, Ablations, and Best Practices

Challenge submissions highlighted:

Adapter and LoRA specialization for each language (via explicit routing or MoE) is consistently superior to monolithic parameter sharing (Xue et al., 24 Jul 2025, Lin et al., 13 Jul 2025).
Language prompts and context prompts drastically reduce cross-lingual confusion and improve both accuracy and generalizability.
Dual encoders (supervised/self-supervised) and learned fusion provide orthogonal gains—semantics from supervised ASR pretraining, phonetic robustness from SSL (Mei et al., 4 Jul 2025, Xue et al., 24 Jul 2025).
Bi-directional context and context masking during training enforce conversational consistency and robustness against noisy hypotheses (Peng et al., 16 Jun 2025).
Chain-of-Thought augmentation and RL (with verifiable sequence-level rewards) induce reasoning and explicit self-correction in ASR LLMs (Li et al., 16 Jun 2025).
Iterative LoRA Training (Focus, Feed Back, Fix) with pseudo-labeling expands coverage and prevents overfitting during adaptation (Meng et al., 11 Jul 2025).
Checkpoint/model averaging offers a low-cost route to variance reduction and final accuracy gains, particularly with full-parameter fine-tuning (Peng et al., 16 Jun 2025).

7. Open Questions and Outlook

While MLC-SLM submissions achieved substantial reductions in error rates through modular architectures, strategic data usage, and robust adaptation, open challenges persist:

Context and dialogue modeling: Better utilization of both short- and long-range conversational context (hierarchical or global dialogue models) beyond fixed-window prompting remains an active research area (Peng et al., 16 Jun 2025, Mu et al., 17 Sep 2025).
Code-switching and low-resource adaptation: Specialized training regimes or dynamic expert routing for utterances containing mixed languages are needed, as is robust semi-supervised learning for data-imbalanced languages (Xue et al., 24 Jul 2025, Meng et al., 11 Jul 2025).
End-to-end vs. cascade integration: Fully end-to-end diarization+ASR SLLMs are promising, but scalable, stable joint modeling of segmentation, speaker attribution, and recognition across languages remains an unsolved task (Saengthong et al., 26 Jun 2025).
Scalability and efficiency: Practical deployment on resource-constrained devices requires architecture distillation, conditional computation, and perhaps model sparsity or MoE pruning (Mei et al., 4 Jul 2025).
Annotation artifacts: Diarization and ASR systems remain sensitive to label noise and misalignment in training data. Label fusion (e.g., VAD augmentation) and explicit noise robustness remain essential (Polok et al., 16 Jun 2025).
Benchmark evolution: Continued releases of more challenging datasets and refinement of evaluation metrics are expected to drive future advances (Mu et al., 17 Sep 2025).

A plausible implication is that future multilingual conversational speech LLMs will require dynamic context modeling, adaptive multilingual parameterization, and robust, large-scale semi-supervised learning to further close the gap to human-level dialogue understanding in unconstrained, real-world spoken interactions.