Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling

Abstract: The rapid progress of LLMs has opened up a new frontier for automatic speech recognition (ASR), making their effective integration a critical and challenging research direction. To this end, this work proposes a projector-based LLM-ASR framework targeting the key challenges of multilingual generalization and modality alignment. Our approach incorporates a Mixture of Experts (MoE) architecture to improve cross-lingual adaptability, and a Continuous Integrate-and-Fire (CIF) mechanism for dynamic downsampling and modality alignment. Experimental results show that the combination of these components yields substantial performance improvements, surpassing strong baseline models. The proposed method represents a step toward building more accurate, robust, and generalizable LLM-based ASR systems.

• The paper introduces a MoE-enhanced projector to improve cross-lingual acoustic-to-text mapping with language-specific expert subnetworks.
• It adopts dynamic CIF-based downsampling to flexibly align audio features with text tokens, effectively adapting to speech rate variations.
• Experimental results demonstrate significant WER reductions across benchmarks, validating the framework's robustness in multilingual settings.

Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling

Introduction

This paper addresses critical challenges in LLM-integrated Automatic Speech Recognition (ASR) systems, focusing on multilingual generalization and modality alignment. The naive integration of LLMs into ASR pipelines has frequently led to suboptimal robustness and performance, especially in multi-language scenarios with high phonetic and lexical diversity. The authors enhance the LLM-ASR framework by introducing a Mixture-of-Experts (MoE) initialized projector to address cross-lingual adaptation and a Continuous Integrate-and-Fire (CIF) mechanism to replace fixed downsampling, aiming to achieve flexible and robust modality alignment between audio features and text tokens.

Baseline Architecture

The foundational architecture utilized in this work is the Encoder–Projector–LLM paradigm, where a pre-trained speech encoder (here, Whisper-large-v3) transforms acoustic signals into intermediate features. A lightweight projector then maps these representations into the token embedding space of a frozen LLM (Qwen-2.5 7B), which subsequently generates the transcription.

Figure 1: Baseline Framework integrating speech encoder, projector, and LLM for end-to-end ASR.

While this baseline achieves end-to-end ASR by leveraging the LLM’s semantic and contextual modeling capabilities, it suffers from three fundamental shortcomings: insufficient modeling of cross-lingual acoustic-to-text mappings by the simple projector, lack of robust modality alignment due to fixed downsampling, and performance sensitivity to prompt design.

MoE-Enhanced Projector for Multilingual Representation

The core architectural innovation is the MoE-enhanced projector, designed to improve adaptability across multiple languages, each with distinct phonotactic and linguistic structures. The MoE structure comprises a shared convolutional backbone and a set of expert subnetworks, one for each language. Routing is performed dynamically via a gating network conditioned on the input acoustic features. The system aggregates expert outputs weighted by their activations, providing a more flexible and expressive mapping from acoustic to textual embeddings.

Figure 2: Structure of the projector module, where a convolutional backbone feeds into language-specific MoE subnetworks selected via gated activation.

The integration of MoE significantly outperforms both baseline LLM-ASR and Whisper-large-v3 in Word Error Rate (WER) across evaluation datasets. The results demonstrate consistent performance improvements for all supported languages, mitigating the pronounced degradation otherwise observed in cross-lingual and domain-shifted settings.

Dynamic Modality Alignment via CIF

Fixed downsampling strategies typically aggregate multiple feature frames into one token-aligned representation at a set ratio. These rigid approaches often fail in the presence of speech rate variation and language-specific phonetic structure, leading to misalignment and information loss.

To address this, the paper adopts Continuous Integrate-and-Fire (CIF) for modality alignment. The CIF predictor sequentially accumulates weighted acoustic features and emits a token representation upon surpassing a threshold, dynamically matching the output length to target token sequences—flexibly adapting to varying speech rates and utterance structures. Notably, the conventional CIF’s tendency for over-compression is remedied by adjusting the objective to output a sequence $n$ times the target token length (with $n=4$ in experiments), improving retention of discriminative information.

Experimental Results

The evaluation spans the MLC-SLM (in-domain), CommonVoice, and FLEURS (out-of-domain) benchmarks, with models trained on both 1,500h and 8,000h multilingual speech corpora. Key results are:

• Baseline LLM-ASR underperforms Whisper-large-v3 in WER on all datasets, confirming difficulties in naive LLM integration.
• MoE Projector reduces WER on MLCSLM-dev from 23.26% (baseline) to 16.10%, consistently improving recognition for individual languages.

  Figure 3: Per-language WER on the MLCSLM-dev set, demonstrating strong gains from MoE and CIF over baseline.

• Standard CIF with original objective degrades performance due to over-compression.
• Modified CIF further reduces WER to 15.27% on MLCSLM-dev, 13.87% on CommonVoice, and 10.46% on FLEURS, representing strong improvements over both baseline and Whisper-large-v3.
• Scaling training data to 8,000h produces substantial OOD generalization gains (WER on CommonVoice-test drops to 9.86%; FLEURS-test to 8.65%), though with a negligible in-domain performance tradeoff.

The combination of MoE and dynamic, length-matched CIF outperforms all benchmarks, confirming the effectiveness of both mechanisms.

Implications and Future Directions

The findings have theoretical significance for the architecture of unified multilingual ASR systems. The MoE approach enables efficient parameter sharing and specialization, reducing negative interference between languages. The modularity of the projection head, coupled with CIF-based adaptive alignment, provides a scalable template for further LLM-ASR integrations, especially as language coverage and domain diversity increase.

Practically, these advances make generalizable, robust, and scalable multilingual speech recognition more tractable, particularly for deployment in real-world environments with high linguistic and domain variance. The decoupling of training between the encoder/projector subsystems and frozen LLM reduces resource costs while retaining extensibility.

Future research may address unsupervised or weakly supervised expert specialization, context- or speaker-conditioned routing, and advanced alignment objectives that further close the gap between token-level and utterance-level semantic fidelity.

Conclusion

This work supplies two principled enhancements—MoE-based projectors and adaptive CIF-based downsampling—to the LLM-ASR framework, resulting in substantial gains in multilingual and cross-domain speech recognition accuracy. These architectural improvements address key limitations of current LLM-integrated ASR pipelines, and provide a robust foundation for future extensions in multilingual, adaptive, and efficient end-to-end speech recognition systems.