Multilingual Experts (MuEx) Overview

• Multilingual Experts (MuEx) are specialized modules that allocate expert capacity across languages within a shared model, mitigating interference and catastrophic forgetting.
• MuEx strategies include explicit language-specific modules and emergent routing approaches that dynamically balance language-exclusive and shared experts for improved performance.
• They are applied in diverse domains such as multilingual ASR, language models, and programming to yield efficiency gains and enhance cross-lingual transfer.

Multilingual Experts (MuEx) denotes a family of multilingual modeling strategies that allocate conditional or modular capacity across languages within a shared backbone. In some works, MuEx refers to explicitly language-specific modules, such as per-language feed-forward experts, adapters, or LoRA branches; in others, especially in MoE LLMs, it refers to an emergent routing structure in which some experts are language-exclusive while others are shared across languages. Across multilingual ASR, multilingual LLMs, continual language expansion, multilingual programming, and speech-driven talking face synthesis, MuEx is used to mitigate negative interference, the curse of multilinguality, and catastrophic forgetting while preserving cross-lingual transfer and sparse inference (Wang et al., 2023, Chen et al., 20 Jan 2026, Zhang et al., 28 May 2025, Su et al., 8 Oct 2025).

1. Conceptual scope and definitions

MuEx is not a single architecture. The term is used broadly for systems that host language-conditioned experts inside a shared model, but the mechanism of specialization varies. In one explicit MoE analysis, multilingual experts are formalized as a heterogeneous pool of language-exclusive experts and shared experts: an expert $i$ at layer $l$ is language-exclusive for language $\ell$ when $W_{l,i}^{(\ell)} > \theta$, while shared experts satisfy $\max_{\ell} W_{l,i}^{(\ell)} \le \theta$ (Chen et al., 20 Jan 2026). Other works use the same underlying idea without the label “MuEx,” for example by identifying language-dominant experts via activation gaps or by allocating experts to language groups rather than individual languages (Khandelwal et al., 28 May 2026, Li et al., 14 Jun 2025).

Paradigm	MuEx realization	Representative papers
Static language experts	Per-language experts or adapters selected by language tags, LID, or deterministic rules	(Ferraz et al., 2023, Wang et al., 2023, Kwon et al., 2023, Zong et al., 18 Jun 2025)
Emergent multilingual experts	Language-exclusive and shared experts identified from routing distributions	(Chen et al., 20 Jan 2026, Bandarkar et al., 6 Oct 2025)
Layer-aware expansion	Expert counts chosen by similarity, neuron diversity, or parameter deviation	(Zhang et al., 28 May 2025, Li et al., 14 Jun 2025, Li et al., 5 Mar 2026)
Cross-modal multilingual experts	Experts routed by pseudo-phonemes and aligned with visemes	(Su et al., 8 Oct 2025)

A recurring distinction is between MuEx and generic MoE. MuEx may use learned token-level routing, but it may also use frame-wise LID routing, utterance-level language tokenizers, or deterministic routing from code fences. Consequently, MuEx is not synonymous with learned anonymous experts. In multilingual programming, for example, experts are pre-assigned to programming languages and routing is deterministic from code-block language tags rather than a learned gate (Zong et al., 18 Jun 2025). In multilingual ASR, LR-MoE and MoLE similarly use language-aware routing rather than unrestricted token-level expert competition (Wang et al., 2023, Kwon et al., 2023).

2. Routing mechanisms and architectural patterns

The most common MuEx instantiation is sparse MoE in Transformer feed-forward sublayers. A router computes expert scores from the token hidden state, applies softmax, selects the top-$K$ experts, and aggregates their outputs. This pattern is used in multilingual MoE LLM analyses, in LayerMoE-style multilingual expansion, and in dynamic multilingual grouping approaches (Chen et al., 20 Jan 2026, Zhang et al., 28 May 2025, Li et al., 14 Jun 2025). In these systems, the added multilingual capacity comes primarily from conditional FFN computation rather than from duplicating the full model.

A second pattern is static or semi-static language routing. LR-MoE replaces the FFN in the last six encoder layers with language-specific expert FFNs, but keeps attention shared and uses a shared frame-level LID pre-router across all expertized layers. The router is trained with an auxiliary LID-CTC loss and uses top-1 frame-wise routing, allowing expert switches at code-switch boundaries while keeping FLOPs at the vanilla baseline (Wang et al., 2023). MoLE uses an utterance-level LSTM “language tokenizer” that selects one language-specific expert and combines it with a language-agnostic expert through the reliability-weighted embedding

$$h = \gamma(X)\, h_{LsE} + \big(1-\gamma(X)\big)\, h_{LaE},$$

where $\gamma(X)$ is the maximum routing posterior (Kwon et al., 2023).

A third pattern uses parameter-efficient experts. Lamer-SSL augments each frozen HuBERT-Large FFN with a Layer-Aware MixturE of LoRA Experts, routed by top-2 sparse softmax and regularized with a load-balancing loss (Xu et al., 13 Feb 2026). Multilingual programming MoLE uses a frozen base model with three LoRA branches in each FFN linear layer: a shared LoRA, a language-specific LoRA, and an NL adapter. For programming-language tokens in language $l$, the effective weight is $W = W_0 + \Delta W_s + \Delta W_l$; for natural-language tokens, $l$0 (Zong et al., 18 Jun 2025).

MuEx can also extend beyond FFNs. BLR-MoE applies multilingual experts not only to FFNs but also to self-attention value and output projections, while reusing an utterance-level LID router for both attention and FFN experts. This is motivated by a decomposition of language confusion into confusion in self-attention and confusion in the router (Ma et al., 22 Jan 2025). In speech–LLM bridging, SMEAR-MoE locates multilingual experts inside a projector rather than inside the backbone model: a gating network computes utterance-level weights, and a virtual expert is constructed by parameter merging,

$l$1

which guarantees dense gradient flow to all experts and avoids collapse (Pandey et al., 27 Jan 2026).

3. Layerwise specialization and multilingual processing

A central finding in recent MuEx analyses is that multilingual specialization is strongly layer-dependent. Across Qwen3-30B-A3B, Phi-3.5-MoE-instruct, GPT-OSS-20B, and related models, routing divergence across languages is typically U-shaped: early and late layers are more language-specific, while middle layers are more aligned across languages (Chen et al., 20 Jan 2026, Bandarkar et al., 6 Oct 2025). One paper characterizes the middle layers as “language-agnostic capacity hubs,” with early layers supporting language-specific understanding and late layers supporting language identity and generation (Chen et al., 20 Jan 2026).

This layerwise decomposition is supported by interventions. In Qwen3-30B-A3B, masking language-exclusive experts in early layers reduces MGSM accuracy from 84.5 to 73.1, middle-layer masking changes it only to 83.6, and late-layer masking reduces it to 78.4. On language consistency, late masking is especially destructive, dropping MGSM consistency from 90.9 to 56.9 and FLORES-200 consistency from 97.2 to 60.9 (Chen et al., 20 Jan 2026). These results localize multilingual understanding and language identity to different depth regions.

Routing alignment in middle layers is also predictive of performance. “Multilingual Routing in Mixture-of-Experts” reports a strong correlation between per-language performance and similarity to English routing in middle layers, and inference-time promotion of English task experts in those layers yields consistent $l$2 gains across tasks, models, and 15+ languages, while steering outside the middle layers degrades performance (Bandarkar et al., 6 Oct 2025). RA-MoE generalizes the same observation into a training-time framework, treating the middle layer range as a language-universal “alignment zone” for task-relevant experts (Deng et al., 27 May 2026).

The same structural picture appears in expansion methods that do not begin from routing analysis. LayerMoE finds that hidden-state similarity is higher in middle and upper layers and lower in very shallow and very deep layers, then allocates fewer experts to high-similarity layers and more experts to low-similarity layers (Zhang et al., 28 May 2025). DMoE similarly ranks layers by monolingual parameter deviation and expertizes the top-$l$3 fraction, with layers near input and output showing larger deviations (Li et al., 14 Jun 2025). NeuronMoE refines this further by measuring cross-lingual neuron diversity and finds language-specific neurons concentrated in early and late layers, with the middle layers relatively sparse (Li et al., 5 Mar 2026). This suggests a common architectural regularity rather than an artifact of a single model family.

4. Training objectives, alignment strategies, and continual adaptation

MuEx training objectives vary according to whether the aim is specialization, alignment, or continual language expansion. In distillation-based formulations, the goal is to compress a stronger multilingual teacher into smaller language-aware students. The supplied description of DistilWhisper frames this as whisper-large-v2 distilling into whisper-small with language-specific experts, but also states that the shared text does not report the exact module placements, parameter counts, hyperparameters, or quantitative gains for the MuEx component (Ferraz et al., 2023).

Several recent methods align internal routing rather than only outputs. SARA aligns dense Softmax routing distributions between low-resource inputs and high-resource “semantic anchors” using a symmetric Jensen–Shannon divergence term added to the task loss,

$l$4

On Global-MMLU, it improves Qwen3-30B-A3B from 70.73 to 71.53 with an English anchor and improves Phi-3.5-MoE-instruct from 41.63 to 42.80 (Dong et al., 24 Jun 2026). RA-MoE uses a different supervision signal: parallel examples are partitioned into $l$5, task experts are identified from English routing, and a KL alignment loss is applied only to $l$6 examples, where English is correct and the target language is incorrect. Averaged over 18 task–language combinations, RA-MoE improves OLMoE from 23.3 under SFT to 24.2, Qwen1.5-MoE from 39.3 to 40.9, and DeepSeek-V2-Lite from 38.1 to 39.1 (Deng et al., 27 May 2026).

Continual multilingual expansion introduces a different problem: preserving old languages while adding new ones. LayerMoE freezes the old expert $l$7, allocates new experts by inverse layer similarity, and uses a second-stage router objective with both language-prior routing and classifier-guided hard routing to $l$8 on high-similarity layers. It reports 60% fewer experts than MoE-LPR in single-expansion settings and 33.3% fewer in lifelong expansion while improving old and new language averages (Zhang et al., 28 May 2025). Lamer-SSL combines layer-aware LoRA experts with replay and load balancing; only 2.14% of parameters are trainable, yet it achieves the best reported CER and LID accuracy among the listed baselines for English, Mandarin, and Cantonese (Xu et al., 13 Feb 2026).

A related but more mechanistic adaptation strategy is Selective and Shared Expert Finetuning. In an English-centric OLMoE model, activation gaps identify language-dominant experts in the final two layers, and top-$l$9 shared experts are updated jointly with them. SSFT updates less than 2% of parameters, reaches 83.6% average on MultiBLiMP versus 84.6% for AEFT and 90.5% for Full-FT, and reduces adaptation cost from about 32 GPU-hours for Full-FT to about 3.3 GPU-hours (Khandelwal et al., 28 May 2026).

5. Domain-specific realizations

In multilingual ASR, MuEx has been used both for language separation and for code-switching. LR-MoE achieves multilingual average TER 6.3 and code-switching MER 8.2 on a four-language setting, compared with 8.8 and 10.4 for the vanilla baseline, while keeping FLOPs at 55.4 GFLOPs (Wang et al., 2023). MoLE improves multilingual CTC CER from 13.34 for a shared multilingual Transformer to 10.51 with language representation loss, a language-agnostic expert, and reliability calibration; notably, Zh improves to 16.18 and Ja to 9.26, both better than the monolingual Transformer values reported in the same table (Kwon et al., 2023). BLR-MoE extends expertization into self-attention and strengthens the LID router, reporting a 16.09% relative WER reduction over LR-MoE, with 3.98% in-domain and 19.09% out-of-domain improvements (Ma et al., 22 Jan 2025). In streaming ASR, a Conformer with MoE layers in the non-causal encoder improves average WER from 11.33 to 9.98, an 11.9% relative gain, while activating only 211M parameters at inference versus 400M for a dense model of comparable quality (Hu et al., 2023).

In speech–LLM and speech SSL settings, MuEx often appears in parameter-efficient forms. SMEAR-MoE uses a stabilized multi-expert projector between a frozen Whisper large-v3 encoder and a frozen Gemma-2-9B LLM, reducing average WER from 30.3 to 28.0 and achieving nearly identical real-time factor to a single projector, 0.198 versus 0.196 (Pandey et al., 27 Jan 2026). Lamer-SSL extends HuBERT-Large to Mandarin and Cantonese with progressive 2/4/6/8 expert allocation, replay, and top-2 routing, reaching average CER 10.50 on CommonVoice and 10.13 on Fleurs, with LID accuracy 99.22% (Xu et al., 13 Feb 2026).

MuEx has also been generalized beyond text and speech. In multilingual talking-face synthesis, MuEx combines a Phoneme-Guided Mixture-of-Experts with Phoneme-Viseme Alignment and a 12-language, 95.04-hour benchmark. The full system reports FVD 171.284, Sync-C 7.536, LSE-D 0.0437, and TMDC 0.756, and human evaluation shows closely matched seen and unseen language averages, 5.68/5.37 and 5.57/5.52, respectively (Su et al., 8 Oct 2025). In multilingual programming, MoLE-48+16 uses deterministic routing to programming-language-specific LoRA experts and reaches Pass@1 averages of 23.75 for summarization, 31.34 for synthesis, and 60.62 for translation, exceeding a single shared LoRA baseline and matching Full-FT on translation (Zong et al., 18 Jun 2025).

6. Efficiency, misconceptions, and unresolved issues

A major attraction of MuEx is that multilingual capacity can scale faster than active compute. LayerMoE reduces expert counts by 60% in single-expansion and 33.3% in lifelong expansion relative to MoE-LPR (Zhang et al., 28 May 2025). DMoE reports +11.4% average perplexity improvement over continued pre-training on 18 languages and beats X-ELM with 3.6× fewer parameters (Li et al., 14 Jun 2025). NeuronMoE achieves approximately 40% average parameter reduction relative to LayerMoE on Llama-3.2-3B and 50% on Qwen-1.5-1.8B while maintaining comparable performance (Li et al., 5 Mar 2026). These results directly contradict the idea that multilingual MoE quality is governed primarily by uniform per-layer expert growth.

Another recurrent clarification is that stronger multilingual performance does not imply more language-specific routing everywhere. Multiple studies report that middle layers are the main locus of shared multilingual processing, while over-specializing those layers can be harmful (Chen et al., 20 Jan 2026, Bandarkar et al., 6 Oct 2025, Deng et al., 27 May 2026). A plausible implication is that effective MuEx systems separate concerns: shallow and deep layers absorb language-specific form, while middle layers preserve access to language-universal task experts.

Several limitations recur across the literature. Static routing by language tag or classifier can fail under language identification errors, unseen languages, or mixed-language utterances (Ferraz et al., 2023, Kwon et al., 2023, Ma et al., 22 Jan 2025). Routing alignment methods depend on the quality of anchors, translations, or English supervision, and may weaken when the anchor language is itself weak or when translation fidelity is poor (Dong et al., 24 Jun 2026, Deng et al., 27 May 2026). One-expert-per-language designs create storage and management pressure as the language inventory grows (Ferraz et al., 2023). Methods built on vocabulary overlap or anchor languages may transfer poorly to language isolates or targets with low overlap to pretrained languages (Khandelwal et al., 28 May 2026).

The field therefore contains two partially competing views. One view treats MuEx primarily as explicit language specialization: dedicate modules to languages, route reliably, and protect them from interference. The other treats MuEx as a structured balance between exclusive and shared experts, with performance depending on how well low-resource languages access the shared multilingual core. The recent routing-analysis and routing-alignment papers support the second view more strongly, but the continued success of language-routed ASR and deterministic expert selection indicates that explicit specialization remains effective when routing supervision is reliable (Wang et al., 2023, Chen et al., 20 Jan 2026, Dong et al., 24 Jun 2026).

MuEx has thus evolved from a parameter-sharing heuristic into a broader design principle for multilingual systems: isolate language-specific computation where necessary, preserve shared capacity where beneficial, and use routing, expert placement, or alignment losses to regulate the boundary between the two.