Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation

Abstract: Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we conduct a systematic analysis of expert routing patterns in MoE models, revealing a phenomenon we term Language Routing Isolation, in which high- and low-resource languages tend to activate largely disjoint expert sets. Through layer-stratified analysis, we further show that routing patterns exhibit a layer-wise convergence-divergence pattern across model depth. Building on these findings, we propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expert subnetworks. RISE applies a tripartite selection strategy, using specificity scores to identify language-specific experts in shallow and deep layers and overlap scores to select universal experts in middle layers. By training only the selected subnetwork while freezing all other parameters, RISE substantially improves low-resource language performance while preserving capabilities in other languages. Experiments on 10 languages demonstrate that RISE achieves target-language F1 gains of up to 10.85% with minimal cross-lingual degradation.

• The paper identifies language routing isolation in multilingual MoE models, showing negligible expert overlap between high- and low-resource languages.
• It introduces the RISE framework for interpretable, layer-aware subnetwork adaptation that selectively updates experts to enhance low-resource performance.
• Empirical experiments demonstrate up to +10.85% F1 improvement on Bengali with RISE, confirming targeted adaptation without degrading performance in other languages.

Unveiling Language Routing Isolation in Multilingual MoE Models: Mechanistic Insights and the RISE Subnetwork Adaptation Framework

Introduction

Mixture-of-Experts (MoE) architectures, by leveraging sparse expert activation, have enabled scaling of LLMs to hundreds of billions of parameters while keeping inference efficient. Despite substantial advances, multilingual MoE models exhibit pronounced performance disparities across languages—most notably, stark gaps between high-resource and low-resource languages. The underlying mechanisms responsible for these gaps, particularly the functional organization of expert routing, have remained elusive. This paper provides a rigorous investigation of routing patterns in multilingual MoE models, uncovers the phenomenon of Language Routing Isolation, and introduces RISE (Routing Isolation-guided Subnetwork Enhancement), a method leveraging these findings for interpretable, minimally invasive adaptation to low-resource languages (2604.03592).

Language Routing Isolation: Empirical Analysis

Systematic analysis at both the global and layer-wise levels demonstrates that MoE models internally segregate expert routing by language resource type. Specifically, the top-activated experts for high-resource languages (e.g., English, Chinese) are almost completely orthogonal to those for low-resource ones (e.g., Bengali, Swahili). This phenomenon is summarized quantitatively using Jaccard similarity over top-K routed experts, revealing negligible overlap between the global expert supports of high- and low-resource groups.

Figure 1: Global-level expert activation overlap across languages in MGSM, illustrating strong isolation between high- and low-resource language routing supports.

Layer-wise analysis reveals a convergence–divergence structure across depth. Shallow and deep layers exhibit language-specific routing (low overlap), while middle layers demonstrate increased cross-lingual overlap—mirroring functional hierarchies in dense Transformers, where shallow and deep layers are associated with language-specific encoding/generation, and middle layers with cross-lingual semantic processing (Figure 2).

Figure 2: Overview of RISE: multilingual routing statistics are analyzed, experts partitioned/selected based on layer-aware specificity/overlap, and only the selected subnet is trained.

Empirical results across multiple MoE architectures (e.g., Qwen3-30B-A3B, Phi-3.5-MoE-Instruct) confirm that such routing isolation and stratification patterns are robust to substantial variations in model width, layer count, and routing capacity.

The RISE Framework: Layer-Aware Expert Subnetwork Adaptation

Building on these findings, RISE implements a three-phase, layer-aware expert selection and adaptation protocol. Routing statistics for all experts and layers are first aggregated across languages, producing language-expert activation affinity matrices. The framework then partitions the MoE layers into shallow, middle, and deep groups.

• In shallow and deep layers, RISE computes a language-specificity score for each expert, selecting those most disproportionately activated by the target low-resource language.
• In middle layers, cross-lingual overlap is quantified by the coefficient of variation in expert activation across languages, and universal (shared) experts are selected.

A composite scoring function, parameterized by an activation scale hyperparameter $\alpha$, combines specificity/overlap with activation frequency to avoid selecting rarely-activated experts, and a budget is allocated asymmetrically—favoring layer groups corresponding to observed divergence. Only the selected expert subnet is then updated with gradients, all others (experts and shared modules) frozen throughout adaptation.

Figure 3: Grouped comparison of layer-wise expert subset combinations, showing the necessity of all three layer groups for optimal adaptation.

Experimental Results

Comprehensive experiments were conducted using the TyDiQA-GoldP (extractive QA, 9 languages) and MGSM (math reasoning, 10 languages) benchmarks. RISE delivers significant gains for low-resource languages with minimal cross-lingual or cross-task degradation. For instance, on Qwen3-30B-A3B, adapting only the RISE-selected subnetwork produces up to +10.85% F1 improvement on Bengali relative to vanilla training, exceeding both random/top-K expert selection and baseline expert specialization methods (e.g., ESFT).

Qualitative error analysis of vanilla/non-adapted models highlights common failure modes in low-resource languages: severe mathematical reasoning errors, premature output truncation, and mixed-language non-answers. RISE adaptation corrects such failures, as evidenced by case studies contrasting answers generated with and without RISE-enabled expert training.

Figure 4: Ablation study on the activation scale factor $\alpha$ in the composite selection score.

A key empirical claim, validated via causal ablation, is that the selected subnetworks are not only associated with but causally responsible for target-language computation. Pruning only the RISE-selected experts leads to catastrophic collapse on the adapted language, while non-target languages are largely unaffected. Layerwise ablation demonstrates that all three expert groups contribute to adaptation, with shallow and deep layers being most critical for low-resource targets.

Figure 5: Global-level expert activation overlap of Qwen3-30B-A3B across languages in TyDiQA, corroborating strong inter-language isolation.

Mechanistic Justification: From Routing Isolation to Gradient Isolation

The paper provides a formal treatment of why routing isolation leads to gradient isolation: since the router seldom (or never) activates target-language experts for non-target languages, updating those experts does not propagate gradients back into representations of other languages. Under perfect isolation (routing supports are strictly disjoint), the forward computation for non-target languages is mathematically invariant—the adaptation is strictly local. In the more realistic setting of near-orthogonal (but not perfectly disjoint) expert routing, a cross-lingual interference bound is derived, scaling linearly with routing overlap probabilities and parameter perturbation magnitudes.

Theoretical and Practical Implications

These contributions clarify several points:

• Functional modularity in MoE models is not a byproduct, but an emergent, measurable property, which can be exploited for targeted parameter-efficient adaptation.
• Equitable adaptation is feasible: significant improvements on low-resource languages can be attained without degradation of general or high-resource language capabilities.
• Mechanistic interpretability now directly informs efficient, robust transfer—not solely post hoc analysis.

RISE, by freezing the non-selected subnet (including experts and shared transformers), also enables significant savings in compute and memory during training.

Future Directions

This work opens new avenues for research on interpretable, modular, and resource-aware adaptation of ultra-large-scale LLMs. Potential directions include:

• Generalization of routing isolation principles to domain- or task-specialized adaptation (beyond multilinguality).
• Dynamic or online expert allocation protocols that autonomously detect and adapt to shifting language/task distributions.
• Integration with parameter-efficient fine-tuning methods (e.g., LoRA applied only to RISE-selected experts).
• Joint optimization of expert routing and subnetwork selection for maximizing capacity utilization across heterogeneous workloads.

  Figure 6: Layer-wise expert selection heatmap for Bengali training on Qwen3-30B-A3B, showing RISE’s preference for middle and deep layers—consistent with strong Bengali routing isolation.

  

  Figure 7: Layer-wise expert selection heatmap for Bengali training on Phi-3.5-MoE-Instruct; RISE consistently concentrates adaptation budget in middle-to-deep layers, even for architectures with fewer, top-2 routed experts.

Conclusion

The paper presents an empirical, methodological, and theoretical advance in understanding and controlling multilingual adaptation in MoE-based LLMs. By articulating and exploiting language routing isolation, the RISE framework enables interpretable, highly effective, and non-destructive subnetwork adaptation for low-resource languages. These contributions have direct implications for scalable, equitable deployment of multilingual LLMs, as well as for future research into self-organizing, mechanism-aware neural architectures.

(2604.03592)