Cross-Lingual Routing Alignment

• Cross-lingual routing alignment is the process by which MoE models route tokens through shared, language-universal experts in their middle layers for effective semantic transfer.
• The study quantifies alignment using entropy-normalized Jensen–Shannon divergence, revealing a strong negative correlation (r ∈ [–0.95, –0.80]) with task accuracy across languages.
• Inference-time interventions that adjust mid-layer routing yield consistent 1–2% gains in multilingual accuracy, highlighting a practical strategy for enhancing low-resource language performance.

Cross-lingual routing alignment refers to the process and mechanisms by which multilingual models—particularly Mixture-of-Experts (MoE) architectures—route representations of tokens from different languages through shared expert subnetworks, enabling generalization and transfer across linguistic boundaries. This phenomenon is closely tied to a model’s ability to leverage language-universal experts (as opposed to language-specialized ones), particularly in the middle layers of the network. The alignment of routing decisions is both a critical explanatory factor for performance gaps across languages and a target for intervention to improve multilingual capabilities (Bandarkar et al., 6 Oct 2025).

1. Routing Patterns and Layerwise Alignment in MoE Models

MoE architectures employ a router mechanism that, for each token, selects a sparse subset of expert networks based on the gating logits. Analysis of routing patterns on parallel multilingual datasets reveals a strong layerwise dynamic:

• Early and late decoder layers: Routing is dominated by language-specific patterns—tokens in non-English languages are assigned to experts very different from those assigned to English.
• Middle layers: A pronounced cross-lingual routing alignment emerges, wherein tokens from different languages, but with similar semantic content, are routed similarly (i.e., to the same set of experts). This mirrors established findings in dense multilingual models, where mid-network representations become maximally language-agnostic.

This routing alignment in middle layers is visualized as a U-shaped curve when plotting the entropy-normalized Jensen–Shannon divergence (Dₕ₋JS) of routing distributions from English across layers, with divergence minimized (alignment maximized) at the network’s center (Bandarkar et al., 6 Oct 2025).

2. Quantifying Routing Alignment and Its Correlation with Performance

The paper quantifies cross-lingual routing alignment using the entropy-normalized Jensen–Shannon divergence between the routing distributions of English and a target language at a given layer: $$D_{h\textrm{-}JS}(\mathbf{q}^{(\textrm{eng},l)} || \mathbf{q}^{(\textrm{lang},l)}) = \frac{1}{F} D_{JS}(\mathbf{q}^{(\textrm{eng},l)} || \mathbf{q}^{(\textrm{lang},l)})$$ where $$F = \log(E) - \frac{H(\mathbf{q}^{(\textrm{eng},l)}) + H(\mathbf{q}^{(\textrm{lang},l)})}{2}$$ and $E$ is the number of experts.

Critically, in the middle layers, there is a strong negative correlation (r ∈ [–0.95, –0.80] in OLMoE) between Dₕ₋JS and language task accuracy (such as on the Belebele benchmark): the more similar the non-English routing is to English in these layers, the higher the model's performance in that language. This correlation is highly interpretable and holds with remarkable consistency across languages, tasks, and architectures (Bandarkar et al., 6 Oct 2025).

3. Inference-time Interventions and Routing Manipulation

Leveraging these findings, the authors propose inference-time interventions targeting the middle layers to induce higher cross-lingual routing alignment. The intervention method modifies the gating logits for experts that are frequently activated on English inputs:

• Soft intervention: For a target expert k, the logit is adjusted as

$z'_k \leftarrow z_k + \lambda \cdot s(\mathbf{z})$

where $s(\mathbf{z})$ is the standard deviation of logits and $\lambda$ calibrates intervention strength ($|λ| ≤ 1$).

• Hard intervention: The logit for the expert is forcibly set to the maximum among all logits ($z'_k \leftarrow \max(\mathbf{z}) + \varepsilon$), overriding normal routing.

Empirically, steering the router in this way to promote activation of English-preferred experts at the middle layers yields robust, consistent gains of 1–2% in accuracy across two tasks (MGSM and medical Global-MMLU), three models, and over 15 non-English languages. Intervening in other layers or on language-specialized experts leads to negligible or negative results (Bandarkar et al., 6 Oct 2025).

Layer Targeted	Routing Alignment	Effect on Performance
Middle	High	+1–2% multilingual gain
Early or Late	Low	Degradation/negligible

4. Generalization Limits and Role of Language-universal Experts

The alignment analysis surfaces a modular structure: early and final layers route tokens using language-specialized experts, presumably for local token processing and output generation, whereas the middle layers rely on a pool of language-universal experts. A non-English input must therefore be “routed” into these shared experts in the middle of the model to benefit from cross-lingual generalization.

When a language fails to achieve routing alignment with English—often due to low resource or domain mismatch—it underutilizes these language-universal experts, resulting in persistently poor performance despite perhaps strong specialization at output layers. The finding that performance is not improved by boosting language-specialized experts or adjusting routing outside the aligned, middle “semantic hub” further supports the criticality of language-universal expert activation (Bandarkar et al., 6 Oct 2025).

5. Implications for Model Design and Multilingual Robustness

These results have several implications:

• Diagnosing Cross-lingual Transfer Gaps: Fine-grained routing analysis explains why performance for non-dominant languages lags; the model’s inability to leverage language-universal experts due to misaligned routing is the principal limitation.
• Design of Future MoE Routers: Routing strategies or auxiliary objectives that explicitly promote mid-layer expert sharing across languages can enhance multilingual generalization, particularly for low-resource languages.
• Layerwise Targeted Interventions: Interventions applied specifically to middle layers are an empirically validated, robust strategy for last-mile performance improvement in deployed systems.
• Overriding Trained Behavior: The consistency of performance gains, even when “overriding” router decisions in extensively-trained models, underscores the persistence of local minima in routing policies and the utility of targeted interventions (Bandarkar et al., 6 Oct 2025).

6. Comparative Findings and Broader Context

Comparative analysis emphasizes that cross-lingual routing alignment, as measured by similarity to English expert patterns in middle layers, is the dominant determinant of a model’s multilingual performance ceiling. Unlike changes to expert allocation in outer layers or intervention on experts merely associated with multilingual specialization, only interventions that reinforce or restore the English-aligned routing signature in the model’s semantic core consistently increase cross-lingual transfer accuracy (Bandarkar et al., 6 Oct 2025).

A plausible implication is that the “semantic bottleneck” formed by language-universal experts in the middle of the network is the locus where most effective knowledge sharing and transfer occurs in large-scale MoE LLMs. Thus, cross-lingual routing alignment serves both as an explanatory lens on model behavior and as a practical tool for manipulating and improving multilingual model generalization.