---
title: 'MoE Speech Projector: Expert Routing in ASR'
url: https://www.emergentmind.com/topics/mixture-of-experts-moe-speech-projector
type: topic
---

# MoE Speech Projector: Expert Routing in ASR

A Mixture of Experts (MoE) speech projector is a speech-side mapping or projector-like module that replaces a single dense transformation with multiple expert subnetworks whose contributions are controlled by a router or an explicit routing rule. In the strict encoder–projector–LLM formulation, the projector is the bridge between a speech encoder and a frozen large language model, converting acoustic frame representations into a shorter or LLM-compatible sequence [2606.10439, 2601.19451]. In a broader speech-recognition sense, closely related projector-like mechanisms appear inside acoustic models such as SpeechMoE and SpeechMoE2, where routed experts mediate the transformation from speech representations to recognition features under varying domains, accents, or other conditions [2105.03036, 2111.11831]. Across these settings, the central design objective is consistent: increase conditional capacity and specialization without increasing inference cost proportionally.

## 1. Conceptual scope

In LLM-based ASR, the projector is explicitly defined as the module connecting the speech encoder to the LLM: it takes frame-level acoustic representations from Whisper-large-v3 and maps them into a shorter sequence in the LLM embedding space so that Qwen-2.5 7B can decode transcripts [2606.10439]. In AVSR, the same bottleneck appears in multimodal form: prior systems such as Llama-AVSR typically use a single dense linear or MLP projector to map pretrained audio and video encoder representations into the LLM token space, and the proposed Sparse Mixture of Projectors replaces that dense projector with a routed expert module [2505.14336]. In code-switching ASR, the MoE-based connector sits between the speech encoder and the LLM and maps or downsamples speech features into the LLM text embedding space [2409.15905].

The term also covers projector-like expert mechanisms that do not appear as a standalone adapter between encoder and LLM. SpeechMoE is a dynamic routing Mixture of Experts acoustic model in which each input frame activates only one expert in each MoE layer, while SpeechMoE2 modifies the router so that expert selection depends on local grapheme embedding together with global domain and accent information [2105.03036, 2111.11831]. EVA provides another variation: visual tokens are first mapped into the speech hidden space by a lightweight linear projection, then multimodal adaptation occurs inside the ASR encoder through MoE layers rather than through a standalone speech-to-text connector [2409.12370].

A common source of terminological confusion is that not every speech MoE module is a projector in the narrow LLM-adapter sense. The supervised S-MoE architecture for speech-to-text modeling explicitly states that it does not introduce a “speech projector” in that narrow sense, but instead inserts supervised experts into Transformer encoder and decoder feed-forward blocks [2508.10009]. This suggests that “MoE speech projector” is best understood as a family of conditional mapping mechanisms centered on speech-side representation transfer, alignment, or specialization rather than a single fixed architectural location.

## 2. Acoustic-model precursors and projector-like routing

SpeechMoE established a speech-specific MoE pattern in which sparse expert routing is conditioned on both a shared embedding and hierarchical layer information. Its router at layer $l$ is
$$
r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),
$$
with routing probabilities
$$
p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},
$$
and selected-expert output
$$
y^{l} = p^{l}_{i} E^{l}_{i}.
$$
The model uses top-1 routing, so only one expert is active per MoE layer per example, and the acoustic backbone contains 30 MoE layers, 30 sequential memory layers, and 3 self-attention layers [2105.03036].

SpeechMoE2 retains the same MoE backbone but changes the router input so that routing depends on local and global utterance-level context:
$$
r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}).
$$
Here, $e^c$ is the local grapheme embedding, $e^a$ is the accent embedding, and $e^d$ is the domain embedding. The accent and domain embeddings are derived from the shared embedding network by temporal average pooling and projection,
$$
e^{a} = W_{a} \cdot \frac{1}{T}\sum_{t=1}^{T} e_{t}^{c}, \qquad
e^{d} = W_{d} \cdot \frac{1}{T}\sum_{t=1}^{T} e_{t}^{c},
$$
so the router sees frame-level content and utterance-level condition information simultaneously [2111.11831].

A closely related condition-aware design appears in aphasic speech recognition, where a Speech Intelligibility Detector provides soft mixture weights over severity-specific experts. The final senone posterior is
$$
y(x)=\sum_{i=0}^N w_i(x)M_i(x),
$$
with four experts corresponding to Healthy, Mild, Moderate, and Severe intelligibility regions defined via the WAB-R Aphasia Quotient [2008.10788]. Although this system is not an LLM connector, it illustrates the same projector-like principle: speech is first mapped into an auxiliary condition space, and that condition estimate governs expert combination.

These precursor systems are important because they show that, in speech, routing often depends on information that is not purely lexical. Domain, accent, severity, channel condition, and utterance-level variability are treated as routing-relevant variables rather than as noise to be absorbed by a single dense model.

## 3. Encoder–projector–LLM realizations

Modern MoE speech projectors appear most clearly in frozen-backbone LLM-ASR and AVSR pipelines. In the multilingual projector-based LLM-ASR framework, Whisper-large-v3 provides acoustic features, the projector maps them into a token-like representation, and Qwen-2.5 7B performs transcription; the baseline projector is two convolutional layers followed by two linear layers with a fixed downsampling factor of 4, and the proposed method replaces the linear projection part with an MoE-enhanced projector while replacing fixed-rate downsampling with Continuous Integrate-and-Fire [2606.10439]. SMEAR-MoE follows the same frozen-backbone pattern but uses frozen Whisper large-v3 and frozen Gemma-2-9B, training only the projector [2601.19451]. In code-switching ASR, the speech-conditioned LLM uses Whisper-large V3 or HuBERT as encoder, an MoE-based connector as bridge, and Qwen2-7B or Baichuan2-7B as decoder-side LLM [2409.15905]. In Llama-SMoP, frozen pretrained audio and video encoders feed a Sparse Mixture of Projectors whose outputs become LLM prefix tokens, while the LLM is fine-tuned with LoRA [2505.14336]. EVA places the main MoE adaptation deeper: CLIP visual tokens are linearly projected into the ASR hidden size, concatenated with speech tokens, and then routed through MoE layers that replace the second FFN of each E-Branchformer block [2409.12370].

| System | Placement | Expert organization |
|---|---|---|
| SpeechMoE2 | Inside the acoustic model routers | Top-1 experts conditioned on grapheme, accent, and domain information |
| MoE-enhanced multilingual projector | Between Whisper-large-v3 and Qwen-2.5 7B | Shared convolutional backbone plus expert sub-networks matched to training languages |
| SMEAR-MoE | Trainable projector between frozen Whisper large-v3 and Gemma-2-9B | Shared downsampler, lightweight expert MLPs, and gating network |
| MoE connector for code-switching ASR | Between speech encoder and LLM | Two language-specialized FFN experts for Mandarin and English |
| Llama-SMoP DEDR | Multimodal projector before the LLM | Separate routers and separate expert pools for audio and video |

Architecturally, these systems differ in what they treat as shared structure. The multilingual projector with CIF keeps a shared convolutional front-end and applies experts to the projection layers [2606.10439]. SMEAR-MoE similarly factors the projector into a shared convolutional downsampler and expert MLPs [2601.19451]. Llama-SMoP uses expert MLPs directly as projectors over modality tokens [2505.14336]. EVA preserves a pretrained speech recognizer and replicates its FFN weights to initialize experts, so multimodal adaptation inherits a strong speech prior [2409.12370]. The code-switching connector emphasizes language specialization first and collaboration later, with two experts explicitly interpreted as Mandarin and English experts [2409.15905].

## 4. Routing, conditioning, and optimization

The defining operation of an MoE speech projector is the routing rule. In sparsely gated projector formulations, the router computes expert probabilities and activates only a subset of experts. Llama-SMoP uses
$$
\mathcal{R}(\mathbf{x}) = \text{Top-K}(\text{Softmax}(\mathbf{x}\cdot \mathbf{W}), K),
$$
and the projected token is
$$
\mathbf{z} = \sum_{i=1}^{\mathsf{N}} \mathcal{R}(\mathbf{x})_i \cdot E_i(\mathbf{x}),
$$
with DEDR separating routers and expert pools by modality [2505.14336]. The code-switching connector computes
$$
\mathcal{P}(x_i)=\mathrm{softmax}(\mathbf{W}x_i + b),
$$
and aggregates experts as
$$
h_i=\sum_{j=1}^n \mathcal{P}(x_i)_j \cdot e_j(x_i),
$$
while the first training stage enforces language-specialized routing and the second stage activates all experts [2409.15905]. EVA uses a linear router $f(x)=W\cdot x$, softmax expert probabilities, and top-$K$ expert activation, with 8 experts and top-4 experts active per token in the reported setup [2409.12370].

Speech-specific conditioning often extends beyond token content. SpeechMoE2 uses local grapheme embedding together with accent and domain embeddings derived from temporal average pooling, and it augments the objective with accent classification loss $L_a$ and domain classification loss $L_d$:
$$
L(x;y)=L_{c}(x;y) +\alpha L_{s}(x) + \beta {L}_{m}(x) + \gamma  L_{e}(x;y) + \eta  L_{a}(x;y_{a}) + \theta L_{d}(x;y_{d}).
$$
The reported MoE hyperparameters are
$$
\alpha=0.05,\quad \beta=0.05,\quad \gamma=0.01,\quad \eta=0.1,\quad \theta=0.1.
$$
To avoid significantly increasing compute, the embedding network is trained in a multi-task manner to jointly predict grapheme, domain, and accent information [2111.11831].

Two optimization issues recur across the literature: expert collapse and speech–text length mismatch. SpeechMoE addresses routing sharpness and load balance with a sparsity $L1$ loss and a mean importance loss, and FastMoE is used to distribute experts across multiple GPUs and multiple nodes while computing router-related losses globally across workers [2105.03036]. SMEAR-MoE targets the same collapse problem more directly. Instead of hard top-$k$ selection, it computes token-level routing probabilities
$$
G = \text{Softmax}(Z_S W_g),
$$
forms utterance-level weights
$$
\bar{g} = \frac{1}{T} \sum_{t=1}^{T} G_t,
$$
and merges expert parameters as
$$
\bar{W} = \sum_{m=1}^{M} \bar{g}_m W_m, \quad \bar{b} = \sum_{m=1}^{M} \bar{g}_m b_m.
$$
The resulting virtual expert ensures dense gradient flow to all experts and is explicitly presented as a way to eliminate expert collapse [2601.19451]. For alignment, the multilingual projector with CIF assigns each acoustic frame a weight in $[0,1]$, fires when the cumulative weight reaches a threshold typically 1, and uses a relaxed target with $n=4$ so that CIF does not over-compress the sequence [2606.10439].

Hierarchical routing appears in AVSR. MoHAVE partitions experts into audio and visual groups and uses an inter-modal router over groups together with intra-modal top-$k$ expert selection within each group. Its hierarchical output is
$$
y = \sum_{i \in \text{top}m(G)} \tilde{q}_i(x) \sum_{j \in \text{top}k(E_i)} \tilde{p}_{ij} E_{ij}(x),
$$
with a group-level load biasing loss that encourages audio experts on audio-only inputs and visual experts on video-only inputs [2502.10447]. This is a more structured form of routing than the flat expert pools used in many earlier projector designs.

## 5. Empirical behavior across tasks

Empirical results consistently report accuracy gains at comparable or nearly comparable runtime. In conventional acoustic ASR, SpeechMoE reports up to 23.0% relative CER improvement on the Read set and 7.0%–11.9% relative CER improvements on Chat, Spon, and AISHELL-2, while keeping inference and training FLOPs comparable to static models because only one expert is active per MoE layer [2105.03036]. SpeechMoE2 reports up to 1.6%–4.8% relative CER reduction on the multi-domain task and 1.9%–17.7% relative CER reduction on the multi-accent task, and increasing the number of experts from 2 to 16 yields average relative gains of 5.1% on domain-specific test sets and 7.7% on accent-specific test sets while FLOPs per frame remain constant [2111.11831].

In projector-based multilingual LLM-ASR, the baseline LLM-ASR system reports 23.26 WER on MLCSLM-dev, 19.57 on CommonVoice-test, and 13.05 on FLEURS-test. Replacing the baseline projector’s linear layers with MoE yields 16.10, 14.48, and 11.06 respectively, and combining MoE with modified CIF gives 15.27, 13.87, and 10.46, which is the best projector design in that study. In the larger 8000-hour setting, the proposed method reports 15.45 on MLCSLM-dev, 9.86 on CommonVoice-test, and 8.65 on FLEURS-test [2606.10439]. SMEAR-MoE reports up to a 7.6% relative WER reduction over the single-projector baseline, with overall WER 28.0 versus 30.3 for the single projector and a real-time factor of 0.198 versus 0.196 for the single projector and 0.243 for the dense ensemble [2601.19451].

Code-switching and multimodal settings show the same pattern. The MoE connector in speech-conditioned LLM ASR improves over a linear connector on Aishell-1 from 5.60 to 5.42 CER, on Librispeech from 2.79 to 2.68 WER, and on the ASRU code-switch benchmark from 8.10 to 7.76 MER; the full proposed system uses 43.4M trainable parameters and reports 7.76% MER [2409.15905]. In Llama-SMoP, SMoP-3 DEDR is the best configuration among the tested variants, improving over Llama-AVSR for 1B, 3B, and 8B LLMs and reporting 0.96 at 8B versus 1.09 for the baseline on the main AVSR comparison, while also improving robustness under babble noise, including 22.9 versus 27.5 for AVSR at $-2.5$ dB [2505.14336]. EVA reports that removing the MoE module degrades recognition from 9.5 to 9.8 on How2, from 14.8 to 15.3 on VisSpeech, and from 57.2 to 59.9 on Ego4D, which the authors interpret as evidence that the MoE improves generalization ability [2409.12370]. MoHAVE further shows that a modality-aware hierarchical MoE can improve noisy AVSR, with MoHAVE-Base reducing noisy LRS3 WER from 6.6 for AV-HuBERT-Base to 5.8 and MoHAVE-Large reducing it from 5.7 to 4.5 [2502.10447].

Across these studies, capacity growth and computational restraint are treated jointly rather than separately. Some systems emphasize constant FLOPs under top-1 routing [2105.03036, 2111.11831], others emphasize nearly unchanged runtime relative to a single projector [2601.19451], and multimodal projector work frames the advantage as “more expert capacity per active FLOP” or as improved performance with smaller LLMs under constrained deployment budgets [2505.14336].

## 6. Limitations, misconceptions, and design implications

A recurrent misconception is that MoE speech projectors are uniformly sparse top-$k$ modules. The literature instead contains at least four distinct routing regimes. There is top-1 routing in SpeechMoE and SpeechMoE2 [2105.03036, 2111.11831]; top-$K$ token-wise routing in SMoP and EVA [2505.14336, 2409.12370]; dense, stabilized parameter merging in SMEAR-MoE [2601.19451]; and supervised deterministic routing in S-MoE, where bandwidth and task tags select experts without training a gating function [2508.10009]. MoE-TTS presents yet another case, with modality-based hard assignment between text and speech experts rather than learned sparse token routing [2508.11326]. The phrase “Mixture of Experts” therefore does not by itself determine the routing semantics.

A second misconception is that adding more experts always improves results. SpeechMoE and SpeechMoE2 report consistent gains as expert count increases within their evaluated ranges [2105.03036, 2111.11831], but Llama-SMoP reports that increasing the number of experts from 4 to 8, 12, and 16 does not improve performance and can slightly degrade it, likely because extra experts become redundant and learn similar token patterns [2505.14336]. This indicates that expert count interacts with data scale, task structure, and the router’s ability to maintain distinct specialization.

Several limitations are explicitly identified in the literature. Standard CIF can degrade performance because it may over-compress and discard information; the relaxed target with $n=4$ was introduced precisely because standard CIF was too aggressive [2606.10439]. Conventional sparse MoE routing in multilingual projector training can produce expert collapse, and load-balancing loss alone is reported as insufficient in low-/mid-resource multilingual settings [2601.19451]. In AVSR, flat AV-MoE baselines are weaker than modality-grouped hierarchical routing, and MoHAVE’s gains are strongest in noisy settings rather than clean ones [2502.10447]. In aphasic speech recognition, training each expert only on its own severity group degrades performance because of insufficient data per expert, whereas the Solo+Neighbor strategy performs best [2008.10788].

Taken together, these results suggest several stable design principles. Local content embeddings alone are often insufficient for routing in speech; explicit condition variables such as domain, accent, modality, language, bandwidth, or intelligibility can materially improve expert selection [2111.11831, 2502.10447, 2008.10788]. The shared structure of the projector matters: many successful systems keep a shared convolutional or pretrained backbone and concentrate MoE capacity in the projection or feed-forward sublayers [2606.10439, 2601.19451, 2409.12370]. Stable optimization is not ancillary but central: sparsity, balance, z-loss regularization, multi-task side objectives, or dense gradient flow mechanisms are repeatedly used to keep expert specialization from degenerating [2105.03036, 2505.14336, 2601.19451]. In that sense, the MoE speech projector is not merely a larger projector; it is a conditional interface whose effectiveness depends on how routing, alignment, and specialization are jointly constrained.

Source: https://www.emergentmind.com/topics/mixture-of-experts-moe-speech-projector