MoE Speech Projector: Expert Routing in ASR

• The MoE speech projector is a conditional module that replaces a single dense mapping with multiple expert subnetworks, converting acoustic frames into LLM-compatible tokens.
• It employs various routing strategies—such as top-1 and top-k selection—and conditions on domain, accent, and language to enhance expert specialization without additional inference cost.
• Empirical evaluations show marked reductions in CER and WER across ASR, AVSR, and code-switching tasks, validating its effective capacity scaling and performance gains.

A Mixture of Experts (MoE) speech projector is a speech-side mapping or projector-like module that replaces a single dense transformation with multiple expert subnetworks whose contributions are controlled by a router or an explicit routing rule. In the strict encoder–projector–LLM formulation, the projector is the bridge between a speech encoder and a frozen LLM, converting acoustic frame representations into a shorter or LLM-compatible sequence (Lin et al., 9 Jun 2026, Pandey et al., 27 Jan 2026). In a broader speech-recognition sense, closely related projector-like mechanisms appear inside acoustic models such as SpeechMoE and SpeechMoE2, where routed experts mediate the transformation from speech representations to recognition features under varying domains, accents, or other conditions (You et al., 2021, You et al., 2021). Across these settings, the central design objective is consistent: increase conditional capacity and specialization without increasing inference cost proportionally.

1. Conceptual scope

In LLM-based ASR, the projector is explicitly defined as the module connecting the speech encoder to the LLM: it takes frame-level acoustic representations from Whisper-large-v3 and maps them into a shorter sequence in the LLM embedding space so that Qwen-2.5 7B can decode transcripts (Lin et al., 9 Jun 2026). In AVSR, the same bottleneck appears in multimodal form: prior systems such as Llama-AVSR typically use a single dense linear or MLP projector to map pretrained audio and video encoder representations into the LLM token space, and the proposed Sparse Mixture of Projectors replaces that dense projector with a routed expert module (Cappellazzo et al., 20 May 2025). In code-switching ASR, the MoE-based connector sits between the speech encoder and the LLM and maps or downsamples speech features into the LLM text embedding space (Zhang et al., 2024).

The term also covers projector-like expert mechanisms that do not appear as a standalone adapter between encoder and LLM. SpeechMoE is a dynamic routing Mixture of Experts acoustic model in which each input frame activates only one expert in each MoE layer, while SpeechMoE2 modifies the router so that expert selection depends on local grapheme embedding together with global domain and accent information (You et al., 2021, You et al., 2021). EVA provides another variation: visual tokens are first mapped into the speech hidden space by a lightweight linear projection, then multimodal adaptation occurs inside the ASR encoder through MoE layers rather than through a standalone speech-to-text connector (Wu et al., 2024).

A common source of terminological confusion is that not every speech MoE module is a projector in the narrow LLM-adapter sense. The supervised S-MoE architecture for speech-to-text modeling explicitly states that it does not introduce a “speech projector” in that narrow sense, but instead inserts supervised experts into Transformer encoder and decoder feed-forward blocks (Jin et al., 5 Aug 2025). This suggests that “MoE speech projector” is best understood as a family of conditional mapping mechanisms centered on speech-side representation transfer, alignment, or specialization rather than a single fixed architectural location.

2. Acoustic-model precursors and projector-like routing

SpeechMoE established a speech-specific MoE pattern in which sparse expert routing is conditioned on both a shared embedding and hierarchical layer information. Its router at layer $l$ is

$$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$

with routing probabilities

$$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$

and selected-expert output

$y^{l} = p^{l}_{i} E^{l}_{i}.$

The model uses top-1 routing, so only one expert is active per MoE layer per example, and the acoustic backbone contains 30 MoE layers, 30 sequential memory layers, and 3 self-attention layers (You et al., 2021).

SpeechMoE2 retains the same MoE backbone but changes the router input so that routing depends on local and global utterance-level context:

$$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e^{c}; e^{a}; e^{d}; o^{l-1}).$$

Here, $e^c$ is the local grapheme embedding, $e^a$ is the accent embedding, and $e^d$ is the domain embedding. The accent and domain embeddings are derived from the shared embedding network by temporal average pooling and projection,

$$e^{a} = W_{a} \cdot \frac{1}{T}\sum_{t=1}^{T} e_{t}^{c}, \qquad e^{d} = W_{d} \cdot \frac{1}{T}\sum_{t=1}^{T} e_{t}^{c},$$

so the router sees frame-level content and utterance-level condition information simultaneously (You et al., 2021).

A closely related condition-aware design appears in aphasic speech recognition, where a Speech Intelligibility Detector provides soft mixture weights over severity-specific experts. The final senone posterior is

$y(x)=\sum_{i=0}^N w_i(x)M_i(x),$

with four experts corresponding to Healthy, Mild, Moderate, and Severe intelligibility regions defined via the WAB-R Aphasia Quotient (Perez et al., 2020). Although this system is not an LLM connector, it illustrates the same projector-like principle: speech is first mapped into an auxiliary condition space, and that condition estimate governs expert combination.

These precursor systems are important because they show that, in speech, routing often depends on information that is not purely lexical. Domain, accent, severity, channel condition, and utterance-level variability are treated as routing-relevant variables rather than as noise to be absorbed by a single dense model.

3. Encoder–projector–LLM realizations

Modern MoE speech projectors appear most clearly in frozen-backbone LLM-ASR and AVSR pipelines. In the multilingual projector-based LLM-ASR framework, Whisper-large-v3 provides acoustic features, the projector maps them into a token-like representation, and Qwen-2.5 7B performs transcription; the baseline projector is two convolutional layers followed by two linear layers with a fixed downsampling factor of 4, and the proposed method replaces the linear projection part with an MoE-enhanced projector while replacing fixed-rate downsampling with Continuous Integrate-and-Fire (Lin et al., 9 Jun 2026). SMEAR-MoE follows the same frozen-backbone pattern but uses frozen Whisper large-v3 and frozen Gemma-2-9B, training only the projector (Pandey et al., 27 Jan 2026). In code-switching ASR, the speech-conditioned LLM uses Whisper-large V3 or HuBERT as encoder, an MoE-based connector as bridge, and Qwen2-7B or Baichuan2-7B as decoder-side LLM (Zhang et al., 2024). In Llama-SMoP, frozen pretrained audio and video encoders feed a Sparse Mixture of Projectors whose outputs become LLM prefix tokens, while the LLM is fine-tuned with LoRA (Cappellazzo et al., 20 May 2025). EVA places the main MoE adaptation deeper: CLIP visual tokens are linearly projected into the ASR hidden size, concatenated with speech tokens, and then routed through MoE layers that replace the second FFN of each E-Branchformer block (Wu et al., 2024).

System	Placement	Expert organization
SpeechMoE2	Inside the acoustic model routers	Top-1 experts conditioned on grapheme, accent, and domain information
MoE-enhanced multilingual projector	Between Whisper-large-v3 and Qwen-2.5 7B	Shared convolutional backbone plus expert sub-networks matched to training languages
SMEAR-MoE	Trainable projector between frozen Whisper large-v3 and Gemma-2-9B	Shared downsampler, lightweight expert MLPs, and gating network
MoE connector for code-switching ASR	Between speech encoder and LLM	Two language-specialized FFN experts for Mandarin and English
Llama-SMoP DEDR	Multimodal projector before the LLM	Separate routers and separate expert pools for audio and video

Architecturally, these systems differ in what they treat as shared structure. The multilingual projector with CIF keeps a shared convolutional front-end and applies experts to the projection layers (Lin et al., 9 Jun 2026). SMEAR-MoE similarly factors the projector into a shared convolutional downsampler and expert MLPs (Pandey et al., 27 Jan 2026). Llama-SMoP uses expert MLPs directly as projectors over modality tokens (Cappellazzo et al., 20 May 2025). EVA preserves a pretrained speech recognizer and replicates its FFN weights to initialize experts, so multimodal adaptation inherits a strong speech prior (Wu et al., 2024). The code-switching connector emphasizes language specialization first and collaboration later, with two experts explicitly interpreted as Mandarin and English experts (Zhang et al., 2024).

4. Routing, conditioning, and optimization

The defining operation of an MoE speech projector is the routing rule. In sparsely gated projector formulations, the router computes expert probabilities and activates only a subset of experts. Llama-SMoP uses

$$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$0

and the projected token is

$$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$1

with DEDR separating routers and expert pools by modality (Cappellazzo et al., 20 May 2025). The code-switching connector computes

$$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$2

and aggregates experts as

$$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$3

while the first training stage enforces language-specialized routing and the second stage activates all experts (Zhang et al., 2024). EVA uses a linear router $$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$4, softmax expert probabilities, and top-$$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$5 expert activation, with 8 experts and top-4 experts active per token in the reported setup (Wu et al., 2024).

Speech-specific conditioning often extends beyond token content. SpeechMoE2 uses local grapheme embedding together with accent and domain embeddings derived from temporal average pooling, and it augments the objective with accent classification loss $$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$6 and domain classification loss $$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$7:

$$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$8

The reported MoE hyperparameters are

$$r^{l} = W_{r}^{l} \cdot \mathrm{Concat}(e; o^{l-1}),$$9

To avoid significantly increasing compute, the embedding network is trained in a multi-task manner to jointly predict grapheme, domain, and accent information (You et al., 2021).

Two optimization issues recur across the literature: expert collapse and speech–text length mismatch. SpeechMoE addresses routing sharpness and load balance with a sparsity $$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$0 loss and a mean importance loss, and FastMoE is used to distribute experts across multiple GPUs and multiple nodes while computing router-related losses globally across workers (You et al., 2021). SMEAR-MoE targets the same collapse problem more directly. Instead of hard top-$$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$1 selection, it computes token-level routing probabilities

$$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$2

forms utterance-level weights

$$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$3

and merges expert parameters as

$$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$4

The resulting virtual expert ensures dense gradient flow to all experts and is explicitly presented as a way to eliminate expert collapse (Pandey et al., 27 Jan 2026). For alignment, the multilingual projector with CIF assigns each acoustic frame a weight in $$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$5, fires when the cumulative weight reaches a threshold typically 1, and uses a relaxed target with $$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$6 so that CIF does not over-compress the sequence (Lin et al., 9 Jun 2026).

Hierarchical routing appears in AVSR. MoHAVE partitions experts into audio and visual groups and uses an inter-modal router over groups together with intra-modal top-$$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$7 expert selection within each group. Its hierarchical output is

$$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$8

with a group-level load biasing loss that encourages audio experts on audio-only inputs and visual experts on video-only inputs (Kim et al., 11 Feb 2025). This is a more structured form of routing than the flat expert pools used in many earlier projector designs.

5. Empirical behavior across tasks

Empirical results consistently report accuracy gains at comparable or nearly comparable runtime. In conventional acoustic ASR, SpeechMoE reports up to 23.0% relative CER improvement on the Read set and 7.0%–11.9% relative CER improvements on Chat, Spon, and AISHELL-2, while keeping inference and training FLOPs comparable to static models because only one expert is active per MoE layer (You et al., 2021). SpeechMoE2 reports up to 1.6%–4.8% relative CER reduction on the multi-domain task and 1.9%–17.7% relative CER reduction on the multi-accent task, and increasing the number of experts from 2 to 16 yields average relative gains of 5.1% on domain-specific test sets and 7.7% on accent-specific test sets while FLOPs per frame remain constant (You et al., 2021).

In projector-based multilingual LLM-ASR, the baseline LLM-ASR system reports 23.26 WER on MLCSLM-dev, 19.57 on CommonVoice-test, and 13.05 on FLEURS-test. Replacing the baseline projector’s linear layers with MoE yields 16.10, 14.48, and 11.06 respectively, and combining MoE with modified CIF gives 15.27, 13.87, and 10.46, which is the best projector design in that study. In the larger 8000-hour setting, the proposed method reports 15.45 on MLCSLM-dev, 9.86 on CommonVoice-test, and 8.65 on FLEURS-test (Lin et al., 9 Jun 2026). SMEAR-MoE reports up to a 7.6% relative WER reduction over the single-projector baseline, with overall WER 28.0 versus 30.3 for the single projector and a real-time factor of 0.198 versus 0.196 for the single projector and 0.243 for the dense ensemble (Pandey et al., 27 Jan 2026).

Code-switching and multimodal settings show the same pattern. The MoE connector in speech-conditioned LLM ASR improves over a linear connector on Aishell-1 from 5.60 to 5.42 CER, on Librispeech from 2.79 to 2.68 WER, and on the ASRU code-switch benchmark from 8.10 to 7.76 MER; the full proposed system uses 43.4M trainable parameters and reports 7.76% MER (Zhang et al., 2024). In Llama-SMoP, SMoP-3 DEDR is the best configuration among the tested variants, improving over Llama-AVSR for 1B, 3B, and 8B LLMs and reporting 0.96 at 8B versus 1.09 for the baseline on the main AVSR comparison, while also improving robustness under babble noise, including 22.9 versus 27.5 for AVSR at $$p^{l}_{i} = \frac{\exp(r^{l}_{i})}{\sum_{j=1}^{n}\exp(r^{l}_{j})},$$9 dB (Cappellazzo et al., 20 May 2025). EVA reports that removing the MoE module degrades recognition from 9.5 to 9.8 on How2, from 14.8 to 15.3 on VisSpeech, and from 57.2 to 59.9 on Ego4D, which the authors interpret as evidence that the MoE improves generalization ability (Wu et al., 2024). MoHAVE further shows that a modality-aware hierarchical MoE can improve noisy AVSR, with MoHAVE-Base reducing noisy LRS3 WER from 6.6 for AV-HuBERT-Base to 5.8 and MoHAVE-Large reducing it from 5.7 to 4.5 (Kim et al., 11 Feb 2025).

Across these studies, capacity growth and computational restraint are treated jointly rather than separately. Some systems emphasize constant FLOPs under top-1 routing (You et al., 2021, You et al., 2021), others emphasize nearly unchanged runtime relative to a single projector (Pandey et al., 27 Jan 2026), and multimodal projector work frames the advantage as “more expert capacity per active FLOP” or as improved performance with smaller LLMs under constrained deployment budgets (Cappellazzo et al., 20 May 2025).

6. Limitations, misconceptions, and design implications

A recurrent misconception is that MoE speech projectors are uniformly sparse top-$y^{l} = p^{l}_{i} E^{l}_{i}.$0 modules. The literature instead contains at least four distinct routing regimes. There is top-1 routing in SpeechMoE and SpeechMoE2 (You et al., 2021, You et al., 2021); top-$y^{l} = p^{l}_{i} E^{l}_{i}.$1 token-wise routing in SMoP and EVA (Cappellazzo et al., 20 May 2025, Wu et al., 2024); dense, stabilized parameter merging in SMEAR-MoE (Pandey et al., 27 Jan 2026); and supervised deterministic routing in S-MoE, where bandwidth and task tags select experts without training a gating function (Jin et al., 5 Aug 2025). MoE-TTS presents yet another case, with modality-based hard assignment between text and speech experts rather than learned sparse token routing (Xue et al., 15 Aug 2025). The phrase “Mixture of Experts” therefore does not by itself determine the routing semantics.

A second misconception is that adding more experts always improves results. SpeechMoE and SpeechMoE2 report consistent gains as expert count increases within their evaluated ranges (You et al., 2021, You et al., 2021), but Llama-SMoP reports that increasing the number of experts from 4 to 8, 12, and 16 does not improve performance and can slightly degrade it, likely because extra experts become redundant and learn similar token patterns (Cappellazzo et al., 20 May 2025). This indicates that expert count interacts with data scale, task structure, and the router’s ability to maintain distinct specialization.

Several limitations are explicitly identified in the literature. Standard CIF can degrade performance because it may over-compress and discard information; the relaxed target with $y^{l} = p^{l}_{i} E^{l}_{i}.$2 was introduced precisely because standard CIF was too aggressive (Lin et al., 9 Jun 2026). Conventional sparse MoE routing in multilingual projector training can produce expert collapse, and load-balancing loss alone is reported as insufficient in low-/mid-resource multilingual settings (Pandey et al., 27 Jan 2026). In AVSR, flat AV-MoE baselines are weaker than modality-grouped hierarchical routing, and MoHAVE’s gains are strongest in noisy settings rather than clean ones (Kim et al., 11 Feb 2025). In aphasic speech recognition, training each expert only on its own severity group degrades performance because of insufficient data per expert, whereas the Solo+Neighbor strategy performs best (Perez et al., 2020).

Taken together, these results suggest several stable design principles. Local content embeddings alone are often insufficient for routing in speech; explicit condition variables such as domain, accent, modality, language, bandwidth, or intelligibility can materially improve expert selection (You et al., 2021, Kim et al., 11 Feb 2025, Perez et al., 2020). The shared structure of the projector matters: many successful systems keep a shared convolutional or pretrained backbone and concentrate MoE capacity in the projection or feed-forward sublayers (Lin et al., 9 Jun 2026, Pandey et al., 27 Jan 2026, Wu et al., 2024). Stable optimization is not ancillary but central: sparsity, balance, z-loss regularization, multi-task side objectives, or dense gradient flow mechanisms are repeatedly used to keep expert specialization from degenerating (You et al., 2021, Cappellazzo et al., 20 May 2025, Pandey et al., 27 Jan 2026). In that sense, the MoE speech projector is not merely a larger projector; it is a conditional interface whose effectiveness depends on how routing, alignment, and specialization are jointly constrained.