---
title: Phoneme-Guided Mixture-of-Experts (PG-MoE)
url: https://www.emergentmind.com/topics/phoneme-guided-mixture-of-experts-pg-moe
type: topic
---

# Phoneme-Guided Mixture-of-Experts (PG-MoE)

Searching arXiv for the cited papers and closely related work on phoneme-guided MoE.
Phoneme-Guided Mixture-of-Experts (PG-MoE) denotes a class of speech architectures in which phonetic or phoneme-derived cues govern the routing of representations through specialized expert subnetworks. In automatic speech recognition, the idea appears in MixNet, which augments a conventional DNN-HMM or LSTM-HMM acoustic model with an input-side MoE keyed to broad phonetic classes and a penultimate MoE keyed to automatically learned acoustic classes [2112.01025]. In text-to-speech, it appears as a dialect-aware MoE driven by IPA embeddings and inserted after the text embedding layer so that dialect-specific phonological realizations are preserved rather than averaged away [2509.22727]. A closely related precursor is Deep Mixture of Experts for speech enhancement, where speech variability mainly caused by phoneme structure motivates expert specialization even though the gating is learned in an unsupervised manner and no phoneme-labeled database is required [1703.09302].

## 1. Conceptual definition

PG-MoE combines two ideas: expert specialization and phonological conditioning. The expert component supplies multiple region-specific transforms or subnetworks; the phoneme-guided component supplies routing information derived from broad phonetic classes, IPA embeddings, or other phoneme-sensitive representations. The immediate objective is not merely parameter scaling, but improved handling of structured heterogeneity in speech representations: overlap among acoustic classes in ASR, phonological differences among dialects in TTS, or speech sub-regimes in enhancement [2112.01025].

In the ASR formulation, the guiding signal is partially supervised at the network input: the first MoE layer is based on pre-defined broad phonetic classes, specifically voiced, unvoiced, and silence. The second MoE is not pre-defined; it is based on automatically learned acoustic classes. This division makes PG-MoE a hybrid of explicit phonetic structure and latent acoustic partitioning. In the TTS formulation, the guiding signal is an IPA sequence processed by a learned gating mechanism; the purpose is to preserve dialect-specific realizations and avoid style homogenization across jointly trained dialects [2509.22727].

A plausible implication is that PG-MoE is best understood not as a single fixed architecture but as a routing principle: phonologically informative features determine how much each expert contributes to the transformed representation. The exact granularity of routing varies across tasks, from frame-wise soft assignments in ASR to one gating decision per sequence in the dialect TTS setting.

## 2. Architectural pattern

Across the cited systems, the recurrent structural motif is a standard speech model into which a comparatively small MoE module is inserted at a strategically chosen layer. In MixNet, the first MoE sits immediately after the LDA-normalized or MFCC input, while a second MoE is inserted just before the final softmax. The input MoE uses $C_a$ experts and a gating network that produces frame-wise soft assignments $\alpha_1(t),\dots,\alpha_{C_a}(t)$; the output MoE uses $C_b$ experts and a softmax gate that produces $\beta_1(t),\dots,\beta_{C_b}(t)$ [2112.01025].

The input-side gating computation is
$$
\alpha(t)=\mathrm{softmax}(W_g x(t)+b_g),
$$
and the resulting transformed feature is
$$
y(t)=\sum_{i=1}^{C_a}\alpha_i(t)\sum_{j=-K}^{K}(A_{i,j}x(t+j)+b_{i,j}),
$$
with $K=1$ in the paper. The penultimate MoE computes
$$
z(t)=\sum_{i=1}^{C_b}\beta_i(t)\,(B_i\,y^{L-1}(t)+b_i),
$$
and this $z(t)$ feeds the final softmax over HMM states. All experts are purely affine, so the nonlinearity in the MoE layers arises from the gating softmax rather than from nonlinear expert blocks [2112.01025].

In DiaMoE-TTS, the integration point is the frontend of F5-TTS. Let $p_{\mathrm{IPA}}=(p_1,\dots,p_T)$ be the input IPA sequence, and let the text encoder map each $p_t$ to $h_t$, collecting $H=[h_1;\dots;h_T]\in\mathbb{R}^{T\times D}$. The gating network first mean-pools:
$$
s=\frac{1}{T}\sum_{t=1}^{T} h_t,
$$
then projects to logits,
$$
z=W_g s+b_g,
$$
and produces mixture weights
$$
g=\mathrm{softmax}(z/\tau), \qquad \sum_{i=1}^{K} g_i=1.
$$
Each expert $E_i$ is a small feed-forward block, and the fused residual is
$$
\mathrm{MoE}(H)_t=\sum_{i=1}^{K} g_i\,E_i(h_t), \qquad H' = H + \mathrm{MoE}(H).
$$
Here the routing decision is sequence-level rather than frame-level, and the residual integration is used to stabilize training [2509.22727].

These designs share a common logic: routing is lightweight, expert computation is specialized, and the MoE output is fused back into the main model rather than replacing it wholesale.

## 3. MixNet as a phoneme-guided MoE for ASR

MixNet is a deep acoustic model for ASR that adds two MoE layers to an otherwise conventional DNN-HMM or LSTM-HMM. The first MoE layer is explicitly phoneme-guided. The authors settle on three interpretable classes at the input: voiced, unvoiced, and silence. The gating network is an auxiliary 3-layer ReLU DNN ending in a linear-plus-softmax layer and is trained to predict these broad phonetic classes from the central frame plus short context [2112.01025].

The expert layer at the input operates on the LDA-normalized MFCC representation. In the full-rank version, with dimension 143 after splicing $\pm 1$, the matrices satisfy $A_{i,j}\in\mathbb{R}^{143\times 143}$. The purpose of this first MoE is to split the raw acoustic feature space into broad phonetic regions before the signal enters the usual stack of fully connected or LSTM hidden layers. The second MoE, inserted at the penultimate layer, operates on the last hidden representation, for example a 1024-dimensional vector, and uses either low-rank factorizations $(1024\times 512)$ or diagonal-plus-off-diagonal blocks to control parameter growth [2112.01025].

The acoustic motivation is stated directly: in natural speech, overlap in distribution across different acoustic classes is inevitable, which leads to inter-class mis-classification. MixNet is developed to account for such overlaps. The paper reports that scatter-diagram analysis verifies that the MoE improves separation between classes and that this translates to better ASR accuracy. In the t-SNE visualizations, raw features colored by voiced, unvoiced, and silence show heavy overlap, whereas after the first MoE transform the three clusters separate cleanly, and the non-speech sil cluster becomes its own tight group [2112.01025].

The reported ASR gains are sequence-discriminative. For the DNN-HMM baseline with 8 fully connected layers and cross-entropy plus sMBR, the word error rate is 14.7%. MixNet-IV with a 3-class input MoE and a 5-class output MoE plus sMBR attains 12.7% WER, corresponding to a 13.6% relative reduction from the baseline. For the LSTM-HMM baseline with 5 layers plus sMBR, the WER is 14.0%, while MixNet-IV LSTM plus sMBR attains 12.6%, corresponding to a 10.0% relative reduction. Against Eigen-DMoE, described as using two output-only MoEs, MixNet gives roughly 1% absolute better WER [2112.01025].

## 4. Training procedures and objective functions

The training workflow in MixNet is staged. First, the auxiliary phonetic-class DNN used by the input MoE gate is trained with cross-entropy to 94% frame accuracy on the three broad classes, and then frozen. Second, the full MixNet, including both MoE layers and the main hidden layers, is trained end-to-end with standard cross-entropy on senone targets. Third, a boosted Minimum Bayes-Risk criterion is applied for sequence-discriminative refinement [2112.01025].

The sMBR objective is given in the usual form. For one utterance $O$ with reference path $r$,
$$
L_{\mathrm{sMBR}}(\theta) = - \sum_{p} P(p \mid O;\theta)\,A(p,r),
$$
normalized by the sum of all path posteriors. In the expanded form across utterances,
$$
L_{\mathrm{sMBR}}(\theta) = - \sum_{u}\sum_{p\in \mathrm{Paths}} P(p\mid O_u;\theta)\,A(p,r_u),
$$
where $A(p,r_u)$ measures alignment accuracy of path $p$ to the reference $r_u$ for utterance $u$. The paper states that minimizing this objective pushes probability mass onto the correct path [2112.01025].

The related DMoE formulation for speech enhancement uses a different joint objective but exhibits the same general principle of end-to-end optimization over gate and experts. Each expert estimates a speech-presence probability vector from a context window of noisy log-spectral frames, while the gating network takes an $L$-dimensional MFCC vector from the same frame. The final speech-presence probability is a gating-weighted average of the expert outputs. The model is trained by maximizing a joint DMoE log-likelihood over all network parameters, and no phoneme labels are ever used; the gating network clusters frames into regimes in an entirely unsupervised way, driven by the maximum-likelihood objective [1703.09302].

A plausible implication is that PG-MoE does not require a unique training recipe. In the cited literature, one finds supervised gate pretraining and frozen routing at the input in ASR, unsupervised joint clustering in enhancement, and auxiliary dialect classification in TTS.

## 5. Extension to dialect TTS

DiaMoE-TTS presents an explicit PG-MoE formulation for dialect text-to-speech. The motivation is that Chinese dialects exhibit large differences in segmental inventories, tone realization, and syllable structure, while a single shared text encoder tends to average out these patterns, leading to blurred or inconsistent pronunciations under joint training. The framework therefore standardizes phonetic representations with IPA and inserts a dialect-aware MoE immediately after the text embedding layer in the F5-TTS frontend [2509.22727].

The number of experts $K$ is set to the number of dialects trained jointly, for example $K=8$ or $10$ in the multilingual setup. Each expert is a 2-layer MLP with GELU or ReLU, hidden dimension $D_{ff}$, optionally with LayerNorm and dropout between layers, and outputs a residual of dimension $D$. The gating network computes one routing weight vector per utterance by mean-pooling the IPA embedding sequence and applying a linear projection followed by softmax, with temperature $\tau=1.0$ in the reported runs. Hidden dimension is $D=256$, and the expert MLP dimension is $D_{ff}=4\cdot D$ [2509.22727].

Training is organized into three stages. Stage 1 jointly trains on Mandarin plus dialect IPA data without the MoE loss, setting $\lambda=0$. Stage 2 enables PG-MoE and optimizes
$$
L_{\mathrm{total}} = L_{\mathrm{OT\text{-}CFM}} + \lambda \cdot L_{\mathrm{dialect}},
$$
with $\lambda=0.1$, where
$$
L_{\mathrm{dialect}} = -\frac{1}{N}\sum_{n=1}^{N}\log \mathrm{softmax}(z^{(n)})_{y_n}.
$$
Stage 3 adapts to new low-resource dialects by freezing the MoE and fine-tuning only LoRA adapters and conditioning adapters in the TTS backbone. Optimization uses AdamW, with peak learning rate $7.5\mathrm{e}{-5}$ in Stages 1 and 2, and $1\mathrm{e}{-5}$ for adapters in Stage 3. The paper states that no additional load-balancing terms are used beyond the dialect-classification loss, because this cross-entropy term sufficiently encourages expert specialization [2509.22727].

The central claim is architectural rather than purely numeric: explicit phonemes plus dialect routing are necessary to model dialect-specific pronunciation. The paper further states that the IPA front-end is essential for correct expert routing and pronunciation modeling.

## 6. Empirical effects, interpretation, and points of caution

The empirical effects reported for PG-MoE are task-specific but directionally consistent in two of the three settings. In ASR, the stated effect is improved class separation and lower WER relative to strong DNN and LSTM baselines. In TTS, the stated effect is the preservation of dialect-specific realizations and improved WER and MOS relative to non-MoE or non-IPA alternatives. In speech enhancement, the reported effect is that an unsupervised DMoE with $m=2$ consistently outperformed a single-expert DNN and state-of-the-art OMLSA, and matched or slightly exceeded a supervised phoneme-DMoE with 39 experts, despite never using phoneme labels in training [1703.09302].

For DiaMoE-TTS, the reported comparison to non-MoE and pinyin baselines is sharp. The paper states that “w/o MoE but with IPA” is strong but less dialect-specific, with higher WER and lower MOS; “with MoE + pinyin” yields catastrophically high WER above 90% and low MOS below 1.3 due to orthography ambiguity; and “MoE + IPA” strikes the best balance. It also states that, across dialects, enabling MoE+IPA consistently reduces WER and raises MOS, with examples including Xi’an from 41.09% to 33.00%, Zhengzhou from 41.54% to 29.59%, and Shijiazhuang from 49.38% to 36.83%, while MOS rises by 0.3–0.8 points [2509.22727].

At the same time, the provided DiaMoE-TTS details contain a Chengdu ablation entry whose signs are internally inconsistent with the surrounding summary: the WER is listed as increasing from 45.08% to 49.01%, and the MOS line is written as “2.46 → 2.22” while marked with an upward arrow. The surrounding bullets nevertheless state that enabling MoE+IPA consistently reduces WER and raises MOS across dialects and that the IPA front-end is essential. The most cautious reading is therefore that the broad conclusion is positive, but one reported Chengdu line is not self-consistent in the supplied summary [2509.22727].

A common misconception is to equate PG-MoE with hard dispatch to a single expert. The cited architectures do not require this. MixNet uses frame-wise soft assignments $\alpha_i(t)$ and $\beta_i(t)$ that sum to one, and DiaMoE-TTS uses a full softmax over experts for each utterance; top-1 sparse gating is mentioned only as something that could be used in very large-$K$ regimes, not as the reported default. Another misconception is that phoneme guidance must require explicit phoneme labels. The enhancement DMoE shows a contrasting case in which speech variability mainly caused by phoneme structure is exploited without any phoneme-labeled database, because the gating network clusters frames in an unsupervised maximum-likelihood sense [2112.01025].

Taken together, these results suggest that PG-MoE is most effective when the routing signal is sufficiently phoneme-sensitive to reflect real substructure in the speech domain being modeled. In the cited ASR system this means voiced, unvoiced, and silence at the input plus learned acoustic classes at the penultimate layer; in the cited TTS system it means IPA-conditioned dialect routing; and in the cited enhancement system it means unsupervised discovery of a small number of speech regimes that behave similarly to broad phonetic partitions.

Source: https://www.emergentmind.com/topics/phoneme-guided-mixture-of-experts-pg-moe