DiaMoE-TTS: Multidialect TTS Framework

• The paper introduces DiaMoE-TTS, a multidialect TTS framework that standardizes input through the International Phonetic Alphabet (IPA) and integrates a dialect-aware Mixture-of-Experts module.
• It leverages residual MoE integration and parameter-efficient transfer via LoRA and Conditioning Adapters to adapt a pretrained TTS backbone to new low-resource dialects.
• Empirical evaluations reveal competitive results in MOS and WER across dialects, underscoring both the potential and limitations of the system’s design for multidialect synthesis.

Searching arXiv for the cited paper and closely related TTS/MoE work to ground the article. DiaMoE-TTS is a multidialect text-to-speech framework that standardizes text input with the International Phonetic Alphabet (IPA), inserts a dialect-aware Mixture-of-Experts (MoE) module into the text representation path, and uses parameter-efficient adaptation to extend a pretrained zero-shot TTS backbone to new low-resource dialects. It is introduced as an open, unified framework for dialect speech synthesis in settings characterized by scarce data, inconsistent orthographies, grapheme-to-phoneme ambiguity, and phonological variation across dialects. The system is built on F5-TTS, retaining its non-autoregressive generation framework, ConvNeXt V2-based text encoding machinery, and DiT-based spectrogram generation process, while adding an IPA frontend, a residual dialect-style MoE, and Stage-3 adaptation with LoRA and Conditioning Adapters (Chen et al., 25 Sep 2025).

1. Problem setting and design rationale

DiaMoE-TTS targets a specific failure mode of conventional multidialect Chinese TTS: written characters do not explicitly resolve pronunciation, pinyin is Mandarin-centric, and joint training over multiple dialects tends to average away dialect-specific phonology and style. The framework therefore replaces character- or pinyin-based conditioning with a shared IPA input space. In the paper’s formulation, all dialects are mapped into a unified IPA phoneme space, so that Mandarin and multiple dialects can be modeled in a harmonized text–IPA–speech alignment framework. The underlying claim is not merely that IPA is linguistically cleaner, but that it directly addresses grapheme-to-phoneme ambiguity and provides a consistent frontend for multidialect training.

At a systems level, the method is designed for two coupled tasks. The first is multidialect training over Mandarin and seen dialects under a shared frontend. The second is rapid transfer to a previously unseen low-resource dialect when only a few hours of speech are available. The paper therefore combines a shared phonetic inventory with dialect-specialized representation learning and parameter-efficient transfer rather than relying on full-model fine-tuning.

A central conceptual distinction is between phonetic standardization and dialect specialization. IPA is used to normalize representation across dialects, while the MoE is introduced because a single shared encoder is said to wash out dialect style during joint training. This suggests a two-level modeling strategy: a common phonetic substrate plus dialect-conditioned specialization in how that substrate is realized.

2. Architecture and formalization

The DiaMoE-TTS pipeline begins with dialect text, converts it into an IPA phoneme sequence $p_{\text{IPA}}$, encodes that sequence using the inherited F5-TTS text path, augments the resulting text representation with a dialect-aware MoE, and then synthesizes speech with the original F5-TTS acoustic generation backbone. The paper writes the text-side transformation as

$$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$

Here $h$ is the original text representation and $h'$ is the dialect-aware representation after residual MoE integration.

The MoE is the main architectural novelty. It is inserted after text embedding and before the rest of the shared TTS backbone. Its gate is computed from a mean-pooled text representation,

$$s = \frac{1}{T}\sum_{t=1}^{T} h_t \in \mathbb{R}^D, \qquad g = \mathrm{Gate}(s) \in \mathbb{R}^K,$$

where $T$ is sequence length, $D$ is hidden size, and $K$ corresponds to the number of experts and appears to match the number of dialects. The paper states that the gate is a linear projection followed by optional activation, but does not specify the activation, whether routing is sample-wise or token-wise, whether top-$k$ routing is used, or the exact sparse aggregation equation. It also claims sparse activation for efficiency, but does not formalize the routing rule (Chen et al., 25 Sep 2025).

To make routing dialect-sensitive, the gate logits are supervised by an auxiliary dialect classification loss: $$\mathcal{L}_{\text{dialect}} = -\frac{1}{N}\sum_{i=1}^{N}\log \frac{\exp(\mathbf{g}_{i,y_i})}{\sum_{j=1}^{K}\exp(\mathbf{g}_{i,j})}.$$ The Stage-2 objective is

$$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$0

with $$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$1 in Stage 2 and $$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$2 elsewhere. No load-balancing loss, entropy regularization, expert-capacity constraint, auxiliary dispatch loss, or router $$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$3-loss is reported.

The architecture leaves several interfaces deliberately underspecified. The paper states that the inherited F5-TTS setup supports reference-speaker conditioning from a reference audio, its transcript, and target text, but does not formalize speaker embedding extraction, prompt encoding, style tokens, prosody controls, or language tags. Likewise, Conditioning Adapters are said to be incorporated into the text embedding module during adaptation, but their architecture and input-output interface are not specified.

Component	Role	Reported detail
IPA frontend	Unified phonetic input	Replaces characters or pinyin
Dialect-aware MoE	Dialect specialization in text representation	Inserted after text embedding
F5-TTS backbone	Acoustic generation	OT-CFM, ConvNeXt V2 text path, DiT spectrogram generation
LoRA	PEFT for new dialects	Rank $$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$4, $$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$5
Conditioning Adapters	Low-resource adaptation	Inserted into text embedding module

3. Training stages and adaptation regime

The training procedure is staged. Stage 0 initializes from a pretrained F5-TTS checkpoint trained on large-scale Mandarin and English data. Stage 1 performs joint training on a unified multidialect IPA dataset covering Mandarin and multiple dialects. Stage 2 continues multidialect IPA training but adds the MoE after text embedding and activates the auxiliary dialect classification loss. Stage 3 adapts the multidialect model to a new low-resource dialect by freezing the backbone and MoE and training only LoRA and Conditioning Adapters, together with simple acoustic augmentation. The overview says the framework is trained in four stages, but only Stages 0–3 are described; this suggests initialization is being counted as one stage (Chen et al., 25 Sep 2025).

The Stage-3 adaptation protocol is intentionally parameter-efficient. LoRA is applied to the query and value projections in the attention layers of both the input embedding and the DiT blocks. The paper reports rank $$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$6 and $$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$7. The Conditioning Adapter is incorporated into the text embedding module, but no bottleneck size, nonlinearity, or insertion depth is given. The main explicit efficiency claim is qualitative rather than quantitative: only LoRA and Conditioning Adapters are updated while the main model remains frozen.

The paper’s use of the phrase “zero-shot” requires care. The framework is said to achieve zero-shot performance on unseen dialects with only a few hours of data, but the Stage-3 protocol is adaptation on the target dialect using a few hours of labeled speech. Strictly construed, this is parameter-efficient few-shot adaptation rather than zero-shot inference in the no-target-data sense. The paper itself contrasts this continual-learning-style procedure with direct fine-tuning, which it says causes substantial forgetting, and keeps the MoE frozen during Stage 3 to preserve previously learned routing and avoid overfitting.

Acoustic augmentation is simple but explicit. For each original utterance, pitch and time-scale modifications are applied with factors $$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$8 and $$h = E_{\text{text}}(p_{\text{IPA}}), \qquad h' = \operatorname{MoE}(h) + h.$$9. Optimization uses AdamW throughout. In Stages 1 and 2, the peak learning rate is $h$0, with 2k warmup steps and linear decay over 200k total steps; batch size is 28k frames per GPU. In Stage 3, the learning rate is $h$1, training lasts 100k steps, and only LoRA plus Conditioning Adapters are updated.

Stage	Operation	Reported settings
Stage 0	Initialize from pretrained F5-TTS	Mandarin and English checkpoint
Stage 1	Joint multidialect IPA training	AdamW, $h$2, 200k steps
Stage 2	Add MoE and dialect loss	$h$3
Stage 3	PEFT adaptation to new dialect	LoRA + Conditioning Adapters, $h$4, 100k steps

4. Data, evaluation, and empirical profile

The multidialect training corpus combines Common Voice Cantonese, the Emilia Mandarin dataset, dialect data from KeSpeech, an open-source Southern Min dataset, and commercially acquired Shanghai and Tianjin dialect speech. The overall scale for Stages 1–2 is summarized as about 0.7k hours of Mandarin plus 0.4k hours of dialect data. For low-resource adaptation, the paper uses approximately 3 hours of professionally recorded Peking Opera recitation covering both Jingbai and Yunbai pronunciation styles, and an equal amount of Nanjing dialect data for comparison. The authors explicitly contrast this scale with systems such as CosyVoice2 at $h$5k hours and Qwen-TTS at $h$6k hours.

Evaluation uses WER with FireRedASR, UTMOSv2, and human MOS. Compared with commercial baselines including Edge TTS, CosyVoice2, and Qwen-TTS, the reported results are mixed. DiaMoE-TTS is competitive in some conditions and covers more dialects and adaptation settings, but it generally trails strong proprietary systems in WER and often in MOS. The reported DiaMoE-TTS scores are: YUE 76.59 / 3.03 / 2.65, SH 76.53 / 1.80 / 2.75, CD 37.12 / 2.22 / 3.33, XA 33.00 / 3.15 / 2.87, ZZ 29.59 / 2.97 / 2.86, TJ 20.18 / 1.66 / 3.10, NAN 92.41 / 2.16 / 3.10, SJZ 36.83 / 2.67 / 2.56, NJ 31.64 / 1.75 / 3.28, Jingbai 39.95 / 3.04 / 1.75, and Yunbai 68.94 / 2.84 / 1.94, in WER / MOS / UTMOSv2 order. The paper notes that DiaMoE-TTS has better UTMOSv2 in YUE and CD than the average of available commercial baselines, but worse MOS in many dialects and much worse WER almost everywhere those baselines are available (Chen et al., 25 Sep 2025).

The strongest evidence for the method’s internal design comes from ablation. Replacing IPA with pinyin while keeping MoE causes performance collapse, with MOS around 1.0–1.23 and WER roughly 90% across Chengdu, Xi’an, Zhengzhou, and Shijiazhuang. This is the clearest experimental support for the paper’s central argument that explicit IPA input is essential for multidialect Chinese TTS. The effect of MoE is more uneven. Relative to an IPA no-MoE baseline, MoE improves MOS and WER in Xi’an, Zhengzhou, and Shijiazhuang, but Chengdu is a counterexample: MOS is 2.22 with MoE versus 2.46 without, and WER is 49.01 with MoE versus 45.08 without. The MoE therefore appears beneficial in most, but not all, ablation settings.

The low-resource adaptation results are most notable in Nanjing and Peking Opera. With only about 3 hours of data, DiaMoE-TTS reports MOS 3.04 for Jingbai and 2.84 for Yunbai. The paper presents this as evidence that the framework extends beyond ordinary conversational dialects to specialized phonological styles. At the same time, WER 39.95 and 68.94 and UTMOSv2 1.75 and 1.94 indicate that these domains remain difficult. The experimental record therefore supports feasibility rather than across-the-board state-of-the-art performance.

5. Interpretive issues, misconceptions, and limitations

Several interpretive caveats are important for accurate classification of DiaMoE-TTS. The first concerns “zero-shot” terminology. Because Stage 3 uses a few hours of target-dialect data and fine-tunes PEFT modules, the regime is better described as low-resource or few-shot adaptation than as strict zero-shot transfer. The second concerns the MoE claim. The paper states that adding MoE will achieve better results, but the ablation evidence is not uniform: Chengdu degrades on both MOS and WER. A more precise reading is that MoE often helps preserve dialect-specific phonological style under joint training, but does not guarantee gains for every dialect-metric pair (Chen et al., 25 Sep 2025).

A third limitation is frontend dependence. The pinyin ablation fails badly, which strengthens the case for IPA but also implies sensitivity to the quality of IPA conversion and lexicon construction. The paper states that it constructs a pronunciation lexicon and aligned text–IPA–speech pairs from open-source corpora, but does not specify the conversion algorithm, tokenizer, lexicon rules, or whether mapping is rule-based, lexicon-based, or manually curated. Because the frontend is demonstrably critical, those omissions matter.

A fourth issue is architectural under-specification. The exact expert architecture, hidden size, number of layers per expert, top-$h$7 routing rule, gate activation, and sparse aggregation equation are all omitted. The system claims sparse activation for efficiency but does not quantify trainable parameter counts, memory savings, inference speed, latency, GPU type, or wall-clock cost. Similarly, Conditioning Adapters are described functionally rather than structurally.

The evaluation scope is also narrow relative to the claims one might want from a dialect synthesis system. The paper does not report CMOS, speaker similarity, dialect classification accuracy on generated speech, CER, F0 RMSE, duration or prosody correlation, or a dedicated pronunciation error rate beyond WER. Nor does it provide analyses of routing entropy, expert specialization, or expert-to-phonology correspondence. A plausible implication is that the framework’s strongest contributions are infrastructural and methodological rather than exhaustive analytical characterization.

6. Position within adjacent TTS research

DiaMoE-TTS sits at the intersection of low-resource dialect TTS, MoE-based specialization, and parameter-efficient transfer. Relative to other MoE-based TTS work, its MoE is dialect-aware and inserted after text embedding, with routing supervised by an auxiliary dialect classification loss. This differs from the modality-based routing in "MoE-TTS: Enhancing Out-of-Domain Text Understanding for Description-based TTS via Mixture-of-Experts," where text tokens use frozen text experts and speech tokens use trainable speech experts under deterministic modality routing (Xue et al., 15 Aug 2025). DiaMoE-TTS also differs from "Lightweight Zero-shot Text-to-Speech with Mixture of Adapters," which uses speaker-conditioned mixtures of adapters in a lightweight FastSpeech2-style system to improve zero-shot speaker adaptation; DiaMoE-TTS instead places a residual dialect-style MoE on text representations and performs adaptation with LoRA plus Conditioning Adapters on top of F5-TTS (Fujita et al., 2024).

Its scope is also distinct from recent emotion-control TTS. "MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt" focuses on multi-modal emotion prompting and disentangled control of content, timbre, emotion, and prosody for expressive zero-shot synthesis rather than dialect phonology (Wu et al., 24 May 2025). "DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speech" targets speaker-independent emotion embeddings for cross-speaker emotion transfer, not multidialect frontend standardization (Cho et al., 26 May 2025). "EMORL-TTS: Reinforcement Learning for Fine-Grained Emotion Control in LLM-based TTS" treats emotion as a controllable sequence-generation problem over discrete speech tokens and emphasizes VAD-based intensity plus emphasis control (Li et al., 7 Oct 2025). These contrasts help delimit DiaMoE-TTS: its novelty is not in fine-grained affect control, multi-modal prompting, or description-based TTS, but in making open, low-resource dialect synthesis practical through a unified IPA frontend, dialect-sensitive expertization, and PEFT-based transfer.

Within that landscape, the main significance of DiaMoE-TTS is methodological. The evidence indicates that IPA is essential in this setting, that dialect-aware expertization can reduce style averaging in joint training, and that freezing the backbone plus adapting only LoRA and Conditioning Adapters can extend the system to new dialects with only a few hours of data. At the same time, the paper’s own results make clear that broader dialect coverage, higher-quality data, fuller analysis of routing behavior, and stronger intelligibility remain open problems.