LEMAS-TTS: Zero-Shot Multilingual TTS Benchmark

• LEMAS-TTS is a benchmark model integrating a 150K-hour multilingual corpus with word-level timestamps to enable zero-shot multilingual synthesis while preserving speaker identity.
• It uses a non-autoregressive flow-matching system with a diffusion transformer, explicit pause tokens, CTC alignment, and GRL-based accent-adversarial training to enhance stability and intelligibility.
• Empirical results demonstrate lower WER and higher speaker similarity across ten languages, establishing LEMAS-TTS as a strong baseline for multilingual TTS research.

Searching arXiv for LEMAS-TTS and directly related multilingual TTS benchmark papers. LEMAS-TTS is the text-to-speech benchmark model introduced alongside LEMAS-Dataset, a 150,000-hour multilingual corpus with word-level timestamps and confidence scores, to validate large-scale prompt-based speech generation. It is built on the F5-TTS family as a non-autoregressive flow-matching system with a Diffusion Transformer backbone, and it targets zero-shot multilingual synthesis: given text and a short reference clip, it generates speech in a target language while preserving speaker identity and natural pronunciation. Within the LEMAS framework, the model is defined not only by its backbone but also by a coordinated set of data, alignment, and conditioning choices intended to improve cross-lingual stability (Zhao et al., 4 Jan 2026).

1. Position within the LEMAS framework

LEMAS-TTS is one of the two benchmark models used to validate the utility of LEMAS-Dataset, the other being LEMAS-Edit for speech editing. The dataset is presented as, to the authors’ knowledge, the largest open-source multilingual speech corpus with word-level timestamps, and the benchmark models are intended to show that this scale and annotation granularity support both synthesis and editing tasks (Zhao et al., 4 Jan 2026).

In this setting, LEMAS-TTS functions as the synthesis-oriented half of a broader prompt-based speech generation program. Its explicit task is zero-shot multilingual TTS rather than closed-set single-language synthesis: the model takes a short reference clip for speaker conditioning and a target-language text sequence, then attempts to synthesize speech with preserved speaker identity and stable pronunciation. The paper frames this as evidence that large-scale multilingual TTS is constrained not primarily by model architecture alone, but by data scale, alignment quality, and cross-lingual stabilization.

A common simplification is to treat LEMAS-TTS as merely another flow-matching TTS system. That characterization is incomplete. In the LEMAS paper, the model is inseparable from the dataset design, timestamp annotations, and auxiliary regularization mechanisms that are used to counter accent leakage and alignment drift. This suggests that LEMAS-TTS should be understood as a benchmarked system design, not only as a backbone choice.

2. Data substrate and multilingual text representation

LEMAS-Dataset covers ten major languages: Chinese, English, Russian, Spanish, Portuguese, German, French, Italian, Indonesian, and Vietnamese. For LEMAS-TTS, the dataset’s scale provides multilingual phonotactics, speaker variation, and cross-lingual transfer, while the word-level timestamps provide a fine-grained temporal scaffold for text-to-speech training (Zhao et al., 4 Jan 2026).

The multilingual front-end maps heterogeneous orthographies into a shared phonetic space. Chinese is represented with tonal Pinyin decomposed into initial-final structure, whereas the other languages are converted into IPA using eSpeak-NG. Language tags such as <zh> and <en> are prepended so that the model can distinguish languages while still operating in a normalized phonetic space. This reduces the burden on the generative model relative to direct mixed-script conditioning.

The word-level timestamps are converted into explicit pause tokens, #1 through #4, corresponding to short, medium, long, and abnormal pauses. These pause tags are inserted into the phoneme sequence and provide explicit prosodic timing cues. In the paper’s account, this improves intelligibility and rhythmic stability by making temporal structure visible to the model rather than leaving pause realization entirely implicit.

This design makes LEMAS-TTS unusual among multilingual TTS systems in that alignment metadata is promoted from a corpus annotation artifact to a modeling primitive. A plausible implication is that the system’s robustness depends on the interaction between phonetic normalization and timestamp-derived temporal structure at least as much as on the capacity of the DiT backbone itself.

3. Core architecture and conditioning pathways

Architecturally, LEMAS-TTS is a non-autoregressive flow-matching model with a Diffusion Transformer backbone derived from the F5-TTS family. The paper contrasts this with classical autoregressive systems such as Tacotron-style models, which decode left-to-right and can suffer from slow inference and error accumulation. By contrast, flow matching learns a continuous transformation from noise to speech and avoids the sequential exposure bias that can make autoregressive systems brittle under long utterances, unfamiliar languages, or mismatched accents (Zhao et al., 4 Jan 2026).

The conditioning stack is multimodal. Text conditioning is supplied through the shared phonetic representation and language tags. Speaker conditioning comes from the short reference clip used for zero-shot synthesis. In addition, the paper introduces a prosody encoder adapted from Seamless Expressive. It extracts 80-bin filterbank features from sub-utterances, encodes them with an ECAPA-TDNN backbone into a 512-dimensional embedding, and injects that embedding into the DiT through linear projections.

The paper reports that speaker contrastive learning and clip-and-shuffle augmentation were explored but then dropped because they hurt stability: strict speaker invariance conflicted with the flow-matching objective, and chunk shuffling disrupted temporal coherence. That negative result is informative because it narrows the intended role of the conditioning pathway. LEMAS-TTS does not attempt maximal disentanglement; instead, it prioritizes a stable compromise among speaker preservation, pronunciation fidelity, and multilingual robustness.

The resulting architecture targets two recurrent failure modes in multilingual zero-shot TTS. The first is cross-lingual accent leakage, in which dominant-language prosody or accent patterns bleed into the target language. The second is alignment drift, which manifests as repetitions, duration errors, and mispronunciations. The rest of the system design is largely organized around these two failure modes.

4. Stabilization objectives and inference strategy

LEMAS-TTS adds two auxiliary objectives on top of the flow-matching loss: a CTC alignment loss and an accent-adversarial loss. The CTC term is applied through a lightweight projection head that maps predicted mel-spectrograms to phone sequences, with the usual objective $$\mathcal{L}_{\text{CTC}} = -\log p_{\theta}(x \mid \hat{y})$$. In the paper’s description, this term encourages monotonic alignment and reduces drift during multilingual synthesis (Zhao et al., 4 Jan 2026).

The accent-adversarial component uses a Gradient Reversal Layer. An accent classifier is attached to the conditioning pathway, and pseudo-labels are produced by an off-the-shelf language identification model. The classifier is trained to predict accent or language, while the encoder is adversarially trained through the GRL to make those predictions difficult. The intended effect is accent-invariant conditioning that preserves speaker identity and linguistic content without leaking a dominant source-language accent into the target utterance.

At inference time, the model uses dynamic classifier-free guidance and a redesigned sway-sampling schedule. The guidance scale is time-dependent, with $g(t)=\lambda(1-t)^2$, so guidance is strongest early in generation and then decays. The sway-sampling schedule reparameterizes the uniform grid as $t_k=s_k^{1+\gamma}$, redistributing sampling resolution across the trajectory. According to the paper, these changes improve perceptual stability, particularly for long multilingual utterances.

Taken together, these mechanisms show that LEMAS-TTS is not simply a base flow-matching model trained on more data. Its stabilization strategy couples explicit monotonicity pressure, adversarial accent suppression, guided sampling, and timestamp-driven pause modeling. This combination is the central methodological claim of the system.

5. Empirical performance and benchmark status

LEMAS-TTS is evaluated on all ten languages against OpenAudio-S1-mini, with WER and speaker similarity as the primary metrics. WER is computed using FunASR Paraformer-zh for Chinese and Whisper-large-v3 for the other languages, while speaker similarity is measured with WavLM-large following the Seed-TTS protocol. The reported average WER decreases from 12.27% for OpenAudio-S1-mini to 8.06% for LEMAS-TTS without prosody and to 6.39% with prosody control. Average speaker similarity improves from 0.480 to 0.547 and 0.539, respectively (Zhao et al., 4 Jan 2026).

The ablation between the prosody-free and prosody-aware variants is nontrivial. Adding the prosody encoder consistently lowers WER, indicating improved pronunciation stability and articulation, but it can slightly reduce similarity in some cases. The paper interprets this as a tradeoff between explicit prosodic conditioning and timbral matching, and it therefore releases both variants. One notable result is that low-resource languages such as Indonesian and Vietnamese show particularly large gains relative to OpenAudio-S1-mini. The paper also notes that Vietnamese WER for the baseline is unusually unstable and excludes it from the average WER computation for fairness.

Later work has treated LEMAS-TTS as a strong multilingual non-autoregressive baseline. The X-Voice paper describes it as a flow-matching based non-autoregressive model trained on 150K hours of MMS force-aligned data and specifically uses the GRL version with accent-adversarial training and CTC loss. In that comparison, X-Voice reports reductions in WER scores for most languages and consistent improvements in SIM-o on the LEMAS-TTS test set, positioning LEMAS-TTS as a serious but surpassable reference point in multilingual zero-shot voice cloning (Xu et al., 7 May 2026).

This benchmark role is important. LEMAS-TTS is not presented in subsequent literature as an obsolete baseline, but as a representative phoneme-based multilingual NAR system whose design choices—especially force-aligned supervision, GRL-based accent mitigation, and CTC regularization—define a meaningful comparison target.

6. Limitations, interpretation, and research significance

The paper identifies several limitations. It does not provide MOS-style subjective evaluation for all languages because native evaluators were unavailable beyond Chinese and English, so the main evaluation remains objective. The model still depends on a high-quality reference clip for zero-shot speaker conditioning, which means it is not a prompt-free generator. The accent-adversarial labels are pseudo-labeled rather than gold-standard. Finally, despite the size of the dataset, language coverage is limited to ten languages and remains imbalanced in utterance length and source domain (Zhao et al., 4 Jan 2026).

These limitations matter for interpretation. LEMAS-TTS demonstrates that multilingual zero-shot synthesis can be stabilized with large-scale aligned data and auxiliary objectives, but it does not establish that such stabilization is complete, universally language-agnostic, or independent of prompt quality. Nor does it eliminate the dependence of multilingual TTS on carefully engineered front-ends and alignment supervision. The later comparison with X-Voice further underscores this point by showing that broader language coverage and transcript-free prompting remain active design frontiers rather than solved problems (Xu et al., 7 May 2026).

The system’s broader significance lies in the specific thesis it advances: prompt-based multilingual TTS improves when dataset design is treated as a first-class modeling variable. In LEMAS-TTS, robustness emerges from the interaction of corpus scale, word-level timestamps, shared phonetic representation, explicit pause modeling, CTC monotonicity regularization, and accent-adversarial conditioning. This suggests a shift in emphasis within multilingual TTS research, from architecture-only optimization toward joint optimization of annotation granularity, linguistic normalization, and stabilization objectives.

In that sense, LEMAS-TTS occupies a distinct place in the evolution of multilingual speech synthesis. It is a benchmark model, but also a methodological argument that high-quality zero-shot multilingual TTS depends on temporally aligned multilingual data and on explicit control over the mechanisms that ordinarily fail in cross-lingual generation.