---
title: 'LEMAS-Dataset: Multilingual Speech Corpus'
url: https://www.emergentmind.com/topics/lemas-dataset
type: topic
---

# LEMAS-Dataset: Multilingual Speech Corpus

Searching arXiv for the primary LEMAS paper and later usage references.

LEMAS-Dataset is the dataset component of **“LEMAS: A 150K-Hour Large-scale Extensible Multilingual Audio Suite with Generative Speech Models”** and is presented as an open multilingual speech corpus for generative speech research, especially multilingual text-to-speech, speech editing, and prompt-based speech generation. The paper describes it, to the authors’ knowledge, as the **largest open-source multilingual speech corpus with rigorous word-level timestamps**, covering **over 150,000 hours across 10 major languages** and constructed through an **efficient data processing pipeline** that combines multilingual forced alignment, confidence-based filtering, and schema unification across heterogeneous public corpora [2601.04233].

## 1. Position within multilingual generative speech research

LEMAS-Dataset is framed as infrastructure rather than as a narrow benchmark. Its intended role is to support **large-scale multilingual generative speech modeling**, with the paper validating that claim through two downstream systems: **LEMAS-TTS**, a non-autoregressive flow-matching multilingual TTS model, and **LEMAS-Edit**, an autoregressive speech editing model that relies directly on precise word-level alignments. The dataset is therefore positioned not only for multilingual synthesis, but also for tasks that require fine-grained temporal grounding, including duration prediction, prompt construction, and localized speech editing [2601.04233].

The paper’s motivation is explicitly data-centric. It argues that prior multilingual speech resources are often monolingual, smaller in scale, domain-limited, weakly filtered, or missing reliable fine-grained timestamps. In that framing, LEMAS-Dataset is defined by the conjunction of four properties: **large scale**, **10-language coverage**, **open availability**, and **word-level temporal supervision**. A plausible implication is that the dataset is intended as a foundation-style corpus for multilingual generative models rather than as a task-specific benchmark.

## 2. Data sources and construction pipeline

LEMAS-Dataset is assembled from **publicly available corpora** rather than from new recordings. The raw sources listed in the paper are **GigaSpeech**, **GigaSpeech2**, **WenetSpeech4TTS**, **Emilia**, **MLS**, **multilingual TEDx**, **Alcaim**, **Golos**, and **YODAS**. These sources span **audiobooks**, **podcasts**, and **in-the-wild conversational speech**, and the comparison table characterizes the overall source mix as both **Audiobook** and **In-the-wild** [2601.04233].

The processing pipeline has six explicit stages. Audio-text pairs are first ingested and normalized into a unified representation. Transcripts are then subjected to **language-specific text normalization** and **romanization**, for example through **Uroman**, so that diverse scripts can be handled in a shared Latin-space representation. Multilingual forced alignment is then performed with the **Multilingual MMS Forced Aligner** from torchaudio, which is based on a **wav2vec-based CTC alignment model**. This stage produces **word-level boundaries** and **token-level posterior probabilities**, and those posteriors are used to derive **word confidence** and **utterance-level average confidence** [2601.04233].

Filtering is a central design component. The paper states that the retained clips must satisfy source-specific **alignment-confidence thresholds** with \(x \in [0.2, 0.5]\), duration limits between **0.5 s** and **30 s**, silence or unaligned-region constraints below **4 s**, language validation against the 10 target languages, and character validation that removes unsupported scripts, emojis, and excessive non-linguistic symbols. It also imposes a language-specific speech-rate constraint of the form
\[
\text{len(text)} / \text{duration} \in [\text{min\_ratio}_L, \text{max\_ratio}_L].
\]
The dataset is finally stored in a **JSON-based format**, with audio converted to **MP3** while preserving **original sampling rates** [2601.04233].

A notable negative fact is that the paper explicitly says **no explicit speech enhancement or background noise removal** is applied during preprocessing. Deduplication, VAD, speaker verification, and speaker clustering are not described as dataset-construction stages.

## 3. Scale, language coverage, and released schema

The training split contains **150.14k h**, **133.71M utterances**, and **1,543.34M words**, with an overall **average duration of 4.18 s** and **average words per utterance of 12.02**. The 10 languages are **Chinese, English, Russian, Spanish, Indonesian, German, Portuguese, Vietnamese, French, and Italian**. The paper states that the dataset is **explicitly curated to balance duration across languages**, although Chinese and English remain substantially larger than the smallest language subsets [2601.04233].

| Language | Total Duration | Utterances |
|---|---:|---:|
| zh | 32.92k h | 17.78M |
| en | 25.35k h | 9.52M |
| ru | 22.92k h | 27.47M |
| es | 21.22k h | 26.41M |
| id | 11.25k h | 11.66M |
| de | 9.84k h | 11.00M |
| pt | 7.38k h | 8.68M |
| vi | 6.60k h | 6.43M |
| fr | 6.54k h | 7.56M |
| it | 6.12k h | 7.21M |

The released unified JSON representation is described as containing a **unique sample key**, **audio file path**, **original transcript**, **normalized transcript**, **word-level timestamps**, and **confidence scores associated with word-level timestamps**. The paper also makes clear that language identity is operationally known throughout processing, although it does not explicitly enumerate language ID as a release field. By contrast, it does **not** state that the released dataset includes phoneme labels, phone-level alignments, speaker labels, demographics, or speaker embeddings [2601.04233].

The evaluation split is curated much more aggressively. It contains **500 utterances per language**, for **5,000 total utterances**, **470.73 minutes** in total, **average duration 5.65 s**, and **78,722 total words**. Selection criteria include **average word-level alignment score \(> 0.9\)**, **word count \(> 5\)**, duration between **3 and 15 seconds**, sentence-end silence trimmed to **\(\le 0.2\) s**, and sampling near each language’s mean character-to-duration ratio; approximately **20%** of the evaluation set is **manually reviewed** to verify that the samples are natural speech [2601.04233].

## 4. Word-level timestamps as the defining annotation

The distinctive annotation of LEMAS-Dataset is the combination of **word-level timestamps** and **confidence scores**. The paper treats these not as an auxiliary convenience but as the defining property that differentiates LEMAS-Dataset from most large multilingual speech corpora. In the comparison table, several large corpora—such as **Libri-Light**, **GigaSpeech**, **GigaSpeech2**, **MLS**, **Emilia**, and **Voxbox**—are listed as lacking timestamps, whereas **LEMAS-Dataset** is reported as providing them [2601.04233].

These timestamps serve multiple roles. First, they support **confidence-aware filtering**, since alignment posterior probabilities can be summarized into utterance-level reliability scores. Second, they make it possible to construct **duration-aware prompts** and insert pause tags in downstream TTS models. Third, and most directly, they enable **word-level speech editing**: the paper states that **LEMAS-Edit** exploits precise word-level alignments to construct training masks and to perform localized infilling with smooth acoustic boundaries [2601.04233].

The paper does not provide a standalone manual timestamp-accuracy study or a direct alignment error table. Instead, it validates timestamp usefulness indirectly through downstream performance, especially editing quality and multilingual TTS results. This suggests that the annotation philosophy is pragmatic: timestamps are justified operationally by what they enable, rather than by an isolated alignment benchmark.

## 5. Validation through LEMAS-TTS and LEMAS-Edit

The dataset is validated through two generative paradigms deliberately chosen to stress different aspects of the corpus. **LEMAS-TTS** is built on **F5-TTS** and uses a **non-autoregressive flow-matching** architecture with a **Diffusion Transformer (DiT) backbone**. It exploits dataset scale and multilinguality for zero-shot multilingual synthesis, while also using timestamp-derived pause tags and auxiliary objectives such as **CTC loss** and **accent-adversarial training**. **LEMAS-Edit** extends **VoiceCraft** into a multilingual editing framework, formulating speech editing as **masked token infilling** and relying directly on aligned word boundaries to define edit spans [2601.04233].

The TTS benchmark is multilingual and cross-lingual in construction. The paper reports that **LEMAS-TTS** consistently outperforms **OpenAudio-S1-mini** on average, improving **WER** from **12.27** to **6.39** when prosody is included, while increasing **SIM** from **0.480** to **0.539**. Reported examples include **de: 3.95 → 0.44 WER**, **en: 4.98 → 1.07**, **id: 32.77 → 6.38**, and **zh: 9.59 → 3.00** [2601.04233].

For editing, the paper uses **20 utterances** sampled from the LEMAS-Dataset evaluation set across **7 languages**, with edits created by replacing one word or phrase and judged by **6 human listeners** in an A/B preference test. The reported outcome is that listener judgments are **balanced** between **LEMAS-TTS** and **LEMAS-Edit**, which the paper interprets as evidence that LEMAS can function as a robust multilingual editing benchmark across architectures [2601.04233].

A later paper, **“X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning”**, treats **LEMAS-TTS** as a substantive multilingual baseline and states that **LEMAS** is included as a source of **Russian** training data. Appendix C of that paper describes the compared **LEMAS-TTS** system as a **flow-matching based non-autoregressive model trained on 150K hours of MMS force-aligned data**, using the **GRL** version with **accent-adversarial training** and **CTC loss** [2605.05611]. This later usage indicates that LEMAS-Dataset has already begun to function as a reusable multilingual speech resource beyond its original paper.

## 6. Relation to prior corpora, strengths, and limitations

The paper situates LEMAS-Dataset against several prominent speech corpora. In the comparison table, **Libri-Light** is listed at **60k h** and **1 language** without timestamps; **GigaSpeech** at **10k h**, **1 language**, no timestamps; **GigaSpeech2** at **28k h**, **3 languages**, no timestamps; **WenetSpeech4TTS** at **13k h**, **1 language**, with timestamps; **MLS** at **51k h**, **8 languages**, no timestamps; **Emilia** at **102k h**, **6 languages**, no timestamps; **Voxbox** at **103k h**, **2 languages**, no timestamps; and **LEMAS-Dataset** at **150k h**, **10 languages**, with timestamps [2601.04233].

On that basis, LEMAS-Dataset’s principal strengths are explicit in the paper: **scale**, **10-language multilingual coverage**, **word-level timestamps**, **confidence scores**, **mixed-domain audio**, **open release**, and an **extensible pipeline** that avoids lexicon-heavy language-specific alignment machinery. The paper repeatedly presents these features as jointly significant, since prior resources generally provide only subsets of them [2601.04233].

The limitations are equally important. The paper does **not** report the **number of speakers**, **gender distribution**, **age**, **accent**, or other demographic statistics. It does not provide a dedicated bias audit, a large-scale manual timestamp verification study, or detailed release-schema documentation beyond the JSON fields listed in prose. It also notes, implicitly or explicitly, that **Chinese and English remain larger**, that source composition is uneven across languages, that **no denoising** is applied, and that alignment quality is filtered rather than manually verified at scale. License terms are also **not stated in the paper** [2601.04233].

These omissions matter for interpretation. They do not diminish the dataset’s stated contribution, but they delimit what can be established from the paper itself. What is firmly documented is that LEMAS-Dataset is a very large, open multilingual corpus with word-level timestamps and confidence scores, designed to support multilingual generative speech systems and already validated through both synthesis and editing workloads.

Source: https://www.emergentmind.com/topics/lemas-dataset