---
title: 'SpeechWeave: Synthetic TTS Data Generation'
url: https://www.emergentmind.com/topics/speechweave
type: topic
---

# SpeechWeave: Synthetic TTS Data Generation

Searching arXiv for the specified SpeechWeave paper and a closely related multilingual TTS paper for context.
SpeechWeave is an end-to-end synthetic data generation pipeline for Text-to-Speech (TTS) training that automatically produces **high-diversity, domain-specific, multilingual text scripts**, their **correctly normalized spoken forms**, and **speaker-standardized speech audio** without relying on costly human transcription and recording workflows [2509.14270]. It is motivated by three recurrent constraints in TTS corpus construction: direct LLM prompting tends to yield repetitive text, post-hoc normalization tools can miss or distort semiotic forms, and large-scale recording by voice artists is difficult to scale. Within multilingual speech synthesis research, this data-centric agenda is complementary to acoustic-model work that removes pronunciation dictionaries through speech-derived phonetic representations rather than target-language lexical resources [2206.00951].

## 1. Problem setting and conceptual scope

SpeechWeave addresses a training-data problem rather than an acoustic-model problem. The paper frames TTS data acquisition as difficult because **domain-specific text is scarce**, **multilingual coverage is required**, **normalization is essential**, and **standardized speaker voices are needed** in commercial settings [2509.14270]. The normalization issue is especially central for written forms that diverge from their spoken realization, including **addresses, phone numbers, email addresses, URLs, dates, times, percentages, person names with salutations, and amounts or currency expressions**.

A central claim is that direct prompting of an LLM does not by itself provide sufficient diversity. For the prompt *“Construct one short sentence in the finance domain”*, the reported outputs remain near-paraphrases even when temperature and top-$p$ vary, including *“Maintaining a budget surplus is essential...”*, *“Maintaining a budget surplus can help...”*, and *“Maintaining a balanced budget is crucial...”* [2509.14270]. This is used to argue that sampling parameters alone do not guarantee diverse corpus generation.

The paper also contrasts SpeechWeave with post-hoc text normalization systems such as **NVIDIA NeMo text normalization**. In the reported analysis, such tools can miss alternate formats of dates, times, and currencies, break names incorrectly, mishandle phone numbers, and overlook semiotic variants not covered by their grammar or model [2509.14270]. SpeechWeave therefore moves normalization closer to data generation itself.

## 2. Pipeline architecture

SpeechWeave is organized as a four-module pipeline: **keyphrase sampler**, **entity sampler with at-source normalization**, **text script generator plus normalization post-processing**, and **speech audio generation plus speaker standardization** [2509.14270]. The paper describes this as a controlled synthetic data factory for generating paired text and audio suitable for TTS training.

| Module | Role | Main mechanisms |
|---|---|---|
| Keyphrase sampler | Increase textual diversity | multi-step prompting, deduplication, keyphrase store |
| Entity sampler | Generate semiotic forms and spoken realizations | locale-sensitive recipes across 9 semiotic classes |
| Text script generator | Produce domain- and language-conditioned scripts | sentence-type prompts, constrained JSON output |
| Audio generation and standardization | Produce speaker-consistent speech | MeloTTS and OpenVoiceV2 tone color converter |

The prompting strategy differs explicitly from a baseline. The baseline prompt is *“Construct one sentence in \{language\} language in \{domain\} domain.”* SpeechWeave instead uses *“Construct one sentence ... with the following words: \{words\}. The following entities should also be present in the text: \{entities\}.”* [2509.14270]. The purpose of this change is not merely stylistic; it constrains generation by **domain**, **sentence type**, **keyphrases**, and **semiotic entities**.

Structured generation is enforced with **lm-format-enforcer**, which constrains intermediate and final outputs to JSON format [2509.14270]. This is used both in keyphrase creation and in script generation.

## 3. Diversity control and normalization strategy

The keyphrase sampler uses a multi-step prompting procedure. The LLM first generates a list of **subdomains** for a target business domain; one subdomain is then sampled; the model writes a **creative paragraph** about that subdomain in the target language; and a subsequent prompt extracts **keyphrases** from that paragraph [2509.14270]. These keyphrases are stored after deduplication in an in-memory keyphrase store that is both domain-specific and language-specific.

Deduplication uses **Token Sort Ratio** and **Levenshtein distance**. The appendix states that a keyphrase is stored only if its similarity to existing phrases is below a threshold, specifically that any keyphrase with **token sort ratio < 0.8** is stored [2509.14270]. The paper notes that Phrase-BERT could be used, but fuzzy search produced better diversity in the reported experiments.

The entity sampler is the main normalization mechanism. Rather than generate raw text and normalize afterward, SpeechWeave generates entities and their normalized spoken forms simultaneously. The reported sampler supports **9 semiotic classes**: **Addresses, Phone numbers, Email addresses, URLs, Dates, Times, Percentages, Person names with salutations, and Amounts or currency expressions** [2509.14270]. Base entities are generated with **Faker**, combined with language-specific recipes.

The examples given in the paper illustrate deterministic source-time normalization. English examples include `Dr. Julie Young` → `Missis Julie Young`, `87%` → `eighty seven percent`, `CA$572` → `five hundred and seventy two Canadian dollars`, and `http://though15.eu` → `h t t p colon slash slash though one five dot e u`. Spanish examples include `CA$572` → `quinientos setenta y dos dólares canadienses`, `02-01-1997` → `el dos de enero de mil novecientos noventa y siete`, and `69.76%` → `sesenta y nueve punto setenta y seis por ciento` [2509.14270].

A post-processing normalizer remains in the pipeline because the LLM may introduce new semiotic classes not present in the prompt. The reported post-processing algorithm **expands acronyms**, **converts numbers to cardinal forms**, **removes hyphens**, **removes underscores**, and **removes brackets** [2509.14270]. The paper notes that this can introduce errors, especially for non-contextual number conversion, although it also states that the short-script regime of up to **50 words** reduces the frequency of such failures.

## 4. Audio generation, speaker standardization, and corpus construction

After normalized text is produced, SpeechWeave synthesizes speech with a **pretrained TTS model** and then applies speaker standardization through voice conversion [2509.14270]. The reported instantiation uses **MeloTTS** as the base synthesis model and **OpenVoiceV2 tone color converter** for voice cloning and standardization. Reference voices come from proprietary voice artists, and the tone color converter is described as **language agnostic**, so an English reference can be used to standardize voices in other languages.

The evaluation corpus contains **3000 datapoints** across **16 business domains**, **5 sentence types**, **9 semiotic classes**, **2 reference speakers** (male and female), and **English and Spanish** [2509.14270]. The sentence types are **Statement, Exclamation, Question, Phrase, and Utterance**. Sentences with fewer than **5 words** or more than **50 words** are excluded and regenerated with a different seed.

For reproducibility, the appendix introduces **secondary seeding**. A primary seed is fixed, and secondary seeds are derived from it so that filtered or regenerated items do not collapse into repeated outputs during iterative generation [2509.14270]. This is a practical design choice for synthetic corpus construction, where regeneration is common.

## 5. Evaluation protocol and empirical results

SpeechWeave is evaluated along three axes: **diversity analysis**, **quality analysis**, and **downstream TTS training impact** [2509.14270]. Diversity is measured with **Diphone Coverage**, **Mean Pairwise Similarity**, and **Token Diversity** through **TTR** and **MATTR**. Mean pairwise similarity is computed from **LaBSE embeddings** using cosine similarity. The grouped formulation reported in the paper is

$$
\text{Grouped Similarity} = \frac{1}{|G|} \sum_{g \in G} \left( \frac{1}{|S_g|(|S_g|-1)} \sum_{i=1}^{|S_g|} \sum_{j=i+1}^{|S_g|} \cos(s_i^g, s_j^g) \right),
$$

while the non-grouped version is

$$
\text{Non Group Similarity} = \frac{1}{M(M-1)} \sum_{j=1}^{M} \sum_{k=j+1}^{M} \cos(s_j, s_k).
$$

The headline results are summarized below.

| Measure | English | Spanish |
|---|---:|---:|
| Diphone coverage vs. baseline | +17.4% | +9.7% |
| Grouped similarity vs. baseline | -45.8% | -44.4% |
| TTR | 0.167 | 0.370 |
| MATTR | 0.803 | 0.979 |
| Normalization accuracy | 0.97 | 0.94 |
| NeMo normalization accuracy | 0.67 | 0.54 |
| SNR (dB) | 59.82 | 53.01 |
| MOS | 4.95 | 4.87 |
| WER (%) | 9.32 | 15.21 |
| Downstream StyleTTS 2 WER | 15.37 → 9.36 | 85.05 → 48.44 |

The diversity results are more detailed than the table alone suggests. Relative to the direct prompting baseline, SpeechWeave reports **48.5% lower** grouped similarity in the most homogeneous English group and **46.7% lower** in the corresponding Spanish group; in non-grouped comparison against public datasets, it reports **58.8% lower mean similarity** for English and **10.7% lower** for Spanish [2509.14270]. For diphone coverage, LibriSpeech still covers **5.7% more** in English and **15.2% more** in Spanish, which the paper attributes to longer utterances.

Normalization accuracy is defined as

$$
\text{Normalization accuracy} = \frac{\text{correctly normalized sentences}}{\text{total evaluated}}.
$$

The manual evaluation covers **500 English** and **500 Spanish** samples [2509.14270]. The reported contrast with NeMo is central to the paper’s claim that normalizing at generation time is more reliable than inferring spoken forms afterward.

Speech audio quality is evaluated with **SNR**, **MOS** estimated using **NISQA**, and **WER**. The reported formulas are

$$
SNR = 10 \log_{10}\left(\frac{P_{\text{signal}}}{P_{\text{noise}}}\right),
$$

and

$$
WER = \frac{S + D + I}{N} \times 100,
$$

where $S$ is substitutions, $D$ is deletions, $I$ is insertions, and $N$ is the number of reference words [2509.14270].

For downstream impact, the paper fine-tunes **StyleTTS 2** from a **LibriTTS checkpoint** for **50 epochs** on SpeechWeave-generated data and evaluates with `stt_en_conformer_ctc_large` for English and `stt_es_conformer_ctc_large` for Spanish [2509.14270]. The reported WER changes are **15.37 → 9.36** in English and **85.05 → 48.44** in Spanish; the paper describes these as **40% relative WER reduction** for English and **27% relative WER reduction** for Spanish.

## 6. Limitations, interpretation, and relation to multilingual TTS

The paper explicitly states several limitations. Normalization quality is limited by the **number of semiotic classes supported** by the entity sampler; unsupported semiotic classes can still trigger post-processing errors; results may vary with the chosen **LLM**, **TTS model**, and **voice conversion model**; evaluation is limited to **English and Spanish**; and the current pipeline **does not address styled speech beyond standardized voice cloning** [2509.14270].

These limitations delimit the scope of the contribution. SpeechWeave is not presented as a universal language-generation or expressive-speech framework; it is a controlled pipeline for producing synthetic TTS training data with improved diversity, normalization, and speaker consistency. This suggests that its primary research significance lies in reducing dependence on manually curated corpora rather than replacing downstream acoustic modeling.

That distinction matters in relation to multilingual synthesis systems such as the dictionary-free method that combines **unsupervised phonetic representations** and **supervised phonetic representations** to eliminate reliance on target-language pronunciation dictionaries [2206.00951]. SpeechWeave addresses corpus generation, whereas that line of work addresses pronunciation modeling and acoustic synthesis. A plausible implication is that multilingual TTS pipelines can reduce dependence on both manually recorded corpora and manually curated pronunciation resources when synthetic data generation and dictionary-free acoustic modeling are treated as complementary components.

Source: https://www.emergentmind.com/topics/speechweave