SpeechWeave: Synthetic TTS Data Generation

• SpeechWeave is an end-to-end pipeline for TTS training that produces diverse, multilingual text scripts and standardized speech audio.
• It integrates a four-module architecture—keyphrase sampling, entity normalization, text script generation, and audio synthesis—to ensure controlled and reproducible data generation.
• The approach improves corpus diversity and normalization accuracy, addressing limitations of direct LLM prompting and post-hoc text processing methods.

Searching arXiv for the specified SpeechWeave paper and a closely related multilingual TTS paper for context. SpeechWeave is an end-to-end synthetic data generation pipeline for Text-to-Speech (TTS) training that automatically produces high-diversity, domain-specific, multilingual text scripts, their correctly normalized spoken forms, and speaker-standardized speech audio without relying on costly human transcription and recording workflows (Dua et al., 15 Sep 2025). It is motivated by three recurrent constraints in TTS corpus construction: direct LLM prompting tends to yield repetitive text, post-hoc normalization tools can miss or distort semiotic forms, and large-scale recording by voice artists is difficult to scale. Within multilingual speech synthesis research, this data-centric agenda is complementary to acoustic-model work that removes pronunciation dictionaries through speech-derived phonetic representations rather than target-language lexical resources (Liu et al., 2022).

1. Problem setting and conceptual scope

SpeechWeave addresses a training-data problem rather than an acoustic-model problem. The paper frames TTS data acquisition as difficult because domain-specific text is scarce, multilingual coverage is required, normalization is essential, and standardized speaker voices are needed in commercial settings (Dua et al., 15 Sep 2025). The normalization issue is especially central for written forms that diverge from their spoken realization, including addresses, phone numbers, email addresses, URLs, dates, times, percentages, person names with salutations, and amounts or currency expressions.

A central claim is that direct prompting of an LLM does not by itself provide sufficient diversity. For the prompt “Construct one short sentence in the finance domain”, the reported outputs remain near-paraphrases even when temperature and top-$p$ vary, including “Maintaining a budget surplus is essential...”, “Maintaining a budget surplus can help...”, and “Maintaining a balanced budget is crucial...” (Dua et al., 15 Sep 2025). This is used to argue that sampling parameters alone do not guarantee diverse corpus generation.

The paper also contrasts SpeechWeave with post-hoc text normalization systems such as NVIDIA NeMo text normalization. In the reported analysis, such tools can miss alternate formats of dates, times, and currencies, break names incorrectly, mishandle phone numbers, and overlook semiotic variants not covered by their grammar or model (Dua et al., 15 Sep 2025). SpeechWeave therefore moves normalization closer to data generation itself.

2. Pipeline architecture

SpeechWeave is organized as a four-module pipeline: keyphrase sampler, entity sampler with at-source normalization, text script generator plus normalization post-processing, and speech audio generation plus speaker standardization (Dua et al., 15 Sep 2025). The paper describes this as a controlled synthetic data factory for generating paired text and audio suitable for TTS training.

Module	Role	Main mechanisms
Keyphrase sampler	Increase textual diversity	multi-step prompting, deduplication, keyphrase store
Entity sampler	Generate semiotic forms and spoken realizations	locale-sensitive recipes across 9 semiotic classes
Text script generator	Produce domain- and language-conditioned scripts	sentence-type prompts, constrained JSON output
Audio generation and standardization	Produce speaker-consistent speech	MeloTTS and OpenVoiceV2 tone color converter

The prompting strategy differs explicitly from a baseline. The baseline prompt is “Construct one sentence in {language} language in {domain} domain.” SpeechWeave instead uses “Construct one sentence ... with the following words: {words}. The following entities should also be present in the text: {entities}.” (Dua et al., 15 Sep 2025). The purpose of this change is not merely stylistic; it constrains generation by domain, sentence type, keyphrases, and semiotic entities.

Structured generation is enforced with lm-format-enforcer, which constrains intermediate and final outputs to JSON format (Dua et al., 15 Sep 2025). This is used both in keyphrase creation and in script generation.

3. Diversity control and normalization strategy

The keyphrase sampler uses a multi-step prompting procedure. The LLM first generates a list of subdomains for a target business domain; one subdomain is then sampled; the model writes a creative paragraph about that subdomain in the target language; and a subsequent prompt extracts keyphrases from that paragraph (Dua et al., 15 Sep 2025). These keyphrases are stored after deduplication in an in-memory keyphrase store that is both domain-specific and language-specific.

Deduplication uses Token Sort Ratio and Levenshtein distance. The appendix states that a keyphrase is stored only if its similarity to existing phrases is below a threshold, specifically that any keyphrase with token sort ratio < 0.8 is stored (Dua et al., 15 Sep 2025). The paper notes that Phrase-BERT could be used, but fuzzy search produced better diversity in the reported experiments.

The entity sampler is the main normalization mechanism. Rather than generate raw text and normalize afterward, SpeechWeave generates entities and their normalized spoken forms simultaneously. The reported sampler supports 9 semiotic classes: Addresses, Phone numbers, Email addresses, URLs, Dates, Times, Percentages, Person names with salutations, and Amounts or currency expressions (Dua et al., 15 Sep 2025). Base entities are generated with Faker, combined with language-specific recipes.

The examples given in the paper illustrate deterministic source-time normalization. English examples include Dr. Julie Young → Missis Julie Young, 87% → eighty seven percent, CA$572` → `five hundred and seventy two Canadian dollars`, and `http://though15.eu` → `h t t p colon slash slash though one five dot e u`. Spanish examples include `CA$572 → quinientos setenta y dos dólares canadienses, 02-01-1997 → el dos de enero de mil novecientos noventa y siete, and 69.76% → sesenta y nueve punto setenta y seis por ciento (Dua et al., 15 Sep 2025).

A post-processing normalizer remains in the pipeline because the LLM may introduce new semiotic classes not present in the prompt. The reported post-processing algorithm expands acronyms, converts numbers to cardinal forms, removes hyphens, removes underscores, and removes brackets (Dua et al., 15 Sep 2025). The paper notes that this can introduce errors, especially for non-contextual number conversion, although it also states that the short-script regime of up to 50 words reduces the frequency of such failures.

4. Audio generation, speaker standardization, and corpus construction

After normalized text is produced, SpeechWeave synthesizes speech with a pretrained TTS model and then applies speaker standardization through voice conversion (Dua et al., 15 Sep 2025). The reported instantiation uses MeloTTS as the base synthesis model and OpenVoiceV2 tone color converter for voice cloning and standardization. Reference voices come from proprietary voice artists, and the tone color converter is described as language agnostic, so an English reference can be used to standardize voices in other languages.

The evaluation corpus contains 3000 datapoints across 16 business domains, 5 sentence types, 9 semiotic classes, 2 reference speakers (male and female), and English and Spanish (Dua et al., 15 Sep 2025). The sentence types are Statement, Exclamation, Question, Phrase, and Utterance. Sentences with fewer than 5 words or more than 50 words are excluded and regenerated with a different seed.

For reproducibility, the appendix introduces secondary seeding. A primary seed is fixed, and secondary seeds are derived from it so that filtered or regenerated items do not collapse into repeated outputs during iterative generation (Dua et al., 15 Sep 2025). This is a practical design choice for synthetic corpus construction, where regeneration is common.

5. Evaluation protocol and empirical results

SpeechWeave is evaluated along three axes: diversity analysis, quality analysis, and downstream TTS training impact (Dua et al., 15 Sep 2025). Diversity is measured with Diphone Coverage, Mean Pairwise Similarity, and Token Diversity through TTR and MATTR. Mean pairwise similarity is computed from LaBSE embeddings using cosine similarity. The grouped formulation reported in the paper is

$$\text{Grouped Similarity} = \frac{1}{|G|} \sum_{g \in G} \left( \frac{1}{|S_g|(|S_g|-1)} \sum_{i=1}^{|S_g|} \sum_{j=i+1}^{|S_g|} \cos(s_i^g, s_j^g) \right),$$

while the non-grouped version is

$$\text{Non Group Similarity} = \frac{1}{M(M-1)} \sum_{j=1}^{M} \sum_{k=j+1}^{M} \cos(s_j, s_k).$$

The headline results are summarized below.

Measure	English	Spanish
Diphone coverage vs. baseline	+17.4%	+9.7%
Grouped similarity vs. baseline	-45.8%	-44.4%
TTR	0.167	0.370
MATTR	0.803	0.979
Normalization accuracy	0.97	0.94
NeMo normalization accuracy	0.67	0.54
SNR (dB)	59.82	53.01
MOS	4.95	4.87
WER (%)	9.32	15.21
Downstream StyleTTS 2 WER	15.37 → 9.36	85.05 → 48.44

The diversity results are more detailed than the table alone suggests. Relative to the direct prompting baseline, SpeechWeave reports 48.5% lower grouped similarity in the most homogeneous English group and 46.7% lower in the corresponding Spanish group; in non-grouped comparison against public datasets, it reports 58.8% lower mean similarity for English and 10.7% lower for Spanish (Dua et al., 15 Sep 2025). For diphone coverage, LibriSpeech still covers 5.7% more in English and 15.2% more in Spanish, which the paper attributes to longer utterances.

Normalization accuracy is defined as

$$\text{Normalization accuracy} = \frac{\text{correctly normalized sentences}}{\text{total evaluated}}.$$

The manual evaluation covers 500 English and 500 Spanish samples (Dua et al., 15 Sep 2025). The reported contrast with NeMo is central to the paper’s claim that normalizing at generation time is more reliable than inferring spoken forms afterward.

Speech audio quality is evaluated with SNR, MOS estimated using NISQA, and WER. The reported formulas are

$$SNR = 10 \log_{10}\left(\frac{P_{\text{signal}}}{P_{\text{noise}}}\right),$$

and

$WER = \frac{S + D + I}{N} \times 100,$

where $S$ is substitutions, $D$ is deletions, $I$ is insertions, and $572` → `five hundred and seventy two Canadian dollars`, and `http://though15.eu` → `h t t p colon slash slash though one five dot e u`. Spanish examples include `CA$0 is the number of reference words (Dua et al., 15 Sep 2025).

For downstream impact, the paper fine-tunes StyleTTS 2 from a LibriTTS checkpoint for 50 epochs on SpeechWeave-generated data and evaluates with stt_en_conformer_ctc_large for English and stt_es_conformer_ctc_large for Spanish (Dua et al., 15 Sep 2025). The reported WER changes are 15.37 → 9.36 in English and 85.05 → 48.44 in Spanish; the paper describes these as 40% relative WER reduction for English and 27% relative WER reduction for Spanish.

6. Limitations, interpretation, and relation to multilingual TTS

The paper explicitly states several limitations. Normalization quality is limited by the number of semiotic classes supported by the entity sampler; unsupported semiotic classes can still trigger post-processing errors; results may vary with the chosen LLM, TTS model, and voice conversion model; evaluation is limited to English and Spanish; and the current pipeline does not address styled speech beyond standardized voice cloning (Dua et al., 15 Sep 2025).

These limitations delimit the scope of the contribution. SpeechWeave is not presented as a universal language-generation or expressive-speech framework; it is a controlled pipeline for producing synthetic TTS training data with improved diversity, normalization, and speaker consistency. This suggests that its primary research significance lies in reducing dependence on manually curated corpora rather than replacing downstream acoustic modeling.

That distinction matters in relation to multilingual synthesis systems such as the dictionary-free method that combines unsupervised phonetic representations and supervised phonetic representations to eliminate reliance on target-language pronunciation dictionaries (Liu et al., 2022). SpeechWeave addresses corpus generation, whereas that line of work addresses pronunciation modeling and acoustic synthesis. A plausible implication is that multilingual TTS pipelines can reduce dependence on both manually recorded corpora and manually curated pronunciation resources when synthetic data generation and dictionary-free acoustic modeling are treated as complementary components.