--- title: Synthetic Parallel Persian-English Speech Generation url: https://www.emergentmind.com/topics/synthetic-parallel-persian-english-speech-generation type: topic --- # Synthetic Parallel Persian-English Speech Generation Synthetic parallel Persian-English speech generation refers to automatic construction of paired spoken utterances in Persian and English that align at the utterance level, supporting data-scarce applications such as direct speech-to-speech translation (S2ST), multilingual TTS, and voice conversion. Recent advances exploit large language models for translation, zero-shot neural text-to-speech (TTS) for audio synthesis, and discrete unit-based representations to create extensible and scalable pipelines, mitigating the paucity of human-recorded parallel resources for low-resource pairs such as Persian–English. These synthetic corpora enable effective supervised or end-to-end neural training for S2ST, evaluation, and benchmarking under low-resource constraints [2511.12690]. ## 1. Motivation and Challenges in Parallel Speech Generation Direct speech-to-speech translation (S2ST) models benefit from parallel speech corpora in both source and target languages; however, such data is rarely available for Persian–English. Synthetic generation addresses: - **Data scarcity:** Parallel Persian–English speech corpora are limited (CVSS Fa–En ≈ 20 hours, ≈11k utterances), restricting supervised S2ST model capacity [2511.12690]. - **Utterance-aligned requirements:** Direct S2ST bypasses intermediate text, but models require large-scale utterance-level alignment of audio pairs—difficult to obtain via manual collection for low-resource languages. - **Speaker and domain diversity:** Human-annotated data often lacks broad coverage; synthetic pipelines can scale to more speakers and content domains. Synthetic parallel generation thus targets three core outcomes: increased data scale, accurate utterance-level alignment, and naturalness sufficient for downstream model training. ## 2. Automated Parallel Speech Generation Pipeline [2511.12690] details a data pipeline that converts monolingual Persian speech corpora into parallel Persian–English speech pairs through sequential neural translation and synthesis: 1. **Collection and Preprocessing:** Starts from cleaned Persian speech utterances and transcriptions (e.g., ∼20k–30k utterances filtered from Common Voice Fa). 2. **Machine Translation Step:** Each Persian transcription is translated into spoken-style English using GPT-4o in a zero-shot or few-shot setup. The system prompt enforces fluent, semantically faithful conversational English. No explicit prompt tuning (e.g., temperature, exemplars) is specified, likely relying on default GPT-4o settings. 3. **Text-to-Speech Synthesis:** The translated English sentence is synthesized into natural-sounding speech using VoiceCraft, a state-of-the-art zero-shot TTS with style-conditioning and zero-shot speaker adaptation via reference-encoder modules. Quality is described as natural-sounding, though no formal metrics (e.g., MOS, MCD) are reported. 4. **Corpus Assembly:** Resulting pairs are aligned at the utterance level, producing synthetic English waveforms paired with the original Persian speech. This workflow scales parallel data availability by approximately sixfold (to ≈120 hours), without manual annotation or human English speech recording. No explicit filtering or evaluation of GPT or TTS outputs is applied beyond initial text quality control. ## 3. Discrete Speech Units and Representation Downstream S2ST models benefit from representing target English speech as sequences of discrete units rather than waveforms or continuous features. [2511.12690] adopts the following encoding: - **Feature extraction:** Frame-level features are computed from target English audio using pretrained HuBERT (hidden-unit BERT) models. - **Quantization:** k-means clustering is applied to HuBERT embeddings to create a codebook of size $K$ (prior work uses $K\in\{100,200,500\}$), mapping each frame to a unit index. - **Unit sequence:** Each English utterance is then encoded as $u=(z_1,z_2,\dots,z_T)$, where $z_t$ is the discrete index at frame $t$. The clustering objective is: $$ \min_{C\in\mathbb{R}^{K\times d},\;z_i\in\{1,\dots,K\}} \sum_{i=1}^N \lVert h_i - C_{z_i} \rVert^2_2 $$ where $h_i$ is the HuBERT embedding, $C_j$ is the $j$th centroid, and $z_i$ is the index assignment. This quantization enables S2ST models to learn a mapping between source features and symbolic target units, facilitating modular waveform decoding. ## 4. Neural Vocoding from Discrete Units A neural vocoder reconstructs waveforms from discrete target unit sequences. [2511.12690] employs HiFi-GAN, a generative adversarial network trained on LJSpeech, to synthesize audio from unit-derived embeddings. - **Network design:** HiFi-GAN uses a generator $G$ (upsampling unit sequences to audio) and multi-period, multi-scale discriminators $D$. - **Losses:** The training objective combines adversarial (LSGAN-style), feature-matching, and mel-spectrogram ($L_1$) reconstruction losses: $$ \mathcal{L}_{\text{vocoder}} = \mathcal{L}_{\text{adv}} + \lambda_{\text{FM}}\,\mathcal{L}_{\text{FM}} + \lambda_{\text{mel}}\,\mathcal{L}_{\text{mel}} $$ with typical $\lambda_{\text{FM}}=10$, $\lambda_{\text{mel}}=45$. The full waveform is synthesized as $\hat{x} = G(u)$. Default HiFi-GAN “V1” hyperparameters are used. This decoupling enables high-quality, speaker-independent waveform synthesis from the unit representation, supporting rapid scalable corpus generation and downstream model training. ## 5. Model Training, Objectives, and Evaluation The synthetic corpus supports direct S2ST training using a conformer-based encoder (initialized from self-supervised pre-training) to map Persian speech to high-level representations, and a causal transformer decoder to predict English discrete units. - **Training loss:** Model training minimizes the negative log-likelihood of the target unit sequence, with optional label smoothing: $$ \mathcal{L}_{\text{S2ST}} = -\sum_{t=1}^T \log p_\theta(z_t \mid z_{