Speech-to-Speech Translation

• Speech-to-Speech Translation (S2ST) is the direct conversion of spoken utterances from one language to another while retaining paralinguistic features such as emotion, rhythm, and speaker identity.
• Modern S2ST systems leverage discrete unit representations, encoder-decoder frameworks, and unit-based vocoders to reduce text dependency and accurately transfer prosody.
• Evaluations based on BLEU scores and human expressiveness ratings demonstrate significant improvements in preserving style and paralinguistic details over traditional cascaded models.

Speech-to-Speech Translation (S2ST) refers to the direct transformation of spoken utterances in one language into spoken utterances in another language, encompassing both linguistic content and, increasingly, paralinguistic features such as emotion, emphasis, intonation, rhythm, and speaker identity. S2ST systems underpin multilingual communication in real-time scenarios (e.g., conversational agents, dubbing, mediation between unwritten languages) and have undergone a dramatic evolution from modular cascades to sophisticated direct neural models capable of paralinguistic transfer and style preservation.

1. Architectures and Discrete Unit Representations

Modern S2ST systems employ multi-stage pipelines with encoder-decoder frameworks, centering on discrete unit representations to mitigate text dependency and facilitate paralinguistic transfer. The canonical model architecture, as exemplified by "A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translation" (Min et al., 1 Feb 2025), comprises three stages:

1. Unit Extraction: Source speech $x_{1:T}$ is encoded into discrete units $u_{1:U}$ using HuBERT pretraining followed by $K$-means clustering ($K=1000$ centroids on 20 ms frame-level embeddings). This clusters continuous speech features into indices, serving as intermediate linguistic representations decoupled from explicit text.
2. Speech-to-Unit Translation: Using a Transformer-based encoder–decoder model (Fairseq S2T), source units are translated into target units, $\hat{u}_{1:\hat{U}}$. This module operates sequence-to-sequence over unit indices.
3. Unit-to-Waveform Synthesis: A unit-HiFi-GAN vocoder generates target speech waveform $\hat{y}$, conditioned on the predicted units and style/prosody embeddings (speaker ID, global style, local pitch).

The unit-based approach reduces dependency on written language, supports unwritten languages, and enables prosody and speaker style modeling. The overall training objective integrates unit prediction, clustering regularization, prosody transfer loss, and auxiliary predictors:

$$L = L_{\text{trans}} + \beta\,L_{\text{unit}} + \gamma\,L_{\text{prosody}}$$

where $L_{\text{trans}}$ is the cross-entropy on unit prediction, $L_{\text{unit}}$ penalizes clustering error, and $L_{\text{prosody}}$ enforces prosody similarity between source and target.

2. Dataset Construction and Alignment for Expressive S2ST

Expressive S2ST requires datasets closely aligned for both textual content and paralinguistic detail. The system introduced in (Min et al., 1 Feb 2025) leverages a curated 300-hour corpus of English-Spanish movie audio pairs, sourced from diverse film and television franchises ("Money Heist," "Elite," Disney, "Shrek," Harry Potter, James Bond, "Poltergeist," etc.). The automatic alignment pipeline consists of:

• Subtitle merging: SRT files merged via ECAPA-TDNN speaker similarity.
• Denoising and filtering: RNNoise, Azure ASR, WER computation, keeping top 80% of segments with WER $\leq$ 40%.
• Duration and speaker criteria: 3 s $\leq$ segment $\leq$ 15 s, cosine similarity of speaker embeddings $<$ 0.5, minimum 5 segments per speaker.

Dataset statistics: | Corpus | Utterances | Avg. Duration | Total Hours | |------------------------|------------|---------------|-------------| | EN/ES Dubbing & Films | 12,610 | 5.1 s | ~300 |

All segments are annotated with pseudo speaker-IDs and reference style descriptors.

3. Paralinguistic Information and Prosody Transfer

For expressive translation, paralinguistic information encompasses emotion (global style), emphasis (local accent), intonation (pitch contour), and rhythm (timing patterns). Key methods include:

• Prosody Extraction and Encoding: Fundamental frequency ($f_t$), energy ($e_t$), and voiced/unvoiced flags ($v_t$) are extracted from reference speech. A reference encoder generates a global style embedding ($s_{\text{ref}}$), while a prosody encoder produces frame-level prosody features $p_t = [f_t, e_t, v_t]$. These are concatenated or added to unit embeddings prior to waveform generation.
• Prosody Transfer Loss:

$$L_{\text{prosody}} = \sum_t \|f_t^{\text{src}} - f_t^{\text{tgt}}\|^2 + \alpha \cdot \|e_t^{\text{src}} - e_t^{\text{tgt}}\|^2$$

with $\alpha$ controlling the tradeoff between pitch and energy fidelity.

• Auxiliary Predictors: Speaker-ID, pitch, and voiced/unvoiced predictors attached to style encoder outputs and GAN discriminators encourage faithful style and prosody transfer.

4. Training Regime and Optimization

Model training in (Min et al., 1 Feb 2025) employs Adam optimizer (learning rate $= 2\times10^{-4}$, inverse-sqrt decay, warmup), dropout of 0.1 in style/prosody encoders, and mixed data comprising the curated movie corpus and 400K hours of high-quality monolingual speech for pseudo-bilingual supervision. Combined loss functions tune weights $\beta$ and $\gamma$ on the development set.

Batch composition and hyperparameters for the style/prosody encoders are determined empirically; details of exact settings are omitted in the paper.

5. Evaluation Metrics and Empirical Results

Evaluation integrates automatic and human judgements:

• BLEU: Calculated via ASR on synthesized audio and comparison to ground-truth transcript. "S2ST input" BLEU utilizes the translation model’s output as input to the vocoder.
• Human Expressiveness Ratings: Ten raters score four paralinguistic dimensions (emotion, emphasis, intonation, rhythm) on a 1–5 scale (from Huang et al. 2023).

System	Emotion	Emphasis	Intonation	Rhythm
Vanilla Unit-TTS	2.03	2.68	2.46	2.30
Holistic Cascade (Ours)	3.58	3.26	3.17	3.56

System	S2ST Input BLEU	GT-input BLEU
Vanilla Unit-TTS	29.2	81.0
Holistic Cascade (Ours)	28.3	74.6
Ideal (Ground Truth)	–	78.2

The model in (Min et al., 1 Feb 2025) achieves 25–55% improvements in expressive ratings compared to vanilla baselines, with a marginal BLEU reduction versus ideal GT-input, indicating that prosody preservation does not severely impact translation accuracy.

6. Significance, Limitations, and Future Directions

Preserving paralinguistic information is vital for human-like interaction, enabling the system to reflect sarcasm, questions, excitement, and nuanced attitudes. Discrete-unit translation obviates text bottlenecks, supports unwritten languages, and mitigates error accumulation inherent to cascaded systems. However, limitations persist in dataset cross-lingual style mismatches (e.g., Spanish pitch bias), noise from movie dubbing sources, and coverage restricted to English↔Spanish.

This suggests that broader expansion to more language pairs (especially unwritten and low-resource languages), hierarchical modeling of local/global prosody, richer emotion embeddings, and end-to-end joint training of discrete unit and waveform synthesis modules may further boost S2ST expressiveness and robustness.

• "A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translation" (Min et al., 1 Feb 2025)
• Huang et al. 2023 (referenced for human expressiveness evaluation criteria)
• Lee et al. 2022 (discrete unit objective formulation)
• Kong et al. 2020 (multi-period, multi-scale GAN architectures)
• Fairseq S2T (Transformer-based translation framework)