Direct Speech-to-Speech Translation

• Direct speech-to-speech translation is an end-to-end approach that maps source speech directly to target speech without intermediate text representation.
• It leverages discrete speech representation, non-autoregressive decoding, and robust vocoding to maintain speaker, style, and prosody details.
• Recent advances in self-supervised pre-training, synthetic data augmentation, and cross-modal learning have propelled S2ST performance close to cascaded systems.

Direct speech-to-speech translation (S2ST) is the family of neural architectures, training methodologies, and evaluation frameworks that enable an end-to-end mapping of source-language speech waveforms directly to intelligible, natural-sounding target-language speech, bypassing any intermediate text representation. This paradigm is positioned in contrast to cascaded approaches (ASR→MT→TTS), and is characterized by its ability to preserve paralinguistic, speaker, and stylistic information, enable lower latency, and unlock translation for unwritten or low-resource languages. Recent advances in self-supervised learning, discrete representation modeling, non-autoregressive decoding, robust vocoding, and cross-modal and multilingual pre-training have led to direct S2ST systems achieving translation quality approaching, and in many cases matching, cascaded pipelines.

1. Direct S2ST: Architectures and Formal Problem Setting

Let $x_\mathrm{src}$ be the source waveform or feature sequence, and $y_\mathrm{tgt}$ be the target waveform. The direct S2ST model aims to map

$$x_\mathrm{src} \xrightarrow{f_\theta} y_\mathrm{tgt}$$

via maximization of the conditional likelihood

$$\mathcal{L}(\theta) = -\sum_{t=1}^{T'} \log P(y_t \mid y_{<t}, x_\mathrm{src}; \theta)$$

with no explicitly decoded text at any intermediate stage (Sarim et al., 3 Mar 2025, Gupta et al., 13 Nov 2024, Lee et al., 2021).

Early end-to-end systems (e.g., Translatotron (Jia et al., 2019), Translatotron 2 (Jia et al., 2021)) are attention-based sequence-to-sequence models that directly map log-mel source spectrograms to target mel-spectrograms, reconstructed to audio via neural vocoders (WaveRNN, HiFi-GAN). Architectures are now dominated by variants of Transformer/Conformer front-ends, discrete speech unit pipelines (speech-to-unit translation, "S2UT"), and hybrid two-pass models with explicit intermediate semantic or phonetic representation layers (Lee et al., 2021, Fang et al., 2023, Min et al., 1 Feb 2025). Parallel innovations in decoding (autoregressive, non-autoregressive (NAR) like CTC (Fang et al., 11 Jun 2024), DAG-based (Fang et al., 2023)) and expressive vocoding (HiFi-GAN, BigVGAN) continue to reduce the quality gap with cascaded systems while delivering large gains in latency, paralinguistic fidelity, and robustness.

2. Discrete Speech Representation and Unit-Based Translation

Recent S2ST models largely avoid direct prediction of dense spectrograms—an approach which is susceptible to over-smoothing, inefficient sequence modeling, and limited speaker/prosody control. Instead, a dominant paradigm utilizes discrete self-supervised representations for target-side speech, discovered by clustering intermediate-layer HuBERT, wav2vec2, or similar SSL model features using k-means (typical codebook sizes: $K = 100$–$1000$), yielding a non-verbal symbolic vocabulary (Lee et al., 2021, Zhang et al., 2022, Fang et al., 2023, Wei et al., 2022, Min et al., 1 Feb 2025). The translation model is cast as a conditional sequence-to-sequence predictor

$P(U_\mathrm{tgt} \mid x_\mathrm{src}; \theta)$

where $U_\mathrm{tgt} = [u^{(t)}_1, \ldots, u^{(t)}_M]$, $u^{(t)}_i \in \{1,\ldots,K\}$.

Model training minimizes cross-entropy:

$$L_\mathrm{trans} = -\sum_{n=1}^M \log P(u_n^{(t)} | U_\mathrm{src}; \theta)$$

followed by high-fidelity speech synthesis via a unit-to-waveform neural vocoder (e.g., HiFi-GAN, BigVGAN, DSPGAN) (Lee et al., 2021, Min et al., 1 Feb 2025).

Unit-based S2ST is robustly extensible to languages lacking orthography, is agnostic to text-based supervision, and naturally supports auxiliary paralinguistic features. Comparative studies show empirical translation quality (BLEU) within 1–2 points of best cascaded ASR–MT–TTS systems for Spanish→English and other language pairs (Lee et al., 2021, Sarim et al., 3 Mar 2025, Min et al., 1 Feb 2025), with substantial improvements in naturalness (MOS), speaker similarity, and expressive prosody transfer.

3. Training Strategies: Pre-training, Data Augmentation, and Weak Supervision

A fundamental challenge in direct S2ST is the paucity of real parallel speech corpora. Direct models are now optimized via synergistic integration of self-supervised pre-training, pseudo-labeling, synthetic corpus generation, and cross-modal multitask learning:

• Self-supervised Pre-training: Large-scale wav2vec 2.0, HuBERT, or AV-HuBERT encoders are pre-trained on 10^4–10^5 hours of unlabeled speech/audio-visual data, dramatically improving phonetic abstraction, cross-lingual generalization, and data efficiency for downstream S2ST (Popuri et al., 2022, Jia et al., 2022, Wei et al., 2022, Huang et al., 2023).
• Joint Pre-training with Bilingual Text: Semi-supervised unit-based models (e.g., Speech2S (Wei et al., 2022)) pre-train on both unpaired speech and parallel text, aligning speech encoder and translation decoder in a common latent unit space, yielding +3–5 BLEU over vanilla encoder-only pre-training (Wei et al., 2022).
• Synthetic Data Generation: Unlabeled monolingual text is back-translated via NMT, synthesized into speech with TTS, and paired into synthetic S2ST data ("Text-aug") (Nguyen et al., 2022). Additional augmentation ("Effects-aug") introduces speed, pitch, and noise perturbation, yielding up to +2 BLEU and dramatic gains (Δ ≈ 27 BLEU) in low-resource or fully unsupervised regimes (Nguyen et al., 2022, Popuri et al., 2022).
• Pseudo-labeling: Cascaded ASR–MT–TTS pipelines create massive weakly-supervised S2ST corpora from speech-only or ASR corpora. Pre-training and fine-tuning strategies integrate both real and synthetic data, controlling overfitting via prompt-tuning or data source conditioning (Dong et al., 2022, Jia et al., 2022).
• Partial Parameter Fine-tuning: Efficient adaptation strategies (e.g., LNA-D, partial-layernorm adaptation) and early encoder freezing enable rapid convergence and mitigate catastrophic forgetting when transferring to new domains (Popuri et al., 2022).
• Non-textual Intermediate Supervision: Bottleneck-feature- or acoustic-unit-based auxiliary losses replace phoneme/text-based regularization, facilitating end-to-end training without any textual annotation (Zhang et al., 2022, Li et al., 2022).

4. Decoding Paradigms: Autoregressive, Non-autoregressive, and Streaming S2ST

While most early systems used fully autoregressive Transformers or LSTMs for unit prediction (or spectrogram generation), multiple recent advances have pushed toward substantially faster, lower-latency architectures:

• Non-autoregressive (NAR) S2UT: CTC-based (e.g., CTC-S2UT (Fang et al., 11 Jun 2024)), DAG-based (DASpeech (Fang et al., 2023)), and FastSpeech-2–style models decouple source/target sequence lengths and enable parallel decoding of token sequences. CTC-S2UT deploys advanced NAR techniques—glancing training (GLAT), non-monotonic latent alignments (NMLA), and knowledge distillation—to achieve ASR-BLEU parity with AR baselines while providing up to 26.8× speedup (Fang et al., 11 Jun 2024). DASpeech models target the full path-integral over alignments in a directed acyclic graph for best-path and expected-path training (Fang et al., 2023).
• Simultaneous/Streaming S2ST: Variational monotonic multihead attention (V-MMA) integrates low-latency, learnable READ/WRITE policies directly into attention; direct streaming models produce discrete units with tight quality-latency control and competitive BLEU under realistic constraints (Ma et al., 2021).
• Pipeline Two-Pass S2ST: Architectures such as ComSpeech (Fang et al., 11 Jun 2024) modularly stitch separately pre-trained S2TT and TTS models via CTC-based vocabulary adaptors, facilitating rapid development, transfer, and even zero-shot S2ST performance solely from S2TT and TTS resources, outperforming cascaded approaches without parallel speech (Fang et al., 11 Jun 2024).
• Audio-visual (AV) S2ST: Models such as AV-TranSpeech (Huang et al., 2023) leverage synchronized visual features (lip motion) for robust speech-to-speech translation in noisy conditions. Multimodal fusion, modality-dropout, and cross-modal SSL pre-training yield consistent gains in extremely adverse acoustic regimes.

5. Prosody, Style, and Paralinguistic Expressivity

Preserving paralinguistic information—including speaker identity, emotional tone, prosody, and style—has emerged as a primary objective of unit-based S2ST research:

• Discrete Unit and Style Separation: Architectures explicitly decouple semantic units (e.g., HuBERT–k-means) from independent prosody or style representations (pitch/statistics, global-style vectors, prosody encoders). Unit-based models can be enhanced with run-length compression and F₀/energy features to permit explicit duration and parody transfer (Min et al., 1 Feb 2025).
• Style Adaptors and Zero-Shot Transfer: Direct S2ST frameworks such as StyleS2ST (Song et al., 2023) and discrete-unit-based style transfer pipelines (Wang et al., 2023) employ powerful, frozen speaker style encoders (e.g., ECAPA-TDNN), or in-context learning via acoustic LLMs, to perform zero-shot style transfer—preserving talker-specific speaking style, timbre, and rhythm cross-lingually, even without parallel speaker data (Song et al., 2023, Wang et al., 2023, Min et al., 1 Feb 2025).
• Voice Preservation and Privacy: Translatotron 2 (Jia et al., 2021) introduces training-time data generation that ensures output always carries the source speaker’s identity, including at speaker-turn boundaries, thereby mitigating privacy and spoofing risks associated with zero-shot voice cloning.
• Expressive Dubbing and Film Alignment: New datasets constructed from movie dubbing (carefully aligned for emotional and rhythmic content) drive the development of models with enhanced expressivity, outperforming baseline unit-TTS on human-rated emotion, emphasis, intonation, and rhythm, approaching the ceiling set by ground-truth (Min et al., 1 Feb 2025).

6. Benchmarking, Metrics, and Empirical Performance

Experimental evaluation of direct S2ST is multi-faceted:

• ASR-BLEU: Standard metric; synthesized speech is transcribed by a strong ASR and evaluated via BLEU against reference translation (Lee et al., 2021, Jia et al., 2021, Fang et al., 2023, Min et al., 1 Feb 2025).
• MOS and SMOS: Mean opinion scores for naturalness, intelligibility, and style similarity; computed by large-scale raters (Jia et al., 2021, Min et al., 1 Feb 2025, Song et al., 2023, Wang et al., 2023).
• Speaker Similarity: Cosine similarity of speaker embeddings demonstrates improved voice preservation in DAG/NAR models over AR baselines (Fang et al., 2023, Wang et al., 2023, Min et al., 1 Feb 2025).
• Paralinguistic Retention: Expressivity axes (emotion, intonation, rhythm) are directly evaluated in dubbing-aligned corpora (Min et al., 1 Feb 2025).
• Latency and Speedup: Direct, NAR, and two-pass models report decoding speedup (CTC-S2UT: up to 26.8×, DASpeech: ~18×) while maintaining competitive BLEU (Fang et al., 2023, Fang et al., 11 Jun 2024).
• Low/zero-resource and robust settings: Models that employ synthetic data augmentation, self-supervised pre-training, or zero-shot style transfer deliver dramatic BLEU gains in low-resource conditions (Δ ≈ 27 BLEU for 10 h Spanish–English with text+effects-augmentation (Nguyen et al., 2022); 17.7 BLEU in zero-supervision (Nguyen et al., 2022); ComSpeech-ZS lags supervised S2ST by only 0.7 BLEU and outperforms cascades (Fang et al., 11 Jun 2024)).

7. Open Challenges and Future Research Directions

Direct S2ST has achieved parity or near-parity with optimized cascaded systems on several benchmarks, but major open research areas remain:

• Data Scarcity: There is a critical need for more large-scale, high-quality, expressive, and multi-speaker S2ST corpora—particularly with paralinguistic alignment and low-resource language coverage (Wei et al., 2022, Sarim et al., 3 Mar 2025, Gupta et al., 13 Nov 2024).
• Robust Unsupervised and Zero-Shot S2ST: Further progress is expected in combining speech mining, back-translated units, and self-supervised pretrained representations to generalize to unwritten and extremely low-resource languages (Li et al., 2022, Zhang et al., 2022, Fang et al., 11 Jun 2024).
• Evaluation: Largely ASR-dependent BLEU is insufficient; speech-based semantics (e.g., BLASER), MOS, SMOS, MEL-cepstral distortion, and style/voice retention metrics must be standardized (Gupta et al., 13 Nov 2024).
• Efficiency and Latency: Progress in non-autoregressive architectures, streaming attention/policy, model quantization, and on-device S2ST is ongoing (Fang et al., 11 Jun 2024, Fang et al., 2023, Ma et al., 2021).
• Paralinguistic and Multimodal Transfer: Achieving robust, fine-grained, controllable transfer of expressive, emotional, and stylistic features across languages, as well as integration of visual cues (lip-reading), remains active (Huang et al., 2023, Song et al., 2023, Min et al., 1 Feb 2025).
• LLM-Integration: Early work is exploring prompting large codec-based speech LMs (SpeechGen, VALL-E, PolyVoice) for instructional or expressive S2ST (Gupta et al., 13 Nov 2024).

In summary, direct S2ST constitutes a foundational shift in cross-lingual speech translation, driven by self-supervised representation learning, discrete-unit modeling, expressive synthesis, non-autoregressive decoding, synthetic/weak supervision, and joint multimodal architectures. It is set to enable real-time, accurate, and expressive speech translation across the world's languages, including those with nonstandard writing or rich oral traditions (Sarim et al., 3 Mar 2025, Gupta et al., 13 Nov 2024, Wei et al., 2022, Nguyen et al., 2022, Lee et al., 2021, Jia et al., 2021, Min et al., 1 Feb 2025, Fang et al., 2023, Fang et al., 11 Jun 2024, Huang et al., 2023, Zhang et al., 2022, Wang et al., 2023, Li et al., 2022, Popuri et al., 2022, Jia et al., 2022, Nguyen et al., 2022, Fang et al., 11 Jun 2024).