Translatotron Models: Direct Speech Translation

Updated 26 November 2025

Translatotron models are a series of neural architectures for direct speech-to-speech translation that bypass the intermediate text representation.
They incorporate evolving techniques—from BLSTM and Conformer encoders to masked autoencoding and cycle-consistent back-translation—to enhance translation quality and alignment.
Recent versions enable unsupervised training and low-latency deployment in streaming and low-resource settings, significantly improving performance metrics.

The Translatotron model family comprises a series of neural architectures for direct speech-to-speech translation (S2ST) that map source-language speech directly into target-language speech, bypassing any explicit intermediate text representation. Unlike cascade-based approaches, which sequentially chain automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS), Translatotron implements an end-to-end sequence-to-sequence framework. Developments from the original Translatotron to Translatotron 3 have yielded substantial improvements in translation quality, latency, voice preservation, and data efficiency, with recent models enabling unsupervised training from monolingual corpora and rapid deployment on resource-constrained or streaming environments.

1. Architectural Evolution

Translatotron 1: Direct Spectral Sequence-to-Sequence

The original Translatotron (Jia et al., 2019, Kala et al., 9 Feb 2025) introduced a proof-of-concept S2ST model, mapping input log-mel spectrogram frames $S^s = [x_1, \dots, x_M]$ through a bi-directional long short-term memory (BLSTM) encoder and an attentional LSTM decoder to predict the target spectrogram $\hat{S}^t$ . The attention mechanism computes alignment energies $e_{t,i}$ and weights $\alpha_{t,i}$ at each decoder time-step $t$ : $e_{t,i} = v^\top \tanh(W_s s_{t-1} + W_h h_i + b_{\rm att}), \qquad \alpha_{t,i} = \mathrm{softmax}_i(e_{t,i}), \qquad c_t = \sum_i \alpha_{t,i} h_i$ The decoder predicts output spectrogram frames as $\hat{y}_t = W_o [s_t; c_t] + b_o$ . Optionally, a pretrained d-vector speaker embedding is concatenated to decoder inputs for voice preservation. The target waveform is synthesized by vocoding these mel-spectrograms.

Translatotron 2: Joint Linguistic and Acoustic Decoding

Translatotron 2 (Jia et al., 2021, Kala et al., 9 Feb 2025) introduces a modular architecture comprising a Conformer-based encoder, a linguistic LSTM decoder predicting target phonemes, a single attention module, and a duration-based acoustic synthesizer generating mel-spectrograms. At each phoneme step $t$ , the decoder state $s_t$ and context vector $c_t$ produce a target phoneme $p_t$ , which along with a predicted duration $\delta_t$ enables temporally aligned upsampling. The acoustic synthesizer operates in a non-attentive fashion, avoiding alignment drift and excessive generation. The entire architecture is trained end-to-end, with a loss function combining phoneme cross-entropy, spectrogram L2 loss, and duration regression.

Translatotron 3: Unsupervised Monolingual Training

Translatotron 3 (Nachmani et al., 2023, Kala et al., 9 Feb 2025) extends the architecture to support unsupervised training from disjoint monolingual speech–text datasets. The shared encoder is trained as a masked autoencoder (MAE), facilitating a multilingual latent space via SpecAugment masking. The encoder output is partitioned, with one half aligned to pre-trained multilingual word embeddings (MUSE loss) and the remainder capturing residual acoustic/para-linguistic features. Translation is achieved through cycle-consistent back-translation: the source speech is encoded, decoded in the target language, re-encoded, then reconstructed in the source language, enforcing semantic consistency and paralinguistic preservation. Dual decoders (one per language) are employed, each with its linguistic and acoustic heads.

2. Model Objectives and Training

Each Translatotron variant is trained under a composite objective:

Model	Spectrogram Loss	Auxiliary/Linguistic Losses	Voice Handling
Translatotron 1	$L_{\rm spec}$ (L1/L2 on mels)	Source/target phoneme decoder (cross-entropy)	d-vector embedding (optional)
Translatotron 2	$L_{\rm spec}$ (L2)	Phoneme cross-entropy, duration regression	Pre-training time TTS voice copy
Translatotron 3	$L_{\rm spec}$ (autoencoder & BT), $L_{\rm MUSE}$	Back-translation cycle, phoneme prediction, duration loss	Latent speaker/prosody transfer

Translatotron 1 stabilizes training with auxiliary phoneme losses, but primarily reconstructs output mels.
Translatotron 2 integrates explicit phoneme prediction and duration modeling into a joint objective $L = \lambda_{\rm ph} L_{\rm phoneme} + \lambda_{\rm s} L_{\rm spec} + \lambda_{\rm d} L_{\rm dur}$ , enabling more stable alignments.
Translatotron 3 employs a multistage training regime: masked autoencoding (MAE), MUSE word embedding alignment, and cycle-consistency for back-translation. Ablation studies show that omitting any of these losses results in training collapse (Nachmani et al., 2023).

3. Voice Preservation and Prosody

Voice retention across translation is a central feature, with distinct mechanisms in each version.

Translatotron 1 utilizes speaker embeddings for prosody and timbre copying but suffers from error propagation and occasional speaker identity drift.
Translatotron 2 eliminates explicit speaker embeddings at inference, relying on synthesized target utterances (via cross-lingual TTS in the source voice) as supervision. This design prevents model misuse for spoofing and enables fine-grained switching in multi-speaker dialogues via ConcatAug data augmentation. The Conformer-based encoder and single attention provide temporal alignment fidelity, preserving intra-utterance prosodic boundaries.
Translatotron 3 preserves paralinguistic cues implicitly: the autoencoder and back-translation cycles require accurate reconstruction of timing, pauses, speaker identity, and prosodic markers. Empirical studies report speaker cosine similarity of $0.6+$ on synthetic test sets, significantly exceeding cascaded system baselines (Nachmani et al., 2023).

4. Empirical Performance Metrics

Performance is typically assessed by BLEU (of ASR transcripts from translated speech), mean-opinion-score (MOS) for naturalness/similarity, and WER.

Model	Data Regime	Spanish→English BLEU	MOS (Intelligibility)	MOS (Speaker Similarity)	Latency
Translatotron 1	Supervised (parallel S2ST)	42.7 (Conv)	4.08	1.85	High (>1.2s)
Translatotron 2	Supervised + TTS augmentation	55.6 (Conv), 42.4 (Fisher)	4.21	2.33	Med (0.6-0.8s)
Cascade (ST→TTS)	Supervised	58.8 (Conv)	4.31	3.30	Med
Translatotron 3	Unsupervised (monolingual S,T)	24.3 (UC), 14.3 (CV11), 10.8 (real)	4.21	0.65	Low (0.4-0.6s)

Translatotron 2 closes nearly the entire BLEU gap to cascaded pipelines while delivering lower latency (Jia et al., 2021). Translatotron 3, despite operating under unsupervised monolingual constraints, outperforms unsupervised cascade systems by $+$ 18 BLEU on generic Spanish–English data (Nachmani et al., 2023).

5. Implementation, Open-Source Toolkits, and Extensions

The ESPnet-ST-v2 toolkit (Yan et al., 2023) provides reference implementations of both the original and second-generation Translatotron models within a unified PyTorch-based framework. The toolkit supports flexible encoder/decoder backbones (including Conformer, Transformer, and LSTM), dual-attention mechanisms for alignment, and multiple vocoding backends. Loss terms in ESPnet-ST-v2 mirror published objectives: $L_{\mathrm{ST}} = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, \mathbf{h}_{\rm enc}), \quad L_{\mathrm{TTS}} = \lambda_1 \|\mathbf{M} - \hat{\mathbf{M}}\|_1 + \lambda_2 \|\mathbf{M} - \hat{\mathbf{M}}\|_2^2 + \lambda_{\mathrm{stop}} \mathrm{BCE}(s, \hat{s})$ Variants such as Translatotron-style spectral multi-decoders and unit-based models (e.g., UnitY) are supported, with reported ASR-BLEU gains and improvements in spectrogram reconstruction errors and real-time factor against earlier code releases.

Further, the SimulTron architecture (Agranovich et al., 4 Jun 2024) adapts the Translatotron framework for low-latency streaming S2ST on mobile hardware, implementing causal Conformer encoders and wait- $k$ attention for fixed-delay online translation, and achieving BLEU on par with or surpassing batch-mode Translatotron 1.

6. Low-Resource Languages and Data Efficiency

A central advance of Translatotron is its applicability under data scarcity and for under-represented languages. Translatotron 2 leverages unsupervised and weakly-supervised data, integrating large-scale pretraining (w2v-BERT, mSLAM) and TTS-based augmentation to boost BLEU score by $+13.6$ (+113% rel.), and gives $+398\%$ relative improvement on low-resource pairs (Jia et al., 2022). Translatotron 3 further relaxes data requirements, learning entirely from monolingual speech–text in each language, and is cited as a promising approach for bridging the language technology gap for African languages (Kala et al., 9 Feb 2025).

7. Limitations, Open Directions, and Comparative Perspective

Translatotron 1 established feasibility but was limited by reduced translation quality and high latency relative to cascades. Translatotron 2 mitigates over-generation and alignment drift, approaches or matches state-of-the-art cascade BLEU, and supports fine-grained voice and prosody transfer, without explicit speaker embeddings at inference (Jia et al., 2021, Kala et al., 9 Feb 2025). Translatotron 3 achieves end-to-end unsupervised S2ST and preserves speaker and prosodic characteristics, but its quality still trails fully supervised models when large parallel corpora are available. Dependency on pre-trained multilingual word embeddings (MUSE) can restrict applicability in languages lacking sufficient resources (Nachmani et al., 2023). Real-time streaming and further prosody modeling—particularly inconjunction with domain adaptation—remain critical open areas. Comparative analysis with non-autoregressive baselines such as DASpeech (Fang et al., 2023) indicates that while Translatotron 2 remains a top-performing autoregressive S2ST model, NAR approaches now offer order-of-magnitude speedup at nearly equivalent BLEU.

References: (Jia et al., 2019, Jia et al., 2021, Jia et al., 2022, Yan et al., 2023, Nachmani et al., 2023, Fang et al., 2023, Agranovich et al., 4 Jun 2024, Kala et al., 9 Feb 2025)