T5Gemma-TTS: High-Quality Autoregressive TTS

• T5Gemma-TTS is a 4B-parameter neural codec that uses an encoder–decoder architecture to maintain persistent text conditioning for long-form TTS synthesis.
• The model integrates Progress-Monitoring Rotary Position Embedding (PM-RoPE) to achieve precise duration control and stable alignment between text and audio tokens.
• Multilingual training on English, Mandarin, and Japanese facilitates robust speaker similarity and zero-shot voice cloning, even for unseen languages like Korean.

T5Gemma-TTS is a 4-billion-parameter autoregressive neural codec LLM designed for high-quality, long-form, zero-shot text-to-speech (TTS) synthesis. Distinct from prior decoder-only architectures, T5Gemma-TTS employs an encoder–decoder backbone to maintain persistent text conditioning throughout synthesis, prevents text–prefix dilution in extended utterances, and incorporates Progress-Monitoring Rotary Position Embedding (PM-RoPE) for robust duration control. The model is trained on a large multilingual dataset encompassing English, Mandarin Chinese, and Japanese and demonstrates significant gains in speaker similarity and intelligibility metrics, particularly for Japanese, while maintaining capacity for zero-shot voice cloning in languages outside the training set, including Korean (Arata et al., 2 Apr 2026).

1. Encoder–Decoder Architecture and Text Representation

T5Gemma-TTS is constructed from the “T5Gemma-2b-2b” UL2-pretrained backbone, comprising a 26-layer encoder and a 26-layer decoder, each with 2 billion parameters (4 billion total). Model dimension is $d = 2304$ for all layers, and each layer employs 32 attention heads. The model processes input using the T5Gemma SentencePiece tokenizer (vocabulary size: 256,000) enabling direct ingestion of English, Chinese (Simplified/Traditional), and Japanese subwords. Each token $x_t$ is mapped to $d$-dimensional embeddings derived from frozen T5Gemma weights, obviating the need for any grapheme-to-phoneme or phonemizer pipeline.

At each decoder layer $\ell \in \{1, \ldots, 26\}$, persistent cross-attention is performed with encoder outputs $H^{\text{enc}} \in \mathbb{R}^{T \times d}$ via a dedicated cross-attention block. This architectural choice ensures that the audio sequence, which can vastly outlength the text (i.e., $S \gg T$ for $S \to 50$ Hz audio tokenization), remains continuously conditioned on bidirectional text representations, avoiding the positional competition and weakening of text context observed in decoder-only frameworks.

2. Progress-Monitoring Rotary Position Embedding (PM-RoPE)

To facilitate accurate duration control and improve long-form alignment, T5Gemma-TTS introduces Progress-Monitoring Rotary Position Embedding (PM-RoPE). PM-RoPE injects a normalized, linear “progress” signal into both the decoder’s cross-attention queries and the encoder keys:

• For decoder step $j \in \{0, \ldots, S-1\}$ and encoder position $i \in \{0, \ldots, T-1\}$:

$$p^{\text{dec}}_j = \frac{j}{S-1} \cdot s, \quad p^{\text{enc}}_i = \frac{i}{T-1} \cdot s$$

where $x_t$0.

• Cross-attention projections are PM-RoPE–rotated:

$x_t$1

Attention weights are computed as $x_t$2.

PM-RoPE is implemented in all 26 decoder cross-attention layers. The monotonic increase of $x_t$3 shifts cross-attention focus over the encoder, conferring a form of internal progress-tracking. This mechanism allows the decoder to “know” its fraction through generation and substantially stabilizes the temporal alignment between the transcript and the audio tokens during inference.

3. Multilingual Training Regimen and Optimization

The model is trained on approximately 170,000 hours of speech audio:

Language	Hours	Corpus
English	≈100,000	LibriHeavy
Mandarin Chinese	≈50,000	Emilia
Japanese	≈20,000	Emilia + additional open-source

The learning objective is next-token cross-entropy for audio tokens $x_t$4:

$x_t$5

Optimization uses AdamW (peak $x_t$6, weight decay $x_t$7), with 2% warmup (≈2,900 steps) and linear decay to zero over ≈143,000 steps. Each GPU processes up to 30,000 tokens, for an effective global batch size of ≈240,000 tokens over 8 AMD MI300X GPUs, with bfloat16 mixed-precision computation and float32 master weights. The model trains for approximately two weeks.

4. Zero-Shot Voice Cloning and Duration Control Mechanisms

T5Gemma-TTS approaches TTS as autoregressive sequence modeling over XCodec2 audio tokens at 50 Hz, $x_t$8, plus 5 special symbols (〈bos〉,〈eos〉,〈pad〉, silence, prompt-separator). During inference, reference audio tokens for a target speaker (3–15 s in duration) are prepended following a prompt-separator, after which the model autoregressively generates the continuation. Having trained with all speaker tokens encountered only as “audio continuation,” the model generalizes to unseen speakers for zero-shot voice cloning.

Duration is controlled by combining persistent cross-attention and PM-RoPE. At inference, a target duration $x_t$9 (in seconds) is estimated via

$d$0

where $d$1 is reference duration and $d$2, $d$3 are the phoneme or character counts. The target number of audio tokens $d$4. Progress signals are computed as $d$5. This provides explicit duration guidance, achieving an empirical duration accuracy (fraction within ±10% of the target) of 79%.

5. Empirical Results and Ablation Studies

The following table summarizes key quantitative results (95% bootstrap confidence intervals) on speaker similarity (SIM) and character error rate (CER):

Metric	T5Gemma-TTS	XTTS v2	Kokoro	F5-TTS	CosyVoice 2
Japanese SIM	0.677 ± 0.016	0.622 ± 0.017	—	—	—
Japanese CER	0.126 ± 0.018	—	0.139 ± 0.016	~1.14	—
Korean SIM (unseen)	0.747 ± 0.029	0.741 ± 0.010	—	—	—
Korean CER (unseen)	0.082 ± 0.026	—	—	—	0.090 ± 0.032

• The Japanese SIM of T5Gemma-TTS is statistically significantly higher than XTTS v2, with non-overlapping confidence intervals.
• The Japanese CER is the lowest among evaluated baselines but has partial CI overlap with Kokoro; rankings should be interpreted with caution.
• Korean speaker similarity scores are highest numerically for T5Gemma-TTS, but CI overlap with XTTS v2 makes the difference not statistically significant; Korean was not included in training.
• Performance of XTTS v2 is notably strong for European languages due to its larger language set.

An ablation disabling PM-RoPE reveals near-complete failure in synthesis: Japanese CER degrades from $d$6 to $d$7, SIM drops from $d$8 to $d$9, UTMOS decreases from $\ell \in \{1, \ldots, 26\}$0 to $\ell \in \{1, \ldots, 26\}$1, and duration accuracy collapses from 79% to 46%. With oracle targets, duration accuracy falls from 100% to 0%. Persistent cross-attention and PM-RoPE jointly are essential for text–audio alignment and length control.

6. Implementation and Availability

T5Gemma-TTS comprises approximately 4 billion parameters and operates on an audio vocabulary of 65,536 plus 5 special tokens. Inference sampling uses top-$\ell \in \{1, \ldots, 26\}$2, top-$\ell \in \{1, \ldots, 26\}$3, and temperature $\ell \in \{1, \ldots, 26\}$4, producing audio at between 0.8$\ell \in \{1, \ldots, 26\}$5 and 2.0$\ell \in \{1, \ldots, 26\}$6 real-time, depending on utterance length, when running with float16 precision on AMD MI300X hardware.

All inference code, pre-trained weights, and evaluation scripts are publicly available at https://github.com/Aratako/T5Gemma-TTS (Arata et al., 2 Apr 2026).

7. Context and Significance in TTS Research

T5Gemma-TTS represents a convergent advance in text-to-speech by integrating the strengths of encoder–decoder architectures (persistent transcript conditioning via cross-attention), subword-level linguistic transfer from large-scale text pretraining (no phoneme conversion), and explicit temporal modeling (PM-RoPE). Its zero-shot voice cloning capability across typologically diverse and even unseen languages, robust duration control, and competitive intelligibility metrics position it as a significant reference baseline for future autoregressive codec language modeling for TTS. Use of PM-RoPE distinguishes T5Gemma-TTS in addressing duration accuracy failure modes previously observed in autoregressive models for long-form synthesis.