--- title: 'T5Gemma-TTS: High-Quality Autoregressive TTS' url: https://www.emergentmind.com/topics/t5gemma-tts type: topic --- # T5Gemma-TTS: High-Quality Autoregressive TTS T5Gemma-TTS is a 4-billion-parameter autoregressive neural codec language model designed for high-quality, long-form, zero-shot text-to-speech (TTS) synthesis. Distinct from prior decoder-only architectures, T5Gemma-TTS employs an encoder–decoder backbone to maintain persistent text conditioning throughout synthesis, prevents text–prefix dilution in extended utterances, and incorporates Progress-Monitoring Rotary Position Embedding (PM-RoPE) for robust duration control. The model is trained on a large multilingual dataset encompassing English, Mandarin Chinese, and Japanese and demonstrates significant gains in speaker similarity and intelligibility metrics, particularly for Japanese, while maintaining capacity for zero-shot voice cloning in languages outside the training set, including Korean [2604.01760]. ## 1. Encoder–Decoder Architecture and Text Representation T5Gemma-TTS is constructed from the “T5Gemma-2b-2b” UL2-pretrained backbone, comprising a 26-layer encoder and a 26-layer decoder, each with 2 billion parameters (4 billion total). Model dimension is $d = 2304$ for all layers, and each layer employs 32 attention heads. The model processes input using the T5Gemma SentencePiece tokenizer (vocabulary size: 256,000) enabling direct ingestion of English, Chinese (Simplified/Traditional), and Japanese subwords. Each token $x_t$ is mapped to $d$-dimensional embeddings derived from frozen T5Gemma weights, obviating the need for any grapheme-to-phoneme or phonemizer pipeline. At each decoder layer $\ell \in \{1, \ldots, 26\}$, persistent cross-attention is performed with encoder outputs $H^{\text{enc}} \in \mathbb{R}^{T \times d}$ via a dedicated cross-attention block. This architectural choice ensures that the audio sequence, which can vastly outlength the text (i.e., $S \gg T$ for $S \to 50$ Hz audio tokenization), remains continuously conditioned on bidirectional text representations, avoiding the positional competition and weakening of text context observed in decoder-only frameworks. ## 2. Progress-Monitoring Rotary Position Embedding (PM-RoPE) To facilitate accurate duration control and improve long-form alignment, T5Gemma-TTS introduces Progress-Monitoring Rotary Position Embedding (PM-RoPE). PM-RoPE injects a normalized, linear “progress” signal into both the decoder’s cross-attention queries and the encoder keys: - For decoder step $j \in \{0, \ldots, S-1\}$ and encoder position $i \in \{0, \ldots, T-1\}$: $$ p^{\text{dec}}_j = \frac{j}{S-1} \cdot s, \quad p^{\text{enc}}_i = \frac{i}{T-1} \cdot s $$ where $s = 2000$. - Cross-attention projections are PM-RoPE–rotated: $$ \tilde{q}_j = \operatorname{RoPE}_{\text{dec}}(W_Q h^{\text{dec}}_j, p^{\text{dec}}_j), \quad \tilde{k}_i = \operatorname{RoPE}_{\text{enc}}(W_K h^{\text{enc}}_i, p^{\text{enc}}_i) $$ Attention weights are computed as $\langle \tilde{q}_j, \tilde{k}_i \rangle$. PM-RoPE is implemented in all 26 decoder cross-attention layers. The monotonic increase of $p^{\text{dec}}_j$ shifts cross-attention focus over the encoder, conferring a form of internal progress-tracking. This mechanism allows the decoder to “know” its fraction through generation and substantially stabilizes the temporal alignment between the transcript and the audio tokens during inference. ## 3. Multilingual Training Regimen and Optimization The model is trained on approximately 170,000 hours of speech audio: | Language | Hours | Corpus | |-------------------|--------------|--------------------------------| | English | ≈100,000 | LibriHeavy | | Mandarin Chinese | ≈50,000 | Emilia | | Japanese | ≈20,000 | Emilia + additional open-source| The learning objective is next-token cross-entropy for audio tokens $y_1, \ldots, y_S$: $$ \mathcal{L} = -\sum_{t=1}^{S} \log p_\theta(y_t \mid y_{