Glow-TTS: Flow-Based Text-to-Speech

• Glow-TTS is a flow-based text-to-speech system that converts text into mel-spectrograms using invertible normalizing flows for precise likelihood estimation.
• It employs a monotonic alignment search combined with a duration predictor to align text tokens and speech frames without external autoregressive models.
• Stochastic modeling of duration and pitch delivers diverse, natural intonation and expressive multi-speaker synthesis, enhancing speech controllability.

Glow-TTS is a flow-based text-to-speech (TTS) system that synthesizes mel-spectrograms from text in parallel, leveraging normalizing flows for exact likelihood training and fast sampling. Distinctively, Glow-TTS obviates the need for external autoregressive aligners by employing an internal monotonic alignment search. Subsequent advancements introduce explicit stochastic modeling of duration and pitch, substantially improving the diversity and naturalness of synthesized speech, especially in the zero-shot multi-speaker regime (Ogun et al., 2023, Kim et al., 2020).

1. Flow-Based Generative Modeling in TTS

Glow-TTS adopts normalizing flows to learn an invertible mapping between mel-spectrograms and a latent Gaussian prior. The transformation $f_\theta$ satisfies

$$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$

where $x$ denotes the mel-spectrogram frames, $z$ the latent variable, and $p_Z$ the isotropic Gaussian density over latents.

The decoder comprises $N$ flow blocks, each stacking ActNorm (channelwise affine scaling), invertible $1 \times 1$ convolutions, and affine coupling layers. This structure enables efficient parallel sampling and exact likelihood computation. During training, the model uses forward mapping to compute likelihoods; during synthesis, the reverse transformation generates spectrograms in a single pass (Kim et al., 2020).

2. Monotonic Alignment Search and Duration Modeling

The core innovation in Glow-TTS is the use of Monotonic Alignment Search (MAS) to solve the token-to-frame alignment problem without reliance on external autoregressive models. The algorithm defines alignment $A$ as a monotonic, surjective mapping from mel frames to tokens. The Viterbi dynamic programming algorithm identifies the alignment $A^*$ that maximizes the likelihood

$$\sum_{j=1}^{T_\mathrm{mel}} \log \mathcal{N}(z_j; \mu_{A(j)}, \sigma_{A(j)})$$

where $$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$0 are Gaussian means output by a 6-layer Transformer text encoder.

Concurrently, a deterministic convolutional duration predictor regresses the per-token frame count. Training employs an MSE objective on ground-truth durations, with gradients stopped on encoder representations to decouple duration learning from flow-based likelihood optimization (Kim et al., 2020).

3. Stochastic Duration and Pitch Prediction

Subsequent research extends Glow-TTS by incorporating stochastic flow-based predictors for both duration and pitch, improving expressivity and diversity in multi-speaker synthesis. The stochastic duration predictor models the distribution of durations via a flow $$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$1, enabling multiple valid rhythm realizations.

Pitch is modeled via a flow-based predictor $$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$2, which learns the distribution of log-F0 sequences $$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$3 for each speaker, conditioned on expanded encoder embeddings. During inference, random draws from these flows allow the generation of more varied pitch contours and durations, addressing the over-regularity found in deterministic systems (Ogun et al., 2023).

Mathematical Formulation

The pitch flow models $$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$4 as

$$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$5

with the likelihood

$$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$6

4. End-to-End Joint Training and Conditioning

The entire architecture—comprising flows for mel-spectrograms, durations, and pitch—is trained end-to-end via the joint log-likelihood objective:

$$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$7

where $$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$8 is the mel-spectrogram, $$z = f_\theta(x), \quad \log p_\theta(x) = \log p_Z(z) + \log \Bigl| \det \left(\frac{\partial f_\theta}{\partial x}\right) \Bigr|$$9 the latent variable, $x$0 the durations, $x$1 the log-F0 contour, and $x$2 the speaker embedding. The decoder’s affine coupling layers are conditioned on both speaker embedding $x$3 and the predicted log-F0 contour $x$4, which itself is projected by a 1D convolution.

Gradients on encoder outputs are stopped when training the pitch predictor, and variational data augmentation (padding $x$5 with Gaussian noise) is used to stabilize the flow (Ogun et al., 2023).

5. Experimental Design and Evaluation Metrics

Training utilizes Mozilla Common Voice v7 English data, filtered to 4,469 speakers over 231 hours. Zero-shot generalization is tested on six unseen VCTK speakers. Glow-TTS variants compared include:

Model	Stochastic Duration	Stochastic Pitch
Glow-TTS	No	No
GlowTTS-STD	Yes	No
GlowTTS-STDP	Yes	Yes

Speech synthesis is assessed with multiple subjective and automatic measures:

• N-MOS: naturalness (short utterances)
• NR-MOS: naturalness (long utterances)
• S-MOS: speaker similarity
• D-MOS: intonation diversity
• WV-MOS: HiFi++ automatic MOS
• Cosine similarity of speaker embeddings
• Log-F0 distributional comparison (Ogun et al., 2023)

Empirical results demonstrate that stochastic pitch prediction (GlowTTS-STDP) achieves the highest naturalness and diversity scores, with N-MOS increasing from 3.13 (baseline) to 3.45, and S-MOS rising from 2.26 to 2.99. The log-F0 distribution of STDP closely matches real speech. Ablation indicates that stochastic duration yields rhythmic improvements, while stochastic pitch further enhances prosodic diversity and naturalness, especially for male speakers.

6. Synthesis Efficiency and Controllability

Glow-TTS decouples synthesis latency from utterance length by parallelizing all flow steps over time and channels—generating mel-spectrograms in approximately 40 ms for long utterances on a V100 GPU, compared with 600–1,000 ms for Tacotron 2. This speed-up is achieved with little loss in quality (Glow-TTS MOS ≈ 4.01 vs Tacotron 2 MOS ≈ 3.88 on LJSpeech).

Control over prosody and speaking rate is afforded via temperature scaling in latent sampling and multiplicative adjustment of predicted durations. Multi-speaker modeling is supported by injecting a 128-dimensional speaker embedding into every coupling layer. Voice conversion is enabled by inverting a mel-spectrogram to $x$6 and re-synthesizing it under a different speaker embedding (Kim et al., 2020).

7. Future Directions and Open Problems

Key future research directions include:

• Joint modeling of energy and pitch flows to increase expressive control.
• End-to-end pitch and waveform integration, as in models like VITS, to further unify acoustic modeling and waveform generation.
• Manipulation of pitch latent variables for TTS style transfer.
• Unsupervised identification of prosodic “style tokens” within the pitch latent space.

Explicit flow-based modeling of both duration and pitch within Glow-TTS offers substantial improvement in zero-shot multi-speaker expressivity, yielding more natural and diverse intonation patterns that closely mirror true speaker distributions (Ogun et al., 2023).