Grad-TTS: Diffusion-Based TTS

• Grad-TTS is a score-based diffusion model that progressively denoises noisy mel-spectrograms conditioned on text to achieve high-quality TTS synthesis.
• It integrates a Transformer-like encoder, a duration predictor with monotonic alignment search, and a U-Net–style score-based decoder within a unified framework.
• Empirical results demonstrate competitive MOS performance with explicit quality–speed trade-offs, though challenges remain for multi-speaker scenarios.

Grad-TTS is a score-based, diffusion probabilistic model for text-to-speech (TTS) synthesis that generates mel-spectrograms by progressively denoising a noisy signal conditioned on text. It is formulated in continuous time, utilizing the framework of stochastic differential equations (SDEs) to enable flexible inference and explicit quality–speed trade-offs. Grad-TTS employs a neural network architecture in which a Transformer-like encoder, a duration predictor with monotonic alignment search, and a U-Net–style score-based decoder are integrated in a unified framework. Grad-TTS is competitive with state-of-the-art neural TTS models in terms of Mean Opinion Score (MOS), and its formulation has further guided subsequent diffusion-based approaches to multi-speaker TTS.

1. Mathematical Foundation: Diffusion Processes for TTS

Grad-TTS leverages diffusion models—originally developed for image generation—which learn to transform complex data distributions into Gaussian noise and then invert this process to generate new samples. For TTS, Grad-TTS treats the mel-spectrogram $x_0 \sim p_0$ as data, modeling a forward (noising) SDE

$$\mathrm{d}x_t \;=\; g(t)\,\mathrm{d}B_t \quad\Longrightarrow\quad x_t\sim\mathcal{N}\bigl(x_0,\;\sigma_t^2I\bigr), \quad x_0\sim p_0$$

where $B_t$ denotes Brownian motion and $\sigma_t^2$ is the noise schedule.

The reverse (denoising) process reconstructs $x_0$ from noise via either the reversed SDE,

$$\mathrm{d}x_t = -g(t)^2\,\nabla_x\log p(x_t,t)\,\mathrm{d}t + g(t)\,\mathrm{d}\bar B_t,$$

or its probability-flow ODE equivalent

$$\mathrm{d}x_t = -\tfrac12\,g(t)^2\,\nabla_x\log p(x_t,t)\,\mathrm{d}t.$$

Since the score function $\nabla_x\log p(x_t,t)$ is intractable, Grad-TTS adopts a neural denoiser $h_\theta(x_t, t)$ and Tweedie's formula: $$\nabla_x\log p(x_t,t) = \frac{h_\theta(x_t, t)-x_t}{\sigma_t^2},$$ with $h_\theta(x_t, t) \approx \mathbb{E}[x_0|x_t]$.

Training minimizes the denoising score matching loss: $$L_{\mathrm{DSM}} = \mathbb{E}_{x_0 \sim p_0} \mathbb{E}_{t \sim [0, T]}\mathbb{E}_{x_t \sim \mathcal{N}(x_0, \sigma_t^2 I)} \|h_\theta(x_t, t) - x_0\|^2.$$ This approach generalizes conventional denoising diffusion probabilistic models to enable speech synthesis and provides a mechanism to control sample quality versus inference speed (Popov et al., 2021).

2. Model Architecture and Alignment

The architecture of Grad-TTS is composed of the following principal modules:

• Encoder: Converts text, represented as characters or phonemes $x_{1:L}$, into contextual features $f_{1:L}$ via a stack of 1D convolutions and Transformer blocks.
• Duration Predictor and Monotonic Alignment Search (MAS): A convolutional duration predictor infers for each input token $i$ the number of mel frames $d_i$. Monotonic Alignment Search is a dynamic programming algorithm that aligns encoder outputs $p_{1:F}$ to target mel-spectrograms $y_{1:F}$ under an $\ell_2$ loss, enforcing both monotonicity and surjectivity.
• Score-based Decoder: A U-Net–style neural architecture serves as the score network (denoiser), conditioned on the aligned encoder outputs $p_{1:F}$, responsible for learning the time-dependent score at every noise level.

The encoder's frame-wise outputs serve as the mean for the diffusion decoder. During training, the alignment between text and frames is re-optimized at each iteration, obviating the need for a separate teacher-forced aligner and ensuring global consistency (Popov et al., 2021).

3. Inference Algorithms and Quality–Speed Control

Grad-TTS inference involves generating an initial noisy mel-spectrogram and solving the reverse-time ODE, conditioned on the predicted durations and aligned encoder features. Explicit steps are:

1. Encode text to embeddings; predict durations; compute alignment to obtain $p_{1:F}$.
2. Sample $X_T \sim \mathcal{N}(p, \tau^{-1} I)$, with temperature $\tau$ controlling the initial Gaussian's width.
3. Numerically integrate the reverse ODE

$$\mathrm{d}x_t = [p - x_t - s_\theta(x_t, t)]\beta_t \mathrm{d}t$$

backward from $t = T$ to $t = 0$ (e.g., $N$-step Euler method).

1. Feed the denoised $X_0$ to a neural vocoder (e.g., HiFi-GAN) to synthesize the waveform.

By varying the number of ODE steps ($N$), Grad-TTS provides a trade-off:

• Small $N$ (e.g., 4–10) yields rapid inference (real-time factor $\approx 0.1–0.3$ on GPU) with only a modest drop in MOS.
• Large $N$ (e.g., 100–1000) yields near ground-truth speech quality (MOS within 0.1 of recordings) at higher compute cost.

Adjustment of $\tau$ enables further stabilization or acceleration of the synthesis process (Popov et al., 2021).

4. Training Procedure and Optimization

The training routine alternates between two steps:

• Alignment Step: Monotonic Alignment Search is performed to find the alignment $A^*$ minimizing the encoder loss $L_{\mathrm{enc}} = \|y_j - p_{A(j)}\|^2$.
• Parameter Update Step: With alignment fixed, parameters of the encoder, duration predictor, and score-based decoder are updated to minimize the total loss: $$L = L_{\mathrm{enc}} + L_{\mathrm{dp}} + L_{\mathrm{diff}}$$ where $L_{\mathrm{dp}}$ penalizes discrepancy in duration prediction and $L_{\mathrm{diff}}$ is the denoising score matching objective.

Empirically, the decoder's ODE-based reverse process exhibited robustness to coarse integration steps, and the architecture's design—particularly conditioning the score network on aligned encoder output—was essential for strong performance.

Key architectural and optimization hyperparameters include:

• Linear noise schedule $\beta_t$ from 0.05 to 20.
• Training horizon $T = 1.0$; inference $T = 1.5$.
• Batch size 16, training duration ∼1.7M iterations on a single 11GB GPU.
• Adam optimizer with learning rate $10^{-4}$.
• U-Net decoder structure with three resolutions.

These considerations collectively enabled Grad-TTS to rival Tacotron 2 at approximately half the inference cost on GPUs (Popov et al., 2021).

5. Empirical Performance and Baseline Comparisons

Subjective evaluation via Mean Opinion Score (MOS) demonstrated that Grad-TTS is competitive with leading TTS models. The continuous-time diffusion formulation, robust monotonic alignment, and score-based denoising conferred advantages in quality, stability, and inference speed.

Further, Grad-TTS became the methodology of reference for subsequent multi-speaker and cross-speaker extensions, as in Multi-GradSpeech. While Grad-TTS showed strong results in single-speaker settings, performance in multi-speaker setups could be degraded due to sampling drift arising from imperfect score approximation and increased data complexity (Xue et al., 2023).

6. Impact, Limitations, and Future Directions

Grad-TTS exemplifies the adaptation of diffusion probabilistic modeling to TTS, introducing continuous-time synthesis and alignment mechanisms that influenced later research. Its main limitations include:

• Sampling Drift in Multi-Speaker Settings: Score mismatch can produce compounding drift in sequential sampling, especially when the target data distribution is multi-modal.
• Alignment Dependency: The quality of monotonic alignment is critical for downstream denoising performance.
• Single-Speaker Orientation: While well-suited for single-speaker data, adaptations are needed for multi-speaker or cross-lingual synthesis.

Emerging methods such as Multi-GradSpeech address these limitations by enforcing additional consistency properties during training and expanding conditioning to handle multi-modal data. Ongoing research investigates efficient consistency regularization, higher-order diffusion constraints, and broader speaker and style generalization (Xue et al., 2023).