FastSpeech 2 Model: Efficient TTS Architecture

• FastSpeech 2 is a non-autoregressive TTS model designed to convert text/phoneme sequences directly into mel-spectrograms and audio waveforms.
• It employs a variance adaptor with dedicated duration, pitch, and energy predictors to enable faster training and robust, prosodically rich inference.
• Extensions like EmoSpeech add emotion and speaker conditioning to mitigate oversmoothing and improve expressiveness in synthesized speech.

FastSpeech 2 is a non-autoregressive neural text-to-speech (TTS) architecture designed to synthesize high-quality speech efficiently by directly mapping input textual/phonemic sequences to mel-spectrograms, and subsequently to audio waveforms. The architecture substantially improves upon its predecessor FastSpeech by eliminating teacher-student distillation pipelines, integrating richer prosodic variation, and enabling significantly faster and more robust training and inference. FastSpeech 2’s methodology has also enabled flexible downstream extensions, such as emotion-conditioned TTS (e.g., EmoSpeech), and has motivated advances in addressing oversmoothing and multimodality in generated audio.

1. Core Architecture of FastSpeech 2

FastSpeech 2 consists of four principal modules: an encoder, a variance adaptor (itself composed of a duration predictor, pitch predictor, energy predictor, and a length regulator), a decoder, and an external neural vocoder such as Parallel WaveGAN for waveform synthesis. The encoder takes a sequence of phoneme inputs of length $n$ and, after token embedding and positional encoding, produces hidden representations $h\in\mathbb{R}^{n\times h_{id}}$.

The variance adaptor predicts phoneme durations ($\hat{d}$), pitch ($\hat{p}$), and energy ($\hat{e}$), using dedicated 2-layer 1D-CNN regressors, each producing a real-valued sequence aligned to phoneme positions. The length regulator uses the predicted durations to expand $h$ into a frame-wise (mel-bin) sequence, broadcasting and summing the predicted pitch and energy embeddings across the relevant frames, yielding a mel-frame-aligned representation of length $m = \sum_{i} d_{i}$.

The decoder, a stack of feed-forward Transformer layers, transforms this expanded sequence into predicted mel-spectrograms $\hat{y} \in \mathbb{R}^{m\times c}$ using multi-head self-attention, pointwise 1D convolutions, residual connections, and LayerNorm.

The overall training loss is: $$L = \|y_{mel} - \hat{y}_{mel}\|_1 + \lambda_{dur} L_{dur} + \lambda_{pitch} L_{pitch} + \lambda_{energy} L_{energy}$$ where the variational terms are MSE losses on the predicted vs. ground-truth duration, pitch (CWT spectrogram), and energy. The default loss weights are $$\lambda_{dur} = \lambda_{pitch} = \lambda_{energy} = 1$$ (Ren et al., 2020).

2. Innovations Over Prior Non-Autoregressive Models

FastSpeech 2 dispenses with the autoregressive teacher-student distillation architecture used in FastSpeech, instead adopting a direct forced-alignment approach for duration targets via Montréal Forced Aligner and training entirely on ground-truth mel-spectrograms. This not only simplifies the pipeline but achieves approximately a 3× speed-up in acoustic model training relative to FastSpeech and substantially reduces inference real-time factors (RTF ≈ 0.02), yielding ~48.5× speedup over autoregressive approaches (Ren et al., 2020).

The key architectural innovations are:

• Direct Prosodic Conditioning: By predicting and embedding duration, pitch, and energy at the phoneme (duration) and frame (pitch/energy) levels, FastSpeech 2 accommodates essential sources of inter- and intra-speaker variability.
• Parallelized Inference: The non-autoregressive design allows full parallelization during inference, avoiding sequence-level dependencies in acoustic mapping.
• FastSpeech 2s: A further extension, FastSpeech 2s, includes a fully parallel waveform decoder, directly generating audio from text without intermediate vocoder steps during inference.

3. Extensions for Expressive and Emotional Speech Synthesis

The modularity of FastSpeech 2 facilitated the development of expressive, emotion-conditioned variants—most notably EmoSpeech (Diatlova et al., 2023). EmoSpeech enhances FastSpeech 2 with the following modules:

• Conditioning Embedding: Concatenates learned speaker ($s\in\mathbb{R}^{smp}$) and emotion ($e\in\mathbb{R}^{emo}$) embeddings, forming $c = [s; e]\in\mathbb{R}^{hid}$, and uses this in various injection strategies.
• eGeMAPS Predictor (EMP): Augments the variance adaptor with a dedicated predictor for two emotion-relevant eGeMAPS features (80th and 50th percentiles of log $F_0$). The MSE loss on these features ($L_{eGeMAPS}$) is added to the training objective.
• Conditional Layer Normalization (CLN): Replaces standard LayerNorm in all encoder/decoder layers with CLN, where scaling and bias are functions of $c$; for input $x$:

$$\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}},\ \gamma(c), \beta(c)$$

$y = \gamma(c) \odot \hat{x} + \beta(c)$

• Conditional Cross-Attention (CCA): Implements an explicit cross-attention mechanism from hidden states $h$ to the emotion-token $c$, inserted after self-attention in each encoder/decoder block.
• Adversarial JCU Discriminator: Introduces a joint conditional-unconditional discriminator to adversarially refine generator outputs, using standard Wasserstein-based losses plus a feature-matching term; the full objective is:

$$L_{total} = L_{rec} + L_{eGeMAPS} + L_{adv}(D) + L_{adv}(G) + \alpha_{fm} L_{fm}$$

4. Oversmoothing and Modeling Multimodality

A notable limitation of the standard FastSpeech 2 training objective is the use of element-wise MSE on mel-spectrograms. This is equivalent to maximum-likelihood estimation under an independent, homoscedastic Gaussian, which forces the model to predict conditional means. With multimodal conditional distributions—particularly common in expressive/complex TTS datasets—this produces over-smooth spectrograms lacking natural local variance, resulting in characteristic perceptual artifacts (“metallic” or “bubbling” audio) in vocoder outputs (Kögel et al., 2023).

To address this, (Kögel et al., 2023) introduced TVC-GMM (Trivariate-Chain Gaussian Mixture Model) decoders, replacing the pointwise MSE decoder head with a mixture of Gaussians modeling local time–frequency dependencies for triplets $[y^{t,f}, y^{t+1,f}, y^{t,f+1}]$ per bin $(t,f)$. This introduces conditional multimodality and reduces oversmoothing. Empirical results demonstrate that TVC-GMM with conditional sampling matches ground-truth spectrogram variability (Laplacian-filtered variance, Var_L ≈ 0.43–0.56 vs. GT 0.36–0.41), and increases both objective and subjective audio quality metrics over baseline FastSpeech 2, especially on expressive datasets.

5. Empirical Performance and Evaluation

FastSpeech 2 and its descendants have set new benchmarks for speed and audio quality in non-autoregressive TTS:

• Mean Opinion Scores (MOS): FastSpeech 2 achieves MOS 3.83 ± 0.08 on LJSpeech (GT 4.30 ± 0.07, FastSpeech: 3.68 ± 0.09), outperforming both FastSpeech and autoregressive models paired with Parallel WaveGAN vocoders. FastSpeech 2s achieves 3.71 ± 0.09 (Ren et al., 2020).
• Training and Inference Efficiency: FastSpeech 2 matches or exceeds its predecessors in voice quality while offering 3× speedup in training (17.0 h vs. 53.1 h for FastSpeech, and 38.6 for Transformer TTS, single V100 GPU) and near 50× faster inference.
• EmoSpeech Ablations: Introducing eGeMAPS predictor, CLN, CCA, and JCU adversarial training successively improves MOS (from 3.74 for baseline to 4.37 for full EmoSpeech), and increases human-judged emotion recognition accuracy (up to 0.85 with CCA, with the final model smoothing extreme emotion distributions to 0.83). These modifications do not impact inference speed (Diatlova et al., 2023).
• Preference Evaluations: In head-to-head comparisons, the full EmoSpeech model is preferred by raters ~10 percentage points more often over baselines.

6. Key Hyperparameters and Implementation Details

General settings include 4 FFT Transformer blocks for both encoder and decoder (hidden size 256, 2 heads, 1D convolutional sublayers), dropout 0.1, Adam optimizer with warmup. Variance predictors use 2-layer Conv1D (kernel 3, 256 channels, ReLU, dropout 0.5). Pitch is modeled with a 10-scale CWT of log-F0; energy via L2-norm of the STFT magnitude. Mel-spectrograms have 80 dims, FFT=1024, hop=256, at 11,025 Hz (LJSpeech). Parameter count: ~23M (FastSpeech 2 acoustic), ~28M (FastSpeech 2s incl. waveform decoder) (Ren et al., 2020).

TVC-GMM integration (for oversmoothing mitigation) replaces the mel-spectrogram decoder head with three parallel heads for mixture weights, component means, and covariance matrices. The mixture model is trained by negative log-likelihood, with sampling strategies to ensure consistent and realistic spectrograms (Kögel et al., 2023).

7. Future Directions and Open Issues

Although FastSpeech 2 has addressed key limitations of non-autoregressive TTS, several challenges persist. Oversmoothing due to conditional mean regression, especially on expressive or highly variable speech, motivates further research into structured probabilistic decoders (e.g., TVC-GMM and higher-order dependencies). Conditional prosody modeling remains an open research frontier, with increasing interest in multimodal embeddings for speaker, style, emotion, and linguistic cues. Model extension to languages with complex phonotactics, code-switching, or underresourced prosody remains only partially addressed. A plausible implication is that as the community adopts richer conditioning and probabilistic decoders, additional architectural innovations will be needed to retain inference speed and scalability on large, heterogeneous TTS datasets.

---

References:

• "FastSpeech 2: Fast and High-Quality End-to-End Text to Speech" (Ren et al., 2020)
• "EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech" (Diatlova et al., 2023)
• "Towards Robust FastSpeech 2 by Modelling Residual Multimodality" (Kögel et al., 2023)