Multi-SpectroGAN: Adversarial TTS Synthesis

• Multi-SpectroGAN is a GAN-based text-to-speech system that generates diverse, high-fidelity mel-spectrograms through adversarial style transfer.
• It integrates a FastSpeech2-inspired generator with phoneme and style encoders to deliver robust multi-speaker modeling and zero-shot style control.
• The approach achieves near-ground-truth naturalness by leveraging adversarial, feature-matching, and variance-prediction losses for stable style interpolation.

Multi-SpectroGAN (MSG) is a generative adversarial network (GAN)–based text-to-speech (TTS) system designed to synthesize high-diversity and high-fidelity mel-spectrograms with adversarial style combination mechanisms. MSG advances neural TTS by enabling robust multi-speaker modeling and style mixing without requiring an explicit reconstruction loss, instead utilizing exclusively adversarial and feature-matching feedback for generator training. This design enables both fine-grained and global style control, and supports zero-shot style transfer capabilities. The resulting model effectively generates mel-spectrograms with naturalness scores approaching that of ground-truth references, and demonstrates stable style interpolation and transfer across seen and unseen speaker/style pairs (Lee et al., 2020).

1. Architectural Framework

The MSG generator is architecturally rooted in FastSpeech2 and consists of a sequence of functional modules:

• Phoneme Encoder $E_p$: Processes the embedded input phoneme sequence summed with a triangular positional encoding, and passes it through four FFT blocks, yielding hidden phoneme states ($\mathcal{H}_{\mathrm{pho}}$).
• Style Encoder $E_s$: Maps the reference mel-spectrogram ($\mathbf{y} \in \mathbb{R}^{T \times F}$) to a fixed-dimensional style vector ($\mathbf{s} = E_s(\mathbf{y})$) via stacked 1D convolutions, a GRU, and final linear projection.
• Variance Adaptor: Injects the style vector $\mathbf{s}$ into the hidden phoneme states. This module predicts frame-level duration, pitch, and energy, with duration realized through a length regulator ($\mathcal{LR}$), and pitch/energy embedded after quantization. The result is $\mathcal{H}_{\mathrm{mel}}$ post-length regulation, with pitch and energy encoded as $\mathbf{p}$ and $\mathbf{e}$.
• Mel-Spectrogram Decoder $g$: Produces the final mel-spectrogram estimate ($\hat{\mathbf{y}}$) from the sum of all frame-level contributions, the style vector, pitch, energy, and positional encoding ($\mathcal{H}_{\mathrm{total}}$).

The discriminator is framed as a multi-scale, frame-level conditional network. Three discriminators ($\{ D_k \}_{k=1}^3$) operate on different context window sizes, each receiving both the mel-spectrogram and the corresponding frame-level condition ($$\mathbf{c} = \mathcal{H}_{\mathrm{mel}} + \mathbf{s} + \mathbf{p} + \mathbf{e}$$), mirroring the approach in MelGAN while extending it with explicit conditional information (Lee et al., 2020).

2. Adversarial Training Objectives and Loss Functions

MSG is trained to optimize a combination of adversarial, feature-matching, and variance-prediction objectives, with an optional reconstruction loss included in certain ablations:

• Variance-Prediction Losses (MSE): Enforce accurate prediction of ground-truth duration, pitch, and energy values from their estimates, aggregated as:

$$\mathcal{L}_{\mathrm{var}} = \mathcal{L}_{\mathrm{Duration}} + \mathcal{L}_{\mathrm{Pitch}} + \mathcal{L}_{\mathrm{Energy}}$$

• Adversarial Loss (LSGAN): For each scale $k$, discriminator and generator objectives are defined as:

$$\mathcal{L}_{D_k} = \mathbb{E}[\| D_k(\mathbf{y},\mathbf{c})-1\|_2^2] + \mathbb{E}[\| D_k(\hat{\mathbf{y}},\mathbf{c})\|_2^2]$$

$$\mathcal{L}_{\mathrm{adv}} = \sum_{k=1}^3 \mathbb{E}[\| D_k(\hat{\mathbf{y}},\mathbf{c})-1\|_2^2]$$

• Feature-Matching Loss: Intermediate discriminator activations from the spectrogram branch are matched for real and fake inputs:

$$\mathcal{L}_{\mathrm{fm}} = \sum_{k=1}^3 \sum_{i=1}^4 \frac{1}{N_i} \mathbb{E}\left[\big\| D_k^{(i)}(\mathbf{y},\mathbf{c})-D_k^{(i)}(\hat{\mathbf{y}},\mathbf{c}) \big\|_1\right]$$

• Optional Reconstruction Loss: MAE between ground truth and generated mel-spectrogram, sometimes included for stabilization.

The overall generator loss is:

$$\mathcal{L}_{\mathrm{MSG}} = \mathcal{L}_{\mathrm{adv}} + \lambda\,\mathcal{L}_{\mathrm{fm}} + \mu\,\mathcal{L}_{\mathrm{var}} + \nu\,(\text{optional } \mathcal{L}_{\mathrm{rec}})$$

where regularization coefficients are as detailed in the original work (Lee et al., 2020).

3. Adversarial Style Combination (ASC)

MSG introduces Adversarial Style Combination (ASC) to enable robust generalization under both seen and interpolated styles:

• Style Extraction and Mixing: Two reference mel-spectrograms yield style vectors $\mathbf{s}_i, \mathbf{s}_j$ via the style encoder. Mixed style vectors are created via linear interpolation:

$$\mathbf{s}_{\mathrm{mix}} = \alpha\,\mathbf{s}_i + (1-\alpha)\mathbf{s}_j,$$

with $\alpha \sim \mathrm{Uniform}(0,1)$.

• Variance and Decoder Input: The mixed style is used to generate mixed duration, pitch, energy, and embeddings ($$\mathbf{p}_{\mathrm{mix}}, \mathbf{e}_{\mathrm{mix}}$$). These are combined with phoneme and positional encodings to form $\mathcal{H}_{\mathrm{mix}}$, which is decoded into a mixed mel-spectrogram ($\hat{\mathbf{y}}_{\mathrm{mix}}$).
• Conditioning the Discriminator: The discriminator receives both real/mixed spectrograms and their respective conditions. The discriminator loss includes an additional term for $\hat{\mathbf{y}}_{\mathrm{mix}}$, while the generator receives a corresponding adversarial term:

$$\mathcal{L}_{\mathrm{mix}} = \sum_{k=1}^3 \mathbb{E}[\, \| D_k(\hat{\mathbf{y}}_{\mathrm{mix}},\mathbf{c}_{\mathrm{mix}})-1 \|_2^2 \,].$$

• Overall ASC Objective: The total objective for training with ASC is:

$$\mathcal{L}_{\mathrm{ASC}} = \mathcal{L}_{\mathrm{adv}} + \lambda \mathcal{L}_{\mathrm{fm}} + \mu \mathcal{L}_{\mathrm{var}} + \nu \mathcal{L}_{\mathrm{mix}}$$

ASC compels the generator to synthesize realistic spectrograms even under blended, previously unseen style embeddings, enhancing diversity and supporting continuous style transition (Lee et al., 2020).

4. Inference, Controllability, and Zero-Shot Synthesis

MSG supports a diverse array of inference paradigms:

• Style Transfer and Zero-Shot Inference: Style vectors extracted from arbitrary reference utterances ($\mathbf{s}^* = E_s(\mathbf{y}^*)$) allow synthesis of input text in the unseen speaker's style.
• Interpolated and Fine-Grained Style Control: Style, pitch, and energy embeddings may be independently mixed using distinct interpolation coefficients ($\alpha_s, \alpha_p, \alpha_e$), providing extensive control over prosodic and stylistic aspects.
• Robustness to Style Mixture: MSG+ASC uniquely maintains stable alignment and prosody in spectrograms when fed blended style references, an area where attention-based TTS models such as GST/Tacotron2 typically fail (Lee et al., 2020).

5. Experimental Evaluation

MSG demonstrates state-of-the-art results in both single- and multi-speaker TTS synthesis:

• Single-Speaker Naturalness: MSG (Mel+PWG) achieves a MOS score of 3.91, nearly indistinguishable from the ground-truth (Mel+PWG 3.94) on the LJ-Speech dataset.
• Ablation Studies: Excluding explicit condition vectors from the discriminator results in objective collapse. Feature-matching loss yields more natural speech than standard MAE. Downsampling factor $\tau=3$ in conditional inputs achieves optimal convergence-quality trade-off.
• Zero-Shot and Multi-Speaker Generalization: MSG+ASC attains a MOS of approximately 3.89 on VCTK seen speakers, and shows improved speaker classification accuracy and MOS for unseen speakers relative to baseline GST- and FastSpeech2-based systems. Objective metrics such as Mel-Cepstral Distortion (MCD), F$_0$ RMSE, and speaker-ID accuracy consistently favor ASC-based MSG (Lee et al., 2020).

6. Strengths, Limitations, and Future Prospects

MSG exhibits multiple technical strengths:

• Enables adversarial-only TTS training for multi-speaker, high-fidelity generation, obviating the need for strict mel-spectrogram reconstruction loss.
• ASC regularizes the generator toward synthesizing realistic, diverse mel-spectrograms for mixed, unseen styles, supporting robust zero-shot generalization.
• Frame-level conditioning within the discriminator facilitates semantically aligned adversarial gradients.
• Demonstrates fidelity and controllability in empirically validated style transfer scenarios, both for seen and unseen speaker conditions.

Limitations include reliance on external neural vocoders (e.g., Parallel-WaveGAN) for waveform generation, sensitivity of LSGAN and feature-matching stability to regularization coefficients, and a style embedding quality bound by the expressiveness of the learned style encoder ($E_s$).

Directions outlined for future work encompass end-to-end waveform generation, few-shot adaptation protocols, cross-lingual TTS via disentangled representations, and advanced frame-level discriminators (e.g., incorporating attention mechanisms) (Lee et al., 2020).