---
title: 'FLY-TTS: Efficient End-to-End TTS'
url: https://www.emergentmind.com/topics/fly-tts
type: topic
---

# FLY-TTS: Efficient End-to-End TTS

FLY-TTS is a fast, lightweight, and high-quality end-to-end text-to-speech (TTS) system that significantly improves computational efficiency and model compression without sacrificing naturalness or synthesis quality. FLY-TTS is architecturally based on the conditional variational autoencoder (VAE) plus flow framework of VITS, incorporating a ConvNeXt block–driven decoder that predicts Fourier coefficients and leverages the inverse short-time Fourier transform (iSTFT) for waveform reconstruction. The system introduces grouped parameter sharing in the text encoder and flow model, and utilizes adversarial loss from a frozen, large-scale pre-trained WavLM model to boost output quality. FLY-TTS achieves a real-time factor (RTF) of 0.0139 on Intel Core i9 CPUs, representing an 8.8× speedup and 1.6× model compression in comparison to the VITS baseline, while maintaining comparable mean opinion scores (MOS) and objective metrics [2407.00753].

## 1. System Architecture and Core Components

FLY-TTS retains the key VITS framework: a conditional VAE prior, a flow-based prior, and a neural decoder, though with extensive architectural modifications. At inference, the model comprises the following stages:

- **Prior Encoder ($E_{\rm prior}$):** Maps input phoneme sequence $c$ to latent code $z$. Consists of a text encoder (Transformer layers with grouped parameter sharing) and a normalizing flow $f_\theta$.
- **Decoder ($G$):** Converts latent $z$ to the target waveform $\hat{y}$ using stacked ConvNeXt blocks followed by a fast iSTFT operation.
- **Discriminators:** Includes the original HiFi-GAN–style multi-period and multi-scale discriminator set ($D$) as well as a fixed, pre-trained WavLM model ($D_{\rm W}$) with a lightweight prediction head, providing adversarial feedback.

**Grouped Parameter Sharing:** Transformer layers in the encoder are organized into $g_1$ groups, each with $m_1$ consecutive layers ($K_1 = g_1\,m_1$), with parameters shared within each group. The normalizing flow contains $K_2 = g_2\,m_2$ steps, partitioned into $g_2$ groups, sharing parameters only in WaveNet-based projection sub-modules within each group.

**ConvNeXt-Based Decoder + iSTFT:**
Waveform synthesis proceeds by having the decoder generate frame-wise Fourier coefficients:
1. Latent sample $z$ yields $S = [s_1, ..., s_T]$, $s_t \in \mathbb{R}^D$.
2. $S$ is projected to match FFT bin size $N$.
3. $L$ stacked ConvNeXt blocks produce amplitude ($M = [m_{t,k}]$) and phase ($P = [p_{t,k}]$) matrices.
4. Time-domain signal is reconstructed via iSTFT:
$$
\hat{y}[n] = \sum_{t=0}^{T-1}\sum_{k=0}^{N-1} m_{t,k} e^{j\,p_{t,k}} w[n - tH] e^{j 2\pi k(n - tH)/N},
$$
where $H$ is hop size, $w[\cdot]$ is the window function, and $N$ is FFT length; batch FFT/iFFT is used for computational efficiency.

## 2. Training Objectives and Optimization

### VAE + Flow Variational Training

FLY-TTS maximizes the evidence lower bound (ELBO) as in VITS:
$$
\log p_\theta(x|c) \geq \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] - \text{KL}(q_\phi(z|x)\,\|\,p_\theta(z|c))
$$
The likelihood $\log p_\theta(x|z)$ is approximated by an L1 or spectrogram reconstruction loss, combined with adversarial and feature-matching losses from HiFi-GAN discriminators.

### WavLM-Based Adversarial Training

The system incorporates a frozen pre-trained WavLM encoder ($D_{\rm W}$), to which a lightweight CNN prediction head is attached. The adversarial losses used are least-squares GAN objectives:
$$
\begin{aligned}
L_{\rm adv}(D_{\rm W}) &= \mathbb{E}_{y\sim p_{\rm real}} [(D_{\rm W}(y)-1)^2] + \mathbb{E}_{z\sim p_z} [D_{\rm W}(G(z))^2] \\
L_{\rm adv}(G) &= \mathbb{E}_{z\sim p_z} [(D_{\rm W}(G(z))-1)^2]
\end{aligned}
$$
The total generator loss $\mathcal{L}_G$ aggregates the VAE + flow terms, HiFi-GAN-based adversarial/feature-matching losses, WavLM adversarial loss, and the auxiliary duration/variance prediction losses as in VITS. All training hyperparameters mirror VITS settings: AdamW with $\beta_1=0.8, \beta_2=0.99$, weight decay 0.01, learning rate $1\times 10^{-4}$, exponential decay $0.999^{1/8}$ per epoch.

## 3. Computational Efficiency and Compression

FLY-TTS achieves substantial gains in CPU inference speed and model size:

| Model         | #Params (M) | RTF-CPU | Speedup vs. VITS-base | Compression Ratio |
|---------------|:-----------:|:-------:|:---------------------:|:-----------------:|
| VITS-base     |    28.11    | 0.1221  |        1.0×           |       1.0×        |
| FLY-TTS       |    17.89    | 0.0139  |        8.8×           |       1.6×        |
| Mini FLY-TTS  |    10.92    | 0.0127  |        9.6×           |       2.6×        |

RTF is defined as the ratio of wall-clock synthesis time to output audio duration, measured on Intel Core i9-10920X @ 3.5 GHz (no further optimizations). FLY-TTS eliminates transposed-convolution upsampling by shifting the majority of computation to fast FFT/iFFT routines. Grouped parameter sharing enables 36%–61% reduction in model size with negligible impact on synthesized speech quality.

## 4. Quality Evaluation and Ablation Analysis

Objective and subjective performance is established through standard metrics:

| Model            | #Params | RTF-CPU | MCD  | WER (%) | MOS ±95% CI    |
|------------------|--------:|--------:|-----:|--------:|---------------:|
| Ground truth     |    –    |    –    |   –  |  1.56   | 4.21 ± 0.10    |
| VITS-base        | 28.11 M | 0.1221  | 5.49 |  1.71   | 4.15 ± 0.09    |
| MB-iSTFT-base    | 27.49 M | 0.0274  | 5.57 |  1.89   | 4.08 ± 0.11    |
| FLY-TTS          | 17.89 M | 0.0139  | 5.56 |  1.77   | 4.12 ± 0.09    |
| Mini FLY-TTS     | 10.92 M | 0.0127  | 5.63 |  2.09   | 4.05 ± 0.09    |

- **MCD**: Mel-cepstral distortion.
- **WER**: Word error rate of TTS output.
- **MOS**: Mean opinion score (naturalness).

FLY-TTS matches or narrowly trails VITS in MCD/ WER, but maintains comparable MOS (4.12 ± 0.09 vs. 4.15 ± 0.09), indicating minimal loss in perceptual quality. Mini FLY-TTS (further compressed) demonstrates only a minor MOS reduction.

Ablation studies show that replacing the ConvNeXt+iSTFT decoder with a multi-band upsampler doubles the RTF and slightly reduces MOS (from 4.12±0.09 to 4.01±0.11). Removal of the WavLM discriminator results in a MOS drop of ~0.14, substantiating its impact on output quality.

## 5. Distinctive Contributions and Methodological Insights

FLY-TTS introduces several impactful design elements:

- **ConvNeXt + iSTFT Decoder:** Removes the dependency on computationally heavy transposed-convolution upsamplers, shifting decoding computation to efficient FFT/iFFT operations and enabling an 8.8× CPU speedup.
- **Grouped Parameter Sharing:** Empirical results show that parameter sharing in contiguous Transformer and flow-step groups is effective, supporting compression rates of up to 61% with marginal performance loss.
- **WavLM-Based Discriminator:** Leveraging a large, fixed self-supervised speech representation via WavLM (with a lightweight head) yields adversarial gradients that significantly enhance speech naturalness, without impacting inference cost.

## 6. Limitations and Potential Research Directions

FLY-TTS is evaluated exclusively in a single-speaker setting (LJSpeech). Extending grouped parameter sharing, ConvNeXt+iSTFT decoding, and WavLM-guided adversarial training to multi-speaker, cross-lingual, or style-conditioned scenarios remains unexplored. Further investigation into more aggressive parameter tying or lighter self-supervised discriminators could provide additional efficiency gains and broader applicability. A plausible implication is that these design elements are orthogonal and can be combined with other lightweight architectures in TTS research [2407.00753].

Source: https://www.emergentmind.com/topics/fly-tts