Incremental FastPitch: Real-Time TTS

• Incremental FastPitch is a text-to-speech model variant that uses chunk-based transformer decoders for incremental, low-latency speech synthesis.
• It employs advanced attention masking and state-caching to maintain high speech fidelity while dramatically reducing end-to-end latency.
• The architecture scales linearly with utterance length, making it ideal for real-time streaming and interactive TTS applications.

Incremental FastPitch is a variant of the FastPitch text-to-speech (TTS) architecture that enables low-latency, high-quality, incremental speech synthesis by re-architecting the decoder to operate on fixed-size Mel-spectrogram chunks using transformer-style chunk-based Feed-Forward Transformer (FFT) blocks, advanced attention masking for receptive-field control, and a state-caching inference regime. The method delivers real-time response suitable for streaming and interactive speech applications, cutting end-to-end latency dramatically compared to fully parallel transformer decoders while preserving output speech fidelity (Du et al., 2024).

1. Architectural Modifications in Incremental FastPitch

The canonical FastPitch system synthesizes speech in a fully parallel manner using transformer FFT blocks that process the upsampled encoder features $\bar u \in \mathbb{R}^{T \times d}$ for the entire utterance in a single pass, yielding the full Mel-spectrogram $Y \in \mathbb{R}^{T \times 80}$. Incremental FastPitch preserves the encoder, duration, pitch, and energy predictors from FastPitch, but restructures the decoder to operate on non-overlapping chunks $\bar u = [\bar u_1; \dots; \bar u_N]$, where each $\bar u_i \in \mathbb{R}^{S_c \times d}$ and $S_c$ is the chunk size.

Each decoder layer comprises a stack of chunk-based FFT blocks, each with:

• Multi-Head Attention (MHA) Sub-layer: At chunk $t$, MHA computes query-key-value attention where keys and values are concatenated from a fixed-size cache of previous keys/values ($pk_i^{t-1}$, $pv_i^{t-1}$, size $S_p$) and the current chunk ($\bar u_i$ projected by $W^K_i$, $W^V_i$). This design ensures the per-chunk inference cost is independent of chunk index $i$ or global sequence length $T$ and supports Mel-continuity.
• Two-Layer Causal Convolutional Feed-Forward Network (FFN): Each FFN layer caches its last $K_j-1$ states ($pc^t_j$ for $j=1,2$) to support causality over the chunk sequence.

The chunk-based FFT computational diagram can be summarized as:

┌────────────────────────────────────────────────────────┐ │ Input chunk: \bar u_i ∈ R^{S_c×d} │ ├─► [MHA w/ past-cache] ──► Add & Norm ──► │ │ • pk_i^{t−1}, pv_i^{t−1} │ │ • produce o^t_M and update pk_i^t, pv_i^t │ ├─► [Causal-Conv FFN w/ past-state] ──► Add & Norm ──► │ │ • pc^{t−1}_{1}, pc^{t−1}_{2} │ │ • produce o^t_{c2} and update pc^t_{1}, pc^t_{2} │ └────────────────────────────────────────────────────────┘

Mathematically, for MHA in head $i$, chunk $t$:

• $$k^t_i = \mathrm{concat}(pk^{t-1}_i,\, \bar u_i W^K_i)$$
• $$v^t_i = \mathrm{concat}(pv^{t-1}_i,\, \bar u_i W^V_i)$$
• $$o^t_i = \mathrm{Attention}(Q=\bar u_i W^Q_i,\, K=k^t_i,\, V=v^t_i)$$
• $o^t_M = \mathrm{concat}_i(o^t_i)\,W^O$
• $pk^t_i = \mathrm{tail\_slice}(k^t_i,\, S_p)$; $pv^t_i = \mathrm{tail\_slice}(v^t_i,\, S_p)$

The causal-conv FFN processes similarly, caching and updating past conv states.

2. Training with Receptive-Field-Constrained Chunk Attention Masks

To ensure the model operates within its inference constraints during training, a strictly block-diagonal attention mask $M \in \{0, -\infty\}^{(S_p+S_c) \times (S_p+S_c)}$ is applied in each decoder layer. The local receptive window per chunk is $L = S_p + S_c$; positions $j=1\ldots S_p$ refer to the past cache and $j=S_p+1\ldots S_p+S_c$ are the current chunk. The attention mask enforces:

$$M_{q,k} = \begin{cases} 0, & \text{if } k \geq \max(1, q-L+1) \wedge k \leq q \ -\infty, & \text{otherwise} \end{cases}$$

Two regimes are explored:

• Static Mask: $(S_c, S_p)$ fixed through training.
• Dynamic Mask: $(S_c, S_p)$ randomly sampled per batch (e.g., $$S_p \in \{0,\, 0.25S_c,\, 0.5S_c,\, 1S_c,\, 2S_c,\, 3S_c,\, \infty\}$$), enabling generalization to varying chunk configurations.

3. State-Caching Inference Algorithm

The inference algorithm maintains, per decoder layer $\ell$, fixed-size past-key/value and past-convolution caches for each processed chunk. For an input of $N = \lceil T/S_c \rceil$ chunks:

1. For each chunk $t$, extract $\bar u_t$.
2. For every decoder layer:
   • Concatenate past caches with the current chunk for MHA.
   • Update caches using $\mathrm{tail\_slice}$ operations post-attention and post-convolution.
   • Normalize and propagate to the next block.
3. Project final output to yield Mel chunk $y_t$ and emit for audio synthesis.

Per-chunk runtime is $O(N_d S_c d^2)$, and the total cost scales linearly with the number of chunks ($T/S_c$); inference cost per chunk is independent of chunk index.

4. Training Objectives and Optimization

The multi-loss objective is retained from FastPitch:

$$\mathcal{L} = \mathcal{L}_{\mathrm{mel}} + \lambda_{\mathrm{dur}} \mathcal{L}_{\mathrm{dur}} + \lambda_{\mathrm{pitch}} \mathcal{L}_{\mathrm{pitch}} + \lambda_{\mathrm{energy}} \mathcal{L}_{\mathrm{energy}}$$

with:

• $$\mathcal{L}_{\mathrm{mel}} = \|Y_{\mathrm{pred}} - Y_{\mathrm{target}}\|_1$$,
• $$\mathcal{L}_{\mathrm{dur}} = \mathrm{MSE}(\log d_{\mathrm{pred}}, \log d_{\mathrm{target}})$$,
• $$\mathcal{L}_{\mathrm{pitch}} = \mathrm{MSE}(p_{\mathrm{pred}}, p_{\mathrm{target}})$$,
• $$\mathcal{L}_{\mathrm{energy}} = \mathrm{MSE}(e_{\mathrm{pred}}, e_{\mathrm{target}})$$.

No extra regularization or loss terms are used; the chunked receptive field is enforced purely by the attention mask.

5. Experimental Results: Speech Quality and Latency

Empirical evaluation demonstrates Incremental FastPitch produces speech of comparable quality to parallel FastPitch at substantially reduced latency.

Table 1: Comparative Metrics

Model	MOS ($\pm$ 95% CI)	Latency (ms)	RTF
Parallel FastPitch	4.185 $\pm$ 0.043	125.8	0.029
Inc. FastPitch (Static Mask)	4.178 $\pm$ 0.047	30.4	0.045
Inc. FastPitch (Dynamic Mask)	4.145 $\pm$ 0.052	30.4	0.045
Ground Truth	4.545 $\pm$ 0.039	—	—

• Mel-Spectrogram Distance: Lowest when $(S_c, S_p)$ match at train/test; dynamic masking generalizes but with an $\approx$8% higher MSD.
• MOS: Incremental FastPitch matches FastPitch (MOS $\approx$ 4.18) while cutting end-to-end latency by $\approx$4x (from 125.8 ms to 30.4 ms).
• RTF: Remains well above real-time requirements ($\approx$1/22).

6. Computational Complexity and Latency Profile

Let $N_d$ = number of decoder layers, $S_c$ = chunk size, $S_p$ = cache size, $d$ = model dimension, $h$ = attention heads, $d_k = d/h$, $d_{ff}$ = FFN hidden dimension, $K_{1,2}$ = conv kernel sizes.

• MHA Cost/Chunk: $O(h (S_p+S_c) S_c d_k) = O(N_d d S_c (S_p + S_c))$
• Conv-FFN Cost/Chunk: $O(N_d (K_1 + K_2) S_c d_{ff})$
• Total Per-Chunk: $$O\left(N_d S_c [d(S_p+S_c) + (K_1+K_2)d_{ff}]\right)$$

These costs are constant with respect to total utterance length ($T$); only the number of chunks and chunk size matter. Start-up latency is determined by $O(N_d S_c d / \mathrm{GPU\_FLOPS})$. With small $S_c$ (e.g., $30$ frames, $\approx1.3$ ms at $22.05$ kHz and hop $=256$), sub-$30$ ms startup latency is achievable.

7. Significance and Applications

Incremental FastPitch integrates chunked processing, cache-based attention, and tailored attention masking to deliver high MOS, low-latency, and scalable synthesis. Its design enables real-time, streaming, and interactive TTS workloads with reduced response times and memory requirements, while retaining the benefits of transformer-based parallel TTS models. The architecture provides a framework for incremental speech synthesis with complexity governed by chunk parameters rather than utterance length, pointing to new possibilities for low-latency, high-continuity TTS systems (Du et al., 2024).