Incremental FastPitch: Real-Time TTS

• Incremental FastPitch is a text-to-speech model variant that uses chunk-based transformer decoders for incremental, low-latency speech synthesis.
• It employs advanced attention masking and state-caching to maintain high speech fidelity while dramatically reducing end-to-end latency.
• The architecture scales linearly with utterance length, making it ideal for real-time streaming and interactive TTS applications.

Incremental FastPitch is a variant of the FastPitch text-to-speech (TTS) architecture that enables low-latency, high-quality, incremental speech synthesis by re-architecting the decoder to operate on fixed-size Mel-spectrogram chunks using transformer-style chunk-based Feed-Forward Transformer (FFT) blocks, advanced attention masking for receptive-field control, and a state-caching inference regime. The method delivers real-time response suitable for streaming and interactive speech applications, cutting end-to-end latency dramatically compared to fully parallel transformer decoders while preserving output speech fidelity (Du et al., 2024).

1. Architectural Modifications in Incremental FastPitch

The canonical FastPitch system synthesizes speech in a fully parallel manner using transformer FFT blocks that process the upsampled encoder features $\bar u \in \mathbb{R}^{T \times d}$ for the entire utterance in a single pass, yielding the full Mel-spectrogram $Y \in \mathbb{R}^{T \times 80}$. Incremental FastPitch preserves the encoder, duration, pitch, and energy predictors from FastPitch, but restructures the decoder to operate on non-overlapping chunks $\bar u = [\bar u_1; \dots; \bar u_N]$, where each $\bar u_i \in \mathbb{R}^{S_c \times d}$ and $S_c$ is the chunk size.

Each decoder layer comprises a stack of chunk-based FFT blocks, each with:

• Multi-Head Attention (MHA) Sub-layer: At chunk $t$, MHA computes query-key-value attention where keys and values are concatenated from a fixed-size cache of previous keys/values ($pk_i^{t-1}$, $pv_i^{t-1}$, size $S_p$) and the current chunk ($\bar u_i$ projected by $Y \in \mathbb{R}^{T \times 80}$0, $Y \in \mathbb{R}^{T \times 80}$1). This design ensures the per-chunk inference cost is independent of chunk index $Y \in \mathbb{R}^{T \times 80}$2 or global sequence length $Y \in \mathbb{R}^{T \times 80}$3 and supports Mel-continuity.
• Two-Layer Causal Convolutional Feed-Forward Network (FFN): Each FFN layer caches its last $Y \in \mathbb{R}^{T \times 80}$4 states ($Y \in \mathbb{R}^{T \times 80}$5 for $Y \in \mathbb{R}^{T \times 80}$6) to support causality over the chunk sequence.

The chunk-based FFT computational diagram can be summarized as:

$pv_i^{t-1}$5

Mathematically, for MHA in head $Y \in \mathbb{R}^{T \times 80}$7, chunk $Y \in \mathbb{R}^{T \times 80}$8:

• $Y \in \mathbb{R}^{T \times 80}$9
• $\bar u = [\bar u_1; \dots; \bar u_N]$0
• $\bar u = [\bar u_1; \dots; \bar u_N]$1
• $\bar u = [\bar u_1; \dots; \bar u_N]$2
• $\bar u = [\bar u_1; \dots; \bar u_N]$3; $\bar u = [\bar u_1; \dots; \bar u_N]$4

The causal-conv FFN processes similarly, caching and updating past conv states.

2. Training with Receptive-Field-Constrained Chunk Attention Masks

To ensure the model operates within its inference constraints during training, a strictly block-diagonal attention mask $\bar u = [\bar u_1; \dots; \bar u_N]$5 is applied in each decoder layer. The local receptive window per chunk is $\bar u = [\bar u_1; \dots; \bar u_N]$6; positions $\bar u = [\bar u_1; \dots; \bar u_N]$7 refer to the past cache and $\bar u = [\bar u_1; \dots; \bar u_N]$8 are the current chunk. The attention mask enforces:

$\bar u = [\bar u_1; \dots; \bar u_N]$9

Two regimes are explored:

• Static Mask: $\bar u_i \in \mathbb{R}^{S_c \times d}$0 fixed through training.
• Dynamic Mask: $\bar u_i \in \mathbb{R}^{S_c \times d}$1 randomly sampled per batch (e.g., $\bar u_i \in \mathbb{R}^{S_c \times d}$2), enabling generalization to varying chunk configurations.

3. State-Caching Inference Algorithm

The inference algorithm maintains, per decoder layer $\bar u_i \in \mathbb{R}^{S_c \times d}$3, fixed-size past-key/value and past-convolution caches for each processed chunk. For an input of $\bar u_i \in \mathbb{R}^{S_c \times d}$4 chunks:

1. For each chunk $\bar u_i \in \mathbb{R}^{S_c \times d}$5, extract $\bar u_i \in \mathbb{R}^{S_c \times d}$6.
2. For every decoder layer:
   • Concatenate past caches with the current chunk for MHA.
   • Update caches using $\bar u_i \in \mathbb{R}^{S_c \times d}$7 operations post-attention and post-convolution.
   • Normalize and propagate to the next block.
3. Project final output to yield Mel chunk $\bar u_i \in \mathbb{R}^{S_c \times d}$8 and emit for audio synthesis.

Per-chunk runtime is $\bar u_i \in \mathbb{R}^{S_c \times d}$9, and the total cost scales linearly with the number of chunks ($S_c$0); inference cost per chunk is independent of chunk index.

4. Training Objectives and Optimization

The multi-loss objective is retained from FastPitch:

$S_c$1

with:

• $S_c$2,
• $S_c$3,
• $S_c$4,
• $S_c$5.

No extra regularization or loss terms are used; the chunked receptive field is enforced purely by the attention mask.

5. Experimental Results: Speech Quality and Latency

Empirical evaluation demonstrates Incremental FastPitch produces speech of comparable quality to parallel FastPitch at substantially reduced latency.

Table 1: Comparative Metrics

Model	MOS ($S_c$6 95% CI)	Latency (ms)	RTF
Parallel FastPitch	4.185 $S_c$7 0.043	125.8	0.029
Inc. FastPitch (Static Mask)	4.178 $S_c$8 0.047	30.4	0.045
Inc. FastPitch (Dynamic Mask)	4.145 $S_c$9 0.052	30.4	0.045
Ground Truth	4.545 $t$0 0.039	—	—

• Mel-Spectrogram Distance: Lowest when $t$1 match at train/test; dynamic masking generalizes but with an $t$28% higher MSD.
• MOS: Incremental FastPitch matches FastPitch (MOS $t$3 4.18) while cutting end-to-end latency by $t$44x (from 125.8 ms to 30.4 ms).
• RTF: Remains well above real-time requirements ($t$51/22).

6. Computational Complexity and Latency Profile

Let $t$6 = number of decoder layers, $t$7 = chunk size, $t$8 = cache size, $t$9 = model dimension, $pk_i^{t-1}$0 = attention heads, $pk_i^{t-1}$1, $pk_i^{t-1}$2 = FFN hidden dimension, $pk_i^{t-1}$3 = conv kernel sizes.

• MHA Cost/Chunk: $pk_i^{t-1}$4
• Conv-FFN Cost/Chunk: $pk_i^{t-1}$5
• Total Per-Chunk: $pk_i^{t-1}$6

These costs are constant with respect to total utterance length ($pk_i^{t-1}$7); only the number of chunks and chunk size matter. Start-up latency is determined by $pk_i^{t-1}$8. With small $pk_i^{t-1}$9 (e.g., $pv_i^{t-1}$0 frames, $pv_i^{t-1}$1 ms at $pv_i^{t-1}$2 kHz and hop $pv_i^{t-1}$3), sub-$pv_i^{t-1}$4 ms startup latency is achievable.

7. Significance and Applications

Incremental FastPitch integrates chunked processing, cache-based attention, and tailored attention masking to deliver high MOS, low-latency, and scalable synthesis. Its design enables real-time, streaming, and interactive TTS workloads with reduced response times and memory requirements, while retaining the benefits of transformer-based parallel TTS models. The architecture provides a framework for incremental speech synthesis with complexity governed by chunk parameters rather than utterance length, pointing to new possibilities for low-latency, high-continuity TTS systems (Du et al., 2024).