Streaming T5-based Text-to-Speech Synthesis with Limited Lookahead

Abstract: Streaming text-to-speech synthesis in cascaded LLM-TTS systems still faces latency challenges as most TTS models require full context before initiating generation. We present S5-TTS, a streaming variant of T5-TTS that enables low-latency, word-by-word incremental speech synthesis through encoder-decoder language modeling and monotonic alignment learning. S5-TTS begins generating speech immediately after receiving the first few words, substantially reducing end-to-end response latency. To maintain quality under limited lookahead, we introduce a lookahead-causal masking mechanism with Conv-based auxiliary attention that preserves intelligibility and speaker similarity, and employ interleaved multi-source distillation to further restore naturalness. Experiments show that S5-TTS achieves comparable quality to full-context T5-TTS, supports zero-shot synthesis with high speaker similarity, and significantly reduces end-to-end latency for practical conversational AI systems.

• The paper introduces S5-TTS, a novel streaming T5-based text-to-speech model that employs limited lookahead, monotonic alignment, and interleaved multi-source distillation to significantly reduce synthesis latency.
• It demonstrates that using a constrained lookahead (k=1 or k=2) yields intelligibility and speaker similarity metrics that match or exceed those of full-context TTS systems, as validated on LibriTTS.
• Extensive ablations confirm that lookahead-causal masks and auxiliary attention are essential for maintaining speech quality, resulting in a real-time factor of 0.609 and minimal first-chunk latency.

Streaming T5-Based Text-to-Speech Synthesis with Limited Lookahead: An Authoritative Review

Motivation and Background

The widespread adoption of LLM-TTS pipelines in practical conversational AI systems exposes critical latency bottlenecks, particularly in scenarios where the TTS module cannot commence synthesis until the entire textual input is available. Conventional TTS architectures—both traditional and recent LLM-based TTS variants—are designed for full-context generation, thereby impeding incremental, low-latency speech output. Previous neural iTTS efforts have demonstrated the utility of limited lookahead (one or two future words) for preserving naturalness and intelligibility, yet existing solutions are either restricted to constrained data/speaker domains or suffer degradation in zero-shot and naturalness metrics. Addressing these deficiencies, the paper proposes S5-TTS: a streaming, encoder-decoder, T5-based text-to-speech model with monotonic alignment learning and specialized architectural innovations for robust, incremental speech synthesis under limited lookahead (2606.21882).

Model Architecture and Streaming Protocol

S5-TTS employs a parallel Transformer encoder and autoregressive Transformer decoder, structured around word-level streaming. The encoder receives a sequence of phonemes (from G2P conversion), processing the current word, preceding words, and $k$ lookahead words. For each decoding step, the decoder generates $K$ new codec tokens across multiple codebooks (for FSQ audio codec reconstruction). Generation proceeds incrementally: after receiving a minimal prefix (three words for $k=2$), S5-TTS initiates speech synthesis and continues chunk-wise generation, with tight coupling between cross-attention alignment and word boundaries.

Figure 1: The left subpanel illustrates the full streaming encoder-decoder architecture; the right highlights lookahead-causal mask matrices and Conv-based auxiliary attention used for monotonic alignment and mask construction.

Word boundary completion is monitored via cross-attention maxima, using:

$$c_s^i = \operatorname{argmax}(\alpha_s) > \Biggl[ \left( \sum_{j=1}^i W_j \right) - 1 \Biggr]$$

Synthesized chunks are then post-processed for chunk overlap/crossfade smoothing.

Lookahead-Causal Masks and Auxiliary Attention

Critical to maintaining intelligibility and preventing exposure bias under streaming conditions, S5-TTS introduces word-indexed lookahead-causal masks applied to both encoder self-attention and decoder cross-attention. Explicit mask construction restricts attention to the current and past context, plus $k$ future words, for both encoder and decoder. The decoder mask leverages a Conv-based auxiliary attention module for word-to-phoneme-to-codec-step mapping during alignment, using MAS and CTC losses to enforce monotonicity and robustness. The auxiliary attention jointly optimizes monotonic path extraction and mask consistency during training, while only reference audio alignment is computed during inference.

Interleaved Multi-Source Distillation

To mitigate naturalness degradation under lookahead constraints, S5-TTS undergoes interleaved multi-source distillation (IMSD) using both parallel text-audio and pure text datasets. The teacher (full-context T5-TTS) supplies soft labels under teacher forcing for audio data and via sampling followed by ASR filtering for text-only data (retention with zero WER). IMSD alternates batches during gradient accumulation, jointly optimizing for decoder hidden state MSE, logits KL divergence, and cross-entropy with hard/soft targets.

Empirical Results: Quality and Latency

Intelligibility and Speaker Similarity

S5-TTS achieves intelligibility and speaker similarity scores equivalent to full-context T5-TTS. For LibriTTS, with $k=1$ and $k=2$, WER is 3.03% and 3.49%, respectively, speaker similarity cosine values exceed 0.932, and UTMOS subjective metrics are within 0.1 of ground-truth TTS. Notably, S5-TTS with $k=1$ yields lower WER than T5-TTS, asserting that reduced lookahead can provide improved focusing on current lexical content (contradicting the common expectation that larger context always improves TTS performance). Increasing $k$ to 3, however, results in substantial intelligibility degradation, affirming the necessity of optimizing lookahead for balance between quality and streaming requirements.

Effectiveness of Key Innovations

Ablations demonstrate that removal of encoder and decoder lookahead-causal masks dramatically impairs intelligibility (WER rises to 40.15% without encoder mask), while the lack of decoder mask shows less severe but still significant degradation. These findings robustly validate architectural choices in S5-TTS for streaming operation.

IMSD distillation substantially elevates speech naturalness and intelligibility (LibriTTS WER from 3.49% to 2.65%, UTMOS from 3.66 to 3.72), bringing subjective MOS within 0.04-0.09 of the full-context T5-TTS, thus restoring perceptual quality under strict streaming constraints.

Comparative Evaluation

Despite being trained on only 4.67K hours (vs. comparators with >100K), S5-TTS surpasses larger AR/NAR TTS models in objective STOI and PESQ, and achieves WER/UTMOS competitive with (or better than) MaskGCT, E2-TTS, CosyVoice, and FireRedTTS. Speaker similarity is preserved across unseen domains and voices, with successful zero-shot synthesis demonstrated.

Efficiency Metrics and Practical Utility

S5-TTS exhibits a real-time factor (RTF) of 0.609 and first-chunk latency (FCL) of 0.169s, nearly halving the end-to-end latency relative to T5-TTS in LLM-TTS cascades (0.343s vs. 0.728s E2E latency). This enables S5-TTS to start synthesis after only a minimal prefix, supporting practical real-time conversational AI scenarios.

Implications and Future Perspectives

The S5-TTS methodology advances incremental TTS with:

• Full encoder-decoder streaming operation
• Robust monotonic alignment under limited lookahead
• Effective quality restoration for both naturalness and intelligibility in zero-shot multi-speaker settings

These contributions could catalyze new architectures for low-latency neural TTS, particularly in modular LLM-driven dialogue systems, phone-level prosody modeling, and scalable multi-lingual synthesis. There remains potential for tighter integration of streaming TTS with streaming LLMs and for enhanced cross-modal prompt-based controls. Additionally, the results challenge assumptions around context size in TTS alignment, motivating deeper investigation into trade-offs between lookahead, alignment monotonicity, and incremental synthesis fidelity.

Conclusion

S5-TTS achieves low-latency, word-by-word streaming text-to-speech synthesis with quality and speaker similarity matching full-context T5-TTS, enabling practical deployment in real-time, modular conversational AI systems. Its architectural and distillation advances demonstrate that meaningful lookahead constraints can be imposed without significant degradation, supporting both theoretical insights and practical innovations in streaming neural TTS (2606.21882).