- The paper introduces a transcript-free framework using WavLM-based self-supervised features mapped into the F5-TTS conditioning space.
- The paper achieves notably lower WER and higher MOS scores on both typical and atypical speaker benchmarks compared to text-conditioned methods.
- The paper employs a robust two-stage training strategy that jointly fine-tunes the SSL projector and diffusion transformer to balance intelligibility with speaker identity preservation.
Transcript-Free Flow-Matching TTS via Speech Feature Conditioning
Motivation and Problem Statement
Zero-shot TTS, especially for atypical speakers (e.g., dysarthric, heavily accented), remains fragile due to reliance on reference transcripts at inference—often obtained via ASR systems prone to substantial errors in challenging speech. Even oracle transcripts can degrade performance, as text-based conditioning may propagate atypical acoustic patterns. This mismatch is particularly problematic for NAR flow-matching architectures like F5-TTS, which infill masked mel-spectrogram segments using concatenated reference and target transcripts. The authors propose removing transcript dependency entirely to improve synthesis quality for non-standard speech.
Approach: RTFree-F5 Architecture
RTFree-F5 replaces text-based reference conditioning with projected self-supervised speech features. Given reference audio, WavLM-encoded SSL features are mapped into the F5-TTS text-conditioning space via a lightweight two-layer MLP projector, which forms "latent text" features for the reference. These are concatenated with target text features and input to the Diffusion Transformer backbone, retaining compatibility with the original architecture and checkpoint.
- Self-supervised feature extraction: WavLM encodes reference audio, providing frame-level representations that intrinsically capture speaker and content characteristics.
- Modality bridging: An MLP projects 1024-dim WavLM features to 512-dim F5-TTS conditioning, with upsampling to match the mel frame rate.
- Two-stage training: Stage 1 aligns SSL and text modalities (projector optimized, backbone frozen). Stage 2 jointly fine-tunes projector and flow-matching backbone (encoders remain frozen) to address cross-utterance distribution shift.
Experimental Protocol
RTFree-F5 builds on pretrained F5-TTS v1 and frozen WavLM-Large. Training leverages cross-utterance pairs from the same speaker to ensure robust speaker representation learning without relying on local acoustic continuity. Evaluation covers typical (LibriSpeech-PC, SeedTTS) and atypical (SAP, L2-ARCTIC) speaker benchmarks, employing WER (Whisper-v3), ECAPA-TDNN speaker similarity, and UTMOS-predicted naturalness.
Results
Typical Speaker Benchmarks
RTFree-F5 (Stage 2) achieves lower WER than oracle and ASR baselines (LibriSpeech-PC: 1.77% vs. 2.08% and 2.17%), with improved MOS (4.13 vs. 3.83/3.84). Speaker similarity is comparable. Stage 1—projector-only training—underperforms (4.68% WER), emphasizing the necessity of joint fine-tuning.
Atypical Speaker Benchmarks
SAP (dysarthric): RTFree-F5 drops WER from 24.62% (original) to 10.39%, outperforming even oracle-transcript-based F5-TTS (20.71%). MOS increases from 2.16 to 2.85. Speaker similarity decreases (0.50 vs. 0.60), revealing a trade-off between intelligibility and strict identity preservation. L2-ARCTIC (non-native): RTFree-F5 reduces WER from 10.75% to 1.44%, surpasses the oracle baseline (2.00%), and achieves higher MOS (4.08).
Analysis of Conditioning Mismatch
Text-based reference conditioning encodes normative phonetic expectations that often conflict with the acoustic context of atypical or non-native speech, leading to poor infilling. SSL features, in contrast, produce conditioning intrinsically aligned with acoustic reality, enabling the flow-matching backbone to synthesize speech with typical pronunciation guided by the target text, while retaining speaker-specific traits. The results demonstrate that transcript-free conditioning via projected SSL features not only removes ASR dependency but fully surpasses text-conditioned setups, even with oracle transcripts.
Implications and Future Directions
Practically, RTFree-F5 enables zero-shot voice cloning and speech reconstruction for atypical speakers without reference transcripts. The approach is poised to enhance accessibility applications, dysarthric speech synthesis, and multi-accent TTS. Theoretically, the results highlight that textual conditioning is fundamentally brittle under pathological or accented speech regimes—a limitation not fully resolved by improved ASR or transcript quality.
Open avenues include refining identity preservation under speech normalization, improved SSL projector architectures, multimodal conditioning (joint SSL-text supervision), and broader exploration of pathological/accented speech for generalized voice synthesis. The interaction between SSL and text features in conditioning flow-matching models warrants further investigation for model interpretability and control.
Conclusion
RTFree-F5 demonstrates that self-supervised speech feature conditioning, mapped into flow-matching TTS architectures, achieves superior intelligibility and naturalness for both typical and atypical speakers, eliminates transcript dependency at inference, and outperforms even oracle transcript setups on challenging speech. This framework marks a shift toward transcript-free, robust TTS for diverse voices and pathologies, with broad implications for accessibility and non-autoregressive neural synthesis.