PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis

Abstract: Building state-of-the-art text-to-speech (TTS) systems typically demands millions of hours of proprietary data and complex multi-stage architectures, creating substantial barriers for resource-constrained research teams. In this report, we present PilotTTS, a lightweight autoregressive TTS system that achieves competitive performance through minimalist architecture and rigorous data engineering. PilotTTS is trained on only 200K hours of data processed entirely with open-source tools. Specifically, our contributions are: (1) a reproducible multi-stage data processing pipeline covering quality assessment, label annotation, and filtering, and (2) a compact model architecture that employs Q-Former-based conditioning to decouple speaker identity from speaking style via cross-sample paired training. Within a unified framework, PilotTTS supports zero-shot voice cloning, emotion synthesis (11 categories), paralinguistic synthesis (4 categories), and Chinese dialect synthesis (14 dialects). On the Seed-TTS Eval benchmark, PilotTTS achieves the lowest WER of 1.50% on test-en, a CER of 0.87% on test-zh, and the highest speaker similarity on both test sets (0.862 and 0.815), outperforming systems trained on significantly larger datasets. We release the complete data pipeline recipe, pretrained weights, and code at https://github.com/AMAPVOICE/PilotTTS.

• The paper demonstrates that strategic data engineering and modular integration enable competitive TTS performance with significantly reduced data requirements.
• It employs a dual conditioning mechanism combining semantic and speaker embeddings to effectively decouple style and identity for enhanced control.
• Experimental results reveal high speaker similarity, low error rates, and advanced capabilities in emotion, paralinguistic, and dialect synthesis.

PilotTTS: A Data-Disciplined Modular Approach to Competitive Speech Synthesis

System Motivation and Challenges

The development of contemporary TTS systems has largely trended toward high data volume and architectural complexity, with leading zero-shot voice cloning and expressivity models requiring datasets spanning millions of hours and intricate networks combining LLMs, codebook quantization, and multi-stage pipelines. This presents a reproducibility and accessibility barrier, especially for research groups lacking proprietary resources or commercial-scale infrastructure. PilotTTS directly addresses this challenge, demonstrating that competitive performance can be attained with strategic data engineering and modular integration based on open-source modules.

Figure 1: Overview of PilotTTS.

Data Processing Pipeline

PilotTTS’s cornerstone is a reproducible, multi-stage pipeline encompassing acoustic quality assessment/enhancement, extensive annotation, and multi-dimensional quality filtering. All components are openly sourced, systematically standardizing input, segmenting speech, assessing quality (DNSMOS, SNR, speech/non-speech classification), and enabling enhancement (denoising). Robust transcription is achieved via cross-system ASR (Paraformer, FireRedASR, Whisper), alongside forced alignment and prosody annotation. Speaker tagging leverages 3D-Speaker-Toolkit, and subsequent quality filtering incorporates criteria for segmentation, synthetic speech detection, and spectral rolloff. The pipeline ultimately yields ∼200K hours of curated data, facilitating downstream model robustness and generalization with substantially lower acquisition and processing cost.

Figure 2: Three-stage data processing pipeline ensuring rigorous data quality and annotation for model training.

Model Architecture and Conditioning Mechanism

PilotTTS adopts a compact yet effective modular architecture with four primary components: a single-codebook speech tokenizer (CosyVoice 3 FSQ), a Qwen3 autoregressive LLM, a Conditional Flow Matching (CFM) decoder (DiT backbone, 300M parameters), and a HiFi-GAN vocoder. The referenced speech input is processed through two distinct conditioning pathways—a Q-Former-based Semantic Content Adapter for dynamic speaking style extraction (cross-attention with w2v-BERT embeddings, Conformer block, and linear projections), and a frozen CAMPPlus speaker encoder for global timbre embedding. This dual-pathway approach enables the model to robustly decouple speaker identity from speaking style, a feature critical for generalization and controllability across zero-shot voice cloning, emotion, paralinguistics, and dialect synthesis.

Figure 3: Architecture of PilotTTS featuring modular components and dual-pathway condition extraction.

The training regimen includes cross-sample paired strategies wherein reference and target utterances are drawn separately from the same speaker, enforcing conditional disentanglement.

Controllability and Extended Capabilities

Beyond zero-shot TTS, PilotTTS leverages its modular conditioning to support advanced synthesis controls:

• Emotion Synthesis: Explicit emotion categories (11 total) are enabled via post-training on 2.2K hours labeled datasets, achieving high accuracy and speaker similarity preservation.
• Paralinguistic Synthesis: Four major phenomena (laughter, breath, crying, cough), plus LAUGH_SPAN (synchronous laughter), are realized through targeted SFT using 200 hours of curated data. Both implicit (text contextual inference) and explicit (onomatopoeic triggers) synthesis modes are supported.
• Dialect Synthesis: Addressing dialectal data scarcity, parallel “dialect–Mandarin” pairs are constructed by model synthesis, with mixed-prompt training allowing transfer from Mandarin reference audio to target dialect while retaining speaker identity.

Experimental Results

On the Seed-TTS Eval benchmark, PilotTTS delivers:

• Speaker Similarity: 0.862 (test-zh), 0.815 (test-en), highest among all reported systems.
• Content Accuracy: CER 0.87% (second only to MiniMax-Speech with a marginal difference), WER 1.50% (lowest).
• Data Efficiency: Achieving these metrics with only 200K hours, PilotTTS outperforms or rivals systems trained on an order of magnitude more data and complexity.

Emotion control evaluation demonstrates a primary emotion success rate of 88.1%, surpassing CosyVoice 3 and other baselines, while maintaining highest speaker similarity under both neutral and emotion-controlled conditions. Paralinguistic synthesis achieves an overall success rate of 85.1% across common categories, with special capability in LAUGH_SPAN (94.6%) and CRY (61.9%) absent in baseline models. Dialect synthesis, evaluated subjectively, achieves >85% accuracy even in cross-dialect scenarios.

Ablation studies confirm the indispensability of Q-Former condition tokens for content accuracy, especially in hard acoustic cases, and the complementary role of CAMPPlus speaker embeddings in maximizing speaker similarity.

Theoretical and Practical Implications

The minimalist, modular pilot architecture validates the hypothesis that thoughtful data engineering and strategic integration of open-source components can rival or surpass trends in data scaling and architectural complexity. The explicit dual-path conditioning significantly advances disentanglement strategies, simplifying subsequent controls for emotion, paralinguistics, and dialect while facilitating robust zero-shot performance.

From a practical deployment perspective, PilotTTS’s approach lowers the entry threshold for competitive TTS system development. The data pipeline is reproducible, scalable, and amenable to rapid adaptation for domain-specific requirements.

On the theoretical front, the architectural ceiling of single-codebook quantization and implicit style modeling hints at future pathways for enhanced expressivity, granularity, and domain transfer—particularly for singing and music. The modular design also provides a sandbox for experimenting with more explicit and high-capacity style encoders, and for advancing workflow toward end-to-end waveform generation as opposed to indirect mel/reconstruction.

Future Research Directions

PilotTTS identifies three core development axes:

1. Explicit Style Modeling: Implementation of style encoders capable of fine-grained, continuous representation beyond Q-Former’s current implicit scope.
2. High-Capacity Quantization: Integration of multi-codebook RVQ or continuous latent tokenization schemes to support more complex generative domains.
3. End-to-End Waveform Generation: Transition from mel spectrogram-based decoders and vocoders to direct waveform synthesis architecture to minimize lossy reconstruction.

These directions are foundational for advancing generalizability, expressivity, and domain extension in LLM-based TTS synthesis.

Conclusion

PilotTTS exemplifies disciplined, reproducible engineering driven by modular integration and rigorous data curation, rather than data scale or architectural novelty. Its performance metrics and controllability features validate its practicality for commercial and research deployment. The approach also establishes a replicable standard for future developments, emphasizing accessibility and extensibility as central design tenets.