PilotTTS: Compact Modular TTS

• The paper introduces PilotTTS, a modular autoregressive TTS system that unifies zero-shot voice cloning, emotion, paralinguistic, and dialect synthesis within a reproducible open-source framework.
• PilotTTS employs a multi-stage data pipeline that standardizes, annotates, and filters raw speech to construct a 200K-hour bilingual corpus with high-quality metrics.
• PilotTTS achieves competitive performance on Seed-TTS Eval by delivering best-in-class speaker similarity and robust content accuracy despite relying on a compact model architecture.

Searching arXiv for the cited PilotTTS paper and related uses of the term. PilotTTS is a lightweight, open-source, autoregressive text-to-speech system presented as a “disciplined modular recipe” for competitive speech synthesis under substantially tighter data and engineering constraints than are typical of many contemporary high-performance TTS systems. The system is trained on about 200K hours of Chinese and English speech data processed entirely with open-source tools, and within a unified framework it supports zero-shot voice cloning, emotion synthesis across 11 categories, paralinguistic synthesis across 4 categories, and Chinese dialect synthesis across 14 dialects. On Seed-TTS Eval, it reports a WER of 1.50% on test-en, a CER of 0.87% on test-zh, and the highest speaker similarity on both sets, with 0.862 on test-zh and 0.815 on test-en (Li et al., 26 May 2026).

1. Research context and stated objective

PilotTTS is framed as a response to three barriers that the paper identifies in modern TTS research: the data barrier, the engineering barrier, and control fragmentation. The data barrier is the dependence of many high-performing systems on millions of hours of proprietary speech data. The engineering barrier is the use of complex multi-stage architectures, including multi-codebook tokenizers, hierarchical prediction modules, and specialized submodels. Control fragmentation refers to the tendency for emotion synthesis, paralinguistic synthesis, and dialect synthesis to be handled by separate systems rather than by a unified framework (Li et al., 26 May 2026).

Within that framing, PilotTTS advances a specific thesis: strong TTS can be obtained from a reproducible, fully open-source data pipeline, a compact autoregressive language-model backbone, decoupled conditioning for speaker identity and speaking style, and targeted post-training for emotion, paralinguistics, and dialects. A common misconception in the area is that competitive performance necessarily depends on extreme data scale and highly bespoke architecture. PilotTTS is presented as a counterexample within the reported experimental setting, although the paper does not claim to solve every controllability or fidelity problem in speech synthesis.

The system is therefore notable less for a single isolated architectural novelty than for an end-to-end recipe that emphasizes disciplined modularity. This includes corpus construction, annotation, filtering, conditioning design, and post-training strategies, all aimed at making a compact system competitive without proprietary infrastructure (Li et al., 26 May 2026).

2. Data processing pipeline and corpus construction

A central component of PilotTTS is its reproducible multi-stage data processing pipeline. The pipeline has three stages: quality assessment and enhancement, label annotation, and quality filtering. Raw audio from heterogeneous sources is standardized to a unified format and sampling rate. Speech segmentation is performed using speech activity detection and speaker change detection. Quality scoring combines DNSMOS for perceptual quality, a speech/non-speech classifier, and SNR estimation. Low-quality segments are identified by criteria such as predicted MOS $\le 3.5$, non-speech labels, and insufficient SNR, and those segments are passed through denoising or enhancement to improve intelligibility (Li et al., 26 May 2026).

The annotation stage constructs rich metadata. ASR transcription is generated from multiple systems, specifically Paraformer, FireRedASR, Whisper, and internal ASR models. The pipeline also includes overlap detection via OSD, forced alignment between text and audio, prosody annotation using Qwen3-Force-Alignment, speaker tagging using 3D-Speaker-Toolkit, and spectral rolloff analysis to remove low-bandwidth recordings. The resulting design treats annotation as a multi-signal inference problem rather than as a single-pass transcription step.

The final filtering stage removes unreliable samples using truncation detection, synthetic speech detection, and joint filtering based on acoustic quality, speech validity, transcription reliability, overlap condition, speaker consistency, truncation risk, synthesis likelihood, and spectral quality. The output is a dataset of about 200,000 hours of clean Chinese/English speech. An important implementation detail is that all excluded data are preserved with metadata, enabling later dataset construction under different quality thresholds. This suggests that the system’s data engineering is intended not only as preprocessing but also as a reusable corpus-management framework (Li et al., 26 May 2026).

3. Model stack, tokenization, and generative formulation

PilotTTS uses a modular stack with four components: a speech tokenizer, an autoregressive text-to-semantic model, a Conditional Flow Matching decoder with a DiT backbone, and a HiFi-GAN vocoder. The speech tokenizer reuses the single-codebook FSQ tokenizer from CosyVoice 3. Its quantization is written as

$$\tilde{H} = \mathrm{ROUND}(\mathrm{Proj}_{down}(H)), \qquad \hat{H} = \mathrm{Proj}_{up}(\tilde{H}),$$

with token index

$$\mu_i = \sum_{j=0}^{D-1} \tilde{h}_{i,j} \cdot (2K+1)^j.$$

The induced codebook size is

$(2K+1)^D = 6561.$

The tokenizer runs at 25 Hz, corresponding to one token per 40 ms, and is pretrained with five supervision signals: ASR, LID, SER, AED, and speaker analysis. The paper characterizes the resulting discrete tokens as semantically rich (Li et al., 26 May 2026).

The autoregressive backbone is Qwen3-0.6B. The generation problem is formulated as conditional autoregressive prediction of audio semantic tokens from text and reference-audio-derived conditions. The paper gives the factorization as

$$p(\mathbf{e}_{\text{Audio}} \mid \mathbf{x}) = \prod_{i=1}^{N_s} p(\mathbf{e}_{\text{Audio},i} \mid \mathbf{x}_{<\text{Audio}}, \mathbf{e}_{\text{Audio},<i}).$$

This places PilotTTS in the class of autoregressive semantic-token TTS systems, but with a deliberately compact backbone and a simplified tokenizer relative to many multi-codebook alternatives.

After semantic token prediction, a CFM decoder with a DiT backbone reconstructs mel spectrograms, and HiFi-GAN converts mels to waveform audio. The decoder is a 300M-parameter module and uses 10-step iterative denoising. Its conditioning includes a reference mel spectrogram, a CAMPPlus speaker embedding, and predicted semantic features. The overall system therefore separates discrete semantic generation from acoustic reconstruction, rather than using end-to-end waveform generation (Li et al., 26 May 2026).

4. Conditioning design, disentanglement strategy, and controllable synthesis

One of the paper’s primary technical contributions is its Q-Former-based conditioning strategy for decoupling speaker identity from speaking style. Speaker identity is captured by a frozen CAMPPlus speaker encoder. Dynamic style and prosody are captured by a Q-Former-based Semantic Content Adapter operating on frozen w2v-BERT embeddings via cross-attention with learnable query vectors, producing a fixed set of style condition tokens. The condition sequence includes speaker identity, 32 Q-Former condition tokens, language and emotion tags, boundary markers, and text and audio token streams (Li et al., 26 May 2026).

The paper motivates this design through a comparison with two prior conditioning patterns. Audio-token continuation is described as rich but fragile to noise and long prompts, while speaker-embedding-only conditioning is described as robust but prone to losing detailed timbre and style cues. PilotTTS combines both pathways: CAMPPlus is intended to provide stable identity information, while the Q-Former tokens encode style cues. To reduce content leakage, the system uses cross-sample paired training: for each training sample, the speaker embedding and style conditions are extracted from a different utterance by the same speaker. The paper states that this forces the conditioner to encode speaker-related attributes independent of content.

This conditioning design underlies the model’s controllable synthesis capabilities. The supported tasks and scopes are as follows.

Capability	Scope	Training note
Zero-shot voice cloning	Arbitrary voices from short reference audio	Evaluated on Seed-TTS Eval
Emotion synthesis	11 categories	Post-training on about 2,200 hours
Paralinguistic synthesis	4 core phenomena plus LAUGH_SPAN	Explicit and implicit control
Chinese dialect synthesis	14 dialects	About 16,000 hours of dialect data

The 11 emotion categories are divided into 7 primary emotions—happy, sad, angry, fear, contempt, serious, and surprise—and 4 extended emotions—concern, blue (melancholy), disgust, and psychology (inner monologue). The 4 core paralinguistic phenomena are laughter, breathing, crying, and coughing, with LAUGH_SPAN defined as a wrapped laughter mode where laughter is temporally coupled with speech. Dialect synthesis uses dialect–Mandarin parallel pairs created by synthesizing Mandarin utterances for dialect speakers, together with mixed-prompt sampling in which the prompt may be Mandarin or dialect. This suggests that PilotTTS treats controllability as an extension of the same conditioning framework rather than as a collection of isolated add-on modules (Li et al., 26 May 2026).

5. Empirical performance on Seed-TTS Eval

PilotTTS reports zero-shot results on the Seed-TTS Eval benchmark for both Chinese and English evaluation sets. The benchmark metrics are CER and speaker similarity on test-zh, and WER and speaker similarity on test-en. The reported PilotTTS results are summarized below (Li et al., 26 May 2026).

Evaluation split	Metric	Result
test-zh	CER	0.87%
test-zh	Speaker similarity	0.862
test-en	WER	1.50%
test-en	Speaker similarity	0.815

The paper identifies several comparison points from the benchmark table. MiniMax-Speech reports the best CER on test-zh at 0.83%. PilotTTS reports the best WER on test-en at 1.50%. It also reports the best speaker similarity on both evaluation sets, at 0.862 for test-zh and 0.815 for test-en. The paper emphasizes that these results are obtained with only about 200K hours of data, whereas several competitors are described as relying on much larger proprietary corpora (Li et al., 26 May 2026).

The significance attributed to these results is fourfold: higher data efficiency, better speaker fidelity, competitive or best-in-class content accuracy, and strong performance despite a compact architecture. The paper particularly links the speaker similarity gains to the dual-pathway conditioning design and the high-quality data pipeline. A plausible implication is that, within this setup, data curation and disentangled conditioning contributed at least as much as raw parameter count or corpus scale.

6. Limitations, reproducibility, and broader usage of the name

PilotTTS is explicit about several limitations. First, it has insufficient explicit style modeling: style is captured implicitly through the Q-Former, and expressive detail may therefore be limited. The paper points to future work on explicit global and local style encoders. Second, the single-codebook FSQ tokenizer is simpler but less expressive than multi-codebook RVQ or continuous latent systems, which may limit scaling to singing or background music. Third, the system uses lossy mel spectrogram generation followed by a vocoder, which can introduce distortion relative to end-to-end waveform generation (Li et al., 26 May 2026).

At the same time, reproducibility is treated as a first-class outcome. The release includes code, pretrained weights, and the complete data pipeline recipe, with the repository hosted at https://github.com/AMAPVOICE/PilotTTS. This is important because the paper’s central claim is not only that the system performs well, but that the recipe is reproducible using open-source tooling rather than proprietary infrastructure.

The name “PilotTTS” is also terminologically non-unique in the supplied literature. In a separate 2022 paper, PilotTTS denotes an AI-based pilot training system for flight simulation, where a behavioral-cloning agent learns from expert demonstrations and provides formative feedback for straight and level flying in X-Plane (Guevarra et al., 2022). A later communications paper does not define PilotTTS as a formal module, but it describes a conceptually related “PilotTTS-like” use of semantics at the receiver side: an LLM compares initially decoded text with corrected text to identify reliable symbols, which are then reused as a semantic pilot for data-aided channel estimation (Park et al., 4 Feb 2026). In current technical usage, however, the most direct and formal use of “PilotTTS” as a named system refers to the modular speech synthesis framework introduced in 2026 (Li et al., 26 May 2026).