MOSS-TTS: Scalable Speech Synthesis Model

• The paper introduces MOSS-TTS as a model built on a scalable discrete-token autoregressive pipeline for end-to-end text-to-speech generation.
• It leverages the MOSS-Audio-Tokenizer for unified semantic-acoustic representations, enabling features like zero-shot voice cloning, token-level duration control, and smooth code-switching.
• The design includes both a single-backbone and a local autoregressive variant, achieving competitive speaker similarity and intelligibility in multilingual and long-context scenarios.

Searching arXiv for MOSS-TTS and closely related papers. MOSS-TTS is a speech generation foundation model built on a scalable recipe: discrete audio tokens, autoregressive modeling, and large-scale pretraining. Built on MOSS-Audio-Tokenizer, a causal Transformer tokenizer that compresses 24 kHz audio to 12.5 fps with variable-bitrate residual vector quantization (RVQ) and unified semantic-acoustic representations, it releases two complementary generators: MOSS-TTS, which emphasizes structural simplicity, scalability, and long-context/control-oriented deployment, and MOSS-TTS-Local-Transformer, which introduces a frame-local autoregressive module for higher modeling efficiency, stronger speaker preservation, and a shorter time to first audio. Across multilingual and open-domain settings, the system supports zero-shot voice cloning, token-level duration control, phoneme-/pinyin-level pronunciation control, smooth code-switching, and stable long-form generation (Gong et al., 18 Mar 2026).

1. Terminology, scope, and design philosophy

MOSS-TTS is presented as an open speech generation foundation model for text-to-speech and broader speech generation. Its central claim is that modern TTS can be built effectively around a simple, scalable recipe—discrete audio tokens, autoregressive modeling, and large-scale pretraining—rather than around many intermediate modules, external semantic teachers, cascaded supervision, refinement stages, or post-hoc alignment. In that framing, the problem is recast as universal next-token prediction over a learned discrete speech interface (Gong et al., 18 Mar 2026).

The naming around the system is slightly inconsistent across papers. In the MOSS-Audio-Tokenizer paper, the downstream TTS model is called CAT-TTS, while the technical report uses MOSS-TTS. The tokenizer paper explicitly states that “MOSS-TTS / CAT-TTS” refers to the same tokenizer-enabled TTS line: a downstream model that predicts CAT or MOSS-Audio-Tokenizer RVQ tokens from text plus a speaker prompt, then reconstructs waveform using the tokenizer decoder. The distinction is therefore terminological rather than architectural (Gong et al., 11 Feb 2026).

This design places MOSS-TTS within the recent wave of speech generation foundation models, but its differentiating emphasis is an end-to-end scalable discrete-token pipeline. A plausible implication is that the authors view tokenizer quality as the central systems problem: if the representation is sufficiently compact, semantically informative, and high fidelity, the generator can remain comparatively simple while still supporting multilinguality, controllability, and long-context robustness (Gong et al., 18 Mar 2026).

2. Representation layer: MOSS-Audio-Tokenizer and the discrete speech interface

The foundational representation layer is MOSS-Audio-Tokenizer, a causal Transformer-based discrete audio tokenizer based on the CAT architecture. It is designed specifically to support large-scale autoregressive audio and speech modeling. The tokenizer uses 68 causal Transformer blocks in both encoder and decoder, a 10-second sliding-window attention mechanism, progressive patchify operations with patch sizes 240, 2, 2, and 2, and 32-layer RVQ with codebook size 1024, factorized vector quantization latent dimension 8, L2-normalized codes, and random quantizer dropout with probability 1.0 during training. The effective result is compression of 24 kHz waveform audio into a discrete sequence at 12.5 fps, with variable bitrate from 0.125 to 4 kbps (Gong et al., 18 Mar 2026).

The tokenizer is not optimized only for waveform reconstruction. It also includes a 0.5B decoder-only causal LLM as a semantic head trained on ASR, multi-speaker ASR, and audio captioning, which is what the report means by unified semantic-acoustic representations. The semantic audio-to-text objective is

$$\mathcal{L}_{\mathrm{sem}} = - \sum_{t=1}^{|\mathbf{s}|} \log p_{\theta_{\mathrm{LLM}}} \left( \mathbf{s}_t \,\middle|\, \mathcal{T},\, \mathbf{q},\, \mathbf{s}_{<t} \right),$$

while the reconstruction term is

$$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$

These terms are combined with commitment, codebook, adversarial, and feature-matching losses in the tokenizer’s multitask training objective (Gong et al., 18 Mar 2026).

For downstream TTS, this interface matters because token length and token semantics jointly determine feasibility. The tokenizer paper argues that strict causality, low frame rate, RVQ depth control, high reconstruction fidelity, and end-to-end learning without pretrained SSL encoders or semantic distillation make the token space a practical target for a single autoregressive TTS model. The model can therefore generate a unified token space that contains both linguistic content and acoustic information, rather than relying on explicit semantic-acoustic disentanglement or a cascaded semantic-to-acoustic stack (Gong et al., 11 Feb 2026).

This representation choice is consistent with earlier work on quantized hierarchical TTS targets. A multi-stage multi-codebook VQ-VAE study showed that replacing direct mel prediction with structured, quantized, multi-resolution acoustic representations can improve quality and parameter efficiency, which helps situate MOSS-TTS within a broader movement away from flat mel-spectrogram targets (Guo et al., 2022).

3. Generator architectures: delay pattern and frame-local autoregression

Both released generators operate on the same tokenizer outputs. With $N_q = 32$ RVQ layers, both architectures predict $N_h = N_q + 1 = 33$ channels per aligned step: one text-or-pad channel and 32 audio channels. Training uses head-wise weighted cross-entropy that up-weights earlier, coarser RVQ layers, reflecting the importance of low-depth RVQ structure for intelligibility and speaker identity (Gong et al., 18 Mar 2026).

The first architecture, MOSS-TTS, uses a delay pattern and a single Transformer backbone. If $\mathbf{a}_{j,t}$ denotes the token at RVQ layer $j$ and frame $t$, the delayed representation is

$$\tilde{\mathbf{a}}_{j,t} = \mathbf{a}_{j, t - (j-1)}.$$

Each delayed time step aggregates per-layer embeddings by summation,

$$\mathbf{h}_{t} = \sum_{j=1}^{N_q} \operatorname{Emb}_j(\tilde{\mathbf{a}}_{j,t}),$$

yielding a sequence of length $T + N_q - 1$ concatenated with text embeddings. The practical meaning of the design, as the report states explicitly, is structural simplicity, scalability, and long-context/control-oriented deployment: a single backbone, no extra within-frame autoregressive module, easier engineering, easier optimized inference backends, and better suitability for long contexts and explicit conditioning tasks such as duration and pronunciation control. The released MOSS-TTS model evaluated for cloning is 8B parameters (Gong et al., 18 Mar 2026).

The second architecture, MOSS-TTS-Local-Transformer, is hierarchical. It dispenses with delay shifting and instead lets the backbone emit a global latent per step, after which a lightweight Local Transformer autoregressively expands that latent into the within-step block $$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$0. Its local input is

$$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$1

The report characterizes this design as more modeling-efficient, with lower per-layer token losses during internal development, stronger speaker preservation, better quality at smaller scale, and shorter time to first audio because it does not need to wait for delayed offsets before producing the first frame. The tradeoff is a heavier steady-state decoding path due to the inner autoregressive loop over channels. The released MOSS-TTS-Local-Transformer is 1.7B parameters (Gong et al., 18 Mar 2026).

In the tokenizer paper, the same downstream line is described using a closely related factorization: a Temporal Transformer over time and a Depth Transformer over RVQ depth within each frame. That paper initializes the Temporal Transformer from Qwen3-1.7B and uses a 4-block Depth Transformer with hidden size 1536 and FFN dimension 8960. The common systems principle is the same: temporal causality across frames and depth-wise causality across RVQ layers, with direct prediction of tokenizer tokens rather than mel frames (Gong et al., 11 Feb 2026).

4. Data pipeline, preprocessing, and curriculum

The scale of pretraining is one of the defining properties of MOSS-TTS. At the tokenizer level, MOSS-Audio-Tokenizer is trained on approximately 3 million hours of speech, sound, and music, including both clean and in-the-wild recordings. For speech generation, the tokenizer paper states that training uses VoxBox plus an internal dataset totaling approximately 200k hours of speech data, whereas the technical report describes the broader MOSS-TTS pretraining corpus as millions of hours in total and does not provide exact hour counts per language or domain in text (Gong et al., 11 Feb 2026, Gong et al., 18 Mar 2026).

The technical report provides a detailed preprocessing and gating pipeline. Noise suppression uses MossFormer2-SE-48K. Speaker diarization uses DiariZen, discarding segments shorter than $$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$2 s and merging consecutive same-speaker segments. A hard one-hour truncation is applied at $$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$3. Transcript generation uses MOSS-Transcribe-Diarize, followed by empty-content filtering, repetition-loop filtering, non-speech-dominance filtering, LLM-based diagnosis and refinement, and single-speaker transcript validation. Joint audio-text filtering then applies DNSMOS $$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$4, Meta AudioBox PQ $$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$5, audio/text language agreement, and transcript-length consistency (Gong et al., 18 Mar 2026).

The curriculum is organized around five subsets: $$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$6, $$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$7, $$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$8, $$\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.$$9, and $N_q = 32$0. Pretraining follows a four-phase WSD-style curriculum. Phase 1 uses $N_q = 32$1 only with max sequence length 32k and learning-rate warmup to $N_q = 32$2. Phase 2 keeps max sequence length 32k, enables all subsets, and strongly upsamples $N_q = 32$3. Phase 3 linearly decays learning rate from $N_q = 32$4 to $N_q = 32$5 while restoring the normal data mixture. Phase 4 extends context from 32k to 64k, holds learning rate at $N_q = 32$6, and heavily upsamples long-form data (Gong et al., 18 Mar 2026).

Some implementation details remain absent. The report explicitly does not provide optimizer type, batch size for TTS generator pretraining, hardware, total step count, exact model hidden sizes and layer counts for the generator backbones, sequence packing details, or inference decoding hyperparameters. Those omissions matter for strict reproducibility (Gong et al., 18 Mar 2026).

5. Conditioning interfaces, control mechanisms, and inference modes

MOSS-TTS conditions on text and, when required, a speech prompt or reference audio. Depending on task, the input may also include the ASR transcript of the prompt in continuation mode, an explicit duration token storing the target audio-token count, tone-marked pinyin for Chinese, or IPA spans enclosed in slashes for English. The report does not provide a fully specified serialized template for every input type, but it makes clear that pronunciation control is implemented on the text side through phonetic replacement rather than by a separate pronunciation head (Gong et al., 18 Mar 2026).

Zero-shot voice cloning is learned through targeted data synthesis. For a target segment $N_q = 32$7, prompt candidates are cropped from other same-speaker segments $N_q = 32$8, each crop at most 30 seconds, with five random crops per segment. Candidates are scored by cosine similarity of speaker embeddings from a fine-tuned WavLM-Large model, and the most similar crop is selected as the prompt. At inference, the report distinguishes Clone, in which a user explicitly provides reference audio, from Continuation, in which the model continues speech after prepending the reference audio and its transcript to the assistant-side speech/text prefix. The tokenizer paper likewise describes inference as continuation-based, concatenating prompt transcription tokens, target text tokens, and prompt audio tokens into a single sequence (Gong et al., 18 Mar 2026, Gong et al., 11 Feb 2026).

Duration control is token-based rather than predictor-based. Since the tokenizer runs at 12.5 audio tokens per second, a target token count $N_q = 32$9 corresponds to

$N_h = N_q + 1 = 33$0

If realized duration is $N_h = N_q + 1 = 33$1, relative duration error is

$N_h = N_q + 1 = 33$2

The key design point is that duration is controlled directly at the audio-token count level, not through a separate duration model of the classical FastSpeech type (Gong et al., 18 Mar 2026).

Variable bitrate is supported through RVQ-depth control. In the tokenizer paper, Progressive Sequence Dropout samples whether to truncate depth during training:

$N_h = N_q + 1 = 33$3

with $N_h = N_q + 1 = 33$4 when dropout is applied. Training then predicts only the retained RVQ prefix, which makes the model robust when inference asks it to generate only the first $N_h = N_q + 1 = 33$5 RVQ layers. That paper reports that larger $N_h = N_q + 1 = 33$6 significantly reduces GPU memory use and chooses $N_h = N_q + 1 = 33$7 for subsequent experiments (Gong et al., 11 Feb 2026).

The report also claims smooth code-switching and stable long-form generation. The long-form capability is attributed to low-frame-rate discrete tokens, autoregressive modeling, the long-context curriculum, and the final context extension from 32k to 64k. The main remaining failure mode is cumulative speaker drift rather than immediate lexical breakdown (Gong et al., 18 Mar 2026).

6. Empirical characteristics and benchmark results

The tokenizer paper reports the headline CAT-TTS objective results on Seed-TTS-Eval as WER 1.89 and SIM 73.1 for English, and CER 1.23 and SIM 78.5 for Chinese. The paper is careful about what this supports: CAT-TTS achieves the highest speaker similarity scores on Seed-TTS-Eval for both English and Chinese among the compared open-source models, but its intelligibility metrics are not universally best across every language and comparison. On Chinese, for example, its strongest claim is on speaker similarity rather than on uniformly best CER (Gong et al., 11 Feb 2026).

The technical report evaluates both released generators under Clone and Continuation settings. For MOSS-TTS (8B), Clone yields EN WER 1.92, EN SIM 69.31, ZH CER 1.46, and ZH SIM 76.21; Continuation yields EN WER 1.84, EN SIM 70.86, ZH CER 1.37, and ZH SIM 76.98. For MOSS-TTS-Local-Transformer (1.7B), Clone yields EN WER 1.87, EN SIM 71.74, ZH CER 1.33, and ZH SIM 77.24; Continuation yields EN WER 1.93, EN SIM 73.28, ZH CER 1.44, and ZH SIM 79.62. The report’s interpretation is explicit: once WER and CER are already below roughly 2, speaker similarity is the more informative metric; Continuation consistently improves speaker similarity over Clone; and MOSS-TTS-Local-Transformer is stronger than MOSS-TTS on speaker preservation despite much smaller scale (Gong et al., 18 Mar 2026).

Beyond Seed-TTS-Eval, the report gives multilingual voice-cloning results on a CV3-Eval subset across zh, en, ja, ko, de, es, fr, it, and ru. Its summary judgment is that MOSS-TTS is competitive on several non-zh/en languages even without benchmark-specific multilingual cloning training, but with larger gaps on harder languages such as Japanese and Korean and in some English continuation cases (Gong et al., 18 Mar 2026).

For explicit control, the duration results are notably tight. Relative duration error for MOSS-TTS is reported as an overall mean of 0.712% for Chinese and 0.723% for English, with similarly low errors across the 10s–1m, 1m–10m, and 10m–30m buckets. Pronunciation control, measured only on the controlled span, yields 1.00 CER for Chinese partial-replace, 1.65 CER for Chinese full-replace, 4.32 WER for English partial-replace, and 5.84 WER for English full-replace (Gong et al., 18 Mar 2026).

Ultra-long generation remains a stress case. The report evaluates internal Chinese and English sets up to roughly one hour and concludes that content fidelity remains usable through most buckets, but degradation becomes severe mainly at the longest horizons. The dominant failure mode is cumulative speaker drift, especially in English; speaker preservation weakens earlier than lexical accuracy; and Continuation helps long-horizon speaker anchoring, especially in Chinese and in some long English buckets (Gong et al., 18 Mar 2026).

7. Related systems, evaluation distinctions, and unresolved issues

MOSS-TTS is part of a broader MOSS speech stack rather than an isolated model family. The clearest extension is MOSS-TTSD, a spoken dialogue synthesis system that follows the same broad architecture as earlier MOSS-TTS but is adapted specifically for long-context, multi-speaker dialogue generation. MOSS-TTSD uses Qwen3-8B-base as the autoregressive backbone and MOSS-Audio-Tokenizer as the speech codec/tokenizer, supports up to 60 minutes of single-pass synthesis, up to 5 speakers, and zero-shot voice cloning from a short reference audio clip, and introduces TTSD-eval for forced-alignment-based dialogue evaluation without reliance on speaker diarization (Zhang et al., 20 Mar 2026).

A common misconception is to conflate MOSS-TTS with MOS prediction or with automated MOS benchmarks. MOSS-TTS itself is a generative model, and its report prioritizes objective metrics such as WER, CER, and speaker similarity. By contrast, TTS evaluation papers treat MOS and side-by-side preference as separate supervised learning problems. Early work such as AutoMOS modeled human naturalness judgments from audio alone and showed that aggregated predictions over multiple utterances can be useful for system ranking (Patton et al., 2016). More recent work proposes NeuralSBS for pairwise preference prediction and improved MOS estimators such as enhanced MOSNet and WhisperBert, reporting RMSE around 0.40 on standardized SOMOS and explicitly arguing that evaluation, rather than generation, has become a bottleneck in modern TTS (Trofimenko et al., 17 Mar 2026).

Several limitations remain explicit in the MOSS-TTS literature. Hard multilingual settings are still challenging, especially some Japanese and Korean cases; ultra-long generation is limited by speaker drift; fine-grained controllability can still be improved; and broader low-resource language coverage is identified as an important next step. The technical report also omits exact TTS backbone dimensions, optimizer and batch size for TTS pretraining, hardware, and inference decoding hyperparameters. The tokenizer paper adds further gaps from a systems-evaluation perspective: it does not report MOS or other human listening tests for TTS, and prompt details remain underspecified. This suggests that the main contribution of MOSS-TTS is architectural and infrastructural—a scalable discrete-token autoregressive recipe—rather than a fully closed reproducibility package or a universally dominant result on every evaluation axis (Gong et al., 18 Mar 2026, Gong et al., 11 Feb 2026).