---
title: 'MOSS-TTS: Scalable Speech Synthesis Model'
url: https://www.emergentmind.com/topics/moss-tts
type: topic
---

# MOSS-TTS: Scalable Speech Synthesis Model

Searching arXiv for MOSS-TTS and closely related papers.
MOSS-TTS is a speech generation foundation model built on a scalable recipe: discrete audio tokens, autoregressive modeling, and large-scale pretraining. Built on MOSS-Audio-Tokenizer, a causal Transformer tokenizer that compresses 24 kHz audio to 12.5 fps with variable-bitrate residual vector quantization (RVQ) and unified semantic-acoustic representations, it releases two complementary generators: MOSS-TTS, which emphasizes structural simplicity, scalability, and long-context/control-oriented deployment, and MOSS-TTS-Local-Transformer, which introduces a frame-local autoregressive module for higher modeling efficiency, stronger speaker preservation, and a shorter time to first audio. Across multilingual and open-domain settings, the system supports zero-shot voice cloning, token-level duration control, phoneme-/pinyin-level pronunciation control, smooth code-switching, and stable long-form generation [2603.18090].

## 1. Terminology, scope, and design philosophy

MOSS-TTS is presented as an open speech generation foundation model for text-to-speech and broader speech generation. Its central claim is that modern TTS can be built effectively around a simple, scalable recipe—discrete audio tokens, autoregressive modeling, and large-scale pretraining—rather than around many intermediate modules, external semantic teachers, cascaded supervision, refinement stages, or post-hoc alignment. In that framing, the problem is recast as universal next-token prediction over a learned discrete speech interface [2603.18090].

The naming around the system is slightly inconsistent across papers. In the MOSS-Audio-Tokenizer paper, the downstream TTS model is called **CAT-TTS**, while the technical report uses **MOSS-TTS**. The tokenizer paper explicitly states that “MOSS-TTS / CAT-TTS” refers to the same tokenizer-enabled TTS line: a downstream model that predicts CAT or MOSS-Audio-Tokenizer RVQ tokens from text plus a speaker prompt, then reconstructs waveform using the tokenizer decoder. The distinction is therefore terminological rather than architectural [2602.10934].

This design places MOSS-TTS within the recent wave of speech generation foundation models, but its differentiating emphasis is an end-to-end scalable discrete-token pipeline. A plausible implication is that the authors view tokenizer quality as the central systems problem: if the representation is sufficiently compact, semantically informative, and high fidelity, the generator can remain comparatively simple while still supporting multilinguality, controllability, and long-context robustness [2603.18090].

## 2. Representation layer: MOSS-Audio-Tokenizer and the discrete speech interface

The foundational representation layer is MOSS-Audio-Tokenizer, a causal Transformer-based discrete audio tokenizer based on the CAT architecture. It is designed specifically to support large-scale autoregressive audio and speech modeling. The tokenizer uses 68 causal Transformer blocks in both encoder and decoder, a 10-second sliding-window attention mechanism, progressive patchify operations with patch sizes 240, 2, 2, and 2, and 32-layer RVQ with codebook size 1024, factorized vector quantization latent dimension 8, L2-normalized codes, and random quantizer dropout with probability 1.0 during training. The effective result is compression of 24 kHz waveform audio into a discrete sequence at 12.5 fps, with variable bitrate from 0.125 to 4 kbps [2603.18090].

The tokenizer is not optimized only for waveform reconstruction. It also includes a 0.5B decoder-only causal language model as a semantic head trained on ASR, multi-speaker ASR, and audio captioning, which is what the report means by unified semantic-acoustic representations. The semantic audio-to-text objective is

$$
\mathcal{L}_{\mathrm{sem}} = - \sum_{t=1}^{|\mathbf{s}|} \log p_{\theta_{\mathrm{LLM}}} \left( \mathbf{s}_t \,\middle|\, \mathcal{T},\, \mathbf{q},\, \mathbf{s}_{<t} \right),
$$

while the reconstruction term is

$$
\mathcal{L}_{\mathrm{rec}} = \sum_{i=5}^{11} \left\| S_{2^i}(\mathbf{x}) - S_{2^i}(\hat{\mathbf{x}}) \right\|_1.
$$

These terms are combined with commitment, codebook, adversarial, and feature-matching losses in the tokenizer’s multitask training objective [2603.18090].

For downstream TTS, this interface matters because token length and token semantics jointly determine feasibility. The tokenizer paper argues that strict causality, low frame rate, RVQ depth control, high reconstruction fidelity, and end-to-end learning without pretrained SSL encoders or semantic distillation make the token space a practical target for a single autoregressive TTS model. The model can therefore generate a unified token space that contains both linguistic content and acoustic information, rather than relying on explicit semantic-acoustic disentanglement or a cascaded semantic-to-acoustic stack [2602.10934].

This representation choice is consistent with earlier work on quantized hierarchical TTS targets. A multi-stage multi-codebook VQ-VAE study showed that replacing direct mel prediction with structured, quantized, multi-resolution acoustic representations can improve quality and parameter efficiency, which helps situate MOSS-TTS within a broader movement away from flat mel-spectrogram targets [2209.10887].

## 3. Generator architectures: delay pattern and frame-local autoregression

Both released generators operate on the same tokenizer outputs. With \(N_q = 32\) RVQ layers, both architectures predict \(N_h = N_q + 1 = 33\) channels per aligned step: one text-or-pad channel and 32 audio channels. Training uses head-wise weighted cross-entropy that up-weights earlier, coarser RVQ layers, reflecting the importance of low-depth RVQ structure for intelligibility and speaker identity [2603.18090].

The first architecture, **MOSS-TTS**, uses a delay pattern and a single Transformer backbone. If \(\mathbf{a}_{j,t}\) denotes the token at RVQ layer \(j\) and frame \(t\), the delayed representation is

$$
\tilde{\mathbf{a}}_{j,t} = \mathbf{a}_{j, t - (j-1)}.
$$

Each delayed time step aggregates per-layer embeddings by summation,

$$
\mathbf{h}_{t} = \sum_{j=1}^{N_q} \operatorname{Emb}_j(\tilde{\mathbf{a}}_{j,t}),
$$

yielding a sequence of length \(T + N_q - 1\) concatenated with text embeddings. The practical meaning of the design, as the report states explicitly, is structural simplicity, scalability, and long-context/control-oriented deployment: a single backbone, no extra within-frame autoregressive module, easier engineering, easier optimized inference backends, and better suitability for long contexts and explicit conditioning tasks such as duration and pronunciation control. The released MOSS-TTS model evaluated for cloning is 8B parameters [2603.18090].

The second architecture, **MOSS-TTS-Local-Transformer**, is hierarchical. It dispenses with delay shifting and instead lets the backbone emit a global latent per step, after which a lightweight Local Transformer autoregressively expands that latent into the within-step block \((y_{0,t+1}, y_{1,t+1}, \dots, y_{N_q,t+1})\). Its local input is

$$
\mathbf{z}_{j,t} =
\begin{cases}
\mathbf{x}_t, & \text{if } j = 0, \\
\operatorname{Emb}^{\mathrm{local}}_{j-1}(y_{j-1,t+1}), & \text{if } 1 \le j \le N_q.
\end{cases}
$$

The report characterizes this design as more modeling-efficient, with lower per-layer token losses during internal development, stronger speaker preservation, better quality at smaller scale, and shorter time to first audio because it does not need to wait for delayed offsets before producing the first frame. The tradeoff is a heavier steady-state decoding path due to the inner autoregressive loop over channels. The released MOSS-TTS-Local-Transformer is 1.7B parameters [2603.18090].

In the tokenizer paper, the same downstream line is described using a closely related factorization: a Temporal Transformer over time and a Depth Transformer over RVQ depth within each frame. That paper initializes the Temporal Transformer from Qwen3-1.7B and uses a 4-block Depth Transformer with hidden size 1536 and FFN dimension 8960. The common systems principle is the same: temporal causality across frames and depth-wise causality across RVQ layers, with direct prediction of tokenizer tokens rather than mel frames [2602.10934].

## 4. Data pipeline, preprocessing, and curriculum

The scale of pretraining is one of the defining properties of MOSS-TTS. At the tokenizer level, MOSS-Audio-Tokenizer is trained on approximately 3 million hours of speech, sound, and music, including both clean and in-the-wild recordings. For speech generation, the tokenizer paper states that training uses VoxBox plus an internal dataset totaling approximately 200k hours of speech data, whereas the technical report describes the broader MOSS-TTS pretraining corpus as millions of hours in total and does not provide exact hour counts per language or domain in text [2602.10934] [2603.18090].

The technical report provides a detailed preprocessing and gating pipeline. Noise suppression uses MossFormer2-SE-48K. Speaker diarization uses DiariZen, discarding segments shorter than \(\tau_{\min}=0.1\) s and merging consecutive same-speaker segments. A hard one-hour truncation is applied at \(t_{\mathrm{lim}} = s_1 + 3600\ \text{s}\). Transcript generation uses MOSS-Transcribe-Diarize, followed by empty-content filtering, repetition-loop filtering, non-speech-dominance filtering, LLM-based diagnosis and refinement, and single-speaker transcript validation. Joint audio-text filtering then applies DNSMOS \(> 2.8\), Meta AudioBox PQ \(> 6.5\), audio/text language agreement, and transcript-length consistency [2603.18090].

The curriculum is organized around five subsets: \(\mathcal{D}_{\mathrm{basic}}\), \(\mathcal{D}_{\mathrm{clone}}\), \(\mathcal{D}_{\mathrm{dict}}\), \(\mathcal{D}_{\mathrm{noise}}\), and \(\mathcal{D}_{\mathrm{phone}}\). Pretraining follows a four-phase WSD-style curriculum. Phase 1 uses \(\mathcal{D}_{\mathrm{basic}}\) only with max sequence length 32k and learning-rate warmup to \(2\times 10^{-4}\). Phase 2 keeps max sequence length 32k, enables all subsets, and strongly upsamples \(\mathcal{D}_{\mathrm{clone}}\). Phase 3 linearly decays learning rate from \(2\times 10^{-4}\) to \(2\times 10^{-6}\) while restoring the normal data mixture. Phase 4 extends context from 32k to 64k, holds learning rate at \(2\times 10^{-6}\), and heavily upsamples long-form data [2603.18090].

Some implementation details remain absent. The report explicitly does not provide optimizer type, batch size for TTS generator pretraining, hardware, total step count, exact model hidden sizes and layer counts for the generator backbones, sequence packing details, or inference decoding hyperparameters. Those omissions matter for strict reproducibility [2603.18090].

## 5. Conditioning interfaces, control mechanisms, and inference modes

MOSS-TTS conditions on text and, when required, a speech prompt or reference audio. Depending on task, the input may also include the ASR transcript of the prompt in continuation mode, an explicit duration token storing the target audio-token count, tone-marked pinyin for Chinese, or IPA spans enclosed in slashes for English. The report does not provide a fully specified serialized template for every input type, but it makes clear that pronunciation control is implemented on the text side through phonetic replacement rather than by a separate pronunciation head [2603.18090].

Zero-shot voice cloning is learned through targeted data synthesis. For a target segment \(s_i\), prompt candidates are cropped from other same-speaker segments \(s_j\), each crop at most 30 seconds, with five random crops per segment. Candidates are scored by cosine similarity of speaker embeddings from a fine-tuned WavLM-Large model, and the most similar crop is selected as the prompt. At inference, the report distinguishes **Clone**, in which a user explicitly provides reference audio, from **Continuation**, in which the model continues speech after prepending the reference audio and its transcript to the assistant-side speech/text prefix. The tokenizer paper likewise describes inference as continuation-based, concatenating prompt transcription tokens, target text tokens, and prompt audio tokens into a single sequence [2603.18090] [2602.10934].

Duration control is token-based rather than predictor-based. Since the tokenizer runs at 12.5 audio tokens per second, a target token count \(n\) corresponds to

$$
T_{\text{target}} = n/12.5.
$$

If realized duration is \(T_{\text{real}}\), relative duration error is

$$
\text{Err}\% = |T_{\text{real}} - T_{\text{target}}|/T_{\text{target}} \times 100\%.
$$

The key design point is that duration is controlled directly at the audio-token count level, not through a separate duration model of the classical FastSpeech type [2603.18090].

Variable bitrate is supported through RVQ-depth control. In the tokenizer paper, **Progressive Sequence Dropout** samples whether to truncate depth during training:

$$
z \sim \mathrm{Bernoulli}(p), \qquad \hat{K} = (1-z)N_q + zK,
$$

with \(K \in \{1,\dots,N_q-1\}\) when dropout is applied. Training then predicts only the retained RVQ prefix, which makes the model robust when inference asks it to generate only the first \(K_{\mathrm{infer}}\) RVQ layers. That paper reports that larger \(p\) significantly reduces GPU memory use and chooses \(p=1.0\) for subsequent experiments [2602.10934].

The report also claims smooth code-switching and stable long-form generation. The long-form capability is attributed to low-frame-rate discrete tokens, autoregressive modeling, the long-context curriculum, and the final context extension from 32k to 64k. The main remaining failure mode is cumulative speaker drift rather than immediate lexical breakdown [2603.18090].

## 6. Empirical characteristics and benchmark results

The tokenizer paper reports the headline CAT-TTS objective results on Seed-TTS-Eval as **WER 1.89** and **SIM 73.1** for English, and **CER 1.23** and **SIM 78.5** for Chinese. The paper is careful about what this supports: CAT-TTS achieves the highest speaker similarity scores on Seed-TTS-Eval for both English and Chinese among the compared open-source models, but its intelligibility metrics are not universally best across every language and comparison. On Chinese, for example, its strongest claim is on speaker similarity rather than on uniformly best CER [2602.10934].

The technical report evaluates both released generators under Clone and Continuation settings. For **MOSS-TTS (8B)**, Clone yields EN WER 1.92, EN SIM 69.31, ZH CER 1.46, and ZH SIM 76.21; Continuation yields EN WER 1.84, EN SIM 70.86, ZH CER 1.37, and ZH SIM 76.98. For **MOSS-TTS-Local-Transformer (1.7B)**, Clone yields EN WER 1.87, EN SIM 71.74, ZH CER 1.33, and ZH SIM 77.24; Continuation yields EN WER 1.93, EN SIM 73.28, ZH CER 1.44, and ZH SIM 79.62. The report’s interpretation is explicit: once WER and CER are already below roughly 2, speaker similarity is the more informative metric; Continuation consistently improves speaker similarity over Clone; and MOSS-TTS-Local-Transformer is stronger than MOSS-TTS on speaker preservation despite much smaller scale [2603.18090].

Beyond Seed-TTS-Eval, the report gives multilingual voice-cloning results on a CV3-Eval subset across zh, en, ja, ko, de, es, fr, it, and ru. Its summary judgment is that MOSS-TTS is competitive on several non-zh/en languages even without benchmark-specific multilingual cloning training, but with larger gaps on harder languages such as Japanese and Korean and in some English continuation cases [2603.18090].

For explicit control, the duration results are notably tight. Relative duration error for MOSS-TTS is reported as an overall mean of **0.712%** for Chinese and **0.723%** for English, with similarly low errors across the 10s–1m, 1m–10m, and 10m–30m buckets. Pronunciation control, measured only on the controlled span, yields **1.00 CER** for Chinese partial-replace, **1.65 CER** for Chinese full-replace, **4.32 WER** for English partial-replace, and **5.84 WER** for English full-replace [2603.18090].

Ultra-long generation remains a stress case. The report evaluates internal Chinese and English sets up to roughly one hour and concludes that content fidelity remains usable through most buckets, but degradation becomes severe mainly at the longest horizons. The dominant failure mode is cumulative speaker drift, especially in English; speaker preservation weakens earlier than lexical accuracy; and Continuation helps long-horizon speaker anchoring, especially in Chinese and in some long English buckets [2603.18090].

## 7. Related systems, evaluation distinctions, and unresolved issues

MOSS-TTS is part of a broader MOSS speech stack rather than an isolated model family. The clearest extension is **MOSS-TTSD**, a spoken dialogue synthesis system that follows the same broad architecture as earlier MOSS-TTS but is adapted specifically for long-context, multi-speaker dialogue generation. MOSS-TTSD uses Qwen3-8B-base as the autoregressive backbone and MOSS-Audio-Tokenizer as the speech codec/tokenizer, supports up to 60 minutes of single-pass synthesis, up to 5 speakers, and zero-shot voice cloning from a short reference audio clip, and introduces TTSD-eval for forced-alignment-based dialogue evaluation without reliance on speaker diarization [2603.19739].

A common misconception is to conflate MOSS-TTS with MOS prediction or with automated MOS benchmarks. MOSS-TTS itself is a generative model, and its report prioritizes objective metrics such as WER, CER, and speaker similarity. By contrast, TTS evaluation papers treat MOS and side-by-side preference as separate supervised learning problems. Early work such as AutoMOS modeled human naturalness judgments from audio alone and showed that aggregated predictions over multiple utterances can be useful for system ranking [1611.09207]. More recent work proposes NeuralSBS for pairwise preference prediction and improved MOS estimators such as enhanced MOSNet and WhisperBert, reporting RMSE around 0.40 on standardized SOMOS and explicitly arguing that evaluation, rather than generation, has become a bottleneck in modern TTS [2604.08562].

Several limitations remain explicit in the MOSS-TTS literature. Hard multilingual settings are still challenging, especially some Japanese and Korean cases; ultra-long generation is limited by speaker drift; fine-grained controllability can still be improved; and broader low-resource language coverage is identified as an important next step. The technical report also omits exact TTS backbone dimensions, optimizer and batch size for TTS pretraining, hardware, and inference decoding hyperparameters. The tokenizer paper adds further gaps from a systems-evaluation perspective: it does not report MOS or other human listening tests for TTS, and prompt details remain underspecified. This suggests that the main contribution of MOSS-TTS is architectural and infrastructural—a scalable discrete-token autoregressive recipe—rather than a fully closed reproducibility package or a universally dominant result on every evaluation axis [2603.18090] [2602.10934].

Source: https://www.emergentmind.com/topics/moss-tts