MOSS-TTSD: Multi-party Dialogue Synthesis

• MOSS-TTSD is a spoken dialogue synthesis model that generates expressive, multi-party conversational speech from dialogue scripts with explicit speaker tags.
• It employs a fully discrete, autoregressive transformer with vector-quantized audio tokens to achieve long-form, single-pass synthesis up to 60 minutes.
• Zero-shot voice cloning and speaker conditioning are integrated via reference audio slots, ensuring high speaker attribution accuracy and consistency across turns.

MOSS-TTSD is a spoken dialogue synthesis model for expressive, multi-party conversational speech across multiple languages. It is designed to generate long-form spoken conversations from dialogue scripts with explicit speaker tags, supports up to 60 minutes of single-pass synthesis, multi-party dialogue with up to 5 speakers, and zero-shot voice cloning from a short reference audio clip (Zhang et al., 20 Mar 2026). In contemporaneous MOSS publications, the same acronym also appears in “MOSS Transcribe Diarize,” a distinct end-to-end Speaker-Attributed, Time-Stamped Transcription system rather than a dialogue-generation model (Yu et al., 4 Jan 2026).

1. Position within the MOSS speech stack

MOSS-TTSD is built on a fully-discrete, LLM-based TTS paradigm. In place of a conventional spectrogram encoder/decoder, it treats audio as a sequence of vector-quantized codebook tokens and uses an autoregressive transformer to generate them. Its discrete representation is produced by MOSS-Audio-Tokenizer, which encodes raw speech into 16 layers of RVQ tokens at 2 kbps with a 12.5 Hz frame rate (Zhang et al., 20 Mar 2026).

Within the broader MOSS speech stack, the technical report on MOSS-TTS describes the same general recipe—discrete audio tokens, autoregressive modeling, and large-scale pretraining—and characterizes MOSS-Audio-Tokenizer as a causal Transformer tokenizer that compresses 24 kHz audio to 12.5 fps with variable-bitrate RVQ and unified semantic-acoustic representations (Gong et al., 18 Mar 2026). This places MOSS-TTSD within a larger family of MOSS models that use discrete audio tokenization as the interface between waveform modeling and LLM-style sequence generation.

The dialogue focus differentiates MOSS-TTSD from single-utterance TTS. The paper frames spoken dialogue generation as requiring accurate turn-taking, cross-turn acoustic consistency, and long-form stability, and further notes that current models often fail on these requirements because they lack dialogue context modeling (Zhang et al., 20 Mar 2026).

2. Input representation and autoregressive architecture

The model uses Qwen3-8B-base as its autoregressive backbone and fine-tunes it to predict the next RVQ token given prior text and audio context (Zhang et al., 20 Mar 2026). The input prompt contains dialogue text with explicit speaker tags, for example a sequence of the form

$$\ldots [S_1]\;\text{“Hello”}\;[S_2]\;\text{“How are you?”}\;\ldots$$

and may also contain optional reference-audio slots for zero-shot cloning, represented as $$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$ in a chat-style template (Zhang et al., 20 Mar 2026).

The full conditioning context—speaker tags, reference tokens, and previously generated audio codes—is flattened into one causal sequence: $$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$ Standard causal self-attention then operates over the combined token stream: $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$ Here $E$ stacks text, speaker-tag, and audio-code embeddings (Zhang et al., 20 Mar 2026).

For audio-token emission, the system adopts the “multi-head delay” pattern from MusicGen. At generation step $t$, only layers $1+(t \bmod H)$ are emitted, cycling through heads and reducing concurrency (Zhang et al., 20 Mar 2026). The result is an architecture that preserves a pure autoregressive decoding interface while controlling per-step computational load.

3. Training objective and curriculum

MOSS-TTSD uses a single training objective: maximizing the likelihood of the correct RVQ token sequence given the available context. Let $\{a_t\}_{t=1}^{T}$ denote the ground-truth RVQ code sequence and $\hat P(a_t\mid \mathbf{c}_{<t})$ the model distribution. The loss is standard cross-entropy: $$\mathcal{L}_{\rm TTS} = -\sum_{t=1}^{T}\log \hat P(a_t\mid \mathbf{c}_{<t}).$$ No auxiliary adversarial loss and no explicit speaker-consistency loss are introduced (Zhang et al., 20 Mar 2026).

The paper states that curriculum learning stages gradually introduce longer multi-speaker contexts while retaining the same $$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$0 objective throughout training (Zhang et al., 20 Mar 2026). It further summarizes this curriculum as scaling from single-speaker to 5-party dialogues. A plausible implication is that the model’s dialogue behavior is intended to emerge from sequence modeling over increasingly complex contexts rather than from task-specific supervisory heads.

This training design contrasts with many modular dialogue-speech pipelines. MOSS-TTSD does not present separate optimization targets for turn-taking, speaker matching, or long-form continuity; these are treated as behaviors that should be recoverable from next-token prediction over discrete audio codes in the presence of dialogue-structured prompts.

4. Long-context dialogue synthesis

The model’s long-form capability is tied directly to its low-bitrate discrete representation. By modeling only 16 RVQ layers at 2 kbps and 12.5 Hz, a 60-minute utterance becomes roughly

$$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$1

audio tokens, which the paper describes as well within the 65,536-token limit of Qwen3-8B’s context window (Zhang et al., 20 Mar 2026). This compression is the basis for the claim that MOSS-TTSD supports up to 60 minutes of single-pass synthesis.

The paper emphasizes three engineering choices for long-form stability: low-bitrate tokenization, the multi-head delay pattern, and single-pass synthesis (Zhang et al., 20 Mar 2026). The multi-head delay pattern reduces the transformer’s per-step computational burden, while causal cache retention preserves historical state across the generated dialogue. Single-pass synthesis means that no chunking or stitching is required; the full prompt-to-audio token sequence is produced end-to-end.

These properties are particularly relevant for dialogue rather than isolated utterances. The stated target scenarios include podcasts, dynamic commentary, entertainment content, and several long-form settings (Zhang et al., 20 Mar 2026). Because the prompt includes speaker tags and prior generated speech within one causal sequence, cross-turn dependencies can in principle be modeled directly rather than reintroduced heuristically at utterance boundaries. This suggests that MOSS-TTSD’s notion of “dialogue context” is not merely textual; it is embodied in the joint history of tags, references, and previously synthesized acoustic codes.

5. Zero-shot voice cloning and speaker conditioning

Zero-shot cloning is implemented by prepending a short reference clip for each speaker. Each reference audio $$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$2 is tokenized into discrete codes

$$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$3

and these tokens are inserted into the prompt under the corresponding speaker label $$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$4 (Zhang et al., 20 Mar 2026). During generation, the transformer uses these reference tokens as conditioning for speaker identity.

The paper also describes an “empirical formula”

$$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$5

to characterize speaker embedding injection (Zhang et al., 20 Mar 2026). In practical terms, the speaker representation is not handled as an external side channel; it is embedded into the same autoregressive prompt that drives dialogue synthesis.

A second mechanism is continuation-based cloning. Beyond explicit reference slots, the autoregressive model can continue past reference audio segments, which the paper states reinforces timbre consistency (Zhang et al., 20 Mar 2026). The ablation summary reports that “voice_clone_and_continuation” yields the best SIM and ACC compared with either “voice_clone” or pure “continuation.” This suggests that speaker preservation in the model is strengthened when reference conditioning and generated-history conditioning are used jointly across turns.

6. TTSD-eval and reported empirical performance

To evaluate spoken dialogue generation, the paper proposes TTSD-eval, an objective framework based on forced alignment that measures speaker attribution accuracy and speaker similarity without relying on speaker diarization tools (Zhang et al., 20 Mar 2026). The workflow is explicit. First, given a reference script $$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$6 and a synthesized waveform, MMS-FA produces word-level timestamps $$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$7. Second, aligned words are grouped into utterance fragments according to punctuation in $$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$8, and each fragment $$\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots$$9 inherits a ground-truth speaker label $$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$0. Third, a fixed speaker-embedding network $$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$1 is applied both to each fragment and to each speaker’s reference clip: $$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$2 Speaker attribution is then

$$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$3

with accuracy

$$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$4

and speaker similarity

$$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$5

Intelligibility is measured by running Whisper-large-v3 on the generated audio and computing WER against $$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$6 (Zhang et al., 20 Mar 2026).

The reported benchmark comprises 50 Chinese and 50 English dialogues, including 30 from seed-tts-eval and 20 with Internet voices, spanning 30–720 seconds, 1–5 speakers, and diverse genres (Zhang et al., 20 Mar 2026). Open-source baselines include Higgs Audio V2, FireRedTTS-2, and VibeVoice in 1.5B and 7B variants. Proprietary baselines include Eleven Labs V3, Gemini-2.5, and Doubao Podcast.

On the objective benchmark, MOSS-TTSD reports $$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$7 ZH ACC, $$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$8 ZH SIM, and $$\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].$$9 ZH WER, alongside $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$0 EN ACC, $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$1 EN SIM, and $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$2 EN WER (Zhang et al., 20 Mar 2026). Under the “eleven voice” condition, it reports $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$3 ZH ACC, $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$4 ZH SIM, and $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$5 ZH WER, with $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$6 EN ACC, $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$7 EN SIM, and $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$8 EN WER (Zhang et al., 20 Mar 2026). The subjective evaluation summary states that MOSS-TTSD leads in speaker attribution, voice similarity, rhythm, and overall quality over open-source systems, and wins 60–80% of head-to-head comparisons against proprietary APIs.

7. Nomenclature and distinction from MOSS Transcribe Diarize

The acronym “MOSS-TTSD” is used for two different MOSS systems. In “MOSS-TTSD: Text to Spoken Dialogue Generation,” it denotes the dialogue-synthesis model described above (Zhang et al., 20 Mar 2026). In “MOSS Transcribe Diarize: Accurate Transcription with Speaker Diarization,” it denotes a unified multimodal LLM for Speaker-Attributed, Time-Stamped Transcription (Yu et al., 4 Jan 2026).

The transcription system operates in the opposite direction. Raw audio of up to 90 minutes is broken into overlapping chunks, passed through a pretrained speech encoder, linearly projected into the text LLM’s embedding space, and interleaved with explicit timestamp markers such as $$Q,K,V = W_QE,\;W_KE,\;W_VE,\quad C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.$$9 (Yu et al., 4 Jan 2026). The decoder then emits sequences of the form

$E$0

so that transcription, speaker attribution, and timestamp prediction are all handled as one autoregressive next-token problem (Yu et al., 4 Jan 2026). The paper explicitly states: no separate diarization pass, no forced alignment, no clustering.

This distinction matters because the two systems occupy neighboring but different parts of the speech stack. The dialogue-generation MOSS-TTSD converts dialogue scripts with speaker tags into spoken multi-party audio, whereas MOSS Transcribe Diarize converts long audio into speaker-attributed, time-stamped transcripts. The shared acronym reflects a naming collision rather than a unified model identity.