---
title: 'MOSS-TTSD: Multi-party Dialogue Synthesis'
url: https://www.emergentmind.com/topics/moss-ttsd
type: topic
---

# MOSS-TTSD: Multi-party Dialogue Synthesis

MOSS-TTSD is a spoken dialogue synthesis model for expressive, multi-party conversational speech across multiple languages. It is designed to generate long-form spoken conversations from dialogue scripts with explicit speaker tags, supports up to 60 minutes of single-pass synthesis, multi-party dialogue with up to 5 speakers, and zero-shot voice cloning from a short reference audio clip [2603.19739]. In contemporaneous MOSS publications, the same acronym also appears in “MOSS Transcribe Diarize,” a distinct end-to-end Speaker-Attributed, Time-Stamped Transcription system rather than a dialogue-generation model [2601.01554].

## 1. Position within the MOSS speech stack

MOSS-TTSD is built on a fully-discrete, LLM-based TTS paradigm. In place of a conventional spectrogram encoder/decoder, it treats audio as a sequence of vector-quantized codebook tokens and uses an autoregressive transformer to generate them. Its discrete representation is produced by MOSS-Audio-Tokenizer, which encodes raw speech into 16 layers of RVQ tokens at 2 kbps with a 12.5 Hz frame rate [2603.19739].

Within the broader MOSS speech stack, the technical report on MOSS-TTS describes the same general recipe—discrete audio tokens, autoregressive modeling, and large-scale pretraining—and characterizes MOSS-Audio-Tokenizer as a causal Transformer tokenizer that compresses 24 kHz audio to 12.5 fps with variable-bitrate RVQ and unified semantic-acoustic representations [2603.18090]. This places MOSS-TTSD within a larger family of MOSS models that use discrete audio tokenization as the interface between waveform modeling and LLM-style sequence generation.

The dialogue focus differentiates MOSS-TTSD from single-utterance TTS. The paper frames spoken dialogue generation as requiring accurate turn-taking, cross-turn acoustic consistency, and long-form stability, and further notes that current models often fail on these requirements because they lack dialogue context modeling [2603.19739].

## 2. Input representation and autoregressive architecture

The model uses Qwen3-8B-base as its autoregressive backbone and fine-tunes it to predict the next RVQ token given prior text and audio context [2603.19739]. The input prompt contains dialogue text with explicit speaker tags, for example a sequence of the form
\[
\ldots [S_1]\;\text{“Hello”}\;[S_2]\;\text{“How are you?”}\;\ldots
\]
and may also contain optional reference-audio slots for zero-shot cloning, represented as \(\langle \text{audio}_1\rangle,\langle \text{audio}_2\rangle,\dots\) in a chat-style template [2603.19739].

The full conditioning context—speaker tags, reference tokens, and previously generated audio codes—is flattened into one causal sequence:
\[
\bigl[T_1,\dots,T_{n_\text{text}}, A^{(1)}_1,\dots,A^{(1)}_{m_1}, A^{(2)}_1,\dots\bigr].
\]
Standard causal self-attention then operates over the combined token stream:
\[
Q,K,V = W_QE,\;W_KE,\;W_VE,\quad
C=\mathrm{Softmax}\!\left(\tfrac{QK^\top}{\sqrt{d}}\right)V.
\]
Here \(E\) stacks text, speaker-tag, and audio-code embeddings [2603.19739].

For audio-token emission, the system adopts the “multi-head delay” pattern from MusicGen. At generation step \(t\), only layers \(1+(t \bmod H)\) are emitted, cycling through heads and reducing concurrency [2603.19739]. The result is an architecture that preserves a pure autoregressive decoding interface while controlling per-step computational load.

## 3. Training objective and curriculum

MOSS-TTSD uses a single training objective: maximizing the likelihood of the correct RVQ token sequence given the available context. Let \(\{a_t\}_{t=1}^{T}\) denote the ground-truth RVQ code sequence and \(\hat P(a_t\mid \mathbf{c}_{<t})\) the model distribution. The loss is standard cross-entropy:
\[
\mathcal{L}_{\rm TTS}
=
-\sum_{t=1}^{T}\log \hat P(a_t\mid \mathbf{c}_{<t}).
\]
No auxiliary adversarial loss and no explicit speaker-consistency loss are introduced [2603.19739].

The paper states that curriculum learning stages gradually introduce longer multi-speaker contexts while retaining the same \(\mathcal{L}_{\rm TTS}\) objective throughout training [2603.19739]. It further summarizes this curriculum as scaling from single-speaker to 5-party dialogues. A plausible implication is that the model’s dialogue behavior is intended to emerge from sequence modeling over increasingly complex contexts rather than from task-specific supervisory heads.

This training design contrasts with many modular dialogue-speech pipelines. MOSS-TTSD does not present separate optimization targets for turn-taking, speaker matching, or long-form continuity; these are treated as behaviors that should be recoverable from next-token prediction over discrete audio codes in the presence of dialogue-structured prompts.

## 4. Long-context dialogue synthesis

The model’s long-form capability is tied directly to its low-bitrate discrete representation. By modeling only 16 RVQ layers at 2 kbps and 12.5 Hz, a 60-minute utterance becomes roughly
\[
60\times 60\times 12.5 \approx 45{,}000
\]
audio tokens, which the paper describes as well within the 65,536-token limit of Qwen3-8B’s context window [2603.19739]. This compression is the basis for the claim that MOSS-TTSD supports up to 60 minutes of single-pass synthesis.

The paper emphasizes three engineering choices for long-form stability: low-bitrate tokenization, the multi-head delay pattern, and single-pass synthesis [2603.19739]. The multi-head delay pattern reduces the transformer’s per-step computational burden, while causal cache retention preserves historical state across the generated dialogue. Single-pass synthesis means that no chunking or stitching is required; the full prompt-to-audio token sequence is produced end-to-end.

These properties are particularly relevant for dialogue rather than isolated utterances. The stated target scenarios include podcasts, dynamic commentary, entertainment content, and several long-form settings [2603.19739]. Because the prompt includes speaker tags and prior generated speech within one causal sequence, cross-turn dependencies can in principle be modeled directly rather than reintroduced heuristically at utterance boundaries. This suggests that MOSS-TTSD’s notion of “dialogue context” is not merely textual; it is embodied in the joint history of tags, references, and previously synthesized acoustic codes.

## 5. Zero-shot voice cloning and speaker conditioning

Zero-shot cloning is implemented by prepending a short reference clip for each speaker. Each reference audio \(x_{\rm ref}^{(i)}\) is tokenized into discrete codes
\[
\{r_j^{(i)}\}=\mathrm{AudioTok}(x_{\rm ref}^{(i)}),
\]
and these tokens are inserted into the prompt under the corresponding speaker label \([S_i]\) [2603.19739]. During generation, the transformer uses these reference tokens as conditioning for speaker identity.

The paper also describes an “empirical formula”
\[
e_{\rm spk}^{(i)} = g_{\rm ref}\bigl(x_{\rm ref}^{(i)}\bigr),
\quad
\text{then prompt}\;\ldots [S_i]\{e_{\rm spk}^{(i)}\}\ldots
\]
to characterize speaker embedding injection [2603.19739]. In practical terms, the speaker representation is not handled as an external side channel; it is embedded into the same autoregressive prompt that drives dialogue synthesis.

A second mechanism is continuation-based cloning. Beyond explicit reference slots, the autoregressive model can continue past reference audio segments, which the paper states reinforces timbre consistency [2603.19739]. The ablation summary reports that “voice_clone_and_continuation” yields the best SIM and ACC compared with either “voice_clone” or pure “continuation.” This suggests that speaker preservation in the model is strengthened when reference conditioning and generated-history conditioning are used jointly across turns.

## 6. TTSD-eval and reported empirical performance

To evaluate spoken dialogue generation, the paper proposes TTSD-eval, an objective framework based on forced alignment that measures speaker attribution accuracy and speaker similarity without relying on speaker diarization tools [2603.19739]. The workflow is explicit. First, given a reference script \(\mathcal{T}=\{w_1,\dots,w_N\}\) and a synthesized waveform, MMS-FA produces word-level timestamps \(\{(t_i^{\rm start},t_i^{\rm end})\}\). Second, aligned words are grouped into utterance fragments according to punctuation in \(\mathcal{T}\), and each fragment \(F_k\) inherits a ground-truth speaker label \(s_k\). Third, a fixed speaker-embedding network \(f_{\rm spk}\) is applied both to each fragment and to each speaker’s reference clip:
\[
\epsilon_k=f_{\rm spk}(F_k),\qquad
\epsilon_{\rm ref}^{(j)}=f_{\rm spk}(x_{\rm ref}^{(j)}).
\]
Speaker attribution is then
\[
\hat s_k=\arg\max_j \cos\bigl(\epsilon_k,\epsilon_{\rm ref}^{(j)}\bigr),
\]
with accuracy
\[
\mathrm{ACC}=\frac{1}{K}\sum_{k=1}^{K}\mathbf{1}\{\hat s_k=s_k\},
\]
and speaker similarity
\[
\mathrm{SIM}=\frac{1}{K}\sum_{k=1}^{K}\cos\bigl(\epsilon_k,\epsilon_{\rm ref}^{(s_k)}\bigr).
\]
Intelligibility is measured by running Whisper-large-v3 on the generated audio and computing WER against \(\mathcal{T}\) [2603.19739].

The reported benchmark comprises 50 Chinese and 50 English dialogues, including 30 from seed-tts-eval and 20 with Internet voices, spanning 30–720 seconds, 1–5 speakers, and diverse genres [2603.19739]. Open-source baselines include Higgs Audio V2, FireRedTTS-2, and VibeVoice in 1.5B and 7B variants. Proprietary baselines include Eleven Labs V3, Gemini-2.5, and Doubao Podcast.

On the objective benchmark, MOSS-TTSD reports \(0.9587\) ZH ACC, \(0.7949\) ZH SIM, and \(4.85\%\) ZH WER, alongside \(0.9626\) EN ACC, \(0.7326\) EN SIM, and \(9.88\%\) EN WER [2603.19739]. Under the “eleven voice” condition, it reports \(0.9736\) ZH ACC, \(0.8165\) ZH SIM, and \(3.91\%\) ZH WER, with \(0.9565\) EN ACC, \(0.7304\) EN SIM, and \(10.05\%\) EN WER [2603.19739]. The subjective evaluation summary states that MOSS-TTSD leads in speaker attribution, voice similarity, rhythm, and overall quality over open-source systems, and wins 60–80% of head-to-head comparisons against proprietary APIs.

## 7. Nomenclature and distinction from MOSS Transcribe Diarize

The acronym “MOSS-TTSD” is used for two different MOSS systems. In “MOSS-TTSD: Text to Spoken Dialogue Generation,” it denotes the dialogue-synthesis model described above [2603.19739]. In “MOSS Transcribe Diarize: Accurate Transcription with Speaker Diarization,” it denotes a unified multimodal large language model for Speaker-Attributed, Time-Stamped Transcription [2601.01554].

The transcription system operates in the opposite direction. Raw audio of up to 90 minutes is broken into overlapping chunks, passed through a pretrained speech encoder, linearly projected into the text LLM’s embedding space, and interleaved with explicit timestamp markers such as \(\langle T=\mathrm{hh:mm:ss}\rangle\) [2601.01554]. The decoder then emits sequences of the form
\[
[S_k]\;\text{word}_1\;\text{word}_2\;\dots\;\text{word}_N\;\langle T=t_i^{\rm end}\rangle,
\]
so that transcription, speaker attribution, and timestamp prediction are all handled as one autoregressive next-token problem [2601.01554]. The paper explicitly states: no separate diarization pass, no forced alignment, no clustering.

This distinction matters because the two systems occupy neighboring but different parts of the speech stack. The dialogue-generation MOSS-TTSD converts dialogue scripts with speaker tags into spoken multi-party audio, whereas MOSS Transcribe Diarize converts long audio into speaker-attributed, time-stamped transcripts. The shared acronym reflects a naming collision rather than a unified model identity.

Source: https://www.emergentmind.com/topics/moss-ttsd