MOSS-Speech: Direct Speech-to-Speech LLM

• MOSS-Speech is a speech-native large language model that directly interprets and generates speech without relying on text intermediates.
• It uses a modality-based layer-splitting architecture with shared lower layers and specialized top layers, built on Qwen3-8B, to maintain cross-modal transfer.
• The model employs a frozen pre-training strategy and a streaming speech tokenizer to preserve paralinguistic information and narrow gaps with text-guided systems.

Searching arXiv for papers on MOSS-Speech and closely related MOSS speech models. MOSS-Speech is a speech-native LLM framework centered on a “true speech-to-speech” formulation in which speech is directly understood and directly generated without relying on text guidance at generation time. In its specific 2025 formulation, it is built on Qwen3-8B, a streaming speech tokenizer and decoder, a modality-based layer-splitting architecture, and a frozen pre-training strategy intended to preserve the reasoning and knowledge of a pretrained text LLM while adding native speech capabilities (Zhao et al., 1 Oct 2025). In the broader MOSS family, adjacent models position “MOSS-Speech” as an ecosystem spanning transcription with diarization, speech generation, spoken dialogue synthesis, and speech front-end modeling (Yu et al., 4 Jan 2026).

1. Definition and scope

MOSS-Speech addresses a limitation shared by cascaded spoken dialogue systems and many recent end-to-end speech models. Cascaded systems follow an ASR $\rightarrow$ text LLM $\rightarrow$ TTS design, while many end-to-end systems still rely on text intermediates or text-guided generation. The stated motivation for MOSS-Speech is that such designs discard paralinguistic cues, constrain expressivity, propagate ASR errors, and impose a text bottleneck on speech interaction (Zhao et al., 1 Oct 2025).

The defining claim of MOSS-Speech is that it is a “true speech-to-speech LLM” that directly understands speech, directly generates speech, and does not require text guidance during speech generation. At the same time, it is explicitly designed not to abandon text competence: the architecture is intended to preserve the reasoning and knowledge of a strong pretrained text model, specifically Qwen3-8B, while adding native speech capabilities (Zhao et al., 1 Oct 2025).

Within the wider MOSS line of work, the label also has an ecosystem meaning. MOSS-TTS is presented as a speech generation foundation model built from discrete audio tokens, autoregressive modeling, and large-scale pretraining; MOSS-TTSD extends this line to long-form spoken dialogue generation; MOSS-VoiceGenerator applies the same family of ideas to instruction-driven voice design; and MOSS Transcribe Diarize addresses speaker-attributed, time-stamped transcription (Gong et al., 18 Mar 2026). This suggests that “MOSS-Speech” denotes both a specific speech-to-speech model and a broader architectural program for speech-native LLM systems.

2. Core architecture

The specific MOSS-Speech model uses Qwen3-8B as its backbone and introduces speech through a modality-based layer-splitting architecture. Qwen3-8B has 36 decoder-only Transformer blocks. MOSS-Speech uses the first 32 layers as shared multimodal layers and splits the final 4 layers into two modality-specific branches: a text branch for text token prediction and a speech branch for speech token prediction (Zhao et al., 1 Oct 2025).

This architectural choice is motivated by a layer-wise analysis of speech–text hidden-state similarity. In an earlier speech model, similarity increases through lower and middle layers and drops sharply in the top layers, which is interpreted as evidence that lower and middle layers are cross-modal and the upper layers are modality-specific. MOSS-Speech operationalizes this by sharing the lower 32 layers and specializing only the top 4 layers (Zhao et al., 1 Oct 2025).

Operationally, speech and text tokens are embedded into a single autoregressive sequence processed by the shared trunk. If the next target token is text, the hidden state is routed through the text branch and text LM head; if it is speech, the hidden state is routed through the speech branch and speech LM head. This preserves a unified causal model while allowing text and speech decoding heads to diverge where modality specialization is most useful (Zhao et al., 1 Oct 2025).

The model is therefore not a pipeline stitched together from separate ASR and TTS modules. It is a single autoregressive Transformer with a shared semantic backbone and late modality specialization. A plausible implication is that this design is intended to preserve cross-modal transfer in the shared layers while reducing interference at the output layers.

3. Speech tokenizer and speech representation

MOSS-Speech depends on a streaming speech tokenizer and decoder. The encoder is based on a modified GLM-4-Voice tokenizer, made fully causal rather than block-causal, and trained only with an ASR objective in a manner described as similar to the CosyVoice 2 semantic tokenizer. Its design requirements are a single-codebook, low-bitrate representation, high semantic content, preservation of paralinguistic information, and full streaming operation (Zhao et al., 1 Oct 2025).

The encoder operates at 12.5 Hz and 175 BPS. On LibriSpeech, evaluated through a 0.6B ASR decoder, it yields an overall WER of 10.80%, compared with 9.17% for GLM-4-Voice, 13.78% for CosyVoice2, and 14.45% for Mimi-8. The paper emphasizes that the MOSS-Speech encoder is fully streaming, unlike GLM-4-Voice’s 2-second chunking (Zhao et al., 1 Oct 2025).

Speech generation uses a flow-matching decoder, described as following CosyVoice 2 while using a compressed chunk size to reduce latency. On Seed-TTS-Eval, the decoder reports English WER 4.14, Chinese WER 2.86, DNSMOS approximately 3.10–3.24, and speaker similarity comparable to CosyVoice 2 (Zhao et al., 1 Oct 2025).

This tokenizer-decoder pair is closely aligned with the tokenizer strategy described for MOSS-TTS. There, MOSS-Audio-Tokenizer is presented as a causal Transformer RVQ-GAN codec that compresses 24 kHz audio to 12.5 fps with variable-bitrate RVQ and unified semantic-acoustic representations, and serves as the token space for downstream autoregressive generators (Gong et al., 18 Mar 2026). This suggests that MOSS-Speech inherits not only a streaming semantic encoder but a family-wide commitment to discrete, low-rate, semantically meaningful audio tokens.

4. Frozen pre-training and training recipe

MOSS-Speech uses a two-stage frozen pre-training strategy on speech data. In Stage 1, all original text LLM parameters are frozen, including the shared backbone, text-specific layers, text embeddings, and text LM head. Only the speech token embeddings, the speech-specific Transformer layers, and the speech LM head are trained. The stated purpose is to align speech tokens with the semantic space of the frozen text LLM without damaging text competence (Zhao et al., 1 Oct 2025).

In Stage 2, speech training is combined with text-only data to preserve knowledge. Three variants are studied: FP–Full, which unfreezes all parameters; FP–Shared, which unfreezes only the shared layers while keeping text-specific layers and text LM head frozen; and FP–Layerwise, which gradually unfreezes shared layers from last to first using a layer-wise delayed warmup–cosine schedule. All three substantially outperform naive non-frozen training (Zhao et al., 1 Oct 2025).

The training corpus begins from about 9M hours of crawler audio, reduced to about 4M hours of speech after VAD with pyannote. This is split into interleaved speech–text pre-training and unsupervised speech pre-training. The interleaved subset is augmented with synthetic speech–text pairs derived from FineWeb-Edu and OpenCSG FineWeb-Edu V2.1 using CosyVoice 2 TTS, explicitly to compensate for the lower knowledge density of raw speech corpora (Zhao et al., 1 Oct 2025).

Supervised fine-tuning then builds on more than 1.5M English and Chinese question–answer pairs derived from text SFT datasets, rewritten by GPT-5 into TTS-friendly spoken style and synthesized using Seed-TTS and MOSS-TTSD. The model is trained in four modality configurations—speech-to-speech, speech-to-text, text-to-speech, and text-to-text—controlled by system prompts, so that one model supports all four interaction patterns (Zhao et al., 1 Oct 2025).

The training objective is standard autoregressive cross-entropy over mixed text and speech token sequences:

$$\mathcal{L} = -\sum_{t=1}^{T} \log p_\theta(y_t \mid y_{<t}),$$

with token routing determined by the target modality (Zhao et al., 1 Oct 2025).

5. Capabilities and empirical behavior

The model supports speech-to-speech, speech-to-text, text-to-speech, and text-to-text interaction in a single architecture. Its central empirical claim is that it narrows the gap between direct speech generation and text-guided speech systems while maintaining competitive text performance (Zhao et al., 1 Oct 2025).

On pre-training evaluations, MOSS-Speech reports stronger text and spoken StoryCloze results than several prior speech-language systems. On the spoken and text StoryCloze variants, it reports 84.87 and 63.17 in English and 90.32 and 71.94 in Chinese, while also reporting MMLU 67.19 and CMMLU 69.53. The paper contrasts these with lower text-task numbers for Moshi, GLM-4-Voice, and SpiritLM (Zhao et al., 1 Oct 2025).

On supervised spoken question answering, MOSS-Speech is evaluated in both speech-to-text and speech-to-speech settings on LLaMA-Question, TriviaQA, and WebQA. In speech-to-speech mode it reports 63.67 on LLaMA-Question, 28.80 on TriviaQA, and 36.71 on WebQA, with UTMOS 4.37. The paper presents these results as competitive with text-guided systems such as GLM-4-Voice while not relying on text guidance during speech generation (Zhao et al., 1 Oct 2025).

The model also aims to preserve paralinguistic information. The argument is architectural rather than based on a specialized auxiliary loss: because speech is directly tokenized, processed in shared hidden space, and regenerated from speech tokens rather than text, timing patterns, hesitations, and other signal-level cues need not be projected into orthographic text before decoding (Zhao et al., 1 Oct 2025). This suggests that the model’s expressivity depends as much on the tokenizer and speech decoder as on the Transformer proper.

The surrounding MOSS ecosystem extends these capabilities in adjacent directions. MOSS-TTSD supports up to 60 minutes of single-pass spoken dialogue generation, up to 5 speakers, and zero-shot voice cloning from short reference clips; MOSS-VoiceGenerator performs voice design from natural language descriptions; and MOSS Transcribe Diarize performs speaker-attributed, time-stamped transcription with a 128k context window for up to 90-minute audio (Zhang et al., 20 Mar 2026). Together, these works indicate that MOSS-Speech is part of a larger attempt to build a complete speech-native stack rather than a standalone speech codec LM.

6. Relations to the broader MOSS speech family

The broader MOSS speech family is organized around a shared set of modeling commitments: discrete audio tokens, autoregressive generation, long-context modeling, and large-scale pretraining. MOSS-TTS formalizes this most explicitly, releasing both a structurally simple delay-pattern generator and a MOSS-TTS-Local-Transformer variant with a frame-local autoregressive module for stronger speaker preservation and shorter time to first audio (Gong et al., 18 Mar 2026).

MOSS-TTSD extends the same discrete-token Qwen-based design to multi-party spoken dialogue. It uses Qwen3-8B-base, MOSS-Audio-Tokenizer, a MusicGen-style multi-head delay pattern over 16 RVQ layers, and a 65,536-token context window. It is designed for dialogue scripts with explicit speaker tags and optional reference audios, enabling long-form, multi-speaker synthesis in a single pass (Zhang et al., 20 Mar 2026).

MOSS-VoiceGenerator shifts the conditioning interface from dialogue scripts or text prompts to free-form voice descriptions. It is a Qwen3-based codec LLM trained on about 25,000 hours of Chinese and English speech, including cinematic content, and is framed as the voice-design and text-to-speech backend of a hypothetical MOSS-Speech stack (Huang et al., 30 Mar 2026).

MOSS Transcribe Diarize provides the inverse direction: end-to-end speaker-attributed, time-stamped transcription. It models SATS as a single autoregressive text sequence containing words, speaker tags, and timestamp tokens, trained under a standard language-model loss. In a wider MOSS-Speech ecosystem, it functions as the transcription and diarization module for long multi-speaker audio (Yu et al., 4 Jan 2026).

At the front-end level, MossFormer and MossFormer2 contribute separation and enhancement building blocks. MossFormer introduces a gated single-head Transformer with convolution-augmented joint local-global self-attention for monaural speech separation, while MossFormer2 adds an FSMN-based recurrent module to better model fine-scale recurrent patterns (Zhao et al., 2023). A plausible implication is that the MOSS family treats front-end enhancement, discrete tokenization, speech generation, and speech understanding as interoperable components rather than isolated tasks.

7. Limitations and open questions

MOSS-Speech does not fully preserve the original text-only competence of Qwen3-8B. Although its MMLU and CMMLU scores remain substantially higher than those of several prior speech-LLMs, they remain below the original Qwen3-8B values reported in the same comparison. The paper therefore presents preservation, not strict retention, of text ability (Zhao et al., 1 Oct 2025).

The current evaluation is concentrated on English and Chinese. The pre-training and SFT pipeline are large-scale, but explicit evidence on low-resource languages, diverse accents, or highly adverse acoustic conditions is limited. This suggests that multilingual generalization beyond the reported settings remains an open empirical question (Zhao et al., 1 Oct 2025).

The system is streaming, but it is still built around an 8B multimodal LLM plus a speech tokenizer and flow-matching decoder. A plausible implication is that latency and deployment cost remain materially higher than in compact cascaded systems, even if the architecture removes text guidance and some forms of pipeline delay.

Finally, the family-wide move toward direct speech generation, voice design, and long-form dialogue raises familiar questions about impersonation, privacy, and controllability. The supplied reports emphasize model capability, data pipelines, and objective evaluation, but do not present a comparably detailed treatment of watermarking or anti-spoofing safeguards (Huang et al., 30 Mar 2026). In encyclopedia terms, MOSS-Speech is therefore best understood as a technically coherent and increasingly complete speech-native LLM program whose most distinctive claim is not merely multimodality, but the removal of text guidance from speech generation while retaining a strong pretrained text backbone.