MOSS-VoiceGenerator: Open-Set Voice Design

• MOSS-VoiceGenerator is an open-source model that generates new speaker timbre and expressive delivery from free-form textual descriptions and transcripts.
• It employs a discrete autoregressive codec language model with delay-pattern multi-codebook decoding, allowing joint processing of instruction and transcript tokens.
• Cinematic-first training on 25,000 hours of expressive speech data ensures natural pauses, hesitations, and varied vocal textures for realistic voice synthesis.

MOSS-VoiceGenerator is an open-source, instruction-driven voice generation model for open-set voice design: it generates a new speaker timbre and expressive delivery directly from a free-form textual description and a target transcript, without requiring reference audio. In the MOSS family, it occupies the text-conditioned voice-design niche, emphasizing natural-language control over timbre, persona, emotion, and delivery style, and it is trained on large-scale expressive speech data sourced from cinematic content rather than only carefully recorded studio speech (Huang et al., 30 Mar 2026).

1. Problem setting and conceptual scope

MOSS-VoiceGenerator addresses voice design from natural language. Its inputs are a natural language instruction $s$ describing timbre, style, or persona, and a text transcript $x$ to be spoken; its output is an audio waveform rendered through discrete audio tokens. The target use cases named for the model include storytelling, game dubbing, role-play agents, and conversational assistants, all of which require controllable voice creation rather than only accurate reading of text (Huang et al., 30 Mar 2026).

The conditioning space is explicitly broad. The instruction can encode gender presentation, age, texture, accent, emotion, tone, speaking rate, pitch, volume, clarity, fluency, and higher-level role or personality attributes. The paper aligns these controllable factors with the attributes assessed by InstructTTSEval: pitch, speed, emotion, gender, age, clarity, fluency, accent, texture, tone, volume, and personality. Control is applied through free-form language rather than through discrete speaker IDs, explicit sliders, or reference-voice enrollment (Huang et al., 30 Mar 2026).

A recurrent misconception is to equate the system with ordinary zero-shot voice cloning. The model is instead framed as open-set voice design: it creates a voice from description alone. This distinguishes it from systems that require prompt audio, and also from preset-voice editing systems that modify an existing timbre rather than synthesize a new one from text alone. The paper’s comparison protocol reflects this distinction by noting that some baselines require reference audio or support only voice editing (Huang et al., 30 Mar 2026).

2. Generative architecture and token representation

MOSS-VoiceGenerator follows a discrete autoregressive codec language modeling design closely aligned with MOSS-TTS. The instruction $s$ and transcript $x$ are concatenated using a fixed chat-style template and tokenized by the native tokenizer of the Qwen3 backbone. There is no separate CLIP-like text encoder, and conditioning is injected by direct token concatenation: the decoder-only LLM attends jointly to instruction tokens, transcript tokens, and generated audio-codec tokens within a single autoregressive sequence (Huang et al., 30 Mar 2026).

The audio side is built on the MOSS-Audio-Tokenizer. In MOSS-VoiceGenerator, audio is discretized into RVQ codebooks, and only the first 16 RVQ layers are used during training and inference. Generation uses a delay-pattern scheme in which the $j$-th codebook layer is shifted forward by $\delta_j = j - 1$ frames relative to layer 1, preserving causality while enabling multi-layer RVQ prediction. After generation reaches EOS, the stacked RVQ token streams are decoded to waveform by the tokenizer’s decoder; no separate mel-spectrogram acoustic model or external neural vocoder is inserted in the path (Huang et al., 30 Mar 2026).

The probabilistic objective is standard next-token prediction over codec tokens:

$$p_\theta(y \mid x, s) = \prod_{t=1}^{T} p_\theta(y_t \mid y_{< t}, x, s),$$

with training loss

$$L_{\mathrm{NTP}}(\theta) = - \sum_t \log p_\theta(y_t \mid y_{< t}, x, s).$$

Here, $y$ denotes the sequence of discrete audio tokens and $\theta$ the model parameters. No auxiliary MSE, adversarial, diffusion, or variational losses are reported for the generator itself (Huang et al., 30 Mar 2026).

Within the broader MOSS stack, the shared tokenizer infrastructure is specified in the MOSS-TTS technical report: MOSS-Audio-Tokenizer is a causal Transformer codec with variable-bitrate RVQ and unified semantic–acoustic supervision that compresses 24 kHz audio to 12.5 fps, so each frame corresponds to 80 ms (Gong et al., 18 Mar 2026). This suggests that MOSS-VoiceGenerator inherits the same discrete-token operating regime even though its task is not cloning or duration control per se, but text-driven voice design.

3. Data curation and instruction construction

The data pipeline is explicitly cinematic-first. Phase 1 mines speech from movies, TV dramas, and episodic series. Speaker diarization is performed with DiariZen; denoising uses MossFormer2_SE_48K; DNSMOS is used for quality control with a threshold of at least 3.0; single-speaker filtering uses MOSS-Transcribe-Diarize; and transcription uses Qwen3-Omni-30B-A3B-Instruct, with Whisper-large-v3 filtering out non-Chinese and non-English material. Detailed speech captions are then generated by Gemini-2.5 Pro and converted into natural-language timbre instructions by Qwen3-32B-A3B-Instruct. This first phase yields about 5,000 hours of data (Huang et al., 30 Mar 2026).

The denoising stage is central to the dataset design. Without denoising, about 5% of cinematic clips pass the quality filter; with denoising, retention increases to roughly 45–50%. The paper explicitly states that this accepts mild artifacts, such as residual breaths or high-frequency smoothing, in order to preserve diversity. The underlying hypothesis is that exposure to real-world acoustic variation produces more perceptually natural voices than reliance on studio-clean corpora alone (Huang et al., 30 Mar 2026).

Phase 2 scales annotation and style coverage. A fine-tuned Qwen3-Omni-30B-A3B-Thinking model becomes an internal speech captioner. A style-aware speech–text embedding model, described as Speech-CLAP-like, maps instructions and audio into a shared space; GPT-5 generates diverse style-instruction queries; and for each instruction the top 50 nearest-neighbor clips are retrieved from internal TTS corpora, with retrieved items removed from the pool to avoid reuse. This style-guided mining contributes roughly 10,000 hours of stylistically rich audio, and additional internal sources include crowdsourced dubbing annotated by the fine-tuned captioner. The final dataset totals approximately 25,000 hours: 18,025 hours in Chinese and 7,047 hours in English (Huang et al., 30 Mar 2026).

The dataset scope is therefore narrower in language than some multilingual TTS systems, but broader in acoustic realism. Languages beyond Chinese and English were filtered out for this release. A plausible implication is that the model prioritizes richness of expressive, everyday, and cinematic acoustic conditions over sheer language count.

4. Training regime and semantics of control

Training begins from Qwen3 checkpoints and proceeds end-to-end on instruction-plus-transcript to codec-token sequences. All parameters are updated; the paper states that no LoRA or PEFT method is used. Two scales, 1.7B and 8B, are compared under the same recipe. The reported outcome is that the 1.7B model achieves comparable instruction-following quality with better generation diversity, and mixing in roughly 10,000 hours of non-instruction TTS-base data does not improve results. The final released model is therefore the 1.7B variant trained exclusively on instruction data (Huang et al., 30 Mar 2026).

The model’s control semantics are implicit and compositional. There is no external parser at inference time, no explicit attribute classifier, and no disentanglement loss reported. Instead, instruction following emerges from end-to-end training on instruction–text–speech triples, where the instruction text is derived from captioners that describe timbre, style, and expressive factors present in audio. The model thus treats persona, age, emotion, accent, pitch, speed, volume, clarity, fluency, and texture as ordinary natural-language conditions expressed in the same token space as the transcript (Huang et al., 30 Mar 2026).

English receives a targeted augmentation because the English subset is smaller than the Chinese subset. Early checkpoints reportedly showed unnatural pauses. To strengthen the English instruction signal, each English audio clip is paired with two semantically equivalent but lexically distinct instruction variants, doubling the English instruction supervision and stabilizing English prosody. This is a training-data intervention rather than an architectural change (Huang et al., 30 Mar 2026).

The paper’s own explanation for the model’s realism rests on four linked design choices: a discrete autoregressive codec LM, delay-pattern multi-codebook decoding, cinematic-first data, and instruction-centric supervision. Together, these are presented as the reason the system can render natural pauses, hesitations, irregular rhythm, and emotionally colored delivery rather than only well-articulated studio-style speech (Huang et al., 30 Mar 2026).

5. Evaluation, empirical behavior, and known limitations

Objective evaluation is conducted on InstructTTSEval, a benchmark of 6,000 cases drawn from cinematic sources. The benchmark comprises three task types—APS, DSD, and RP—across two languages with 1,000 samples per task-language combination. APS specifies 12 explicit acoustic attributes; DSD rewrites the same attributes into freer descriptions with omissions; and RP uses abstract role-play scenarios. The authors report corpus exclusion and fuzzy transcript matching to avoid overlap with training data (Huang et al., 30 Mar 2026).

The reported instruction-following accuracy percentages are as follows:

Language	APS	DSD	RP
English	68.2	82.0	68.7
Chinese	78.0	80.0	74.0

These figures place the model as competitive among open-source systems and particularly strong on DSD and RP, where the instruction signal is less templatic and more natural-language-like (Huang et al., 30 Mar 2026).

Subjective evaluation uses 100 instruction–text items, split evenly between Chinese and English, with three comparison dimensions: Overall Performance, Instruction Following, and Naturalness. Each item is annotated by three independent crowd workers, with majority vote determining the final label; each worker scores only one dimension per pair. Baselines in the pairwise study are Qwen3-TTS-VD, MiniMax Voice Design, and MiMo-Audio-7B-Instruct. The text states that MOSS-VoiceGenerator consistently wins across all three dimensions against all baselines, although the exact Win/Tie/Lose percentages are not given in the text. Qualitative analysis emphasizes natural pauses and hesitations, age-related vocal color, and spontaneous expressions such as laughter, sobbing, awkwardness, and anger (Huang et al., 30 Mar 2026).

The limitations are also explicit. The release covers only Chinese and English. English prosody is weaker in some styles because the English subset is smaller, despite augmentation. Denoising improves retention but can introduce mild artifacts, including residual breath and high-frequency smoothing. The paper also notes occasional generation instability. On safety and governance, it does not detail safeguards such as speaker identity protection, consent handling, watermarking, or policy enforcement, and treats these as issues outside the scope of the reported model (Huang et al., 30 Mar 2026).

6. Relation to adjacent MOSS systems and broader significance

MOSS-VoiceGenerator is one component of a broader MOSS speech stack rather than a standalone design philosophy. Related systems in the family occupy adjacent but distinct problem settings:

System	Primary conditioning	Reported capability
MOSS-VoiceGenerator	Natural-language instruction + transcript	Open-set voice design from description
MOSS-TTS	Text, optional prompt audio, optional duration field	Zero-shot voice cloning, duration and pronunciation control, long-form generation
MOSS-TTSD	Speaker-tagged dialogue script, optional per-speaker references	Multi-speaker spoken dialogue generation up to 60 minutes
MOSS-Speech	Speech context without text guidance during inference	True speech-to-speech generation without text intermediates

MOSS-TTS provides the closest infrastructural precedent. It uses the same MOSS-Audio-Tokenizer, supports zero-shot voice cloning, token-level duration control, phoneme- and pinyin-level pronunciation control, smooth code-switching, and stable long-form generation, with two released generators—MOSS-TTS and MOSS-TTS-Local-Transformer—trading off structural simplicity against stronger speaker preservation and shorter time to first audio (Gong et al., 18 Mar 2026). MOSS-VoiceGenerator departs from this cloning-and-control framing by replacing prompt-audio conditioning with natural-language voice specification.

MOSS-TTSD extends the family toward dialogue-native synthesis. It is built on Qwen3-8B-base and the MOSS-Audio-Tokenizer, models the first 16 RVQ layers at 2 kbps and 12.5 Hz, supports up to five speakers, and can synthesize up to 60 minutes in a single pass from scripts with explicit speaker tags and optional per-speaker reference audio. In that sense, MOSS-TTSD can be read as the multi-speaker, dialogue-centric complement to MOSS-VoiceGenerator’s single-request voice-design focus (Zhang et al., 20 Mar 2026).

MOSS-Speech pushes in a different direction entirely: it is a true speech-to-speech LLM that directly understands and generates speech without relying on text guidance during inference. Its architecture fuses speech and text representations in a shared transformer trunk, then specializes the final layers for speech or text generation, with the explicit aim of removing the text bottleneck that constrains prosody, timbre, emotion, laughter, hesitation, and other non-verbal vocalizations (Zhao et al., 1 Oct 2025). Relative to that system, MOSS-VoiceGenerator remains text-conditioned, but it shares the larger MOSS program of treating speech generation as native token generation rather than as a cascade of ASR, text reasoning, and TTS.

The significance of MOSS-VoiceGenerator therefore lies less in introducing a new codec-LM paradigm than in specializing that paradigm for instruction-driven voice creation. It is positioned at the intersection of controllable TTS, expressive speech synthesis, and natural-language-conditioned persona design. Its defining claim is not merely that it can synthesize speech, but that it can map free-form descriptions of character, affect, and delivery into plausible new timbres without reference audio, while preserving the “lived-in” acoustic qualities associated with cinematic, in-the-wild speech (Huang et al., 30 Mar 2026).