UltraVoice: A Style-Controlled Speech Dataset

• UltraVoice is a large-scale synthetic speech-dialogue dataset designed with fine-grained style control over emotion, speed, volume, accent, language, and composite styles.
• It bridges the gap between generic TTS and dialogue systems by integrating natural-language style instructions with matched audio responses across 832 hours of diverse speech.
• The dataset’s robust synthetic pipeline and empirical gains in IFR and MOS demonstrate significant improvements in expressive, controllable conversational AI.

Searching arXiv for recent and directly relevant papers on UltraVoice and adjacent voice-control systems. UltraVoice is a large-scale synthetic speech-dialogue dataset created to train and evaluate spoken dialogue models that can control how they speak, not just what they say. It is presented as the first dataset specifically engineered for multiple fine-grained speech style control in dialogue, with 100,770 speech dialogue samples and 832.92 hours covering emotion, speed, volume, accent, language, and composite styles. Its central purpose is to bind together dialogue context, linguistic response content, natural-language style instructions, and matched stylized response speech so that spoken dialogue models can learn instruction-following over paralinguistic dimensions rather than only semantic response generation (Tu et al., 26 Oct 2025).

1. Definition and scope

UltraVoice addresses a specific gap in spoken dialogue modeling: existing dialogue datasets are mostly text conversations rendered into speech with generic TTS, which preserves lexical content but strips away meaningful paralinguistic variation, while existing controllable TTS datasets are not conversational and thus do not train real dialogue behavior. UltraVoice is designed to close that gap by constructing a dataset that is simultaneously conversational, instruction-rich, acoustically expressive, and suitable for explicit style conditioning (Tu et al., 26 Oct 2025).

In this framework, “fine-grained style control” means explicit control over specific speech attributes at the level of individual response utterances, using natural-language instructions. The six targeted dimensions are emotion, speed, volume, accent, language, and composite styles. The control is fine-grained in two senses: it is dimension-specific, such as “respond in a happy tone” or “answer with a slow pace,” and sub-dimension-specific, because each dimension is decomposed into concrete categories or realizations.

A central implication is that UltraVoice is a data resource rather than a waveform generation architecture. This distinguishes it from systems such as OpenVoice, which is a decoupled instant voice cloning system that separates a base speaker TTS model from a tone color converter (Qin et al., 2023), and from UniVoice, which is a unified speech and singing voice generation framework based on conditional flow matching (Zheng et al., 4 Jun 2026). UltraVoice instead supplies supervised training material for spoken dialogue models and controllable TTS models.

2. Style ontology and dataset composition

UltraVoice operationalizes six control dimensions through explicit category sets and natural-language instruction design. Emotion includes seven categories: Neutral, Happy, Sad, Angry, Surprised, Fearful, and Disgusted. Speed includes Slow, Fast, and Normal. Volume includes Low volume, High volume, and Normal volume. Accent covers six English accents: Australian (AU), Canadian (CA), British/United Kingdom (GB), Indian (IN), Singaporean (SG), and South African (ZA). Language covers Chinese, Japanese, and Korean. Composite styles combine speed, volume, and emotion into a single natural descriptive style instruction rather than enumerating the three attributes directly (Tu et al., 26 Oct 2025).

The final dataset contains 84,832 samples with explicit style conditioning and 15,938 general English QA samples without style prompts. The reported mean CER is 5.93%, the mean UTMOS is 4.00, and the average dialogue length is 29.35 s.

Dimension	Categories or scope	Samples / hours
Emotion	7 categories	21,209 / 182.53 h
Volume	High, Low, Normal	11,154 / 91.37 h
Speed	Fast, Normal, Slow	10,334 / 85.28 h
Accent	AU, CA, GB, IN, SG, ZA	26,839 / 253.31 h
Language	Chinese, Japanese, Korean	11,153 / 93.84 h
Composite	Speed + volume + emotion	4,143 / 33.47 h
General QA	No style prompts	15,938 / 93.12 h

The compositional design is especially important. Composite prompts are converted into short natural descriptions such as “Barely contained rage spilling through sharp speech,” “Soft warmth with slow, deliberate rhythm,” and “Urgent energy rising in a loud, fast tone.” The paper treats this as a more realistic multi-attribute control setting than rigid symbolic tuples.

3. Construction pipeline

UltraVoice is built with a four-stage synthetic pipeline. The first stage is text corpus curation. The base text source is UltraChat, and the selected dialogues come mainly from “Question About the World” and “Creation and Generation.” Filtering removes dialogues containing URLs, academic citations, and long quoted text. After filtering, about 200,000 clean and natural question-answer pairs are retained as the textual foundation (Tu et al., 26 Oct 2025).

The second stage is style injection and stylized response generation. GPT-4o is used both for instruction rewriting and for stylized textual response generation. The rewritten instructions are spoken-dialogue-style requests that naturally embed control directives. The response text is generated to be semantically aligned with the question and stylistically compatible with the requested control. The paper also reports several TTS-oriented rewriting rules: numerals are converted to words, technical or code-like requests are rephrased into more speakable language, responses are kept concise to fit speech lengths of roughly 15–30 seconds, and text structures hard for TTS, such as lists or parentheses, are avoided.

The third stage is stylized speech synthesis. Instruction audio simulates realistic user speech by sampling speaker timbres from seedtts_testset_en, which contains diverse speakers and real-world background noise. Response audio is synthesized using a single fixed timbre across the dataset so that style analysis is not conflated with speaker identity. The response-side model selection is dimension-specific: accent uses Edge TTS plus voice conversion with CosyVoice-300M; composite, emotion, speed, and volume use GPT-4o-audio-preview; language and general QA use CosyVoice 300M. The accent pipeline is distinctive because Edge TTS does not support custom speaker timbre, so accented speech is first synthesized and then voice-converted into the fixed target voice.

The fourth stage is quality control and filtering. Whisper-large-v3 transcribes each instruction and response audio, and only samples with CER < 20% and duration < 30 seconds are retained. This quality filter removes low-quality synthesis, transcription-mismatched samples, and abnormally long utterances.

4. Dialogue format, training usage, and evaluation protocol

UltraVoice is an instruction-response speech dialogue dataset with asymmetric speaker design. The instruction is spoken user audio derived from a rewritten question that often includes style guidance. The response is spoken system audio satisfying both semantic content and style constraints. On the user side, speaker timbres are diverse and noisy; on the assistant side, the response voice is fixed. This asymmetry is deliberate: it introduces realistic user variability while keeping assistant-side style control analytically clean (Tu et al., 26 Oct 2025).

The dataset is used for supervised fine-tuning of spoken dialogue models. The reported models are SLAM-Omni-0.5B, VocalNet-1B, VocalNet-7B, and VocalNet-8B. The paper also reports transfer to controllable TTS by fine-tuning a pre-trained EmoVoice-0.5B checkpoint into UltraVoice-0.5B-SFT. The manuscript repeatedly states that models are trained with supervised fine-tuning, but it does not provide an explicit mathematical training objective.

The evaluation protocol has two layers. First, an internal UltraVoice test set is constructed by randomly sampling 100 examples from each fine-grained dimension across the six major control categories, yielding 2,300 test samples total. Second, URO-Bench is used to test whether style fine-tuning harms or improves general speech dialogue ability in Oral Conversation, Understanding, and Reasoning under Basic and Pro settings.

The core metrics are operational rather than human-listener based. MOS is generated automatically by Gemini-2.5-Flash and asks the evaluator to score from 1 to 5 how well the audio satisfies the instruction, considering content accuracy, emotion, speed, volume, accent, style description, language change, and overall naturalness/alignment. IFR, or Instruction Following Rate, is also computed by Gemini-2.5-Flash and is binary at the sample level: the evaluator outputs 1 if all specified acoustic controls are followed and 0 if any specified control is not followed. WER is computed using Whisper-large-v3. Emotional expressiveness uses emotion2vec embeddings through Emotion Similarity and Recall Rate. UTMOS is used for naturalness.

A common misconception is to read the reported MOS as a standard human MOS. In UltraVoice it is explicitly an automatic score generated by an audio-LLM judge, not a human listening study.

5. Empirical findings

The main empirical result is that fine-tuning on UltraVoice substantially improves fine-grained style controllability without degrading core conversational ability. The reported IFR gains range from +14.61 to +40.09 points, and the reported relative average MOS gains range from +29.12% to +42.33% across the four spoken dialogue models (Tu et al., 26 Oct 2025).

Model	IFR: base → SFT	MOS: base → SFT
SLAM-Omni-0.5B	28.30 → 68.39	2.15 → 3.06
VocalNet-1B	36.28 → 55.91	2.62 → 3.45
VocalNet-7B	41.30 → 55.96	2.73 → 3.59
VocalNet-8B	44.74 → 59.35	2.85 → 3.68

The improvements are especially strong for emotion, accent, volume, and composite description control. Examples reported in the paper include SLAM-Omni-0.5B emotion IFR improving from 13.86 to 58.71, SLAM-Omni-0.5B accent IFR improving from 42.67 to 88.33, and VocalNet-8B composite IFR improving from 28.00 to 65.00. On the MOS side, VocalNet-7B emotion improves from 2.42 to 3.79, and VocalNet-8B composite improves from 2.86 to 4.07.

The clearest weakness is language control. Language remains weak for LLaMA-based models, with VocalNet-1B at 0.33 → 0.33 IFR and VocalNet-8B at 0.67 → 1.00 IFR, while Qwen-based models improve more, such as SLAM-Omni-0.5B at 0.00 → 30.33 and VocalNet-7B at 24.00 → 34.00. The paper attributes this to limited multilingual pretraining exposure and insufficient multilingual diversity and volume in the fine-tuning data.

On URO-Bench, the average improvements are +10.84% on Basic and +7.87% on Pro. This indicates that UltraVoice fine-tuning does not merely specialize models for style following; it often improves broader speech dialogue ability as well. The strongest UltraVoice-trained dialogue model is VocalNet-7B SFT, which reaches Basic avg 81.56 and Pro avg 52.70.

The dataset also transfers to controllable TTS. On UltraVoice in-domain emotional TTS evaluation, UltraVoice-0.5B-SFT reaches WER 3.97, Emo_Sim 0.95, Emo_Recall 0.39, and UTMOS 4.46. Compared with EmoVoice-0.5B-Pre-trained, it improves emotion control from MOS 2.52 to 3.08 and IFR 50.29 to 67.43, accent control from MOS 3.62 to 4.10 and IFR 74.67 to 88.33, and composite control from MOS 3.59 to 3.92 and IFR 76.00 to 86.00. This supports the paper’s claim that the dataset’s utility extends beyond spoken dialogue into controllable speech synthesis.

6. Interpretation, limitations, and relation to adjacent research

UltraVoice’s strongest claim is that expressive, instruction-conditioned spoken dialogue is fundamentally a data problem as much as a modeling problem. If spoken dialogue models are trained only on semantically correct but acoustically neutral supervision, they do not learn controllable paralinguistics. UltraVoice contributes the missing supervision: dialogue context, natural-language style instructions, and matched stylized speech (Tu et al., 26 Oct 2025).

Several limitations are explicit. First, the language-control subset is relatively small and spans only Chinese, Japanese, and Korean. Second, the dataset is largely single-turn QA/dialogue, which the paper links to the one notable regression, where SLAM-Omni-0.5B Pro Reasoning drops from 24.72 to 20.07 on URO-Bench. Third, the dataset is fully synthetic, generated via GPT-4o plus several TTS systems, which avoids privacy issues but may introduce stylistic distribution biases, limited realism in human spontaneity, and dependence on what current TTS systems can render well. Fourth, the response side uses a single fixed timbre, which isolates style but may limit robustness across assistant voices. Fifth, accent and language coverage remain narrow.

UltraVoice also sits in a broader landscape of controllable and modular speech systems. VoiceSculptor is an instruction-driven voice design and voice cloning framework that maps natural-language descriptions into designed prompt voices and then uses CosyVoice2 for downstream speech synthesis (Hu et al., 15 Jan 2026). OpenVoice separates a base speaker TTS model from a tone color converter to achieve instant voice cloning with independent style control (Qin et al., 2023). UniVoice unifies speech and singing voice generation through factorized conditioning over content, melody, timbre, and task (Zheng et al., 4 Jun 2026). This suggests that UltraVoice primarily occupies the supervision layer: it provides style-rich conversational training data that could, in principle, complement architecture-level systems for voice design, cloning, or unified generation.

A further source of confusion is the name itself. Despite the “Ultra” prefix, UltraVoice as defined here is not an ultrasound sensing or silent-speech system. That role is served by systems such as USpeech, which uses pseudo-ultrasound acoustic sensing on commodity phones for speech enhancement (Yu et al., 2024), and SottoVoce, which maps under-jaw ultrasound imaging to synthesized speech for silent interaction (Kimura et al., 2023). UltraVoice is instead a synthetic speech-dialogue corpus for style-controlled spoken dialogue models.

In sum, UltraVoice is best understood as a large-scale, instruction-conditioned conversational speech dataset that operationalizes six dimensions of fine-grained style control and demonstrates that such supervision materially improves both style following and, in many cases, general spoken dialogue capability. Its most durable contribution is not a new decoder or vocoder, but a training resource that makes expressive spoken dialogue a directly learnable target.