--- title: 'UniVocal: Unified Speech-Singing Synthesis' url: https://www.emergentmind.com/topics/univocal type: topic --- # UniVocal: Unified Speech-Singing Synthesis UniVocal is a unified framework for **Speech-Singing Code-Switching (SCS) Synthesis**, a task in which a model generates a single continuous vocal stream that switches between speech and singing within one utterance, with transitions driven by the semantics of the input text rather than explicit mode tags or manual segmentation [2606.01677]. In this framework, speech and singing are treated as distinct vocal modes, yet are generated by a single system that performs standard text-to-speech, standard singing generation, and semantically driven mixed-mode synthesis while preserving speaker identity across modes [2606.01677]. ## 1. Definition and scope UniVocal addresses the problem of generating speech and singing within the same utterance under a single generative framework, with switching decisions inferred implicitly from textual context [2606.01677]. The target behavior is a continuous output in which speech-like and singing-like segments emerge where the text suggests them, rather than where a user inserts explicit control tags [2606.01677]. The framework is positioned against two established but limited classes of systems. Conventional text-to-speech systems are trained on spoken data and therefore learn spoken prosody rather than stable melody, while singing systems are designed for musically constrained generation and do not ordinarily produce natural speech [2606.01677]. Earlier unified systems can generate either speech or singing, or can mix them only when provided with explicit switching cues; UniVocal instead performs switching from text semantics alone, using only a high-level instruction for the overall task type [2606.01677]. The paper uses the term “vocal modes” for speech and singing, and frames SCS synthesis as an attempt to mimic the way humans may shift from conversation into humming or sung phrases and then back to speech, with transitions triggered by context [2606.01677]. This suggests a conception of code-switching that is not linguistic in the usual multilingual sense, but modal: the system must decide when text should be realized with conversational prosody and when with singing-like melodic structure. ## 2. Unified architecture and generation process UniVocal is built on **CosyVoice 2** and uses a single architecture for speech, singing, and SCS generation [2606.01677]. Its core consists of a **Text-to-Vocal Language Model**, a **flow-matching acoustic model**, and a **HiFi-GAN vocoder** [2606.01677]. The language model is a **24-layer causal Transformer** with approximately **0.5B parameters** [2606.01677]. Its input is text plus, when required, a natural-language instruction that sets the global task: SCS instructions such as “Generate a monologue./podcast/audiobook. <|endofprompt|>”, singing instructions describing style, or plain text for standard TTS [2606.01677]. The model outputs an interleaved sequence of **refined cent tokens** and **semantic tokens**, both at **25 Hz** [2606.01677]. The flow-matching acoustic model, modified from CosyVoice 2, conditions on semantic tokens, refined cent tokens, and optional speaker or reference conditioning to produce mel-spectrograms [2606.01677]. A pretrained HiFi-GAN vocoder then converts these spectrograms into waveform audio [2606.01677]. The same acoustic model and vocoder are used for speech, singing, and SCS, which is central to the system’s claim of unification [2606.01677]. At the probabilistic level, the language model factorizes output generation frame by frame as first a pitch-planning step and then a content step: \[ P(\mathbf{Y} \mid \mathbf{X}) = \prod_{t=1}^{T} P(c_t \mid \mathbf{X}, \mathbf{Y}_{” followed by lyrics [2606.01677]. The language model is trained autoregressively on interleaved cent and semantic tokens, and the acoustic model is trained with flow matching to reconstruct spectrograms [2606.01677]. In **Stage 2**, the aligned model is fine-tuned for SCS using a balanced mixture of synthetic mixed-mode samples and reduced single-mode data [2606.01677]. The data include **11,769 synthetic SCS samples, 262 hours**, plus approximately **200 hours** of speech from LibriTTS and approximately **200 hours** of singing from Suno with **10 hours** from GTSinger [2606.01677]. The mixture ratio is **1:1:1** for SCS, speech, and singing [2606.01677]. SCS samples are formatted as first-person monologues, podcasts, or audiobook-like scripts with scenario instructions such as “Generate a monologue. <|endofprompt|>”, but without explicit segment-level speech/sing tags [2606.01677]. The paper reports that removing curriculum learning and training directly on the joint mixture substantially degrades SCS performance: the ablation labeled **“w/o CL”** reduces SCSBench-Mixed F1 from **0.716** to **0.496** [2606.01677]. This indicates that the alignment-first, switching-second training order is not merely convenient but functionally important for acquiring semantically controlled mixed-mode generation. Optimization uses **AdamW** with \(\beta_1 = 0.9\), \(\beta_2 = 0.95\), weight decay **0.1**, and gradient clipping **1.0** [2606.01677]. Stage 1 uses a learning rate linearly decayed from \(2\times10^{-4}\) to 0 over **70k steps** with **5k warmup steps**, while Stage 2 uses a constant learning rate of \(1\times10^{-4}\) for **30k steps** [2606.01677]. Training uses dynamic batching up to **4.5 minutes** of audio per batch and was run on **4 × NVIDIA A800 GPUs**; Stage 1 required approximately **5 days**, Stage 2 approximately **1 day**, and flow-matching fine-tuning approximately **2 days** [2606.01677]. ## 4. Refined cent tokens and prosody planning A central technical contribution of UniVocal is the introduction of **refined cent tokens** and a **Chain-of-Thought (CoT) generation** mechanism for planning prosody before content generation [2606.01677]. The motivation is that semantic tokenizers preserve high-level linguistic content but do not adequately encode fine acoustic detail such as subtle prosody and melody, whereas acoustic tokenizers tend to entangle content, prosody, and timbre [2606.01677]. Pitch is represented in the **cent scale** relative to A4, using \[ f_{cent} = 1200 \times \log_2\left(\frac{f_{Hz}}{440}\right). \] It is then discretized into a single octave with **1200 possible bins**, plus a special unvoiced token: \[ I(f_{cent}) = \begin{cases} \lceil f_{cent} \bmod 1200 \rceil & \text{if } f_{Hz} \neq 0 \\ -1 & \text{if } f_{Hz} = 0 \end{cases} \] yielding a total vocabulary of **1201 tokens** [2606.01677]. The paper states that this gives a quantization error of at most **1 cent** and preserves local melodic contour while discarding absolute octave height [2606.01677]. The CoT mechanism is implemented by predicting the refined cent token \(c_t\) before the semantic token \(s_t\) at each frame [2606.01677]. In effect, the model first produces a frame-level prosodic or melodic plan and then generates content consistent with that plan. The paper explicitly interprets this as a “plan-then-generate” mechanism adapted from Chain-of-Thought reasoning ideas [2606.01677]. The paper reports that predicted and realized cent trajectories have **Spearman and Pearson correlations greater than 0.6**, indicating that the cent token stream genuinely tracks output pitch [2606.01677]. Ablations comparing **12**, **480**, and **1200** pitch bins show that the **1200-bin** representation yields the best expressive speech and competitive singing performance [2606.01677]. The main trade-off is that CoT improves expressive quality more than it improves switching accuracy. Removing CoT slightly improves SCSBench mode-switching F1 but degrades empathetic speech and singing quality [2606.01677]. On the textual empathy test, removing CoT reduces **E-MOS** from **2.26** to **2.03** and **P-MOS** from **2.22** to **1.84**; on Fullsong, **M-MOS** falls from **2.18** to **1.86** [2606.01677]. This suggests that CoT contributes primarily to prosodic richness and musicality rather than to the discrete decision boundary between speech and singing. ## 5. Synthetic SCS data pipeline and SCSBench UniVocal is trained and evaluated on a synthetic SCS corpus constructed by a dedicated pipeline intended to preserve both semantic plausibility and acoustic naturalness [2606.01677]. The pipeline has three main stages. First, **Gemini 2.5 Pro** is prompted to generate first-person scripts in three scenarios: **monologues**, **personal podcasts**, and **audiobooks** [2606.01677]. These scripts are speech-dominant but contain naturally inserted singing or humming segments [2606.01677]. The paper distinguishes **implicit cues**, where lyrical or repetitive language suggests singing, from **explicit cues**, where transitional phrases such as “And that reminds me of a lyric…” or “Let me try to recall that jingle for you…” are inserted immediately before a singing segment [2606.01677]. Approximately **50%** of samples contain such explicit cues [2606.01677]. Second, the **Stage-1 model** is used to synthesize the audio [2606.01677]. Scripts are segmented into speech versus singing or humming parts, but all segments within a sample share the same speaker embedding [2606.01677]. Speech segments are conditioned on emotion-specific reference audio drawn from **Expresso** and **EmoVoice-DB**, with emotions including **confused, happy, sad, angry, surprised, fearful, disgusted, default,** and **laughing** [2606.01677]. Singing segments are conditioned only on the target speaker embedding [2606.01677]. Segments are concatenated with **0.25 s silence** inserted at boundaries [2606.01677]. Third, the samples are filtered for quality using **Whisper v3** [2606.01677]. Samples with **WER ≥ 20%** are discarded, while samples with **WER between 10% and 20%** are retained but their text is replaced by ASR transcripts [2606.01677]. Approximately **15%** of initial samples are removed by this filtering stage [2606.01677]. The resulting dataset contains **11,769 samples** and **262 hours** of audio [2606.01677]. A held-out subset of approximately **1,210 samples** forms **SCSBench**, divided into **SCSBench-Implicit**, **SCSBench-Explicit**, and **SCSBench-Mixed**, balanced across cue type and scenario [2606.01677]. The remainder is used for Stage-2 training [2606.01677]. A plausible implication is that UniVocal’s performance depends not only on architectural unification but also on the availability of training data in which semantics, speaker consistency, and cross-mode continuity are already aligned. The article’s own ablations and real-world tests are consistent with that interpretation [2606.01677]. ## 6. Empirical performance, related systems, and limitations UniVocal is evaluated against cascaded baselines on SCSBench and against strong systems on standard speech and singing tasks [2606.01677]. The main SCS baselines are **Gemini + Bark** and **Gemini + CosyVoice 2 + LeVo** [2606.01677]. In the first, Gemini segments the text and Bark synthesizes segments; in the second, Gemini segments the text, CosyVoice 2 generates speech, and LeVo generates singing [2606.01677]. On **SCSBench-Mixed**, UniVocal reaches **F1(O) = 0.871** and **F1(S) = 0.810**, compared with **0.607 / 0.566** for Gemini+CosyVoice 2+LeVo and **0.465 / 0.199** for Gemini+Bark [2606.01677]. The paper also states that UniVocal achieves the **lowest WER** and **highest UTMOS** across SCSBench subsets, while its speaker similarity is slightly lower than Gemini+CosyVoice 2+LeVo but its **intra-sample speaker consistency** is substantially better [2606.01677]. The latter follows from using a single unified generator rather than stitching together outputs from separate speech and singing systems [2606.01677]. On **SeedTTS-EN**, UniVocal is compared with **F5-TTS**, **CosyVoice 2**, and **Vevo 1.5** [2606.01677]. It achieves **WER 2.69**, **SIM 0.703**, and **UTMOS 4.21**, with the best UTMOS among the systems listed [2606.01677]. On a dedicated textual empathy test, UniVocal reaches **E-MOS 2.26**, **P-MOS 2.22**, and **WER 0.32**, compared with **1.78 / 1.74** for CosyVoice 2 and **2.30 / 2.47** for ElevenLabs multilingual-v2 [2606.01677]. On singing benchmarks, UniVocal is evaluated on **GTSinger** and **Fullsong** [2606.01677]. On GTSinger it obtains **WER 18.07** and **QUA 10.70**, both reported as best among the compared systems, with **SIM 0.703** [2606.01677]. On Fullsong it achieves **WER 35.88**, **SIM 0.72**, **QUA 7.75**, **M-MOS 2.18**, and **N-MOS 2.23**, outperforming or matching systems such as **Vevo 1.5**, **YuE**, and **LeVo** on the listed metrics [2606.01677]. The paper nevertheless identifies several limitations. The singing corpus is partly derived from Suno-generated material and processed through source separation and ASR, which introduces artifacts such as **“electric tone” residuals**, lyric misalignment, and noisy style tags [2606.01677]. SCS training data are synthetic and centered on idealized monologue, podcast, and audiobook settings, producing a domain gap relative to spontaneous real-world SCS [2606.01677]. On approximately **30 minutes** of real human SCS recordings, UniVocal achieves **F1 ≈ 0.201** without explicit cues, but this rises to **0.730** when a single explicit semantic cue is added per sample [2606.01677]. Purely implicit lyric-like text can still be realized as speech rather than singing, while humming is easier because it is textually distinct [2606.01677]. These results place UniVocal among unified speech-singing systems but differentiate it by the absence of segment-level switching tags. The paper explicitly contrasts it with **Bark**, which can mix speech and singing but requires explicit control tags and exhibits unstable transitions, and with **UniSyn**, **UniAudio**, and **Vevo2**, which can generate one mode at a time but not semantically inferred intra-utterance switching [2606.01677]. ## 7. Significance, reproducibility, and outlook UniVocal’s principal significance lies in combining three capabilities in one framework: competitive text-to-speech, competitive singing generation, and semantically driven speech-singing switching without explicit segment annotations [2606.01677]. The system unifies these behaviors in a single language-model-plus-acoustic-decoder pipeline, rather than delegating them to separate specialized generators [2606.01677]. The released resources include code, dataset assets, and pretrained checkpoints at the project repository, and the paper states that code and dataset are released [2606.01677]. The implementation includes the integration of refined cent tokens and CoT generation into a CosyVoice-like pipeline, along with scripts for the SCS data synthesis process [2606.01677]. The paper also notes a restrictive license intended to prohibit commercial misuse and illegal impersonation [2606.01677]. Several future directions are suggested by the reported results. Better singing data could reduce artifacts inherited from synthetic training material; more realistic SCS corpora could reduce the gap between synthetic benchmarks and spontaneous mixed-mode vocal behavior; and finer-grained controllability could complement the present emphasis on implicit inference [2606.01677]. The paper also points toward multilingual extensions and interactive applications such as assistants that can sing or hum within otherwise spoken responses [2606.01677]. In that sense, UniVocal marks a shift from single-mode generation and manually controlled switching toward a model in which speech and singing are treated as context-sensitive realizations of one vocal generation problem. Its central technical claim is not merely that speech and singing can coexist in one architecture, but that the decision boundary between them can be learned from text semantics and implemented through a shared token-planning and acoustic-rendering stack [2606.01677]. Source: https://www.emergentmind.com/topics/univocal