UniVocal: Unified Speech-Singing Synthesis

• UniVocal is a unified framework for speech-singing code-switching that infers mode transitions implicitly from text cues, eliminating explicit control tags.
• It integrates a 24-layer Transformer, a flow-matching acoustic model, and a HiFi-GAN vocoder to generate refined cent tokens and plan prosody before content synthesis.
• A two-stage curriculum learning strategy with a synthetic SCS dataset underpins its high performance in mixed-mode synthesis, achieving competitive F1 scores and speaker consistency.

UniVocal is a unified framework for Speech-Singing Code-Switching (SCS) Synthesis, a task in which a model generates a single continuous vocal stream that switches between speech and singing within one utterance, with transitions driven by the semantics of the input text rather than explicit mode tags or manual segmentation (Shi et al., 1 Jun 2026). In this framework, speech and singing are treated as distinct vocal modes, yet are generated by a single system that performs standard text-to-speech, standard singing generation, and semantically driven mixed-mode synthesis while preserving speaker identity across modes (Shi et al., 1 Jun 2026).

1. Definition and scope

UniVocal addresses the problem of generating speech and singing within the same utterance under a single generative framework, with switching decisions inferred implicitly from textual context (Shi et al., 1 Jun 2026). The target behavior is a continuous output in which speech-like and singing-like segments emerge where the text suggests them, rather than where a user inserts explicit control tags (Shi et al., 1 Jun 2026).

The framework is positioned against two established but limited classes of systems. Conventional text-to-speech systems are trained on spoken data and therefore learn spoken prosody rather than stable melody, while singing systems are designed for musically constrained generation and do not ordinarily produce natural speech (Shi et al., 1 Jun 2026). Earlier unified systems can generate either speech or singing, or can mix them only when provided with explicit switching cues; UniVocal instead performs switching from text semantics alone, using only a high-level instruction for the overall task type (Shi et al., 1 Jun 2026).

The paper uses the term “vocal modes” for speech and singing, and frames SCS synthesis as an attempt to mimic the way humans may shift from conversation into humming or sung phrases and then back to speech, with transitions triggered by context (Shi et al., 1 Jun 2026). This suggests a conception of code-switching that is not linguistic in the usual multilingual sense, but modal: the system must decide when text should be realized with conversational prosody and when with singing-like melodic structure.

2. Unified architecture and generation process

UniVocal is built on CosyVoice 2 and uses a single architecture for speech, singing, and SCS generation (Shi et al., 1 Jun 2026). Its core consists of a Text-to-Vocal LLM, a flow-matching acoustic model, and a HiFi-GAN vocoder (Shi et al., 1 Jun 2026).

The LLM is a 24-layer causal Transformer with approximately 0.5B parameters (Shi et al., 1 Jun 2026). Its input is text plus, when required, a natural-language instruction that sets the global task: SCS instructions such as “Generate a monologue./podcast/audiobook. <|endofprompt|>”, singing instructions describing style, or plain text for standard TTS (Shi et al., 1 Jun 2026). The model outputs an interleaved sequence of refined cent tokens and semantic tokens, both at 25 Hz (Shi et al., 1 Jun 2026).

The flow-matching acoustic model, modified from CosyVoice 2, conditions on semantic tokens, refined cent tokens, and optional speaker or reference conditioning to produce mel-spectrograms (Shi et al., 1 Jun 2026). A pretrained HiFi-GAN vocoder then converts these spectrograms into waveform audio (Shi et al., 1 Jun 2026). The same acoustic model and vocoder are used for speech, singing, and SCS, which is central to the system’s claim of unification (Shi et al., 1 Jun 2026).

At the probabilistic level, the LLM factorizes output generation frame by frame as first a pitch-planning step and then a content step:

$$P(\mathbf{Y} \mid \mathbf{X}) = \prod_{t=1}^{T} P(c_t \mid \mathbf{X}, \mathbf{Y}_{<t}) \cdot P(s_t \mid \mathbf{X}, \mathbf{Y}_{<t}, c_t).$$

Here $c_t$ denotes the refined cent token and $s_t$ the semantic token at frame $t$ (Shi et al., 1 Jun 2026). Logit masking enforces the interleaved structure by allowing only cent-token outputs at cent positions and only semantic-token outputs at semantic positions (Shi et al., 1 Jun 2026).

A notable property of the architecture is that it contains no explicit per-segment speech/singing mode variable. Mode selection is instead encoded implicitly in the token sequence distribution learned by the LLM from mixed-mode training data (Shi et al., 1 Jun 2026). The global instruction determines the task regime, while segment-level switching arises from textual semantics and the learned association between semantic patterns and acoustic token trajectories (Shi et al., 1 Jun 2026).

3. Curriculum learning and training strategy

UniVocal is trained with a two-stage curriculum learning strategy that first aligns speech and singing in a common latent space and then teaches the model to perform speech-singing switching (Shi et al., 1 Jun 2026).

In Stage 1, the objective is latent representation alignment across single-mode tasks. The training data comprise LibriTTS, 960 hours of speech, and approximately 3,700 hours English singing extracted from the Suno dataset plus GTSinger (Shi et al., 1 Jun 2026). The singing-to-speech ratio is set to 4:1 in training batches (Shi et al., 1 Jun 2026). Speech uses the default CosyVoice 2 formatting, while singing samples are prefixed with style instructions such as “Generate a song in the {style} style. <|endofprompt|>” followed by lyrics (Shi et al., 1 Jun 2026). The LLM is trained autoregressively on interleaved cent and semantic tokens, and the acoustic model is trained with flow matching to reconstruct spectrograms (Shi et al., 1 Jun 2026).

In Stage 2, the aligned model is fine-tuned for SCS using a balanced mixture of synthetic mixed-mode samples and reduced single-mode data (Shi et al., 1 Jun 2026). The data include 11,769 synthetic SCS samples, 262 hours, plus approximately 200 hours of speech from LibriTTS and approximately 200 hours of singing from Suno with 10 hours from GTSinger (Shi et al., 1 Jun 2026). The mixture ratio is 1:1:1 for SCS, speech, and singing (Shi et al., 1 Jun 2026). SCS samples are formatted as first-person monologues, podcasts, or audiobook-like scripts with scenario instructions such as “Generate a monologue. <|endofprompt|>”, but without explicit segment-level speech/sing tags (Shi et al., 1 Jun 2026).

The paper reports that removing curriculum learning and training directly on the joint mixture substantially degrades SCS performance: the ablation labeled “w/o CL” reduces SCSBench-Mixed F1 from 0.716 to 0.496 (Shi et al., 1 Jun 2026). This indicates that the alignment-first, switching-second training order is not merely convenient but functionally important for acquiring semantically controlled mixed-mode generation.

Optimization uses AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay 0.1, and gradient clipping 1.0 (Shi et al., 1 Jun 2026). Stage 1 uses a learning rate linearly decayed from $2\times10^{-4}$ to 0 over 70k steps with 5k warmup steps, while Stage 2 uses a constant learning rate of $1\times10^{-4}$ for 30k steps (Shi et al., 1 Jun 2026). Training uses dynamic batching up to 4.5 minutes of audio per batch and was run on 4 × NVIDIA A800 GPUs; Stage 1 required approximately 5 days, Stage 2 approximately 1 day, and flow-matching fine-tuning approximately 2 days (Shi et al., 1 Jun 2026).

4. Refined cent tokens and prosody planning

A central technical contribution of UniVocal is the introduction of refined cent tokens and a Chain-of-Thought (CoT) generation mechanism for planning prosody before content generation (Shi et al., 1 Jun 2026). The motivation is that semantic tokenizers preserve high-level linguistic content but do not adequately encode fine acoustic detail such as subtle prosody and melody, whereas acoustic tokenizers tend to entangle content, prosody, and timbre (Shi et al., 1 Jun 2026).

Pitch is represented in the cent scale relative to A4, using

$$f_{cent} = 1200 \times \log_2\left(\frac{f_{Hz}}{440}\right).$$

It is then discretized into a single octave with 1200 possible bins, plus a special unvoiced token:

$$I(f_{cent}) = \begin{cases} \lceil f_{cent} \bmod 1200 \rceil & \text{if } f_{Hz} \neq 0 \ -1 & \text{if } f_{Hz} = 0 \end{cases}$$

yielding a total vocabulary of 1201 tokens (Shi et al., 1 Jun 2026). The paper states that this gives a quantization error of at most 1 cent and preserves local melodic contour while discarding absolute octave height (Shi et al., 1 Jun 2026).

The CoT mechanism is implemented by predicting the refined cent token $c_t$0 before the semantic token $c_t$1 at each frame (Shi et al., 1 Jun 2026). In effect, the model first produces a frame-level prosodic or melodic plan and then generates content consistent with that plan. The paper explicitly interprets this as a “plan-then-generate” mechanism adapted from Chain-of-Thought reasoning ideas (Shi et al., 1 Jun 2026).

The paper reports that predicted and realized cent trajectories have Spearman and Pearson correlations greater than 0.6, indicating that the cent token stream genuinely tracks output pitch (Shi et al., 1 Jun 2026). Ablations comparing 12, 480, and 1200 pitch bins show that the 1200-bin representation yields the best expressive speech and competitive singing performance (Shi et al., 1 Jun 2026).

The main trade-off is that CoT improves expressive quality more than it improves switching accuracy. Removing CoT slightly improves SCSBench mode-switching F1 but degrades empathetic speech and singing quality (Shi et al., 1 Jun 2026). On the textual empathy test, removing CoT reduces E-MOS from 2.26 to 2.03 and P-MOS from 2.22 to 1.84; on Fullsong, M-MOS falls from 2.18 to 1.86 (Shi et al., 1 Jun 2026). This suggests that CoT contributes primarily to prosodic richness and musicality rather than to the discrete decision boundary between speech and singing.

5. Synthetic SCS data pipeline and SCSBench

UniVocal is trained and evaluated on a synthetic SCS corpus constructed by a dedicated pipeline intended to preserve both semantic plausibility and acoustic naturalness (Shi et al., 1 Jun 2026). The pipeline has three main stages.

First, Gemini 2.5 Pro is prompted to generate first-person scripts in three scenarios: monologues, personal podcasts, and audiobooks (Shi et al., 1 Jun 2026). These scripts are speech-dominant but contain naturally inserted singing or humming segments (Shi et al., 1 Jun 2026). The paper distinguishes implicit cues, where lyrical or repetitive language suggests singing, from explicit cues, where transitional phrases such as “And that reminds me of a lyric…” or “Let me try to recall that jingle for you…” are inserted immediately before a singing segment (Shi et al., 1 Jun 2026). Approximately 50% of samples contain such explicit cues (Shi et al., 1 Jun 2026).

Second, the Stage-1 model is used to synthesize the audio (Shi et al., 1 Jun 2026). Scripts are segmented into speech versus singing or humming parts, but all segments within a sample share the same speaker embedding (Shi et al., 1 Jun 2026). Speech segments are conditioned on emotion-specific reference audio drawn from Expresso and EmoVoice-DB, with emotions including confused, happy, sad, angry, surprised, fearful, disgusted, default, and laughing (Shi et al., 1 Jun 2026). Singing segments are conditioned only on the target speaker embedding (Shi et al., 1 Jun 2026). Segments are concatenated with 0.25 s silence inserted at boundaries (Shi et al., 1 Jun 2026).

Third, the samples are filtered for quality using Whisper v3 (Shi et al., 1 Jun 2026). Samples with WER ≥ 20% are discarded, while samples with WER between 10% and 20% are retained but their text is replaced by ASR transcripts (Shi et al., 1 Jun 2026). Approximately 15% of initial samples are removed by this filtering stage (Shi et al., 1 Jun 2026).

The resulting dataset contains 11,769 samples and 262 hours of audio (Shi et al., 1 Jun 2026). A held-out subset of approximately 1,210 samples forms SCSBench, divided into SCSBench-Implicit, SCSBench-Explicit, and SCSBench-Mixed, balanced across cue type and scenario (Shi et al., 1 Jun 2026). The remainder is used for Stage-2 training (Shi et al., 1 Jun 2026).

A plausible implication is that UniVocal’s performance depends not only on architectural unification but also on the availability of training data in which semantics, speaker consistency, and cross-mode continuity are already aligned. The article’s own ablations and real-world tests are consistent with that interpretation (Shi et al., 1 Jun 2026).

6. Empirical performance, related systems, and limitations

UniVocal is evaluated against cascaded baselines on SCSBench and against strong systems on standard speech and singing tasks (Shi et al., 1 Jun 2026). The main SCS baselines are Gemini + Bark and Gemini + CosyVoice 2 + LeVo (Shi et al., 1 Jun 2026). In the first, Gemini segments the text and Bark synthesizes segments; in the second, Gemini segments the text, CosyVoice 2 generates speech, and LeVo generates singing (Shi et al., 1 Jun 2026).

On SCSBench-Mixed, UniVocal reaches F1(O) = 0.871 and F1(S) = 0.810, compared with 0.607 / 0.566 for Gemini+CosyVoice 2+LeVo and 0.465 / 0.199 for Gemini+Bark (Shi et al., 1 Jun 2026). The paper also states that UniVocal achieves the lowest WER and highest UTMOS across SCSBench subsets, while its speaker similarity is slightly lower than Gemini+CosyVoice 2+LeVo but its intra-sample speaker consistency is substantially better (Shi et al., 1 Jun 2026). The latter follows from using a single unified generator rather than stitching together outputs from separate speech and singing systems (Shi et al., 1 Jun 2026).

On SeedTTS-EN, UniVocal is compared with F5-TTS, CosyVoice 2, and Vevo 1.5 (Shi et al., 1 Jun 2026). It achieves WER 2.69, SIM 0.703, and UTMOS 4.21, with the best UTMOS among the systems listed (Shi et al., 1 Jun 2026). On a dedicated textual empathy test, UniVocal reaches E-MOS 2.26, P-MOS 2.22, and WER 0.32, compared with 1.78 / 1.74 for CosyVoice 2 and 2.30 / 2.47 for ElevenLabs multilingual-v2 (Shi et al., 1 Jun 2026).

On singing benchmarks, UniVocal is evaluated on GTSinger and Fullsong (Shi et al., 1 Jun 2026). On GTSinger it obtains WER 18.07 and QUA 10.70, both reported as best among the compared systems, with SIM 0.703 (Shi et al., 1 Jun 2026). On Fullsong it achieves WER 35.88, SIM 0.72, QUA 7.75, M-MOS 2.18, and N-MOS 2.23, outperforming or matching systems such as Vevo 1.5, YuE, and LeVo on the listed metrics (Shi et al., 1 Jun 2026).

The paper nevertheless identifies several limitations. The singing corpus is partly derived from Suno-generated material and processed through source separation and ASR, which introduces artifacts such as “electric tone” residuals, lyric misalignment, and noisy style tags (Shi et al., 1 Jun 2026). SCS training data are synthetic and centered on idealized monologue, podcast, and audiobook settings, producing a domain gap relative to spontaneous real-world SCS (Shi et al., 1 Jun 2026). On approximately 30 minutes of real human SCS recordings, UniVocal achieves F1 ≈ 0.201 without explicit cues, but this rises to 0.730 when a single explicit semantic cue is added per sample (Shi et al., 1 Jun 2026). Purely implicit lyric-like text can still be realized as speech rather than singing, while humming is easier because it is textually distinct (Shi et al., 1 Jun 2026).

These results place UniVocal among unified speech-singing systems but differentiate it by the absence of segment-level switching tags. The paper explicitly contrasts it with Bark, which can mix speech and singing but requires explicit control tags and exhibits unstable transitions, and with UniSyn, UniAudio, and Vevo2, which can generate one mode at a time but not semantically inferred intra-utterance switching (Shi et al., 1 Jun 2026).

7. Significance, reproducibility, and outlook

UniVocal’s principal significance lies in combining three capabilities in one framework: competitive text-to-speech, competitive singing generation, and semantically driven speech-singing switching without explicit segment annotations (Shi et al., 1 Jun 2026). The system unifies these behaviors in a single language-model-plus-acoustic-decoder pipeline, rather than delegating them to separate specialized generators (Shi et al., 1 Jun 2026).

The released resources include code, dataset assets, and pretrained checkpoints at the project repository, and the paper states that code and dataset are released (Shi et al., 1 Jun 2026). The implementation includes the integration of refined cent tokens and CoT generation into a CosyVoice-like pipeline, along with scripts for the SCS data synthesis process (Shi et al., 1 Jun 2026). The paper also notes a restrictive license intended to prohibit commercial misuse and illegal impersonation (Shi et al., 1 Jun 2026).

Several future directions are suggested by the reported results. Better singing data could reduce artifacts inherited from synthetic training material; more realistic SCS corpora could reduce the gap between synthetic benchmarks and spontaneous mixed-mode vocal behavior; and finer-grained controllability could complement the present emphasis on implicit inference (Shi et al., 1 Jun 2026). The paper also points toward multilingual extensions and interactive applications such as assistants that can sing or hum within otherwise spoken responses (Shi et al., 1 Jun 2026).

In that sense, UniVocal marks a shift from single-mode generation and manually controlled switching toward a model in which speech and singing are treated as context-sensitive realizations of one vocal generation problem. Its central technical claim is not merely that speech and singing can coexist in one architecture, but that the decision boundary between them can be learned from text semantics and implemented through a shared token-planning and acoustic-rendering stack (Shi et al., 1 Jun 2026).