UniVocal: Unified Speech-Singing Code-Switching Synthesis

Abstract: We propose UniVocal, a unified framework that implicitly infers vocal modes from text context to pioneer Speech-Singing Code-Switching (SCS) Synthesis - a task where transitions are autonomously driven by textual semantics, akin to seamless human language blending. Unlike single-mode generation or systems relying on switching-control tags, our proposed UniVocal implicitly infers vocal modes solely from text context. To achieve this, we employ a data-efficient two-stage curriculum learning strategy that progressively trains a competitive TTS system to acquire the desired SCS capability. Addressing data scarcity, we introduce a scalable pipeline to synthesize diverse code-switching data that is both semantically and acoustically natural, alongside a new multi-scenario benchmark, SCSBench. To address limitations of semantic tokenizers in capturing acoustic details, we also introduce refined cent token and Chain-of-Thought (CoT) generation for planning prosody before content generation, effectively enhancing empathetic speech generation and singing melody. Experimental results demonstrate that UniVocal achieves state-of-the-art performance on SCSBench while maintaining competitive performance on regular speech and singing tasks. Audio samples are available at https://project-univocal-demo.github.io/demo/. The code and dataset are released at https://github.com/FunAudioLLM/FunResearch/tree/main/UniVocal.

• The paper introduces UniVocal, a unified model that synthesizes seamless speech-singing code-switching via autoregressive planning with refined cent tokens.
• It employs an innovative two-stage curriculum with scalable data augmentation to align speech and singing modalities for expressive vocal outputs.
• Experimental evaluations demonstrate significant gains in switching accuracy, naturalness, and speaker consistency over traditional cascaded baselines.

UniVocal: A Framework for Unified Speech–Singing Code-Switching Synthesis

Problem Definition and Motivation

Conventional audio generation approaches compartmentalize speech and singing synthesis into distinct tasks—TTS models exclusively generate speech while SVS and music generation models are restricted to song. However, natural human vocal behavior fluidly blends speech and singing within the same utterance, switching modes according to context. Existing unified frameworks either require explicit control signals (e.g., tag-based triggers) or produce only one mode per prompt, prohibiting seamless intra-sequence transitions. The task defined as Speech–Singing Code-Switching (SCS) Synthesis is the generation of vocal outputs that autonomously alternate between speech and singing, driven solely by textual semantics. This is previously unsolved by contemporary audio generation pipelines.

Figure 1: Common audio generation tasks, categorized into specialized (left) and unified (right) tasks. UniVocal is the first to support semantically-driven code-switching between speech and singing within a single sequence.

Methodology and Design Choices

Unified Model Architecture with Autoregressive Planning

UniVocal adapts the CosyVoice 2 architecture to support SCS, introducing critical architectural innovations:

• An instruction-driven paradigm, leveraging natural language prompts to globally define the generation task (monologue, podcast, audiobook) without segment-level switching tags.
• At each generation timestep, UniVocal autoregressively emits a refined cent token (high-resolution pitch representation) followed by a semantic token (linguistic–phonetic content). This "plan-then-generate" mechanism—reminiscent of Chain-of-Thought (CoT) prompting—enables explicit prosodic/melodic planning prior to content realization.

  Figure 2: UniVocal overview. Text and task instruction are processed by the autoregressive model, which outputs interleaved refined cent and semantic tokens. These are conditioned on prompt audio and synthesized into the final waveform output.

• The refined cent token adopts a 1200-bin discretization (one semitone = 100 cents, with tokens wrapping within one octave) for microtonal precision suitable for both expressive speech and fine-grained singing melody, with negligible quantization error.
• During generation, cent-token steps are logit-masked to prevent emission of non-cent tokens (and vice versa), enforcing strict alternation.

Scalable Data Augmentation Pipeline

Data scarcity for natural SCS is addressed by synthesizing a 262-hour code-switching corpus. Scripts are generated via LLMs (Gemini 2.5 Pro) using a spectrum of cue strategies:

• Explicit cues: Transitional phrases (e.g., "let me sing") inserted before singing segments.
• Implicit cues: Semantic contrasts—prose vs. lyric—without surface markers.

Each code-switching sample is synthesized via a stage-1 model (with speaker and emotion conditioning for acoustic consistency), concatenated, and filtered by ASR-based WER to enforce semantic and acoustic alignment.

Curriculum Learning for Mode Alignment and SCS Mastery

Model adaptation proceeds in two distinct training phases:

1. Latent Representation Alignment: Continued pretraining to unify speech and singing in a shared latent space via a 4:1 singing-to-speech data ratio.
2. SCS Specialization: Fine-tuning with balanced mixtures of regular speech, singing, and synthetic code-switching data to develop robust mode-switching capability while mitigating catastrophic forgetting.

Experimental Evaluation

Benchmarks and Evaluation Protocol

Three capability domains are targeted:

• SCS Synthesis: SCSBench—a held-out stratified set—evaluates accuracy of text-driven switching, using F1 scores for segment-wise mode labeling (by both an LLM and human raters).
• Expressive and Empathetic Speech: SeedTTS and a curated empathy test set quantify intelligibility, speaker similarity, naturalness (UTMOS), and subjective empathy/prosody (E-MOS/P-MOS).
• Singing Generation: Short-phrase (GTSinger) and long-form (Fullsong) datasets test melodic accuracy, musicality (M-MOS), and timbral/naturalness (N-MOS/QUA/AES).

Strong Numerical Results and Contradictory Claims

On SCSBench-Mixed, UniVocal achieves F1(O)=0.871 (LLM-based) and F1(S)=0.810 (human), outperforming cascaded baselines (e.g., Gemini+Bark, Gemini+Cosy2+LeVo, with best F1(S)=0.566), supporting the claim that explicit segment tagging and model cascades are insufficient for robust semantic-driven switching.

In single-mode tasks, UniVocal attains state-of-the-art UTMOS (4.21) on SeedTTS-EN, matching or exceeding commercial and open baselines in expressive and empathetic speech, and surpasses Vevo 1.5 in both musicality (M-MOS 2.18 vs. 2.08) and naturalness (N-MOS 2.23 vs. 2.17) on singing benchmarks.

Notably, while cascaded systems can achieve higher global speaker similarity, UniVocal preserves far superior intra-sample speaker consistency (minimizing timbre drift across code-switches), as evidenced by heatmap visualization:

Figure 3: Pairwise similarity heatmap across temporal segments in generated samples, demonstrating that UniVocal maintains consistent speaker identity throughout code-switching, outperforming cascaded baselines.

Ablation and Analytical Insights

Systematic ablations demonstrate:

• The CoT/refined cent token mechanism is essential for maximizing expressive generation (substantially increasing MOS/EMOS/PMOS), despite minor trade-offs in segmentation F1.
• The two-stage curriculum is necessary for latent alignment; joint single-stage training (w/o curriculum) drastically reduces switching accuracy (F1 drops from 0.810 to 0.496).
• Combining emotionally diverse singing data with prosodic planning "unlocks" latent empathy in synthetic speech, without dependence on explicit instructions or handcrafted cues.
• Explicit cues in the text (trigger phrases) are strongly preferred for reliable mode switching. UniVocal’s performance degrades in purely implicit-cue scenarios, highlighting current limitations in fully semantic-derived switching.

Implications and Future Directions

UniVocal constitutes the first demonstration of text-driven, semantically-anchored intra-utterance switching between speech and singing in neural audio generation. By modeling prosody/mechanical structure explicitly and leveraging model-agnostic curriculum learning, it fills a critical gap in the capabilities of both discrete token-based and LLM-driven speech/music models.

Practical applications immediately arise in dubbing, audiobooks, conversational agents, and expressive TTS systems where mixed vocalization is natural. The approach of interleaved structural planning with high-resolution discrete tokens may generalize to other multi-modal, multi-style autoregressive architectures (e.g., speech–rap, hybrid musical improvisation, or dialog–song blends).

Several limitations persist:

• Reliance on LLM-generated and synthetic SCS data constrains real-world coverage. UniVocal currently generalizes best when small explicit semantic anchors are present. Handling subtle, fully implicit switching points remains unresolved.
• Singing data quality, dominated by synthetic and source-separated tracks, bounds the upper limits of timbre realism and lyric–melody alignment.
• Mode-switch evaluation at fine-grained temporal resolution remains challenging; current F1 assessments may lack precision in short or ambiguous segments.

Conclusion

UniVocal advances neural audio synthesis with a unified model handling speech, singing, and their semantically-driven interleaving. Key contributions include the integration of a refined cent token for explicit prosodic planning, a scalable data synthesis pipeline for SCS, and a curriculum-based training protocol. UniVocal achieves the highest reported SCSBench scores and highly competitive metrics across single-mode tasks, offering a new paradigm for unified, expressive, human-like vocal synthesis and laying groundwork for future multi-modal, multi-style generative systems.