SCSBench: Speech-Singing Code-Switching Benchmark

• SCSBench is a benchmark for Speech-Singing Code-Switching Synthesis, evaluating models' ability to interleave spoken and sung segments based solely on textual semantics.
• It features a multi-scenario evaluation suite across monologue, podcast, and audiobook settings, with implicit, explicit, and mixed cue regimes.
• The benchmark employs both objective metrics (e.g., mode-switch F1, WER, UTMOS) and subjective human evaluations to assess synthesis quality and intra-utterance consistency.

Searching arXiv for the cited SCSBench/UniVocal paper to ground the article and ensure up-to-date citation. SCSBench is a benchmark for Speech-Singing Code-Switching (SCS) Synthesis, introduced with UniVocal as a multi-scenario evaluation suite for systems that generate a single vocal stream in which spoken and sung or hummed segments are automatically interleaved, with transitions determined solely by textual semantics rather than explicit control tags (Shi et al., 1 Jun 2026). It is designed to test whether a model can recognize cues in text that signal when to remain in speech, when to switch into singing or humming, and how to time those speech↔sing transitions correctly. In the UniVocal study, SCSBench serves as the held-out test benchmark for semantic-driven mode switching, while accompanying objective and subjective metrics quantify intelligibility, naturalness, speaker consistency, and mode-transition fidelity (Shi et al., 1 Jun 2026).

1. Task formulation and benchmark scope

SCSBench is grounded in the task definition of Speech-Singing Code-Switching Synthesis: generating one continuous vocal output in which speech and singing or humming are interleaved without explicit markup such as <sing> or <speech> tags (Shi et al., 1 Jun 2026). The system must infer the appropriate vocal mode from the text alone. In this setup, prose-like language cues speech, while lyrical forms or non-lexical patterns such as “hmm-hmm” cue singing or humming.

The benchmark is stratified into three everyday narrative scenarios: monologue, personal podcast, and audiobook (Shi et al., 1 Jun 2026). These scenarios are intended to cover settings in which transitions between narration and song can plausibly occur within the same utterance. The benchmark further organizes evaluation by three cue regimes. In the implicit regime, transitions are driven only by textual style shifts. In the explicit regime, sung regions are preceded by trigger phrases such as “That reminds me of a tune…”. In the mixed regime, both implicit and explicit cues are present (Shi et al., 1 Jun 2026).

This structure makes mode-switch recognition central to benchmark performance. A potential misconception is that SCSBench is primarily a benchmark for generic TTS or singing synthesis quality. The benchmark does measure intelligibility, speaker similarity, and naturalness, but its defining objective is whether a system can correctly detect and realize semantically motivated transitions between speech and singing (Shi et al., 1 Jun 2026).

2. Corpus composition, held-out split, and annotations

SCSBench is derived from an English synthetic code-switching corpus generated for the UniVocal study (Shi et al., 1 Jun 2026). The text scripts are automatically generated by Gemini 2.5 Pro under constrained prompting so that speech and singing styles remain clearly differentiated. Audio is synthesized by UniVocal’s stage-1 model, described as an adapted CosyVoice 2, using 9 speaker embeddings and, for speech segments, 9 emotion-conditioned reference audios (Shi et al., 1 Jun 2026).

The full synthetic code-switching dataset used for training contains 11,769 utterances, 261.9 hours in total, and an average duration of 80.1 s (Shi et al., 1 Jun 2026). Its scenario breakdown is: monologue with 6,247 utterances and 84.3 h total duration, podcast with 2,432 utterances and 87.2 h, and audiobook with 3,090 utterances and 90.4 h (Shi et al., 1 Jun 2026). The corresponding average durations are 48.6 s for monologue, 129.1 s for podcast, and 105.3 s for audiobook.

The held-out SCSBench test set contains approximately 1,210 samples, corresponding to roughly 10% of the full data (Shi et al., 1 Jun 2026). It is balanced across the three cue-type subsets—Implicit, Explicit, and Mixed—with each subset containing about 403 samples. Within each cue subset, samples are equally divided among monologue, podcast, and audiobook, at approximately 135 samples per scenario (Shi et al., 1 Jun 2026).

Annotation and transcription follow two parallel processes. First, Whisper-v3 is used for ASR-based quality filtering. Samples with WER ≥ 20% are discarded; samples with 10–20% WER are retained using the ASR transcript for forced alignment; and samples below that threshold retain the original text (Shi et al., 1 Jun 2026). Second, human annotators label each audio segment as speech or singing, with humming counted as singing for the purposes of mode-switching evaluation (Shi et al., 1 Jun 2026).

3. Data synthesis pipeline

The benchmark data are produced through a three-stage synthesis pipeline (Shi et al., 1 Jun 2026). The first stage is semantic text generation. Gemini 2.5 Pro is used to generate scripts for each scenario, with text constrained into two modes: natural prose narrative for speech, and lyrical lines or “hmm-hmm” sequences for singing or humming. In 50% of scripts, explicit trigger phrases are prepended before singing segments (Shi et al., 1 Jun 2026).

The second stage is unified acoustic synthesis. Each script is split into speech and singing or humming segments. Speech segments are synthesized with the stage-1 UniVocal model conditioned on both a speaker embedding and an emotion reference audio. Singing segments are synthesized using only the speaker embedding. The resulting segments are concatenated with a fixed 0.25 s silence between them (Shi et al., 1 Jun 2026).

The third stage is quality control. The paper specifies a deterministic filtering procedure driven by ASR-derived WER. If a generated sample has WER ≥ 0.20, it is discarded. If 0.10 ≤ WER < 0.20, it is retained but paired with the ASR transcript. Otherwise it is retained with the original text (Shi et al., 1 Jun 2026). This pipeline couples semantic control, acoustic rendering, and post hoc filtering within a single benchmark construction process. A plausible implication is that SCSBench emphasizes robustness not only to vocal-mode generation but also to script-level ambiguity and synthesis noise introduced by long-form, mixed-mode utterances.

4. Evaluation protocol and metrics

SCSBench uses both objective and subjective evaluation (Shi et al., 1 Jun 2026). The primary objective metric for transition fidelity is mode-switching accuracy, reported as F1(O) and computed by an ICL-calibrated Gemini 2.5 Pro evaluator. The component definitions are standard: $$\mathrm{Precision} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}, \qquad \mathrm{Recall} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}},$$

$$\mathrm{F1} = \frac{2 \cdot \mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.$$

Additional objective metrics quantify auxiliary properties of the generated audio. Word Error Rate (WER) is defined as

$\mathrm{WER} = \frac{S + D + I}{N_{\mathrm{ref}}},$

where $S$, $D$, and $I$ denote substitution, deletion, and insertion counts (Shi et al., 1 Jun 2026). Speaker similarity (SIM) is measured as cosine similarity between embeddings extracted by ERes2Net, and UTMOS is used as a predicted MOS naturalness score from ClearerVoice on a 1–5 scale (Shi et al., 1 Jun 2026).

Subjective evaluation is reported as F1(S), a human-labeled mode-switching score. Each sample is evaluated by at least 3 raters, and three annotators per sample classify each segment as speech or singing; the average F1 is then computed across samples (Shi et al., 1 Jun 2026). Inter-rater agreement is reported as Fleiss’ $\kappa = 0.684$.

The benchmark also specifies summary definitions for macro- and micro-aggregation: $$\mathrm{F1}_{\mathrm{macro}} = \frac{1}{M}\sum_{i=1}^{M} \mathrm{F1}_i,$$

$$\mathrm{F1}_{\mathrm{micro}} = \frac{2\sum_i \mathrm{TP}_i} {2\sum_i \mathrm{TP}_i + \sum_i \mathrm{FP}_i + \sum_i \mathrm{FN}_i}.$$

These definitions place mode-switch recognition at the center of evaluation while retaining standard speech-synthesis diagnostics as secondary dimensions (Shi et al., 1 Jun 2026).

5. Baseline systems and quantitative results

The UniVocal study reports baseline results on SCSBench for Gemini + Bark, Gemini + Cosy2 + LeVo, and UniVocal (Shi et al., 1 Jun 2026). For mode-switching performance on the Implicit subset, Gemini + Bark attains F1(O) = 0.414 and F1(S) = 0.142, Gemini + Cosy2 + LeVo attains 0.752 and 0.685, and UniVocal attains 0.626 and 0.595 (Shi et al., 1 Jun 2026). On the Explicit subset, the corresponding values are 0.533/0.250, 0.572/0.489, and 0.714/0.635. On the Mixed subset, they are 0.465/0.199, 0.607/0.566, and 0.871/0.810 (Shi et al., 1 Jun 2026).

These results show that performance depends materially on cue type. UniVocal is strongest on the Mixed subset, where it substantially exceeds the cascaded baselines in both automatic and human mode-switch F1. The data also show that the best-performing system varies by subset: the Implicit subset favors Gemini + Cosy2 + LeVo in F1, whereas the Explicit and especially Mixed subsets favor UniVocal (Shi et al., 1 Jun 2026). This suggests that the benchmark distinguishes between recognizing stylistic cues alone and handling combined stylistic and explicitly signaled transitions.

For speech-quality metrics, the paper reports WER, SIM, and UTMOS for the Implicit and Mixed subsets (Shi et al., 1 Jun 2026). On the Implicit subset, Gemini + Bark records WER 21.83% and UTMOS 3.41, Gemini + Cosy2 + LeVo records WER 17.97%, SIM 0.758, and UTMOS 3.42, and UniVocal records WER 5.83%, SIM 0.650, and UTMOS 4.36. On the Mixed subset, the corresponding values are WER 29.60% and UTMOS 3.31 for Gemini + Bark, WER 12.43%, SIM 0.773, and UTMOS 3.54 for Gemini + Cosy2 + LeVo, and WER 10.90%, SIM 0.652, and UTMOS 4.36 for UniVocal (Shi et al., 1 Jun 2026).

The paper’s summary states that UniVocal yields the lowest WER and highest naturalness (UTMOS) on all subsets, while its global SIM is slightly lower; at the same time, its unified architecture maintains far better intra-sample speaker consistency across mode switches (Shi et al., 1 Jun 2026). That distinction is important because SCSBench is sensitive not only to aggregate speaker embedding similarity but also to within-utterance coherence across mode changes.

6. Methodological significance and interpretive considerations

SCSBench occupies a specific niche in speech and singing synthesis evaluation: it measures semantic-driven speech↔singing switching under scenario variation and cue variation, rather than evaluating speech synthesis and singing synthesis as isolated tasks (Shi et al., 1 Jun 2026). The benchmark’s three-scenario design and three cue subsets provide a controlled way to probe whether a model can respond to prose-versus-lyric structure, explicit trigger phrases, or both. This suggests a decomposition of the problem into at least two sub-capabilities: recognizing switch cues from text and rendering stable transitions acoustically.

A second methodological feature is the combination of synthetic benchmark construction, objective LLM-based evaluation, and human segment labeling (Shi et al., 1 Jun 2026). The synthetic pipeline enables scale and balancing across scenarios and cue regimes. The ICL-calibrated evaluator supports automated mode-switch scoring, while human F1(S) and Fleiss’ $\kappa$ provide a check on subjective validity. A plausible implication is that SCSBench is intended not merely as a demonstration set for UniVocal, but as a reusable testbed for future systems that claim implicit control over mixed speech-and-singing generation.

Finally, SCSBench should not be interpreted as a benchmark for explicit-control switching systems alone. Its defining constraint is the absence of switching-control tags in the task formulation (Shi et al., 1 Jun 2026). Systems that rely on explicit markup may still be evaluated if adapted, but the benchmark’s central research question is whether mode transitions can be inferred from textual semantics without external control symbols. In that sense, SCSBench formalizes a benchmark target that is narrower than generic expressive TTS and broader than conventional single-mode speech or singing synthesis.