---
title: 'SCSBench: Speech-Singing Code-Switching Benchmark'
url: https://www.emergentmind.com/topics/scsbench
type: topic
---

# SCSBench: Speech-Singing Code-Switching Benchmark

Searching arXiv for the cited SCSBench/UniVocal paper to ground the article and ensure up-to-date citation.
SCSBench is a benchmark for **Speech-Singing Code-Switching (SCS) Synthesis**, introduced with UniVocal as a multi-scenario evaluation suite for systems that generate a single vocal stream in which spoken and sung or hummed segments are automatically interleaved, with transitions determined solely by textual semantics rather than explicit control tags [2606.01677]. It is designed to test whether a model can recognize cues in text that signal when to remain in speech, when to switch into singing or humming, and how to time those speech↔sing transitions correctly. In the UniVocal study, SCSBench serves as the held-out test benchmark for semantic-driven mode switching, while accompanying objective and subjective metrics quantify intelligibility, naturalness, speaker consistency, and mode-transition fidelity [2606.01677].

## 1. Task formulation and benchmark scope

SCSBench is grounded in the task definition of **Speech-Singing Code-Switching Synthesis**: generating one continuous vocal output in which speech and singing or humming are interleaved without explicit markup such as `<sing>` or `<speech>` tags [2606.01677]. The system must infer the appropriate vocal mode from the text alone. In this setup, prose-like language cues speech, while lyrical forms or non-lexical patterns such as “hmm-hmm” cue singing or humming.

The benchmark is stratified into three everyday narrative scenarios: **monologue**, **personal podcast**, and **audiobook** [2606.01677]. These scenarios are intended to cover settings in which transitions between narration and song can plausibly occur within the same utterance. The benchmark further organizes evaluation by three cue regimes. In the **implicit** regime, transitions are driven only by textual style shifts. In the **explicit** regime, sung regions are preceded by trigger phrases such as “That reminds me of a tune…”. In the **mixed** regime, both implicit and explicit cues are present [2606.01677].

This structure makes mode-switch recognition central to benchmark performance. A potential misconception is that SCSBench is primarily a benchmark for generic TTS or singing synthesis quality. The benchmark does measure intelligibility, speaker similarity, and naturalness, but its defining objective is whether a system can correctly detect and realize semantically motivated transitions between speech and singing [2606.01677].

## 2. Corpus composition, held-out split, and annotations

SCSBench is derived from an English synthetic code-switching corpus generated for the UniVocal study [2606.01677]. The text scripts are automatically generated by **Gemini 2.5 Pro** under constrained prompting so that speech and singing styles remain clearly differentiated. Audio is synthesized by UniVocal’s stage-1 model, described as an adapted **CosyVoice 2**, using **9 speaker embeddings** and, for speech segments, **9 emotion-conditioned reference audios** [2606.01677].

The full synthetic code-switching dataset used for training contains **11,769 utterances**, **261.9 hours** in total, and an **average duration of 80.1 s** [2606.01677]. Its scenario breakdown is: **monologue** with **6,247** utterances and **84.3 h** total duration, **podcast** with **2,432** utterances and **87.2 h**, and **audiobook** with **3,090** utterances and **90.4 h** [2606.01677]. The corresponding average durations are **48.6 s** for monologue, **129.1 s** for podcast, and **105.3 s** for audiobook.

The held-out **SCSBench** test set contains approximately **1,210 samples**, corresponding to roughly **10% of the full data** [2606.01677]. It is balanced across the three cue-type subsets—**Implicit**, **Explicit**, and **Mixed**—with each subset containing about **403 samples**. Within each cue subset, samples are equally divided among monologue, podcast, and audiobook, at approximately **135 samples per scenario** [2606.01677].

Annotation and transcription follow two parallel processes. First, **Whisper-v3** is used for ASR-based quality filtering. Samples with **WER ≥ 20%** are discarded; samples with **10–20% WER** are retained using the ASR transcript for forced alignment; and samples below that threshold retain the original text [2606.01677]. Second, human annotators label each audio segment as **speech** or **singing**, with **humming counted as singing** for the purposes of mode-switching evaluation [2606.01677].

## 3. Data synthesis pipeline

The benchmark data are produced through a three-stage synthesis pipeline [2606.01677]. The first stage is **semantic text generation**. Gemini 2.5 Pro is used to generate scripts for each scenario, with text constrained into two modes: natural prose narrative for speech, and lyrical lines or “hmm-hmm” sequences for singing or humming. In **50% of scripts**, explicit trigger phrases are prepended before singing segments [2606.01677].

The second stage is **unified acoustic synthesis**. Each script is split into speech and singing or humming segments. Speech segments are synthesized with the stage-1 UniVocal model conditioned on both a speaker embedding and an emotion reference audio. Singing segments are synthesized using only the speaker embedding. The resulting segments are concatenated with a fixed **0.25 s silence** between them [2606.01677].

The third stage is **quality control**. The paper specifies a deterministic filtering procedure driven by ASR-derived WER. If a generated sample has **WER ≥ 0.20**, it is discarded. If **0.10 ≤ WER < 0.20**, it is retained but paired with the ASR transcript. Otherwise it is retained with the original text [2606.01677]. This pipeline couples semantic control, acoustic rendering, and post hoc filtering within a single benchmark construction process. A plausible implication is that SCSBench emphasizes robustness not only to vocal-mode generation but also to script-level ambiguity and synthesis noise introduced by long-form, mixed-mode utterances.

## 4. Evaluation protocol and metrics

SCSBench uses both objective and subjective evaluation [2606.01677]. The primary objective metric for transition fidelity is **mode-switching accuracy**, reported as **F1(O)** and computed by an **ICL-calibrated Gemini 2.5 Pro evaluator**. The component definitions are standard:
\[
\mathrm{Precision} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}, \qquad
\mathrm{Recall} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}},
\]
\[
\mathrm{F1} = \frac{2 \cdot \mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.
\]

Additional objective metrics quantify auxiliary properties of the generated audio. **Word Error Rate (WER)** is defined as
\[
\mathrm{WER} = \frac{S + D + I}{N_{\mathrm{ref}}},
\]
where \(S\), \(D\), and \(I\) denote substitution, deletion, and insertion counts [2606.01677]. **Speaker similarity (SIM)** is measured as cosine similarity between embeddings extracted by **ERes2Net**, and **UTMOS** is used as a predicted MOS naturalness score from **ClearerVoice** on a **1–5** scale [2606.01677].

Subjective evaluation is reported as **F1(S)**, a human-labeled mode-switching score. Each sample is evaluated by **at least 3 raters**, and **three annotators per sample** classify each segment as speech or singing; the average F1 is then computed across samples [2606.01677]. Inter-rater agreement is reported as **Fleiss’ \(\kappa = 0.684\)**.

The benchmark also specifies summary definitions for macro- and micro-aggregation:
\[
\mathrm{F1}_{\mathrm{macro}} = \frac{1}{M}\sum_{i=1}^{M} \mathrm{F1}_i,
\]
\[
\mathrm{F1}_{\mathrm{micro}} =
\frac{2\sum_i \mathrm{TP}_i}
{2\sum_i \mathrm{TP}_i + \sum_i \mathrm{FP}_i + \sum_i \mathrm{FN}_i}.
\]
These definitions place mode-switch recognition at the center of evaluation while retaining standard speech-synthesis diagnostics as secondary dimensions [2606.01677].

## 5. Baseline systems and quantitative results

The UniVocal study reports baseline results on SCSBench for **Gemini + Bark**, **Gemini + Cosy2 + LeVo**, and **UniVocal** [2606.01677]. For mode-switching performance on the **Implicit** subset, Gemini + Bark attains **F1(O) = 0.414** and **F1(S) = 0.142**, Gemini + Cosy2 + LeVo attains **0.752** and **0.685**, and UniVocal attains **0.626** and **0.595** [2606.01677]. On the **Explicit** subset, the corresponding values are **0.533/0.250**, **0.572/0.489**, and **0.714/0.635**. On the **Mixed** subset, they are **0.465/0.199**, **0.607/0.566**, and **0.871/0.810** [2606.01677].

These results show that performance depends materially on cue type. UniVocal is strongest on the **Mixed** subset, where it substantially exceeds the cascaded baselines in both automatic and human mode-switch F1. The data also show that the best-performing system varies by subset: the **Implicit** subset favors Gemini + Cosy2 + LeVo in F1, whereas the **Explicit** and especially **Mixed** subsets favor UniVocal [2606.01677]. This suggests that the benchmark distinguishes between recognizing stylistic cues alone and handling combined stylistic and explicitly signaled transitions.

For speech-quality metrics, the paper reports **WER**, **SIM**, and **UTMOS** for the **Implicit** and **Mixed** subsets [2606.01677]. On the Implicit subset, Gemini + Bark records **WER 21.83%** and **UTMOS 3.41**, Gemini + Cosy2 + LeVo records **WER 17.97%**, **SIM 0.758**, and **UTMOS 3.42**, and UniVocal records **WER 5.83%**, **SIM 0.650**, and **UTMOS 4.36**. On the Mixed subset, the corresponding values are **WER 29.60%** and **UTMOS 3.31** for Gemini + Bark, **WER 12.43%**, **SIM 0.773**, and **UTMOS 3.54** for Gemini + Cosy2 + LeVo, and **WER 10.90%**, **SIM 0.652**, and **UTMOS 4.36** for UniVocal [2606.01677].

The paper’s summary states that UniVocal yields the **lowest WER** and **highest naturalness (UTMOS) on all subsets**, while its global SIM is slightly lower; at the same time, its unified architecture maintains **far better intra-sample speaker consistency across mode switches** [2606.01677]. That distinction is important because SCSBench is sensitive not only to aggregate speaker embedding similarity but also to within-utterance coherence across mode changes.

## 6. Methodological significance and interpretive considerations

SCSBench occupies a specific niche in speech and singing synthesis evaluation: it measures **semantic-driven speech↔singing switching** under scenario variation and cue variation, rather than evaluating speech synthesis and singing synthesis as isolated tasks [2606.01677]. The benchmark’s three-scenario design and three cue subsets provide a controlled way to probe whether a model can respond to prose-versus-lyric structure, explicit trigger phrases, or both. This suggests a decomposition of the problem into at least two sub-capabilities: recognizing switch cues from text and rendering stable transitions acoustically.

A second methodological feature is the combination of **synthetic benchmark construction**, **objective LLM-based evaluation**, and **human segment labeling** [2606.01677]. The synthetic pipeline enables scale and balancing across scenarios and cue regimes. The ICL-calibrated evaluator supports automated mode-switch scoring, while human F1(S) and Fleiss’ \(\kappa\) provide a check on subjective validity. A plausible implication is that SCSBench is intended not merely as a demonstration set for UniVocal, but as a reusable testbed for future systems that claim implicit control over mixed speech-and-singing generation.

Finally, SCSBench should not be interpreted as a benchmark for explicit-control switching systems alone. Its defining constraint is the absence of switching-control tags in the task formulation [2606.01677]. Systems that rely on explicit markup may still be evaluated if adapted, but the benchmark’s central research question is whether mode transitions can be inferred from textual semantics without external control symbols. In that sense, SCSBench formalizes a benchmark target that is narrower than generic expressive TTS and broader than conventional single-mode speech or singing synthesis.

Source: https://www.emergentmind.com/topics/scsbench