Code-Switching FLEURS: Multilingual Speech Benchmarks

• CS-FLEURS is a collection of multilingual code-switched speech benchmarks derived from FLEURS, offering diverse datasets for ASR, SLI, and S2TT research.
• It features multiple variants created using distinct generation paradigms, including controlled substitution with UniCoM and LLM-based prompting with alignment strategies.
• The benchmarks deliver practical insights with detailed metrics such as CER, MOS, CMI, and specialized evaluation protocols to guide multilingual experimentation.

Code-Switching FLEURS (CS-FLEURS) designates code-switched speech resources derived from FLEURS and used for multilingual automatic speech recognition (ASR), spoken language identification, and speech-to-text translation (S2TT). In recent literature, the name has been applied to multiple related but non-identical resources: a 1,600-hour, 253-language-pair intra-sentential code-switching corpus constructed with UniCoM; a massively multilingual dataset with one training subset and four test subsets covering 113 unique code-switched language pairs across 52 languages; and a publicly released Mandarin–English code-switching ASR benchmark with CS-FLEURS-XTTS and CS-FLEURS-READ splits (Lee et al., 21 Aug 2025, Yan et al., 17 Sep 2025, Nguyen et al., 5 Jun 2026). The term is therefore context-dependent: all variants inherit the FLEURS premise of parallel, multilingual speech, but they differ in generation pipeline, scale, validation regime, and intended downstream task.

1. Terminological scope and dataset variants

The most important point for precise usage is that CS-FLEURS is not a single canonical release in the current literature. One line of work, associated with Yonsei University’s UniCoM pipeline, presents CS-FLEURS as a 1,600-hour corpus spanning 23 in-domain languages and 253 distinct ordered pairs, with 654,700 utterances and 73 “n-way” parallel sentences across all 23 languages for future CS-S2TT research (Lee et al., 21 Aug 2025). A second line of work presents CS-FLEURS as a massively multilingual benchmark organized into one training subset and four test subsets, all built from the same 2,009 FLEURS utterances recast as code-switched sentences under a Matrix-Language Frame model; this version reports 113 unique code-switched pairs across 52 languages and provides a 128-hour generative training set across 16 X-English language pairs (Yan et al., 17 Sep 2025). A third usage, in downstream ASR work, refers specifically to a Mandarin–English benchmark comprising CS-FLEURS-XTTS for training and CS-FLEURS-READ for testing, with English “embedded” spans occurring within a Mandarin matrix (Nguyen et al., 5 Jun 2026).

Variant	Scope	Key properties
UniCoM CS-FLEURS	253 ordered pairs, 1,600 h	Intra-sentential corpus generated by SWORDS and style unification
Massively multilingual CS-FLEURS	113 unique pairs across 52 languages	One training subset and four test subsets, mixing human-read and synthetic speech
cmn-eng CS-FLEURS benchmark	Mandarin–English	CS-FLEURS-XTTS training and CS-FLEURS-READ test splits

Within all variants, “matrix language” denotes the base language of the utterance and “embedded language” the language inserted into that structure. In the Mandarin–English benchmark, the English material is explicitly described as the embedded span within a Mandarin matrix (Nguyen et al., 5 Jun 2026). In the multilingual benchmark, X-English pair design indicates a non-English matrix language with English as embedded language in the X-English subsets (Yan et al., 17 Sep 2025).

2. Generation paradigms

The UniCoM version of CS-FLEURS is generated by a three-stage pipeline: preprocessing, intra-sentential source-mixing via SWORDS, and style unification (Lee et al., 21 Aug 2025). Preprocessing applies a band-pass filter that removes frequencies below 80 Hz and above 7 kHz, followed by RMS amplitude normalization. SWORDS then replaces selected words from a matrix-language utterance with semantically equivalent words from a parallel utterance in the embedded language while preserving original syntax and meaning. The permissible parts of speech are restricted to $$P=\{\text{NOUN}, \text{VERB}, \text{INTERJECTION}\}$$, and substitutions are limited to at most $K=3$ words per sentence. In the stated formalization,

$$SWORDS(w;L_{mat},L_{emb}) = \begin{cases} \tau(w) & \text{if } POS(w)\in P \text{ and } w \text{ selected for substitution} \ w & \text{otherwise} \end{cases}$$

where $\tau(w)$ is the translation of $w$ into the embedded language. Word pairs are mapped by POS using GPT-4o-mini, and forced alignment is performed with MMS-FA to clip the corresponding audio segments (Lee et al., 21 Aug 2025).

The massively multilingual dataset uses a different generation strategy for different subsets (Yan et al., 17 Sep 2025). For cs-fleurs-read, code-switched text is generated with GPT-4o under three prompting strategies: GPT-Base, GPT-EC, and GPT-Pred. GPT-EC adds “equivalence-constrained” English keywords, and GPT-Pred adds predicted embedding candidates. An in-context morphological example is provided for 10 of 14 languages to encourage intra-word mixing. Human readers rejected approximately 10% of generations, and fluency was rated on a 0–2 scale. For cs-fleurs-xtts and cs-fleurs-mms, code-switched text is generated by an “align-then-swap” procedure: AwesomeAlign is used for X↔English word alignment; Stanza is used for POS tagging; and 30% of nouns, verbs, adjectives, and adverbs are randomly selected for swap with Poisson sampling $p=0.3$. For character-based languages such as Mandarin and Japanese, segmentation is also performed with Stanza (Yan et al., 17 Sep 2025).

These two paradigms represent distinct assumptions about realistic code-switch generation. The UniCoM pipeline enforces POS-specific substitution and speaker-style unification at the waveform level, whereas the massively multilingual benchmark explicitly mixes human-read and synthetic conditions and employs both LLM prompting and alignment-based substitution. A plausible implication is that “CS-FLEURS” now refers less to a single construction recipe than to a class of FLEURS-derived code-switching benchmarks.

3. Corpus composition and linguistic control

The UniCoM release emphasizes broad pair coverage and controlled code-mixing statistics (Lee et al., 21 Aug 2025). Its 23 in-domain languages are described as mostly Indo-European and Latin-script, yielding 253 distinct ordered pairs. The dataset reports a mean Code-Mixing Index (CMI) of approximately 0.11 and a mean I-Index of approximately 0.19, with distribution peaks at CMI around 5–10% and I-Index around 10–20%. Maximum substitutions per utterance are capped at three words, and the empirical tuning of $K$ and $P$ is stated to be based on linguistic studies of authentic code-switching. The representative examples show word-level intra-sentential switches within clauses, with POS-guided selection intended to preserve natural syntax in SVO word-order languages (Lee et al., 21 Aug 2025).

The massively multilingual benchmark is structured more explicitly around train/test partitioning (Yan et al., 17 Sep 2025). The cs-fleurs-read subset covers 14 X-English pairs, with 2,009 utterances per matrix language, for 28,126 utterances and 17 hours in total; the speakers are 21 bilingual readers with an overall 2:1 male:female ratio, and all recordings are human-validated. The cs-fleurs-xtts training split covers 16 X-English pairs and contains 128 hours and 889,000 tokens; the corresponding dev split has 15 hours and 105,000 tokens; test1 has 36 hours and 257,000 tokens; and test2 covers 60 non-English pairs of the form {Arabic, Mandarin, Hindi, Spanish}–X with 42 hours and 300,000 tokens. The cs-fleurs-mms split covers 45 X-English pairs and contains 56 hours and 315,000 tokens. Same-script and distinct-script pairings are explicitly tracked across these splits, with read approximately balanced, xtts-test2 heavily distinct-script, and mms nearly balanced (Yan et al., 17 Sep 2025).

The multilingual benchmark also gives the explicit CMI definition used for cs-fleurs-read:

$$CMI = \frac{1}{N}\sum_{s=1}^{N}\Bigl(1 - \max_{\ell\in L}\frac{f_{\ell}(s)}{\sum_{m\in L}f_{m}(s)}\Bigr)\times100,$$

where $N$ is the number of sentences, $K=3$0 is the set of languages, and $K=3$1 is the number of tokens in language $K=3$2 for sentence $K=3$3 (Yan et al., 17 Sep 2025). This formalization makes code-switch frequency directly measurable across typologically diverse pairs.

4. Speech synthesis, speaker consistency, and quality control

Speaker consistency is a central issue in synthetic code-switched speech because the audio often splices material originating in different languages, speakers, or synthesizers. In UniCoM CS-FLEURS, this problem is addressed by a style-unification stage based on kNN-VC and a HiFi-GAN neural vocoder (Lee et al., 21 Aug 2025). The stated rationale is that retrieval of speaker self-supervised features stabilizes speaker timbre across languages, while HiFi-GAN provides fast waveform generation with real-time factor approximately 0.01. The post-processing chain inherits from FLEURS-R and includes band-pass filtering, RMS normalization, forced alignment at 16 kHz for segmentation, and voice conversion at 24 kHz/16-bit with HiFi-GAN synthesis (Lee et al., 21 Aug 2025).

In the massively multilingual dataset, the synthesis stack depends on the subset (Yan et al., 17 Sep 2025). XTTS-v2 is used for the generative subsets, with a GPT-2 encoder, VQ-VAE units, a HiFi-GAN decoder, prepended language-ID tokens, and single-utterance voice conversion from FLEURS speakers; outputs are generated at 24 kHz and down-sampled to 16 kHz. The concatenative cs-fleurs-mms subset instead uses MMS-TTS, described as VITS-style single-speaker models producing monolingual segments concatenated with 100 ms silence, which results in speaker discontinuities. Quality filtering is performed uniformly with MMS-ZS forced alignment scores, and the bottom 5% of utterances within each language pair are discarded. The reported effect is substantial on xtts-test1, where CER decreases from 32.95% to 18.46% and UTMOS increases from 2.46 to 2.64 after filtering; on mms-test, CER decreases from 43.10% to 40.07 while UTMOS increases only marginally from 3.02 to 3.04, and Speaker Change Detection increases (Yan et al., 17 Sep 2025).

The Mandarin–English benchmark used in POI-aware contrastive training is public and separates synthetic training from human-read evaluation: CS-FLEURS-XTTS contains synthetic and human-read Mandarin utterances containing English insertions, while CS-FLEURS-READ is a held-out test set of human-read code-switched speech (Nguyen et al., 5 Jun 2026). Its significance is stated directly: it is one of the first large-scale, general-domain datasets where English embedded spans occur within a Mandarin matrix, making it a standard challenge for end-to-end CS-ASR research.

5. Evaluation protocols and downstream modeling

The three CS-FLEURS lines support partially overlapping but distinct evaluation protocols. UniCoM evaluates corpus quality with Romanized Character Error Rate (RER), Speaker Identity Consistency (SECS), Mean Opinion Score (MOS), and Speaker Identity Score (SIS) (Lee et al., 21 Aug 2025). RER is defined analogously to phoneme error rate:

$K=3$4

where $K=3$5, $K=3$6, and $K=3$7 are substitutions, deletions, and insertions, and $K=3$8 is the number of reference characters (Lee et al., 21 Aug 2025).

The massively multilingual benchmark evaluates ASR with case-insensitive, unpunctuated Character Error Rate (CER) and speech translation to English with case-insensitive BLEU. The baseline systems are OpenAI Whisper-Large-v3 for ASR/ST and a self-conditioned XLSR-based recipe in ESPnet for augmentation experiments (Yan et al., 17 Sep 2025).

The Mandarin–English benchmark introduces a more localized error measure around code-switch boundaries (Nguyen et al., 5 Jun 2026). Given an utterance with audio $K=3$9 and reference transcript $$SWORDS(w;L_{mat},L_{emb}) = \begin{cases} \tau(w) & \text{if } POS(w)\in P \text{ and } w \text{ selected for substitution} \ w & \text{otherwise} \end{cases}$$0, embedded-language spans are collected into $$SWORDS(w;L_{mat},L_{emb}) = \begin{cases} \tau(w) & \text{if } POS(w)\in P \text{ and } w \text{ selected for substitution} \ w & \text{otherwise} \end{cases}$$1, switch-boundary neighborhoods are expanded with radius $$SWORDS(w;L_{mat},L_{emb}) = \begin{cases} \tau(w) & \text{if } POS(w)\in P \text{ and } w \text{ selected for substitution} \ w & \text{otherwise} \end{cases}$$2, and the union of those neighborhoods defines the point-of-interest set $$SWORDS(w;L_{mat},L_{emb}) = \begin{cases} \tau(w) & \text{if } POS(w)\in P \text{ and } w \text{ selected for substitution} \ w & \text{otherwise} \end{cases}$$3. For cmn-eng, English POIs are detected simply by Latin-script tokens. Evaluation then reports both Word Error Rate (WER) and Point-of-Interest Error Rate (PIER):

$$SWORDS(w;L_{mat},L_{emb}) = \begin{cases} \tau(w) & \text{if } POS(w)\in P \text{ and } w \text{ selected for substitution} \ w & \text{otherwise} \end{cases}$$4

On this benchmark, POI-aware contrastive training fine-tunes Whisper-small with LoRA using a POI-weighted cross-entropy anchor objective and a multi-negative contrastive ranking loss, with near-miss negatives generated from N-best hypotheses and expanded offline by Gemini 2.5 Pro under acoustic, phonemic, and textual constraints (Nguyen et al., 5 Jun 2026).

6. Empirical findings, applications, and limitations

UniCoM reports overall CS-FLEURS averages of RER = 31.6%, MOS = 4.44, and SIS = 4.74 (Lee et al., 21 Aug 2025). In its in-domain comparisons, CS-FLEURS (de-en) reports RER 30.1, MOS 4.36, SIS 4.83, versus SWC 25.8, 4.51, and 4.90; CS-FLEURS (es-en) reports RER 28.9, MOS 4.00, SIS 4.89, versus Miami-Bangor 56.9, 4.20, and 4.40. In out-of-domain comparison, CS-FLEURS-O reports RER 40.8, MOS 3.68, SIS 4.69, versus MUCS2021 values of RER 48.3 for hi and 57.6 for bn. For ASR impact, combining human corpora with CS-FLEURS improves performance: in en–de, SWC + CSF yields CER 23.0 on SWC compared with 26.7 for SWC only; in en–es, MBC + CSF yields CER 35.8, whereas MBC only produces blank outputs (Lee et al., 21 Aug 2025).

The massively multilingual benchmark quantifies the recognition cost of code-switching directly (Yan et al., 17 Sep 2025). For ASR, monolingual FLEURS versus CS-FLEURS yields 8.44 to 19.83 CER on read-test, 5.71/7.56 to 18.46 on xtts-test1, 12.50/14.26 to 36.06 on xtts-test2, and 22.03/22.33 to 40.07 on mms-test. For ST BLEU, read-test shows 30.23 to 47.27, while xtts-test1 shows 28.31/27.04 to 23.81, xtts-test2 shows 24.30/26.35 to 20.66, and mms-test shows 23.89/21.80 to 16.58. Distinct-script pairs yield approximately three times higher ASR CER than same-script pairs. Augmenting FLEURS with cs-fleurs-xtts improves CER from 31.77 to 26.24 on seen pairs and from 29.62 to 27.77 on unseen pairs (Yan et al., 17 Sep 2025).

The Mandarin–English benchmark demonstrates how CS-FLEURS can function as a stress test for code-switch-specific training objectives (Nguyen et al., 5 Jun 2026). On CS-FLEURS-READ, standard LoRA fine-tuning with CE gives WER 16.67 and PIER 17.25. WCE + CL with tri-level gate reduces these to WER 14.06 and PIER 15.10, corresponding to absolute reductions of 2.61 and 2.15 points. The stated interpretation is that simple POI reweighting yields only marginal gains, whereas acoustically and phonemically plausible near-misses provide a stronger signal around code-switch boundaries.

Several limitations recur across CS-FLEURS variants. Synthetic speech may lack natural prosody; concatenative MMS introduces speaker discontinuities; some synthetic subsets are not human-validated beyond alignment filtering; and the massively multilingual benchmark is restricted to Wikipedia-style read text rather than conversational or spontaneous speech (Yan et al., 17 Sep 2025). At the same time, the consistent use of code-switch-aware controls such as CMI, I-Index, matrix/embedded language structure, and POI-localized evaluation makes CS-FLEURS a useful substrate for controlled multilingual experimentation. This suggests that the principal value of CS-FLEURS lies not only in scale, but in providing a reproducible spectrum of code-switching conditions—from human-read to generative and concatenative synthesis—on top of a shared FLEURS foundation.