Universal Code-Mixer (UniCoM)
- The paper introduces UniCoM to generate semantically faithful, natural code-switching speech through precise word-level substitutions.
- UniCoM’s pipeline integrates preprocessing, intra-sentential source-mixing (SWORDS), and retrieval-based voice conversion (kNN-VC) to maintain speaker consistency.
- The framework underpins the CS-FLEURS corpus, which improves ASR and speech-to-text translation performance by providing controlled, multilingual code-switching data.
Universal Code-Mixer (UniCoM) is a pipeline for generating high-quality, natural code-switching speech without altering sentence semantics. Introduced in "UniCoM: A Universal Code-Switching Speech Generator" (Lee et al., 21 Aug 2025), it targets a central bottleneck in multilingual speech technology: the scarcity of suitable code-switching datasets. UniCoM combines preprocessing, intra-sentential source-mixing, and style unification to synthesize multilingual utterances in which selected words from a matrix language are replaced by translated counterparts from an embedded language. The same framework is used to construct Code-Switching FLEURS (CS-FLEURS), a multilingual corpus for automatic speech recognition (ASR) and speech-to-text translation (S2TT).
1. Problem formulation and design objective
Code-switching (CS) is the alternation between two or more languages within a single speaker's utterances. In real-world conversation it is common, but systems capable of handling it remain underexplored, primarily because suitable data are scarce. UniCoM is proposed as a response to that data bottleneck, with the explicit goal of generating high-quality, natural CS samples while preserving the semantics of the original sentence (Lee et al., 21 Aug 2025).
The design objective is narrower than generic multilingual speech synthesis. UniCoM does not merely juxtapose content from different languages; it performs intra-sentential mixing under lexical and grammatical constraints. This suggests that the system is intended not only to increase corpus size, but also to approximate naturally occurring code-mixing patterns closely enough to support downstream CS-ASR and CS-S2TT training.
A recurrent simplification in synthetic CS generation is to concatenate larger spans of bilingual material. UniCoM rejects that strategy. Its core assumption is that semantically faithful and grammatically plausible code-switching is better approximated by controlled word-level substitution than by phrase-level substitution, especially when source languages differ in word order.
2. Pipeline architecture
UniCoM consists of three stages: preprocessing, intra-sentential source-mixing through the SWORDS module, and style unification through voice conversion (Lee et al., 21 Aug 2025).
In preprocessing, the audio undergoes band-pass filtering in the range 80–7 kHz to remove recording artifacts, followed by amplitude normalization through peak normalization to equalize volume across utterances. These steps standardize source material before linguistic manipulation and segment recombination.
The second stage, intra-sentential source-mixing, is the main generative component. Rather than concatenating whole sentences, UniCoM makes fine-grained substitutions at the word level, conditioned on part of speech (POS). The paper names this algorithm Substituting WORDs with Synonyms (SWORDS), although the operational mechanism described is replacement of selected words with their translations.
The third stage, style unification, addresses a direct consequence of recombining segments from two utterances: speaker identity may fluctuate within the generated sample. UniCoM uses kNN-VC, described as retrieval-based voice conversion, to map all segments to a single reference speaker embedding. The paper further states that kNN-VC uses a pretrained self-supervised front end and a HiFi-GAN vocoder for fast, language-agnostic conversion.
3. SWORDS and semantic preservation
SWORDS operates in four steps (Lee et al., 21 Aug 2025). First, it samples equivalent sentence pairs from an -way parallel corpus, such as FLEURS-R, selecting pairs and that are translations of each other. Second, it uses GPT-4o-mini to generate a bilingual dictionary of equivalent word pairs organized by POS. Third, it applies MMS-FA for forced alignment so that the speech segment corresponding to each word pair can be clipped from each utterance. Fourth, it completes source-mixing by recombination: within the matrix-language utterance, it randomly selects up to three word pairs from allowed POS categories and replaces each chosen word by its counterpart from the embedded language, reassembling both speech and text.
Several constraints are imposed to preserve semantics and naturalness. Substitution is restricted to single words rather than entire phrases, because phrase-level substitution often violates language-specific word order, including contrasts such as SVO and SOV. Only POS categories empirically common in natural CS—nouns, verbs, and interjections—are allowed. The number of substitutions is capped at three per sentence to match real-world code-mixing intensity.
The paper attributes semantic stability to two interacting mechanisms: the use of true translation pairs and the retention of the original grammatical scaffold. By operating at the POS level and replacing only selected lexical items, UniCoM aims to preserve original meaning while keeping sentence structure intact. A plausible implication is that the method treats code-switching as constrained lexical insertion into an otherwise stable syntactic frame rather than as unrestricted bilingual rewriting.
4. CS-FLEURS corpus construction
Using UniCoM, the authors construct Code-Switching FLEURS (CS-FLEURS), a multilingual CS corpus designed for ASR and S2TT (Lee et al., 21 Aug 2025). The baseline source is FLEURS-R, described as 102 languages denoised via Miipher. For in-domain selection, the construction process retains 23 European languages in Latin script that overlap with VoxPopuli. Source mixing via SWORDS yields 253 distinct ordered language pairs.
The reported scale and annotation structure are central to the resource.
| Property | Value |
|---|---|
| Duration | ~1,600 h |
| Utterances | 654.7 k |
| Tokens | 179.6 k |
| Code-Mixing Index (CMI) | 0.11 (avg) |
| I-Index | 0.19 (avg) |
| Fully parallel (-way) | 73 sets |
Forced alignment is also used for annotation: each replaced word is segmented, and the metadata include matrix-language versus embedded-language labels, CMI, and I-Index. This makes CS-FLEURS not only a training corpus but also a structured resource for analysis of code-mixing patterns.
The paper also distinguishes an out-of-domain setting, CS-FLEURS-O, involving zh, hi, and bn. This separation is important because later evaluation shows that the pipeline behaves differently in-domain and out-of-domain, especially with respect to naturalness and intelligibility.
5. Evaluation protocol and empirical results
The evaluation uses Romanized Character Error Rate (RER), Mean Opinion Score (MOS), and Speaker Identity Score (SIS) (Lee et al., 21 Aug 2025). RER is defined as
where , , and are substitutions, insertions, and deletions over the Romanized transcription of length 0. MOS is a 1–5 naturalness score, and SIS is a 1–5 score for within-utterance speaker consistency.
For in-domain language pairs, CS-FLEURS is compared with SWC for de-en and MBC for es-en. The reported results are as follows.
| Dataset | Metric summary |
|---|---|
| SWC (de-en) | RER 25.8, MOS 4.51, SIS 4.90 |
| CS-FLEURS (de-en) | RER 30.1, MOS 4.36, SIS 4.83 |
| MBC (es-en) | RER 56.9, MOS 4.20, SIS 4.40 |
| CS-FLEURS (es-en) | RER 28.9, MOS 4.00, SIS 4.89 |
These results support the paper’s statement that CS-FLEURS performs comparably to existing datasets on both objective and subjective metrics. In de-en, SWC has lower RER and slightly higher MOS and SIS than CS-FLEURS. In es-en, CS-FLEURS improves substantially on MBC in RER and SIS, while MOS is slightly lower.
For out-of-domain evaluation, Table 4b compares CS-FLEURS-O with ASCEND and MUCS2021. The reported examples are MUCS2021 (bn): RER 57.6, MOS 3.87, SIS 4.28; CS-FLEURS-O (bn): RER 42.7, MOS 2.88, SIS 4.71; ASCEND (zh): MOS 4.85, SIS 4.66; and CS-FLEURS-O (zh): RER 41.5, MOS 4.50, SIS 4.37. The paper’s own synthesis of these findings is that SWORDS plus kNN-VC yields highly intelligible CS speech with average RER 31.6%, and natural and speaker-consistent speech with average MOS 4.44 and SIS 4.74.
6. Downstream impact, limitations, and research use
UniCoM is evaluated not only as a generation pipeline but also as a data-generation method for downstream ASR (Lee et al., 21 Aug 2025). Fine-tuning XLS-R on in-domain pairs with and without CS-FLEURS produces the following reported outcomes: for en-de, SWC alone yields CER 26.7%, while SWC + CS-FLEURS yields 23.0%; for en-es, MBC alone yields CER 100% with model collapse, MBC + CS-FLEURS yields 35.8%, and CS-FLEURS alone yields 20.1%. These numbers support the paper’s claim that CS-FLEURS can function both as augmentation and as primary training data.
The stated applications are training data for code-switching ASR and speech-to-text translation, data augmentation in low-resource CS settings, and linguistic analysis of code-mixing patterns through the included CMI and I-Index metadata. Because the corpus records matrix and embedded language labels at the replaced-word level, it also supports more granular empirical study of intra-sentential CS behavior.
The limitations are explicit. Voice conversion is currently limited to in-domain languages characterized as Latin-script and Indo-European. Out-of-domain languages yield lower MOS and sometimes lower intelligibility. SWORDS also depends on high-quality forced alignment through MMS-FA and on the availability of parallel text. These constraints delimit the generality of the present system. A plausible implication is that the strongest performance regime for UniCoM is one in which parallel multilingual corpora, robust forced alignment, and compatible voice-conversion conditions all coexist.
Taken together, UniCoM is best understood as a constrained synthesis and corpus-construction framework: SWORDS provides linguistically bounded intra-sentential substitution, kNN-VC enforces speaker-style consistency, and the resulting CS-FLEURS resource demonstrates utility for both benchmark comparison and downstream CS-ASR training.