---
title: 'Code-Switching ASR: Methods & Challenges'
url: https://www.emergentmind.com/topics/code-switching-automatic-speech-recognition-cs-asr
type: topic
---

# Code-Switching ASR: Methods & Challenges

Code-Switching Automatic Speech Recognition (CS-ASR) denotes automatic speech recognition for speech that contains two or more alternating languages within a conversation, often within a single utterance. In contemporary work, the task is typically framed as unified recognition over a mixed vocabulary rather than as a pipeline of monolingual recognizers, and its central difficulties are language confusion, accent bias that blurs phonetic boundaries, spontaneous intra-sentence switching, and the scarcity of annotated code-switching speech and text [2306.08753] [2412.16507] [2509.24310]. Research on CS-ASR spans bilingual settings such as Mandarin-English, Frisian-Dutch, Egyptian Arabic-English, and Arabic-English, as well as multilingual settings with more than two languages, and increasingly includes joint recognition and language identification, synthetic data generation, and large-language-model-based correction.

## 1. Problem characteristics

CS-ASR differs from monolingual ASR because the recognizer must determine not only what token was spoken, but also which language it belongs to, while language identity can change inside the sentence. Mandarin-English work on SEAME and the ASRU challenge repeatedly describes the core failure mode as **language confusion**, especially when speakers produce a non-native language with accent-induced phonetic overlap and when switches are intra-sentential rather than utterance-level [2210.14567] [2412.16507]. The same literature also emphasizes that bilingual vocabularies increase the number of competing token alternatives, making acoustic and language modeling jointly harder than in monolingual recognition [2403.05887] [2509.24310].

The severity of the problem depends on the language pair and corpus. Frisian-Dutch broadcast speech adds confusion from related languages, orthographic similarity, and shared vocabulary, so mixed-segment performance and switch detection become central evaluation targets [1807.10945] [1808.00521]. Egyptian Arabic-English introduces a different source of difficulty: dialectal Arabic is orthographically unstandardized and morphologically rich, and code-switching can appear not only between words and sentences but also inside a word through Arabic affixes attached to English stems [2108.12881]. This suggests that CS-ASR is not a single uniform task; the dominant bottleneck may be lexical sparsity, switch-boundary ambiguity, accent bias, orthographic instability, or morphological mixing, depending on the corpus.

A recurrent misconception in the recent literature is that multilingual pre-training alone solves code-switching. Comparative analyses using Whisper-small show the opposite: zero-shot or near-direct application of a strong multilingual pretrained model is markedly worse than adapted CS-ASR systems, especially on SEAME, where spontaneous switching is frequent and fluid [2412.16507] [2509.24310]. A second misconception is that the constituent languages being high-resource implies the mixed-language task is also high-resource; several studies explicitly note that even when both monolingual languages are individually well resourced, annotated code-switching speech remains scarce [2309.15674] [2509.24310].

## 2. Corpora, benchmarks, and evaluation

CS-ASR research is organized around a small set of benchmark corpora and a larger set of corpus-construction efforts. SEAME is the standard Mandarin-English benchmark: one study describes it as a **112 hours** Mandarin-English code-switching conversational corpus with **61.5% code-switching**, and reports evaluation on **eval\(_{\text{man}}\)** and **eval\(_{\text{sge}}\)**, which differ in matrix language and difficulty [2006.10414]. The ASRU 2019 Mandarin-English challenge defines a larger training regime with a **500 h Mandarin-only training set**, **200 h intra-sentence English-Mandarin CS training set**, **40 h intra-sentence English-Mandarin CS development set**, and **20 h intra-sentence English-Mandarin CS test set** [2309.16953]. Frisian-Dutch work centers on FAME!, a broadcast archive containing Frisian-only, Dutch-only, and mixed segments [1807.10945]. Egyptian Arabic-English work introduces ArzEn, a **12-hour** spontaneous corpus with explicit treatment of inter-sentential, intra-sentential, and intra-word code-switching [2108.12881]. Recent synthetic-data work adds CS-FLEURS, a multilingual resource spanning **253 language pairs**, **1.6k hours**, and **654.7k utterances** [2508.15244].

| Corpus | Language setting | Reported role |
|---|---|---|
| SEAME | Mandarin-English | Conversational benchmark with eval\(_{\text{man}}\) and eval\(_{\text{sge}}\) [2006.10414] |
| ASRU 2019 CS challenge | Mandarin-English | 500 h Mandarin-only, 200 h CS train, 40 h dev, 20 h test [2309.16953] |
| FAME! | Frisian-Dutch | Broadcast archive with monolingual and mixed segments [1807.10945] |
| ArzEn | Egyptian Arabic-English | 12-hour spontaneous corpus with intra-word CS [2108.12881] |
| CS-FLEURS | 253 language pairs | 1.6k-hour synthetic multilingual CS corpus [2508.15244] |

Evaluation is correspondingly heterogeneous. Mandarin-English work most often uses **MER**, which combines English **WER** and Mandarin **CER** [2210.14567] [2309.16953]. Frisian-Dutch work uses **WER**, often reported separately for Frisian-only, Dutch-only, and mixed segments, and also studies switch detection with **DET curves** and **EER** [1807.10945] [1906.08003]. SEAME augmentation work uses **MER** for mixed scoring and also reports Mandarin **CER** and English **WER** on subsets [2309.15674]. Synthetic-data work on CS-FLEURS uses **CER** for CS-ASR impact experiments and separate quality measures such as **RER**, **MOS**, **SIS**, **CMI**, and **I-index** [2508.15244]. This metric diversity reflects a substantive issue rather than a reporting preference: CS-ASR papers often evaluate transcription quality, language assignment quality, switch propensity, and dataset realism as partially distinct targets.

## 3. Architectural paradigms

A large fraction of CS-ASR systems use a unified end-to-end recognizer with a shared bilingual or multilingual output space. A canonical formulation is the hybrid **CTC/attention** architecture with a **Conformer encoder** and **Transformer decoder**, used in several Mandarin-English studies and extended with auxiliary language-aware tasks [2210.14567] [2309.16953] [2403.05887]. In this family, the central design question is whether the mixed-language sequence should be modeled by a single shared representation or whether language identity should be injected explicitly.

One response is to preserve language-specific processing inside an otherwise unified recognizer. The **Multi-Encoder-Decoder (MED)** Transformer uses two symmetric language-specific encoders and language-specific source-target attention modules in the decoder, with monolingual pretraining on AISHELL-2 and LibriSpeech before SEAME fine-tuning; it reports **10.2%** and **10.8%** relative error-rate reductions on the two SEAME evaluation sets [2006.10414]. A decoding-side variant appears in **multi-graph decoding** for Frisian-Dutch, where one shared acoustic model is paired with the union of bilingual and monolingual WFST graphs, allowing larger monolingual Dutch language models to compete with the bilingual search space without harming mixed-segment accuracy [1906.07523]. These methods assume that language-specific constraints are useful at recognition time, but they also increase structural complexity.

A second response is to factorize the problem rather than split the architecture into fixed monolingual streams. In zero-shot Mandarin-English CS-ASR, one proposal lets monolingual modules transcribe **all** speech segments indiscriminately in their own script, effectively producing transliterations; a later bilingual module then resolves the final mixed-language sequence by using both transliterations together with a bilingual language model [2211.01458]. This formulation was introduced precisely because prior monolingual modules were overburdened with implicit switch-point detection in the zero-shot setting [2211.01458].

A third line of work uses explicitly phonological intermediate representations. The **Two-Stage Phoneme-Centric** model for Vietnamese-English performs **speech-to-phone** and **phone-to-text** in sequence, using an extended Vietnamese phoneme set to represent both Vietnamese speech and English words as realized in Vietnamese-accented code-switching. On its CS test set, the best variant reports **20.8% WER**, compared with **27.9%** for PhoWhisper-base [2509.05983]. This suggests that phoneme-mediated modeling can be advantageous when pronunciation adaptation across languages is itself a major source of ambiguity.

Foundation-model adaptation has become a separate paradigm. Whisper-based CS-ASR studies fine-tune or adapt pretrained multilingual speech models through LoRA, full fine-tuning, encoder refinement, and language-aware decoding. One adaptation study introduces an **encoder refiner** and **two sets of language-aware adapters** in the decoder, improving Whisper on SEAME to **14.0 MER** on `dev_man` and **20.6 MER** on `dev_sge`, with strongest gains on the non-native language in each subset [2412.16507]. A broader comparative study concludes that for large pretrained multilingual models, auxiliary language-aware supervision can be more effective than language-specific modularization, while still noting that full fine-tuning remains the strongest Whisper configuration in its ASRU and SEAME experiments [2509.24310].

## 4. Data scarcity, augmentation, and synthetic corpora

Data-centric work addresses the fact that naturally transcribed code-switching speech is scarce and often domain-specific. A foundational Frisian-Dutch study enlarged the in-domain CS-related acoustic pool from **11.5 hours** of manually annotated speech by adding **125.5 hours** of automatically annotated broadcast data, and then showed that adding **442.5 hours** of Dutch and **307.5 hours** of Flemish became beneficial only after the in-domain CS pool had been sufficiently expanded [1807.10945]. The same work attacked textual scarcity by generating **10M**, **25M**, **50M**, and **75M** words of synthetic CS text with a recurrent LM, adding **3M** words of automatic transcripts and **8.5M** words of translated Dutch text, ultimately reaching a **107M-word** LM corpus [1807.10945]. This is one of the clearest demonstrations that CS-ASR data scarcity is both acoustic and textual.

Synthetic CS speech generation from monolingual resources has since become a major subfield. **Speech Collage** constructs code-switched speech by splicing monolingual segments matched to target CS text, uses an overlap-add procedure with a Hamming window and energy normalization, and supports both in-domain text-driven generation and zero-shot generation from synthesized CS text [2309.15674]. The paper reports up to **34.4%** relative reduction in **MER** in the in-domain scenario and **16.2%** relative reduction in **WER** in the zero-shot scenario, while also noting that CS augmentation reduces monolingual bias and raises output **CMI** toward the reference [2309.15674]. The same report records that its visible SEAME table is not numerically consistent with the **34.4%** statement, which makes this one of the few explicit result discrepancies in the CS-ASR literature [2309.15674].

Large-scale multilingual synthesis is represented by **UniCoM** and **CS-FLEURS**. UniCoM performs word-level cross-lingual substitution under part-of-speech constraints, aligns source segments with **MMS-FA**, and uses **kNN-VC** for style unification; the released corpus spans **253 language pairs**, **1.6k hours**, and **654.7k utterances** [2508.15244]. In ASR experiments, CS-FLEURS functions both as a standalone training source and as augmentation: for English-German, adding CS-FLEURS to SWC reduces CER from **26.7** to **23.0**, and for English-Spanish, adding CS-FLEURS to MBC reduces CER from **100** to **35.8** [2508.15244]. At the same time, the same study explicitly documents strong synthetic-to-real domain mismatch, so the corpus is presented as augmentation and benchmarking infrastructure rather than a full substitute for natural CS speech [2508.15244].

Recent work has sharpened the distinction between acoustic mismatch and textual mismatch. A model- and data-centric analysis using CosyVoice2 TTS shows that **textual mismatch hurts much more than accent mismatch**: replacing SEAME-style text with ASRU-style text while keeping SEAME-like acoustics produces far larger degradation than changing the prompt accent, and simple TTS over original CS text helps SEAME more than ASRU because SEAME benefits more from added diversity in switching patterns [2509.24310]. To address this, the same paper proposes **SECT**, a simplified equivalence-constraint-theory prompting strategy for LLM-based code-switching text generation; when SECT-generated text is converted to speech by TTS and mixed with real ASRU data, Whisper-small improves from **8.8** to **8.3** MER-equivalent test performance on ASRU [2509.24310].

A more lightweight variant of the same data-centric theme appears in the abstract of a unified ASR/LID model that introduces a new method for creating CS-ASR datasets from purely monolingual data sources and a **Concatenated Tokenizer** that reuses monolingual tokenizers while emitting language ID per text token [2306.08753]. Semi-supervised learning has also been proposed: **LLM-Filter** places monolingual unlabeled speech into a noisy-student loop and uses prompt-based LLM correction for monolingual data selection and pseudo-label refinement [2407.04219].

## 5. Language-aware supervision and integrated language identification

A major research direction treats CS-ASR as recognition under explicit language supervision rather than as multilingual ASR with a larger vocabulary. In a representative hybrid CTC/attention formulation, the ASR loss is augmented by a token-level language diarization objective,
$$
\mathcal{L}_{\mathrm{joint}}=\alpha \mathcal{L}_{\mathrm{ctc}}+(1-\alpha)\mathcal{L}_{\mathrm{att}}+\beta \mathcal{L}_{\mathrm{ld}},
$$
and the decoder is biased with token-level language posteriors through
$$
\mathbf{w}'_{n-1}=\mathrm{Concat}\big(\mathbf{w}_{n-1},\mathbf{p}(l_{n-1}\mid w_{1:n-1},\mathbf{X})\big).
$$
This **language posterior bias** formulation improved a Conformer-Transformer baseline on SEAME, with the best **+LD+LPB** system reaching **16.1** and **22.8** MER on the two test sets when used with an LM [2210.14567].

The same idea has been extended to both frame and token resolutions. **Interactive Language Biases (ILB)** concatenates frame-level language posteriors to encoder states and token-level language posteriors to decoder inputs, allowing the encoder, decoder, and optionally the CTC branch to share multi-level language information [2309.16953]. On the ASRU challenge set, the best **encoder + decoder + CTC LPB** configuration reduces MER from **12.8** to **11.8**, and the paper reports that an external LM then degrades performance, interpreting the result as evidence that language bias already strengthens internal language modeling [2309.16953].

A closely related proposal is **Language Alignment Loss (LAL)**, which learns pseudo frame-level language labels from decoder cross-attention and aligns encoder hidden states to those labels without any manual frame-level language annotation [2403.05887]. LAL adds only a single linear layer, keeps parameter count effectively unchanged relative to the baseline, and improves the ASRU Conformer baseline from **12.8** to **11.7** MER when language weighting is applied, corresponding to an **8.6%** relative improvement [2403.05887]. The same paper argues that LAL is especially effective in primary-language-dominant data because class weights can upweight minority-language frames [2403.05887].

Frame-level language information is also useful for switch detection in its own right. A Frisian-Dutch study replaces language labels derived from 1-best word alignment with **ASR-generated language posteriors** obtained by summing phone posteriors per language. This reduces EER from **9.7%** to **8.7%** on development and from **6.3%** to **4.8%** on test, while reducing short spurious switch segments and false alarms [1906.08003]. The result is important because it shows that language-aware signals can improve not only transcription but also the temporal localization of switches.

Tokenizer design provides a further integration point between ASR and language identification. The **Concatenated Tokenizer** model is explicitly described as enabling an ASR system to generate language ID for each emitted text token while reusing existing monolingual tokenizers, and the same work reports **98%+** spoken language-identification accuracy on out-of-distribution FLEURS in addition to state-of-the-art Miami Bangor CS-ASR results for English-Hindi and English-Spanish [2306.08753]. This suggests that token inventory design can itself serve as a language-aware inductive bias.

## 6. Decoding, correction, and system combination

Because bilingual decoding spaces remain difficult even after acoustic modeling improves, CS-ASR has developed a substantial literature on post-recognition correction and search-space control. In WFST-based systems, **multi-graph decoding** uses the union of a bilingual CS graph and monolingual graphs under a shared acoustic model, allowing monolingual and bilingual hypotheses to compete in one search. For Frisian-Dutch, the best rescored **\(cs \cup nl++\)** system reaches **21.9%** total WER and improves Dutch WER over the best single-graph rescored baseline from **23.1%** to **20.4%** on development and from **18.8%** to **16.3%** on test, without degrading Frisian or mixed speech [1906.07523]. The same study also shows that a Frisian monolingual graph does not help, which illustrates that language-specific backends are useful only when monolingual resources are asymmetrically richer [1906.07523].

Large language models have been introduced into CS-ASR primarily as **generative error correction** rather than conventional rescoring. **GER** first collects N-best lists from multiple ASR systems and then trains a LoRA-adapted **Chinese-Llama2-7B** to map the combined hypotheses to the reference transcript, explicitly treating error correction as hypotheses-to-transcription generation rather than selection from the list [2310.13013]. On the ASRU Mandarin-English dataset, the best ensemble-hypothesis configuration improves MER from **11.0** 1-best to **8.3**, with strong data efficiency: even **1.3h** of in-domain H2T supervision yields noticeable gains [2310.13013]. This is one of the clearest demonstrations that post-ASR LLMs can exploit complementary fragments across hypotheses in code-switching speech.

LLM-based correction has also been combined with language-aware ASR features. A study built on **LAL** introduces a **linguistic hint** derived from LAL outputs and decoded hypotheses to guide prompting for generative error correction [2403.05887]. The paper reports **14.1%** relative improvement on ASRU and **5.5%** relative improvement on SEAME for LLM-based correction with linguistic hints, while also documenting that inaccurate hints can harm performance on conversational SEAME, especially around colloquial interjections [2403.05887]. This makes the method both promising and fragile: the value of the hint depends on how accurately the ASR stage summarizes utterance-level language identity.

System combination remains relevant in low-resource settings where different recognizers have complementary strengths. In Egyptian Arabic-English ArzEn, the best **CNN-TDNN** hybrid reaches **32.1% WER**, the best Transformer E2E system reaches **32.7%**, and a combination procedure that merges their outputs at sentence and word levels reduces overall WER to **30.6%**, a **4.7%** relative improvement over the baseline and **4.8%** relative improvement on intra-sentential code-mixed sentences [2108.12881]. The paper shows that the complementarity is linguistically meaningful rather than accidental: the E2E system is more robust to Arabic orthographic variation and morphology, while the hybrid system is stronger on sparse English words and lexicon-supported OOV handling [2108.12881].

## 7. Limitations, tradeoffs, and current directions

The contemporary CS-ASR literature converges on a strongly corpus-dependent view of the task. Comparative analysis across ASRU and SEAME argues that effective CS-ASR strategies must be aligned with the linguistic characteristics of the target corpus: model-centric language-aware methods are most effective when matrix-language structure is relatively stable, while data-centric augmentation becomes more important when switching is more spontaneous and varied [2509.24310]. This suggests that there is no universally dominant architectural intervention.

Several broader lessons recur across otherwise different language pairs. First, **multilingual pre-training is not sufficient by itself**: Whisper-small is extremely weak zero-shot on code-switching and requires either fine-tuning or adaptation to become competitive [2412.16507] [2509.24310]. Second, **more monolingual data is not automatically better**: Frisian-Dutch studies show that high-resource Dutch data became beneficial only after in-domain CS speech had been expanded, and Egyptian Arabic-English experiments show that monolingual-only training leads to catastrophic English performance in mixed speech and that adding too much English data can eventually harm recognition [1807.10945] [2108.12881]. Third, **synthetic data are useful but not equivalent to real code-switching**: CS-FLEURS improves downstream training but exhibits large synthetic-to-real mismatch, and SECT-based TTS augmentation helps most when it adds linguistically valid switching diversity rather than merely revoicing existing text [2508.15244] [2509.24310].

A final misconception concerns decoding support. Several papers show that an external LM is not guaranteed to help; in language-biased systems it can even degrade performance, apparently because explicit language bias already strengthens internal bilingual sequence modeling [2309.16953]. Conversely, LLM-based correction can be highly effective, but only when the N-best pool or language hint contains enough recoverable information [2310.13013] [2403.05887]. The field therefore continues to move toward systems that combine acoustically grounded language awareness, data augmentation tuned to corpus-specific switching patterns, and post-ASR correction that treats code-switching as structured bilingual generation rather than monolingual transcription with occasional foreign insertions.

Source: https://www.emergentmind.com/topics/code-switching-automatic-speech-recognition-cs-asr