Code-Switching ASR: Methods & Challenges

• CS-ASR is the process of recognizing speech containing alternating languages, addressing challenges like language confusion and spontaneous intra-sentence switching.
• It leverages unified end-to-end models and language-aware decoding techniques to jointly process bilingual or multilingual vocabularies effectively.
• Data augmentation with synthetic code-switched speech and integrated language identification methods significantly improve overall recognition performance.

Code-Switching Automatic Speech Recognition (CS-ASR) denotes automatic speech recognition for speech that contains two or more alternating languages within a conversation, often within a single utterance. In contemporary work, the task is typically framed as unified recognition over a mixed vocabulary rather than as a pipeline of monolingual recognizers, and its central difficulties are language confusion, accent bias that blurs phonetic boundaries, spontaneous intra-sentence switching, and the scarcity of annotated code-switching speech and text (Dhawan et al., 2023, Zhao et al., 2024, Liu et al., 29 Sep 2025). Research on CS-ASR spans bilingual settings such as Mandarin-English, Frisian-Dutch, Egyptian Arabic-English, and Arabic-English, as well as multilingual settings with more than two languages, and increasingly includes joint recognition and language identification, synthetic data generation, and large-language-model-based correction.

1. Problem characteristics

CS-ASR differs from monolingual ASR because the recognizer must determine not only what token was spoken, but also which language it belongs to, while language identity can change inside the sentence. Mandarin-English work on SEAME and the ASRU challenge repeatedly describes the core failure mode as language confusion, especially when speakers produce a non-native language with accent-induced phonetic overlap and when switches are intra-sentential rather than utterance-level (Liu et al., 2022, Zhao et al., 2024). The same literature also emphasizes that bilingual vocabularies increase the number of competing token alternatives, making acoustic and language modeling jointly harder than in monolingual recognition (Liu et al., 2024, Liu et al., 29 Sep 2025).

The severity of the problem depends on the language pair and corpus. Frisian-Dutch broadcast speech adds confusion from related languages, orthographic similarity, and shared vocabulary, so mixed-segment performance and switch detection become central evaluation targets (Yılmaz et al., 2018, Yılmaz et al., 2018). Egyptian Arabic-English introduces a different source of difficulty: dialectal Arabic is orthographically unstandardized and morphologically rich, and code-switching can appear not only between words and sentences but also inside a word through Arabic affixes attached to English stems (Hamed et al., 2021). This suggests that CS-ASR is not a single uniform task; the dominant bottleneck may be lexical sparsity, switch-boundary ambiguity, accent bias, orthographic instability, or morphological mixing, depending on the corpus.

A recurrent misconception in the recent literature is that multilingual pre-training alone solves code-switching. Comparative analyses using Whisper-small show the opposite: zero-shot or near-direct application of a strong multilingual pretrained model is markedly worse than adapted CS-ASR systems, especially on SEAME, where spontaneous switching is frequent and fluid (Zhao et al., 2024, Liu et al., 29 Sep 2025). A second misconception is that the constituent languages being high-resource implies the mixed-language task is also high-resource; several studies explicitly note that even when both monolingual languages are individually well resourced, annotated code-switching speech remains scarce (Hussein et al., 2023, Liu et al., 29 Sep 2025).

2. Corpora, benchmarks, and evaluation

CS-ASR research is organized around a small set of benchmark corpora and a larger set of corpus-construction efforts. SEAME is the standard Mandarin-English benchmark: one study describes it as a 112 hours Mandarin-English code-switching conversational corpus with 61.5% code-switching, and reports evaluation on eval$_{\text{man}}$ and eval$_{\text{sge}}$, which differ in matrix language and difficulty (Zhou et al., 2020). The ASRU 2019 Mandarin-English challenge defines a larger training regime with a 500 h Mandarin-only training set, 200 h intra-sentence English-Mandarin CS training set, 40 h intra-sentence English-Mandarin CS development set, and 20 h intra-sentence English-Mandarin CS test set (Liu et al., 2023). Frisian-Dutch work centers on FAME!, a broadcast archive containing Frisian-only, Dutch-only, and mixed segments (Yılmaz et al., 2018). Egyptian Arabic-English work introduces ArzEn, a 12-hour spontaneous corpus with explicit treatment of inter-sentential, intra-sentential, and intra-word code-switching (Hamed et al., 2021). Recent synthetic-data work adds CS-FLEURS, a multilingual resource spanning 253 language pairs, 1.6k hours, and 654.7k utterances (Lee et al., 21 Aug 2025).

Corpus	Language setting	Reported role
SEAME	Mandarin-English	Conversational benchmark with eval$_{\text{man}}$ and eval$_{\text{sge}}$ (Zhou et al., 2020)
ASRU 2019 CS challenge	Mandarin-English	500 h Mandarin-only, 200 h CS train, 40 h dev, 20 h test (Liu et al., 2023)
FAME!	Frisian-Dutch	Broadcast archive with monolingual and mixed segments (Yılmaz et al., 2018)
ArzEn	Egyptian Arabic-English	12-hour spontaneous corpus with intra-word CS (Hamed et al., 2021)
CS-FLEURS	253 language pairs	1.6k-hour synthetic multilingual CS corpus (Lee et al., 21 Aug 2025)

Evaluation is correspondingly heterogeneous. Mandarin-English work most often uses MER, which combines English WER and Mandarin CER (Liu et al., 2022, Liu et al., 2023). Frisian-Dutch work uses WER, often reported separately for Frisian-only, Dutch-only, and mixed segments, and also studies switch detection with DET curves and EER (Yılmaz et al., 2018, Wang et al., 2019). SEAME augmentation work uses MER for mixed scoring and also reports Mandarin CER and English WER on subsets (Hussein et al., 2023). Synthetic-data work on CS-FLEURS uses CER for CS-ASR impact experiments and separate quality measures such as RER, MOS, SIS, CMI, and I-index (Lee et al., 21 Aug 2025). This metric diversity reflects a substantive issue rather than a reporting preference: CS-ASR papers often evaluate transcription quality, language assignment quality, switch propensity, and dataset realism as partially distinct targets.

3. Architectural paradigms

A large fraction of CS-ASR systems use a unified end-to-end recognizer with a shared bilingual or multilingual output space. A canonical formulation is the hybrid CTC/attention architecture with a Conformer encoder and Transformer decoder, used in several Mandarin-English studies and extended with auxiliary language-aware tasks (Liu et al., 2022, Liu et al., 2023, Liu et al., 2024). In this family, the central design question is whether the mixed-language sequence should be modeled by a single shared representation or whether language identity should be injected explicitly.

One response is to preserve language-specific processing inside an otherwise unified recognizer. The Multi-Encoder-Decoder (MED) Transformer uses two symmetric language-specific encoders and language-specific source-target attention modules in the decoder, with monolingual pretraining on AISHELL-2 and LibriSpeech before SEAME fine-tuning; it reports 10.2% and 10.8% relative error-rate reductions on the two SEAME evaluation sets (Zhou et al., 2020). A decoding-side variant appears in multi-graph decoding for Frisian-Dutch, where one shared acoustic model is paired with the union of bilingual and monolingual WFST graphs, allowing larger monolingual Dutch LLMs to compete with the bilingual search space without harming mixed-segment accuracy (Yılmaz et al., 2019). These methods assume that language-specific constraints are useful at recognition time, but they also increase structural complexity.

A second response is to factorize the problem rather than split the architecture into fixed monolingual streams. In zero-shot Mandarin-English CS-ASR, one proposal lets monolingual modules transcribe all speech segments indiscriminately in their own script, effectively producing transliterations; a later bilingual module then resolves the final mixed-language sequence by using both transliterations together with a bilingual LLM (Yan et al., 2022). This formulation was introduced precisely because prior monolingual modules were overburdened with implicit switch-point detection in the zero-shot setting (Yan et al., 2022).

A third line of work uses explicitly phonological intermediate representations. The Two-Stage Phoneme-Centric model for Vietnamese-English performs speech-to-phone and phone-to-text in sequence, using an extended Vietnamese phoneme set to represent both Vietnamese speech and English words as realized in Vietnamese-accented code-switching. On its CS test set, the best variant reports 20.8% WER, compared with 27.9% for PhoWhisper-base (Nguyen et al., 7 Sep 2025). This suggests that phoneme-mediated modeling can be advantageous when pronunciation adaptation across languages is itself a major source of ambiguity.

Foundation-model adaptation has become a separate paradigm. Whisper-based CS-ASR studies fine-tune or adapt pretrained multilingual speech models through LoRA, full fine-tuning, encoder refinement, and language-aware decoding. One adaptation study introduces an encoder refiner and two sets of language-aware adapters in the decoder, improving Whisper on SEAME to 14.0 MER on dev_man and 20.6 MER on dev_sge, with strongest gains on the non-native language in each subset (Zhao et al., 2024). A broader comparative study concludes that for large pretrained multilingual models, auxiliary language-aware supervision can be more effective than language-specific modularization, while still noting that full fine-tuning remains the strongest Whisper configuration in its ASRU and SEAME experiments (Liu et al., 29 Sep 2025).

4. Data scarcity, augmentation, and synthetic corpora

Data-centric work addresses the fact that naturally transcribed code-switching speech is scarce and often domain-specific. A foundational Frisian-Dutch study enlarged the in-domain CS-related acoustic pool from 11.5 hours of manually annotated speech by adding 125.5 hours of automatically annotated broadcast data, and then showed that adding 442.5 hours of Dutch and 307.5 hours of Flemish became beneficial only after the in-domain CS pool had been sufficiently expanded (Yılmaz et al., 2018). The same work attacked textual scarcity by generating 10M, 25M, 50M, and 75M words of synthetic CS text with a recurrent LM, adding 3M words of automatic transcripts and 8.5M words of translated Dutch text, ultimately reaching a 107M-word LM corpus (Yılmaz et al., 2018). This is one of the clearest demonstrations that CS-ASR data scarcity is both acoustic and textual.

Synthetic CS speech generation from monolingual resources has since become a major subfield. Speech Collage constructs code-switched speech by splicing monolingual segments matched to target CS text, uses an overlap-add procedure with a Hamming window and energy normalization, and supports both in-domain text-driven generation and zero-shot generation from synthesized CS text (Hussein et al., 2023). The paper reports up to 34.4% relative reduction in MER in the in-domain scenario and 16.2% relative reduction in WER in the zero-shot scenario, while also noting that CS augmentation reduces monolingual bias and raises output CMI toward the reference (Hussein et al., 2023). The same report records that its visible SEAME table is not numerically consistent with the 34.4% statement, which makes this one of the few explicit result discrepancies in the CS-ASR literature (Hussein et al., 2023).

Large-scale multilingual synthesis is represented by UniCoM and CS-FLEURS. UniCoM performs word-level cross-lingual substitution under part-of-speech constraints, aligns source segments with MMS-FA, and uses kNN-VC for style unification; the released corpus spans 253 language pairs, 1.6k hours, and 654.7k utterances (Lee et al., 21 Aug 2025). In ASR experiments, CS-FLEURS functions both as a standalone training source and as augmentation: for English-German, adding CS-FLEURS to SWC reduces CER from 26.7 to 23.0, and for English-Spanish, adding CS-FLEURS to MBC reduces CER from 100 to 35.8 (Lee et al., 21 Aug 2025). At the same time, the same study explicitly documents strong synthetic-to-real domain mismatch, so the corpus is presented as augmentation and benchmarking infrastructure rather than a full substitute for natural CS speech (Lee et al., 21 Aug 2025).

Recent work has sharpened the distinction between acoustic mismatch and textual mismatch. A model- and data-centric analysis using CosyVoice2 TTS shows that textual mismatch hurts much more than accent mismatch: replacing SEAME-style text with ASRU-style text while keeping SEAME-like acoustics produces far larger degradation than changing the prompt accent, and simple TTS over original CS text helps SEAME more than ASRU because SEAME benefits more from added diversity in switching patterns (Liu et al., 29 Sep 2025). To address this, the same paper proposes SECT, a simplified equivalence-constraint-theory prompting strategy for LLM-based code-switching text generation; when SECT-generated text is converted to speech by TTS and mixed with real ASRU data, Whisper-small improves from 8.8 to 8.3 MER-equivalent test performance on ASRU (Liu et al., 29 Sep 2025).

A more lightweight variant of the same data-centric theme appears in the abstract of a unified ASR/LID model that introduces a new method for creating CS-ASR datasets from purely monolingual data sources and a Concatenated Tokenizer that reuses monolingual tokenizers while emitting language ID per text token (Dhawan et al., 2023). Semi-supervised learning has also been proposed: LLM-Filter places monolingual unlabeled speech into a noisy-student loop and uses prompt-based LLM correction for monolingual data selection and pseudo-label refinement (Xi et al., 2024).

5. Language-aware supervision and integrated language identification

A major research direction treats CS-ASR as recognition under explicit language supervision rather than as multilingual ASR with a larger vocabulary. In a representative hybrid CTC/attention formulation, the ASR loss is augmented by a token-level language diarization objective,

$$\mathcal{L}_{\mathrm{joint}}=\alpha \mathcal{L}_{\mathrm{ctc}}+(1-\alpha)\mathcal{L}_{\mathrm{att}}+\beta \mathcal{L}_{\mathrm{ld}},$$

and the decoder is biased with token-level language posteriors through

$$\mathbf{w}'_{n-1}=\mathrm{Concat}\big(\mathbf{w}_{n-1},\mathbf{p}(l_{n-1}\mid w_{1:n-1},\mathbf{X})\big).$$

This language posterior bias formulation improved a Conformer-Transformer baseline on SEAME, with the best +LD+LPB system reaching 16.1 and 22.8 MER on the two test sets when used with an LM (Liu et al., 2022).

The same idea has been extended to both frame and token resolutions. Interactive Language Biases (ILB) concatenates frame-level language posteriors to encoder states and token-level language posteriors to decoder inputs, allowing the encoder, decoder, and optionally the CTC branch to share multi-level language information (Liu et al., 2023). On the ASRU challenge set, the best encoder + decoder + CTC LPB configuration reduces MER from 12.8 to 11.8, and the paper reports that an external LM then degrades performance, interpreting the result as evidence that language bias already strengthens internal language modeling (Liu et al., 2023).

A closely related proposal is Language Alignment Loss (LAL), which learns pseudo frame-level language labels from decoder cross-attention and aligns encoder hidden states to those labels without any manual frame-level language annotation (Liu et al., 2024). LAL adds only a single linear layer, keeps parameter count effectively unchanged relative to the baseline, and improves the ASRU Conformer baseline from 12.8 to 11.7 MER when language weighting is applied, corresponding to an 8.6% relative improvement (Liu et al., 2024). The same paper argues that LAL is especially effective in primary-language-dominant data because class weights can upweight minority-language frames (Liu et al., 2024).

Frame-level language information is also useful for switch detection in its own right. A Frisian-Dutch study replaces language labels derived from 1-best word alignment with ASR-generated language posteriors obtained by summing phone posteriors per language. This reduces EER from 9.7% to 8.7% on development and from 6.3% to 4.8% on test, while reducing short spurious switch segments and false alarms (Wang et al., 2019). The result is important because it shows that language-aware signals can improve not only transcription but also the temporal localization of switches.

Tokenizer design provides a further integration point between ASR and language identification. The Concatenated Tokenizer model is explicitly described as enabling an ASR system to generate language ID for each emitted text token while reusing existing monolingual tokenizers, and the same work reports 98%+ spoken language-identification accuracy on out-of-distribution FLEURS in addition to state-of-the-art Miami Bangor CS-ASR results for English-Hindi and English-Spanish (Dhawan et al., 2023). This suggests that token inventory design can itself serve as a language-aware inductive bias.

6. Decoding, correction, and system combination

Because bilingual decoding spaces remain difficult even after acoustic modeling improves, CS-ASR has developed a substantial literature on post-recognition correction and search-space control. In WFST-based systems, multi-graph decoding uses the union of a bilingual CS graph and monolingual graphs under a shared acoustic model, allowing monolingual and bilingual hypotheses to compete in one search. For Frisian-Dutch, the best rescored $cs \cup nl++$ system reaches 21.9% total WER and improves Dutch WER over the best single-graph rescored baseline from 23.1% to 20.4% on development and from 18.8% to 16.3% on test, without degrading Frisian or mixed speech (Yılmaz et al., 2019). The same study also shows that a Frisian monolingual graph does not help, which illustrates that language-specific backends are useful only when monolingual resources are asymmetrically richer (Yılmaz et al., 2019).

LLMs have been introduced into CS-ASR primarily as generative error correction rather than conventional rescoring. GER first collects N-best lists from multiple ASR systems and then trains a LoRA-adapted Chinese-Llama2-7B to map the combined hypotheses to the reference transcript, explicitly treating error correction as hypotheses-to-transcription generation rather than selection from the list (Chen et al., 2023). On the ASRU Mandarin-English dataset, the best ensemble-hypothesis configuration improves MER from 11.0 1-best to 8.3, with strong data efficiency: even 1.3h of in-domain H2T supervision yields noticeable gains (Chen et al., 2023). This is one of the clearest demonstrations that post-ASR LLMs can exploit complementary fragments across hypotheses in code-switching speech.

LLM-based correction has also been combined with language-aware ASR features. A study built on LAL introduces a linguistic hint derived from LAL outputs and decoded hypotheses to guide prompting for generative error correction (Liu et al., 2024). The paper reports 14.1% relative improvement on ASRU and 5.5% relative improvement on SEAME for LLM-based correction with linguistic hints, while also documenting that inaccurate hints can harm performance on conversational SEAME, especially around colloquial interjections (Liu et al., 2024). This makes the method both promising and fragile: the value of the hint depends on how accurately the ASR stage summarizes utterance-level language identity.

System combination remains relevant in low-resource settings where different recognizers have complementary strengths. In Egyptian Arabic-English ArzEn, the best CNN-TDNN hybrid reaches 32.1% WER, the best Transformer E2E system reaches 32.7%, and a combination procedure that merges their outputs at sentence and word levels reduces overall WER to 30.6%, a 4.7% relative improvement over the baseline and 4.8% relative improvement on intra-sentential code-mixed sentences (Hamed et al., 2021). The paper shows that the complementarity is linguistically meaningful rather than accidental: the E2E system is more robust to Arabic orthographic variation and morphology, while the hybrid system is stronger on sparse English words and lexicon-supported OOV handling (Hamed et al., 2021).

7. Limitations, tradeoffs, and current directions

The contemporary CS-ASR literature converges on a strongly corpus-dependent view of the task. Comparative analysis across ASRU and SEAME argues that effective CS-ASR strategies must be aligned with the linguistic characteristics of the target corpus: model-centric language-aware methods are most effective when matrix-language structure is relatively stable, while data-centric augmentation becomes more important when switching is more spontaneous and varied (Liu et al., 29 Sep 2025). This suggests that there is no universally dominant architectural intervention.

Several broader lessons recur across otherwise different language pairs. First, multilingual pre-training is not sufficient by itself: Whisper-small is extremely weak zero-shot on code-switching and requires either fine-tuning or adaptation to become competitive (Zhao et al., 2024, Liu et al., 29 Sep 2025). Second, more monolingual data is not automatically better: Frisian-Dutch studies show that high-resource Dutch data became beneficial only after in-domain CS speech had been expanded, and Egyptian Arabic-English experiments show that monolingual-only training leads to catastrophic English performance in mixed speech and that adding too much English data can eventually harm recognition (Yılmaz et al., 2018, Hamed et al., 2021). Third, synthetic data are useful but not equivalent to real code-switching: CS-FLEURS improves downstream training but exhibits large synthetic-to-real mismatch, and SECT-based TTS augmentation helps most when it adds linguistically valid switching diversity rather than merely revoicing existing text (Lee et al., 21 Aug 2025, Liu et al., 29 Sep 2025).

A final misconception concerns decoding support. Several papers show that an external LM is not guaranteed to help; in language-biased systems it can even degrade performance, apparently because explicit language bias already strengthens internal bilingual sequence modeling (Liu et al., 2023). Conversely, LLM-based correction can be highly effective, but only when the N-best pool or language hint contains enough recoverable information (Chen et al., 2023, Liu et al., 2024). The field therefore continues to move toward systems that combine acoustically grounded language awareness, data augmentation tuned to corpus-specific switching patterns, and post-ASR correction that treats code-switching as structured bilingual generation rather than monolingual transcription with occasional foreign insertions.