Code-Switching ASR Advances

Updated 2 September 2025

Code-switching ASR is an advanced field developing systems to transcribe speech that fluidly alternates between languages, addressing linguistic variability and data scarcity.
State-of-the-art methods leverage hybrid and end-to-end architectures, combining monolingual and multilingual language models to reduce error rates.
Recent advances employ mixture-of-experts, attention-guided adaptations, and retrieval-based models to enhance language boundary detection and transcription accuracy.

Code-switching automatic speech recognition (ASR) refers to the development and deployment of speech recognition systems capable of transcribing utterances in which speakers fluidly alternate between two or more languages within or across utterances. The code-switching phenomenon manifests as insertional, intra-sentential, inter-sentential, or even intra-word language changes, significantly complicating both the acoustic and linguistic modeling processes in ASR systems.

1. Linguistic Complexity and Corpus Construction

Code-switching ASR research is challenged by the high variability and diversity of language mixing patterns, the scarcity of naturally occurring transcribed code-switched data, and the often spontaneous nature of code-switching speech. Notable corpus compilation efforts address these challenges by:

Curating language-balanced datasets, e.g., a 14.3-hour five-language South African soap opera corpus with high speech rate and spanned English, isiZulu, isiXhosa, Setswana, and Sesotho (Yılmaz et al., 2018).
Employing tools such as ELAN for precise manual annotation of both monolingual and mixed-language segments, and capturing duration statistics per language to ensure representativity—essential for robust evaluation.
Addressing intra-sentential, insertional, and rare intra-word code-switching (e.g., English roots with Bantu affixes), which can confound modeling by blurring phonological and lexical boundaries.

Across the literature, a substantial proportion of resources and benchmarks concern Mandarin–English, Hindi–English, and Arabic–English code-switching, alongside low-resource combinations such as Frisian–Dutch or Malay–English (Agro et al., 10 Jul 2025).

2. Model Architectures and LLMing Techniques

To account for the intricacies of code-switching, a wide spectrum of ASR architectures has been explored, including:

Hybrid HMM-DNN and TDNN-HMM systems with carefully designed lexicons that merge phonetic symbol sets from multiple languages (e.g., 284 phonetic symbols for a five-language South African ASR (Yılmaz et al., 2018)).
Deep neural networks (DNNs), Time Delay Neural Networks (TDNNs), and hybrid TDNN-(Bi)LSTM architectures, with feature augmentation (e.g., pitch for tonal languages (Yılmaz et al., 2018)).
End-to-end models, particularly transformer-CTC, transformer-transducer, and hybrid CTC/attention encoder–decoders with joint loss functions, multi-task learning components, and cross-lingual information infusion (Dalmia et al., 2020, Yang et al., 27 Feb 2024, Luo et al., 2018, Liu et al., 2023).

LLMing in CS-ASR generally requires the interpolation of code-switched n-gram LLMs with large monolingual LMs. For instance, weights of 0.85–0.9 on the CS-LM component have been empirically shown to minimize perplexity in multilingual South African ASR (Yılmaz et al., 2018). Advanced methods fuse monolingual LMs with code-switched LMs or leverage neural (subword or word-piece) LMs to better capture cross-language contexts (Diwan et al., 2021, Dalmia et al., 2020, Chowdhury et al., 2021).

3. Training Protocols, Evaluation, and Language Recognition

Robust code-switching ASR systems employ a multi-stage training regimen:

Initial GMM-HMM alignment followed by DNN/TDNN or hybrid model training, often sequence-trained with sMBR or LF-MMI objectives (Yılmaz et al., 2018).
Data augmentation strategies, such as 3-fold speed perturbation (Yılmaz et al., 2018) or SpecAugment (Hamed et al., 2021), to mitigate resource scarcity and enhance robustness.
Transfer learning from larger monolingual corpora, semi-supervised and noisy-student training paradigms, and synthetic data generation via text-to-speech (TTS), phrase-mixing, or self-training (Sharma et al., 2020, Nguyen et al., 17 Jun 2025, Slottje et al., 2021, Xi et al., 5 Jul 2024).
Explicit evaluation of word/character error rate (WER/CER), mixed error rate (MER), and transliteration error metrics (T-WER, TW) for script-mixed or dialectal evaluation (Dalmia et al., 2020, Diwan et al., 2021).
Implicit or explicit language recognition is assessed using language confusion matrices and language tag error rates, revealing confusability between linguistically similar pairs, e.g., isiZulu–isiXhosa (Yılmaz et al., 2018), and the benefits of language identification (LID) via auxiliary loss or language tags (Luo et al., 2018, Dalmia et al., 2020, Liu et al., 2023).

4. Data Scarcity, Augmentation, and Synthetic Generation

The lack of real-world code-switched speech remains a central barrier:

Synthetic data generation, such as phrase-level mixing with 10–30% token/phrase replacement per sentence, translation with alignment, and audio splicing (e.g., for Malay-English, Mandarin-Malay, Tamil-English) simulates naturalistic switching patterns and reduces data-collection costs (Nguyen et al., 17 Jun 2025).
TTS-augmented approaches use high-quality synthetic speech matched to CS text to expand training resources; mixup strategies interpolate TTS with real speech to counter distributional mismatch and regularize ASR models (Sharma et al., 2020).
Semi-supervised learning, leveraging pseudo-labels from monolingual ASR or large pre-trained models refined via LLM filtering (prompt-based data quality assessment) further bridges the resource gap (Slottje et al., 2021, Xi et al., 5 Jul 2024).

Performance improvements from these methods are consistent, e.g., up to 5% absolute WER reduction using TTS and mixup augmentation for Hindi–English (Sharma et al., 2020), and relative WER reduction of 83% for BM-EN (Malay-English) in Singaporean benchmarks using phrase-mixing (Nguyen et al., 17 Jun 2025).

5. Advanced Adaptation and Handling Language Interactions

Recent research introduces sophisticated mechanisms to address language interaction, confusion, and the rapidity of code-switching:

Mixture-of-experts (MoE) models with explicitly disentangled language-specific encoder streams, combined with gating networks dynamically weighting language contributions frame-wise, enhance representation separability and reduce MER relative to dual-encoder baselines (Yang et al., 27 Feb 2024).
Deep language posterior injection, non-peaky CTC losses, and intermediate LID blocks are shown to improve language boundary alignment, mitigate acoustic and semantic confusion, and yield further significant MER reductions on SEAME Mandarin–English (Yang et al., 26 Nov 2024).
Attention-guided adaptation strategies in parameter-efficient settings (adapters comprising only 5.6% additional parameters) guide selected decoder attention heads to attend to language-identity tokens, resulting in MER of 14.2%—surpassing prior SOTA—on SEAME (Aditya et al., 2023).
Retrieval-augmented models (kNN-CTC) with dual monolingual datastores and gated selection mechanisms prevent cross-language contamination in retrieval, further improving zero-shot CS-ASR (Zhou et al., 6 Jun 2024).
Multi-graph decoding strategies construct union WFSTs of monolingual and bilingual LMs, enabling flexible adaptation to high- and low-resource language combinations with improved WER for monolingual segments (Yılmaz et al., 2019, Ali et al., 2021).

6. Challenges, Benchmarking, and Future Directions

The field recognizes several persistent challenges:

Severe data imbalances, particularly for non-English-dominant languages and low-resource dialects (e.g., Sesotho with only 0.2M words of training text (Yılmaz et al., 2018)).
Linguistic similarity causing high confusability, insertional and intra-word CS phenomena, and orthographic/morphological variation (especially in dialectal Arabic and low-resource Indian languages) (Diwan et al., 2021, Hamed et al., 2021).
Lack of standardized metrics or evaluation protocols. Mixed (MER), transliterated (T-WER), and code-mixing index (CMI) measures have been proposed, but adoption is uneven and complicates cross-paper comparison (Agro et al., 10 Jul 2025).
Catastrophic forgetting and the challenge of maintaining monolingual performance when adapting for code-switching settings.

Research opportunities include expanding and diversifying public datasets (covering more language pairs and domains), improving cross-domain adaptation, designing domain-robust and efficient neural architectures, advancing data synthesis techniques, and establishing more unified evaluation benchmarks (Agro et al., 10 Jul 2025).

7. Practical Implications and Prospects

Practical CS-ASR deployments stand to benefit from:

Cost-effective strategies for low-resource and under-represented language pairs using synthetic data and semi-supervised adaptation (Nguyen et al., 17 Jun 2025, Xi et al., 5 Jul 2024).
Modular and scalable ASR architectures (adapter-based, multi-graph, MoE, retrieval-augmented) with strong performance on both monolingual and code-switched test sets (Nguyen et al., 17 Jun 2025, Aditya et al., 2023).
Methods that explicitly model and leverage language-specific cues (via LID, attention guidance, or posterior injection) to ensure precise boundary detection and correct transcription at language transitions.

Future research directions are expected to focus on integrating more nuanced prosodic, contextual, and subword cues into both training and decoding, more precise and adaptable language recognition and switching mechanisms, and continually reducing the reliance on expensive real-world code-switched data through improved synthesis and transfer strategies.

This overview summarizes the principal technical dimensions, methodological evolutions, and continuing challenges in code-switching ASR, as reflected across a wide array of empirical and methodological studies encompassing diverse language scenarios, modeling paradigms, and resource environments.