Progressive Code-Switching Strategies

• Progressive Code-Switching (PCS) is a curriculum-based approach that gradually introduces mixed-language data by progressively increasing the code-switching difficulty.
• PCS employs methods such as bucketization by resource dominance, relevance-based token replacement, and staged prompt scheduling for effective cross-lingual adaptation.
• Empirical results show that PCS improves performance in low-resource scenarios with gains of up to 13.1 points, optimizing tasks like sentiment analysis and QA.

Progressive Code-Switching (PCS) refers to a family of curriculum-driven strategies for cross-lingual transfer and adaptation, wherein models are exposed to increasingly challenging code-switched inputs—blending lexical, syntactic, or contextual information from resource-rich and low-resource languages—either during fine-tuning, in-context learning, or continued pretraining. PCS systematically organizes the introduction of code-switched data using measurable “difficulty” (e.g., fraction of resource-rich tokens, token-level replacement score, or phase of switching within context), gradually shifting the model from easier, resource-rich dominated data toward harder, low-resource dominated or fully target language settings. PCS has been formally defined and studied in several contexts, including curriculum self-training for sentiment analysis, code-switching in-context learning (ICL) for LLMs, zero-shot transfer via controlled code-switch augmentation, and staged curriculum learning for LLM adaptation. Empirical results across these paradigms demonstrate that progressive curricula materially improve transfer to low-resource and code-switched scenarios compared to static or randomly mixed code-switching baselines (Ranjan et al., 2022, Yoo et al., 7 Oct 2025, Li et al., 2024, Yoo et al., 2024).

1. Formal Definitions and Core Principles

PCS strategies instantiate the following core principle: models are presented with input distributions that incrementally increase in code-switch “difficulty,” where difficulty may be defined by linguistic dominance, replacement impact, or code-switching granularity. Key PCS instantiations are:

• Curriculum Self-Training (Sentiment Analysis): Given a labeled corpus $S$ in a resource-rich language (e.g., English) and an unlabeled code-switched corpus $T$ (mixing with a low-resource language), a base model is fine-tuned on $S$, pseudo-labels are assigned to $T$, and samples in $T$ are partitioned into $K$ buckets according to the fraction $$f_\text{eng}(x_i) = n_\text{eng}(x_i) / n_\text{words}(x_i)$$ (i.e., English dominance). Retraining occurs in $K$ stages, cumulatively incorporating buckets from most to least English-dominated (Ranjan et al., 2022).
• Difficulty-Controlled Data Augmentation: PCS in the context of zero-shot transfer employs a relevance-based code-switching mechanism—Layer-Wise Relevance Propagation (LRP) measures token impact on prediction, a temperature variable $\tau$ controls number of tokens replaced, and a dynamic scheduler orders data from easy (low-impact replaced) to hard (high-impact or many replacements) (Li et al., 2024).
• Prompt Engineering for LLM In-Context Learning: PCS in ICL explicitly scaffolds reasoning by stepping the model through prompts and demonstrations that shift from the target language to English. Each demonstration phase $i$ covers a fraction $f_i = i/K$ in English, the remainder in $L_0$ (target), advancing via inter-sentential code-switching (Yoo et al., 7 Oct 2025).
• Curriculum Pretraining (CSCL): LLMs are continually pretrained in three successive phases—token-level code-switching, sentence-level interleaving, finally monolingual mix—mirroring human second-language acquisition, without intermediate mixing (Yoo et al., 2024).

These frameworks employ various forms of code-switching (token-level, sentence-level, inter-sentential, and difficulty-aware) and structure the curriculum phase according to measurable linguistic properties. All rely on either explicit curriculum schedules or adaptive schedulers, with difficulty or dominance as the organizing principle.

2. Methodologies for Progressive Code-Switching

PCS incorporates several operational mechanisms, each grounded in measurable properties:

• Bucketization by Resource Dominance: Code-switched corpora are partitioned by proportion of resource-rich tokens or dominance measures; buckets are created such that each represents a defined range of $f_\text{eng}$, sorted high to low (Ranjan et al., 2022). Subsequent curriculum phases incorporate one more bucket at each step, progressively exposing the model to more challenging (low-resource-heavy) distributions.
• Relevance-Based Token Replacement: Difficulty of code-switched samples is determined by the relevance of each word to model prediction (via LRP). During each curriculum step, tokens with lowest relevance are replaced first, progressively increasing replacement difficulty via temperature $\tau$, ultimately moving to high-relevance tokens. Dynamic schedulers decide phase advances and occasionally revisit earlier (easier) curricula to prevent catastrophic forgetting (Li et al., 2024).
• Prompt-Phase Scheduling in ICL: Demonstrations are rewritten in $K+1$ versions, each increasing the English fraction, with explicit inter-sentential switches. Instructions are similarly scheduled, prompting explicit multi-phase translation and reasoning (Yoo et al., 7 Oct 2025).
• Staged Curriculum with Synthetic Corpora: For LLM pretraining, three sequential data phases are generated: (i) token-level code-switching (mixed tokens within sentences, sampled via Bernoulli mask), (ii) sentence-level code-switching (alternating English/target-language sentences), (iii) monolingual (half English, half target language), with millions to billions of tokens per phase (Yoo et al., 2024).

3. Experimental Frameworks and Results

PCS has been validated across multiple architectures, tasks, and languages:

Paper	Model(s)	Setting / Tasks	Main PCS Gains
(Ranjan et al., 2022)	mBERT	Sentiment analysis (ZS/ST/No-PT)	+1.2–4.0 Macro-F1 on code-switched
(Li et al., 2024)	mBERT, XLM-R	Zero-shot PAWS-X, MLDoc, XTOD	+6.3~13.1 pts accuracy/F1
(Yoo et al., 2024)	Qwen, Gemma	QA, MT, safety (Korean/Japanese)	+3.8–11.9 pts QA, +1.9 COMET
(Yoo et al., 7 Oct 2025)	Qwen3/Grok	LLM ICL (Global MMLU, QA, MT)	+6.0 pts on target, +4.8 unseen

PCS consistently narrows the gap to fully supervised or monolingual code-switched training, and is especially effective for low-resource or typologically distinct languages. Statistical significance for improvements is consistently reported (e.g., $p < 0.001$ or 95% bootstrap CI).

4. Ablation Analyses and Sensitivity

Ablation studies constitute a core validation approach for PCS:

• Source Data Removal: Omitting $S$ (labeled resource-rich data) during progressive phases degrades stability and Macro-F1 (Ranjan et al., 2022).
• Flat or Randomized Curriculum: Training on all code-switched data (no progression) or without LRP-based token scoring lowers transfer performance by 1-1.5 pts (Li et al., 2024).
• Scheduler Ablation: Disabling dynamic revisiting leads to catastrophic forgetting on easier distributions, confirming the need for adaptive curricula (Li et al., 2024).
• Directionality: Progressing from easy (resource-rich) to hard (low-resource) consistently outperforms anti-curriculum (Li et al., 2024).
• Instruction and Demonstration Design: Both gradual code-switching demonstrations and phased translation instructions contribute additively, with maximal gain when paired (Yoo et al., 7 Oct 2025).
• Code-Switching Granularity: Token-level and sentence-level code-switching amplify cross-lingual gains, and their combination outperforms either alone (Yoo et al., 2024).

A plausible implication is that the efficacy of PCS arises from minimizing overfitting to noisy, high-difficulty code-switched examples early while smoothly adapting to challenging distributions as model competence increases.

5. Implementation and Engineering Considerations

PCS variants span both supervised and self-supervised regimes and are compatible with a variety of model families:

• Labeling and Pseudo-Labeling: PCS self-training pipelines rely on high-confidence pseudo-label selection, often stratified per class to avoid output skew (Ranjan et al., 2022).
• Token Replacement Pipelines: LRP-based scoring and temperature schedule automate replacement difficulty; multi-target code-switching (e.g., EN→ES/DE) can yield better contextual word alignment (Li et al., 2024).
• Demo Generation in ICL: High-capacity LLMs (e.g., GPT-5, GPT-4O) serve to generate code-switched demonstration corpora in compliance with the Matrix Language Frame theory; explicit inter-sentential switching is favored for current LLMs (Yoo et al., 7 Oct 2025, Yoo et al., 2024).
• Hyperparameter Schedules: PCS implementations use empirically robust settings, e.g., $K=2$ for bucketization, $\delta \in [0.4, 0.6]$ for selection ratio, $\tau$ step of $0.1$ for difficulty schedule, and standard fine-tuning/training routines.
• Evaluation and Metrics: Experiments employ Macro-F1, accuracy, exact-match, and COMET for translation, always cross-validating with baseline, ablation, and alternative CS strategies.

PCS is lightweight, does not fundamentally alter model architectures, and can be integrated with chain-of-thought, self-consistency, or external retrieval techniques without conflict (Yoo et al., 7 Oct 2025).

6. Theoretical Rationale and Empirical Significance

PCS draws on principles of curriculum learning, with measurable effects:

• Noise Mitigation: By staging code-switched inputs from “easy” to “hard,” PCS suppresses label noise introduced by high-ambiguity, low-resource data at early stages (Ranjan et al., 2022, Li et al., 2024).
• Representation Bridging: Gradual alignment via PCS narrows the internal representational gap between resource-rich anchors and code-switched or low-resource regimes, substantiated by improved OOD (out-of-distribution) detection and adaptation (Ranjan et al., 2022).
• Latent Reasoning Scaffold: In LLMs, PCS removes the translation barrier by aligning latent reasoning with English rather than relying on implicit, lossy translation (Yoo et al., 7 Oct 2025).
• Efficiency and Safety: PCS achieves cross-lingual gains with limited (1B tokens) target-language data; safety alignment is improved, reducing attack success rate and spurious correlations between resource and safety (Yoo et al., 2024).

7. Limitations and Open Directions

PCS, while broadly effective, presents certain constraints and avenues for further research:

• Code-Switching Granularity: Most current PCS frameworks employ inter-sentential or token-level switching; intra-word or phrase-level switching remains unexplored (Yoo et al., 7 Oct 2025).
• Language Coverage: Empirical validation spans up to ten languages, but typological and script diversity are not yet exhaustively studied (Yoo et al., 7 Oct 2025, Yoo et al., 2024).
• Human Evaluation: No systematic human assessment of code-switching quality exists; fidelity and fluency of generated code-switch samples merit further investigation (Yoo et al., 7 Oct 2025).
• Dynamic Curricula: Future work may dynamically adjust code-switching rate/phase based on model proficiency or task error signals rather than fixed schedules (Yoo et al., 2024).
• Instruction Tuning and Low-Resource Extension: Extending PCS into instruction-tuning and extremely low-resource, morphologically rich, or script-divergent languages is an ongoing pursuit (Yoo et al., 2024).
• Paraphrastic Controls: PCS gains are not solely due to longer or more diverse prompts—paraphrastic controls do not replicate the code-switching benefit (Yoo et al., 7 Oct 2025).

PCS constitutes a principled, curriculum-based method, empirically validated across multilingual pretraining, self-training, ICL, and cross-lingual transfer, and achieving state-of-the-art performance in several zero-shot and few-shot regimes (Ranjan et al., 2022, Li et al., 2024, Yoo et al., 7 Oct 2025, Yoo et al., 2024).