CSICL: Code-Switching In-Context Learning

• CSICL is a prompting strategy that progressively translates a target language query into English via structured code-switching to mitigate the translation barrier.
• It employs a five-shot demonstration with gradual code-switching, exposing intermediate translation states to improve multilingual reasoning.
• Empirical results indicate significant accuracy gains on low-resource and unseen languages compared to monolingual demonstrations.

Code-Switching In-Context Learning (CSICL) is a prompting strategy for cross-lingual in-context learning in which few-shot demonstrations gradually move from a target language to English via code-switching, while the instruction explicitly asks the model to “gradually translate” the query into English, think in English, and then answer in the target language. In the formulation introduced for LLMs, CSICL is a test-time, training-free method intended to mitigate the “translation barrier,” namely the failure mode in which multilingual reasoning depends on an implicit internal translation into English that can be incorrect or incomplete (Yoo et al., 7 Oct 2025). More broadly, CSICL belongs to a family of code-switching methods that treat language alternation not as incidental noise but as a structured bridge between a model’s high-resource latent space and lower-resource inputs; related evidence comes from work on English–Korean “knowledge activation,” code-switched comprehension benchmarks, code-switching-aware pre-training, and curriculum-based multilingual transfer (Kim et al., 2024, Mohamed et al., 16 Jun 2025, Wang et al., 2 Apr 2025, Yoo et al., 2024).

1. Definition, linguistic assumptions, and the translation barrier

CSICL is defined around a directional curriculum in the prompt. Let $L_{\text{tgt}}$ denote a target language and $\text{En}$ English. A CSICL demonstration is a gradual code-switching sequence

$q^{(0)}, q^{(25)}, q^{(50)}, q^{(75)}, q^{(100)},$

where $q^{(0)}$ is the query in $L_{\text{tgt}}$ only, $q^{(25)}, q^{(50)}, q^{(75)}$ are inter-sentential code-switched variants with English as embedded language and target as matrix language, and $q^{(100)}$ is the fully English equivalent. The associated instruction requires the model to output an explicit five-step translation process and then provide the final answer in the target language (Yoo et al., 7 Oct 2025).

This formulation adopts the standard matrix/embedded language distinction used throughout the code-switching literature. In the relevant prompt regime, the target language supplies the syntactic frame, while English is progressively inserted as embedded material. That design follows the Matrix Language Frame perspective used in several adjacent studies, including English–Korean code-switched question answering and synthetic code-switching generation, where the matrix language preserves word order and functional structure and the embedded language contributes semantically salient content (Kim et al., 2024).

The central rationale is the translation barrier. The CSICL paper builds on prior findings that multilingual LLMs often internally translate non-English text into English and then “think in English.” When this latent pivoting fails, performance in non-English languages deteriorates sharply. CSICL externalizes that process: instead of relying on a hidden, brittle translation step, it scaffolds a visible progression from target language to English inside both demonstrations and response format (Yoo et al., 7 Oct 2025).

Independent evaluation of code-switched comprehension supplies a complementary asymmetry. When foreign tokens are inserted into English text, reasoning accuracy degrades; when English is embedded into other languages, comprehension often improves, especially for weaker matrix languages. That asymmetry is consistent with an English-centric latent space and helps explain why CSICL is directional rather than symmetric: it moves the model toward English rather than away from it (Mohamed et al., 16 Jun 2025).

2. Prompt architecture and demonstration construction

In the canonical implementation, CSICL uses a 5-shot setup. Five test-set instances are randomly sampled as demonstrations and removed from the test set. Each instance is transformed into a gradual, inter-sentential code-switching sequence from the target language into English. The demonstrations therefore contain the original target-language query, several intermediate code-switched variants, the final English equivalent, the stepwise translation pattern, and the final answer (Yoo et al., 7 Oct 2025).

The instruction is highly constrained. The paper specifies the following mandatory opening sentence:

“Let's gradually translate this non-English query into English, then think in English, and finally answer the question.”

The model is then required to show the transformation from the original query to English in five steps, with each step progressively replacing more non-English words with English until the query is fully natural English. The final answer must end with the exact format “The answer is X,” where $X$ is a single uppercase letter (Yoo et al., 7 Oct 2025).

Demonstration generation is itself a two-stage process. First, given a parallel pair, GPT-5 generates a code-switched version under Matrix Language Frame constraints: the matrix language preserves word order and syntax, and approximately half of words or phrases are replaced by equivalents from the other language. Second, given the original sentence pair and one code-switched sentence, GPT-5 produces a five-sentence gradual progression spanning 100% target language, intermediate mixtures, and 100% English. The appendix example for English–Korean specifies the reverse direction explicitly—100% English, 75% English + 25% Korean, 50/50, 25% English + 75% Korean, 100% Korean—while CSICL uses the target-to-English direction for inference (Yoo et al., 7 Oct 2025).

This architecture differs from simpler multilingual prompting in two ways. First, the prompt embodies a curriculum: it does not merely juxtapose two languages but orders them in a sequence that progressively aligns the target-language input with English. Second, the response is not answer-only; the model is required to expose the intermediate translation states. A plausible implication is that CSICL turns a latent alignment problem into an explicit sequence modeling problem inside the context window.

3. Empirical evaluation and quantitative results

The primary CSICL evaluation spans 4 multilingual LLMs, 6 datasets, and 10 languages. The models are Qwen3-32B, deepseek-chat-v3.1, grok-4-fast, and Gemini 2.5 Flash. The main benchmark is Global MMLU with 36,000 samples across 6 subject categories and 10 languages: English; 3 target languages—French, Korean, Yoruba; and 6 unseen languages—Chinese, Spanish, Indonesian, Turkish, Swahili, Telugu. Additional tasks include FLORES+, MedExpQA, PolyMath, BLEnD, and MBBQ (Yoo et al., 7 Oct 2025).

Across all 4 models, 6 datasets, and 10 languages, CSICL improves over X-ICL baselines by 3.1 percentage points in target languages and 1.9 percentage points in unseen languages. In low-resource settings, the reported gains rise to 14.7% in target languages and 5.3% in unseen languages. English performance remains essentially unchanged, within approximately 0.2 percentage points in the Global MMLU analysis (Yoo et al., 7 Oct 2025).

On Global MMLU with Qwen3, the low-resource effects are particularly explicit. For Yoruba, monolingual target-language demonstrations yield 55.5 accuracy, while CSICL reaches 64.9, a gain of 9.4 percentage points. For unseen low-resource languages, the paper reports Swahili increasing from approximately 44.6 under monolingual English demonstrations to 51.2 with CSICL, and Telugu from approximately 37.7 to 43.4. The paper also reports that, relative to monolingual target-language demonstrations, CSICL is +6.0 percentage points on target languages and +4.8 percentage points on unseen low-resource languages for Qwen3 (Yoo et al., 7 Oct 2025).

Task-specific results exhibit a similar pattern. On FLORES+, CSICL improves COMET from 76.6 to 83.4 in the target language and from 72.3 to 75.3 in the unseen language. On the reasoning-oriented tasks MedExpQA and PolyMath, average exact-match gains over monolingual baselines are +5.4 percentage points for target and +3.8 percentage points for unseen languages. On the knowledge-intensive tasks BLEnD and MBBQ, the gains are smaller but positive: +2.6 percentage points for target and +1.1 percentage points for unseen languages (Yoo et al., 7 Oct 2025).

These results align with earlier English–Korean evidence that code-switched inputs can “faithfully activate” language-specific knowledge, especially in domains such as History and Tradition, where Korean lexical anchors substantially improve knowledge identification and leveraging relative to English-only formulations (Kim et al., 2024).

4. Baselines, ablations, and prompting dynamics

The CSICL study evaluates several comparison regimes: zero-shot learning, few-shot monolingual demonstrations in English or the target language, parallel demonstrations, and zero-shot chain-of-thought translation baselines that translate from the target language into English or a random language and then answer. It also includes ablations with simple inter-sentential code-switching demonstrations, reverse-direction gradual code-switching, and gradual translation instruction without code-switched demonstrations (Yoo et al., 7 Oct 2025).

The central ablation is additive. Gradual code-switching demonstrations alone already improve target and unseen performance over monolingual and parallel baselines by approximately +3.4 percentage points. Gradual translation instruction alone yields approximately +3.8 percentage points over monolingual. Combining both—the full CSICL configuration, namely Gradual CS (Tgt→En) plus Gradual Translation (Tgt→En)—produces the best result: 76.8 for target versus 72.0 for Monolingual Tgt, and 46.0 for unseen low versus 41.2 for Monolingual Tgt (Yoo et al., 7 Oct 2025).

Directionality matters. Gradual CS (Tgt→En) is better than Gradual CS (En→Tgt), and when both demonstrations and instruction are gradual, the target-to-English direction outperforms the reverse. This is one of the strongest pieces of evidence that CSICL is not merely “mixed-language prompting.” The effective signal is a curriculum that aligns the query with English, rather than a generic bilingual prompt (Yoo et al., 7 Oct 2025).

The paper also controls for prompt length by replacing each monolingual demonstration with five paraphrases. That control yields only small gains, whereas CSICL remains clearly superior: averaged over Qwen3 and Gemini, Paraphrasing (En) gives target 71.0 and unseen low 41.6, Paraphrasing (Tgt) gives target 72.3 and unseen low 39.3, while CSICL gives target 76.8 and unseen low 46.0. The benefit therefore does not reduce to “more sentences per demonstration” (Yoo et al., 7 Oct 2025).

Zero-shot gradual translation without demonstrations is insufficient. The model often collapses the requested five-step progression into an abrupt two-step transition, for example outputting two sentences in Korean and then three fully in English. The difference between zero-shot translation and zero-shot gradual translation is negligible, which shows that the demonstrations teach the model what “gradual” means operationally (Yoo et al., 7 Oct 2025).

A distinct but relevant benchmark study found that minimal CSW-aware instructions are highly model-dependent: Qwen variants often improve, sometimes surpassing monolingual English baselines, whereas LLaMA and Mistral can degrade under the same style of instruction. That result does not contradict CSICL, but it clarifies that prompt sensitivity remains architecture-dependent and that not all code-switch-aware prompting behaves like a well-structured gradual curriculum (Mohamed et al., 16 Jun 2025).

5. Relation to training-time code-switching, switch localization, and multilingual representation learning

CSICL is explicitly an inference-time method, but much of its conceptual basis is shared with training-time work on code-switching. “Code-Switching Curriculum Learning for Multilingual Transfer in LLMs” introduces a continual pre-training strategy with three stages—token-level code-switching, sentence-level code-switching, and monolingual corpora—and reports that both token- and sentence-level code-switching significantly enhance cross-lingual transfer, while curriculum learning amplifies these effects. In the Qwen 2 Korean setting, full CSCL improves K-MMLU, HAE-RAE, CLIcK, and translation quality over monolingual continual pre-training while better preserving English (Yoo et al., 2024).

That training-time result is echoed at pre-training scale. “Investigating and Scaling up Code-Switching for Multilingual LLM Pre-Training” argues that natural code-switching in pre-training corpora is a key driver of multilingual alignment and shows that synthetic code-switching, especially replacement-style code-switching in English, is markedly more efficient than simply adding monolingual data. In the En–Zh setting, En-Token-Repl 100M reaches a Chinese average accuracy comparable to adding 2000M monolingual Chinese tokens, and the best synthetic mixture improves Chinese average accuracy from 41.4 to 45.7 while strengthening MEXA alignment and downstream En→Zh translation and zero-shot XNLI transfer (Wang et al., 2 Apr 2025).

These results make the English-anchor design of CSICL more legible. Replacement-style code-switching in a high-resource language forces the model to interpret low-resource tokens inside a familiar semantic and syntactic scaffold. CSICL does not alter weights, but its demonstrations reproduce a related alignment geometry inside the context window: target-language material is not isolated; it is progressively attached to English.

A finer-grained perspective comes from work on switch localization. “Adapting Language Balance in Code-Switching Speech” treats code-switching points as points of interest, defines token-level script labels $s_{i,t} \in \{0,1\}$, and up-weights embedded-language tokens with a weighted cross-entropy objective. The paper’s purpose is speech recognition rather than CSICL, but its explicit treatment of where switching is hard—rare embedded tokens, switch points, and context bias during generation—suggests that prompt-side CSICL may benefit from explicit localization of switch positions rather than undifferentiated bilingual text (Ugan et al., 21 Oct 2025).

Earlier language-modeling work already emphasized that code-switching is structured rather than random. A Hindi–English factored RNNLM with POS and a binary CS-factor reduces perplexity dramatically relative to word-only models, and the best configuration combines POS and explicit switch information. Likewise, evaluation on code-switched NER and POS shows that generic multilingual pre-training does not automatically yield high-quality representations, whereas hierarchical meta-embeddings can match or exceed much larger multilingual models at far lower parameter cost. Taken together, these studies indicate that CSICL should be understood not as arbitrary language mixing, but as structured in-context side information about where switching occurs and how cross-lingual alignment should proceed (Sreeram et al., 2017, Winata et al., 2021).

6. Scope, limitations, and open problems

The current CSICL formulation is deliberately narrow. The 2025 CSICL paper explores inter-sentential code-switching only; it does not investigate intra-sentential, intra-word, or tag switching. Evaluation uses automatic metrics only—accuracy, exact match, and COMET—and does not include human evaluation of translation quality or prompt naturalness. The study covers 10 languages, which is broad but not comprehensive, and the additional domain tasks fix Spanish as the target language and vary unseen languages by dataset. The method also assumes English as the anchor language, and its five-step demonstrations plus five-step response format impose a nontrivial token overhead (Yoo et al., 7 Oct 2025).

Other related work adds further caveats. The English–Korean “knowledge switch” study is explicitly a single case study on Korea-specific knowledge and relies on a synthetic dataset, EnKoQA, constructed with GPT-3.5 translation and code-switch synthesis followed by human selection. Its positive findings therefore support, but do not universally establish, the broader CSICL hypothesis (Kim et al., 2024). The benchmark study on code-switched comprehension is limited to English, Arabic, German, French, and Chinese, and to three task families—Belebele, MMLU, and XNLI—while showing that prompting-based mitigation can be unstable across model families (Mohamed et al., 16 Jun 2025).

Several open directions follow directly from these limitations. One is extending CSICL beyond inter-sentential progression to intra-sentential and token-level switching, especially since both training-time curriculum learning and pre-training analyses report strong benefits from token-level replacement and mixed token/sentence regimes (Yoo et al., 2024, Wang et al., 2 Apr 2025). Another is replacing the fixed English pivot with alternative or multiple pivots, a question left open by the explicit English-centric framing of the current method (Yoo et al., 7 Oct 2025). A third is mechanistic analysis: the CSICL paper offers behavioral evidence for reduced translation-barrier effects, but does not directly probe attention, hidden-state alignment, or language-specific neuron dynamics. Finally, explicit switch-point marking, PIER-style evaluation restricted to code-switched moments, and feedback concentrated at those locations remain plausible extensions, but they are extrapolations from adjacent speech and code-switching literature rather than established components of CSICL proper (Ugan et al., 21 Oct 2025).

In its present form, CSICL is best understood as a principled inference-time bridge between multilingual surface input and an English-centric reasoning substrate. Its distinctiveness lies not in bilingual prompting per se, but in the ordered progression from target language to English, the explicit exposure of intermediate translation states, and the use of code-switching as a structured alignment device rather than a stylistic flourish (Yoo et al., 7 Oct 2025).