Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers

Abstract: Code-switching is a pervasive linguistic phenomenon in global communication, yet modern information retrieval systems remain predominantly designed for, and evaluated within, monolingual contexts. To bridge this critical disconnect, we present a holistic study dedicated to code-switching IR. We introduce CSR-L (Code-Switching Retrieval benchmark-Lite), constructing a dataset via human annotation to capture the authentic naturalness of mixed-language queries. Our evaluation across statistical, dense, and late-interaction paradigms reveals that code-switching acts as a fundamental performance bottleneck, degrading the effectiveness of even robust multilingual models. We demonstrate that this failure stems from substantial divergence in the embedding space between pure and code-switched text. Scaling this investigation, we propose CS-MTEB, a comprehensive benchmark covering 11 diverse tasks, where we observe performance declines of up to 27%. Finally, we show that standard multilingual techniques like vocabulary expansion are insufficient to resolve these deficits completely. These findings underscore the fragility of current systems and establish code-switching as a crucial frontier for future IR optimization.

• The paper demonstrates that code-switching severely degrades performance across statistical, bi-encoder, and cross-encoder retrieval models.
• It constructs detailed benchmarks (CSR-L and CS-MTEB) to assess the impacts of code-mixing on various IR tasks across multiple language pairs.
• The study shows that vocabulary expansion improves results, yet fails to fully overcome the semantic gap seen in monolingual performance.

Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers

Introduction and Motivation

The paper "Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers" (2604.17632) delivers a comprehensive examination of information retrieval (IR) under the pervasive linguistic phenomenon of code-switching—a setting where queries naturally mix languages such as English and Chinese or Japanese, reflecting the real-world habits of a multilingual global user base. While recent advancements in IR have focused extensively on scaling semantic representation methods and extending coverage to numerous languages, the code-switching regime remains critically underexplored. The study identifies, quantifies, and anatomizes the persistent robustness gap exhibited by both monolingual and multilingual retrieval models when exposed to code-switched input, regardless of model scale, architecture class, or training strategy.

Figure 1: The workflow proceeds from benchmark construction (CSR-L), through large-scale evaluation across tasks (CS-MTEB), to controlled interventions (vocabulary expansion) dissecting the embedding space discrepancy between monolingual and code-switched queries.

Benchmark Construction: CSR-L and CS-MTEB

To enable systematic evaluation, the authors introduce two benchmark suites:

• CSR-L (Code-Switching Retrieval Benchmark-Lite): Built via rigorous human annotation, it recasts queries from established IR datasets (Touché 2020, HumanEval, TRECCOVID, FollowIR) into code-switched English–Chinese and English–Japanese forms, ensuring naturalistic mixing while preserving information need and linguistic salience. This benchmark enables controlled experimentation across statistical, dense/bi-encoder, cross-encoder, and late-interaction retrieval paradigms.
• CS-MTEB: To generalize code-switching assessment beyond simple retrieval, CS-MTEB scales the evaluation using LLM-assisted automated rewriting, covering 11 task types across 7 categories (instruction reranking, retrieval, clustering, classification, semantic textual similarity, reranking, and pair classification). The language coverage includes mixtures of English with nine typologically diverse tongues (Chinese, Japanese, German, Spanish, etc.), facilitating cross-task and cross-linguistic analysis at scale.

The CSR-L construction methodology emphasizes quality and naturalness via double annotation, explicit guidelines, and query-level fidelity—in contrast to many prior efforts using either automatic code-mixing or synthetic data augmentation. The CS-MTEB approach, leveraging LLMs and carefully engineered prompts, scales this philosophy for large-batch settings and supports thorough ablation and robustness assessment across state-of-the-art retrievers.

Empirical Findings: Robustness Under Code-Switching

Retrieval Paradigms and Model Classes

Experiments consistently demonstrate that query-side code-switching introduces severe performance degradation across all evaluated IR families:

• Statistical Baselines (BM25): Suffer significant drops due to term mismatch in mixed-language queries.
• Bi-encoders (e.g., e5-large-v2, all-MiniLM-L12-v2): Experience absolute drops of 12–15 nDCG@10 points on general retrieval tasks when moving from monolingual to code-switched input, even for models specifically trained on multilingual data.
• Multilingual Models (e.g., Arctic-Embed-m-v2.0, Qwen3-Embedding series): Retain relative robustness, especially in high-capacity settings, but still manifest a substantial gap—scaling helps but is insufficient to close the deficit.
• Cross-Encoders and Late-Interaction (e.g., ColBERT v2): Display no notable improvement in resilience to code-mixing compared to more efficient architectures.

Moreover, supplementary analysis using non-English monolingual baselines confirms that these effects are not artifacts of English-centric evaluation, but instead reflect a general disruption in semantic modeling induced by code-switching.

Embedding Space Analysis

Visualization of the embedding geometry reveals that code-switched queries are displaced within the representational space. For English-centric bi-encoders, code-switched and monolingual queries form well-separated, tight clusters, indicating a breakdown of semantic alignment. In contrast, multilingual models display decreased centroid distance and increased intersection, but not convergence, between distributions.

Figure 2: PCA of e5 on Touché 2020 dataset demonstrates the divergence between monolingual and code-switched query embeddings, elucidating the source of retrieval failures.

Broad Task Coverage: CS-MTEB Results

CS-MTEB evaluations substantiate the universality of this effect:

• For English-centric models, average drops range from 10–15 points regardless of language pair.
• Multilingual models, while more robust, still lose 4–7 points (e.g., Arctic-Embed-m-v2.0: -5 points in English–Spanish).
• Task sensitivity analysis reveals that reranking and high-precision ranking tasks are most susceptible, with performance in Japanese switching scenarios plummeting more than 30 points for reranking.
• Notably, lexical proximity between code-switched languages yields no clear mitigation, invalidating the hypothesis that linguistic similarity correlates with robustness in pre-trained embedding models.

Qualitative and quantitative analysis across multiple non-standard retrieval datasets—such as AILACaseDocs and MIRACL Chinese—confirm that the effect is pervasive and not benchmark-dependent.

Controlled Intervention: Vocabulary Expansion

To test whether tokenization limitations are predominantly responsible for the observed brittleness, the study adapts English-only retrievers using a lexicon-based vocabulary expansion technique. This method augments the model's subword vocabulary with high-frequency tokens from the partner language, initializing their embeddings via semantic averaging from bilingual lexicons:

• The expansion consistently delivers measurable gains (e.g., e5-large-v2 improves from 35.32 to 43.50 on CSR-L-Chinese), but never restores parity with monolingual performance. The largest relative improvement occurs in structurally complex queries with significant code-switching, yet substantial deficits persist.
• This finding indicates that mere surface-level token coverage is insufficient; deeper architectural or representational adaptation is necessary to resolve the semantic challenges induced by code-switching.

Discussion and Implications

• Semantic Alignment Versus Relevance Modeling: While multilingual models excel at cross-lingual semantic alignment tasks—such as bitext mining—their ability to preserve retrieval relevance under code-switching is far weaker. The failure of foundation models on code-switched reranking tasks, despite strong monolingual and cross-lingual generalization, reinforces this dichotomy.
• Limits of Multilingual Training and Scaling: Architectural complexity or increased parameter count in foundation models (up to 8B parameters) cannot compensate for the semantic disruptions caused by fluid mixed-language input. Code-switching must be treated as a distinct modality, rather than an interpolation between monolingual benchmarks.
• Recommendations for Future Work: Achieving robust retrieval under code-switching demands architectural innovation, explicit pre-training (or fine-tuning) on naturally occurring code-switched data, and further investigation into representation learning that preserves cross-language semantic relations in hybrid language environments.

This work positions code-switching IR as a crucial open problem in the next generation of retrieval systems, matching user-generated search behavior and supporting retrieval-augmented systems in truly global, linguistically diverse contexts.

Conclusion

The study provides the most holistic and rigorous analysis to date of retrieval model failure under code-switching. The empirical evidence establishes a strong, consistent robustness gap across all model classes and task types, with only partial mitigation from existing multilingual training or vocabulary expansion techniques. These findings underscore the necessity of treating code-switching as a first-class challenge in IR, with direct implications for the design of retrieval-augmented NLP, search, and agent systems. Progress on this frontier will require benchmarks, pre-trained models, and theory that account for the unique properties of naturally mixed-language input and its representation in embedding spaces (2604.17632).