When English Isn't the Best Teacher: Source Language Effects in Cross-Lingual In-Context Learning

Abstract: Cross-lingual transfer in multilingual NLP has been widely explored in supervised fine-tuning contexts, where factors like data availability and linguistic similarity largely determine transfer quality. As the field shifts toward few-shot In-Context Learning (ICL), it is often presumed that insights from fine-tuning carry over unchanged. Yet this assumption has not been rigorously evaluated, leaving open the question of how to choose source languages for cross-lingual ICL. We conduct a broad empirical study of cross-lingual transfer in ICL spanning seven tasks, six models, and a typologically diverse set of languages. We further analyze language confusion, a key obstacle for generative tasks in cross-lingual ICL. Our results show that conventional fine-tuning-based expectations do not consistently apply in the ICL regime and point to alternative heuristics for selecting source languages effectively.

• The paper shows that the target language is optimal in only about 24% of cases while English underperforms in roughly 16% of source-target pairs.
• It demonstrates that internal representation alignment (up to r≈0.6) predicts transfer success, contrary to expectations based on linguistic similarity.
• It uncovers that low-resource, non-Latin-script languages serve as effective donor sources, offering new operational guidelines for ICL.

Source Language Selection and Transfer Dynamics in Cross-Lingual In-Context Learning

Introduction

This paper presents a comprehensive empirical analysis of cross-lingual In-Context Learning (ICL) in multilingual LLMs, challenging several conventional assumptions originating from supervised fine-tuning regimes. Focusing on seven tasks, six recent LLMs, and 18 typologically and script-diverse languages, the study interrogates the effectiveness of source language selection for ICL-based transfer, the role of linguistic similarity, cross-lingual representation alignment, language confusion phenomena, and the effect of resource imbalance.

Key Empirical Findings

Source Language Is Not Always the Best Teacher

A central finding is that the target language itself is the optimal source language for transfer in only approximately 24% of cases. Moreover, English, despite its dominance in pretraining data, acts as the worst source language in about 16% of source-target pairs. The donor-recipient language dynamics reveal a robust negative correlation (Pearson $r \approx -0.93$); strong target languages (e.g., English, Spanish, German) are consistently weak sources, while low-resource, non-Latin-script languages (e.g., Thai, Telugu, Bengali) are disproportionately effective source languages. This breaks with fine-tuning-based paradigms, where self-source and high-resource languages typically drive optimal transfer.

Linguistic Similarity Is Not Predictive

Contrary to established literature—where linguistic similarity (syntactic, genetic, featural, phonological) is a prime predictor for transfer efficacy under fine-tuning—the study finds no systematic correlation between source-target similarity (as measured by URIEL/lang2vec vectors) and ICL transfer success. Weak or inconsistent correlations are observed across tasks and models, indicating that surface linguistic properties are mostly uninformative for source language selection in ICL.

Cross-Lingual Representation Alignment Is Critical

Instead, cross-lingual internal alignment learned through pretraining, as quantified using Centered Kernel Alignment (CKA) on last-layer representations from FLORES-200 data, robustly predicts transfer performance (statistically significant correlations up to $r \approx 0.6$ across models/tasks). Thus, representational proximity in model-internal space, not superficial similarity, governs transfer in the ICL regime.

Script Type and Resource Level Effects

Statistical analyses uncover two additional strong predictors: (1) non-Latin-script languages and (2) low-resource languages are significantly superior as source languages for cross-lingual transfer—effects that are additive and not interactive (as per ANOVA and post-hoc Tukey HSD tests, both $p < 0.001$). The role reversal is salient: languages that are recipients rather than donors in fine-tuned transfer paradigms are the most effective donors in ICL.

Language Confusion in Generative ICL

Evaluation on generative tasks using the Language Confusion Benchmark, focusing on the Line-level Pass Rate (LPR) metric, demonstrates that language confusion (model output in an unintended language) is a persistent failure mode. Non-Latin scripts are penalized as target languages (higher confusion), but when used as source languages, they induce less confusion in target predictions compared to Latin-script source languages. Language confusion patterns are only moderately correlated with task transfer performance and show little relationship with linguistic similarity or resource status.

Theoretical and Practical Implications

Paradigm Shift from Fine-Tuning Heuristics

These findings demand a departure from transferring fine-tuning insights to the ICL regime. The ineffectiveness of linguistic similarity, coupled with the reversed donor-recipient roles and source language asymmetries, suggests fundamentally different mechanisms for cross-lingual generalization in ICL. Parameter-free inference and pretraining-driven representational clustering override structural or lexical proximity.

Rethinking Cross-Lingual Source Selection Heuristics

The unexpectedly strong donor effect of low-resource, non-Latin-source languages offers a new operational heuristic for practitioners constructing cross-lingual ICL benchmarks or applications. Nevertheless, caution is warranted: as model coverage of these languages improves, this effect could diminish or change qualitatively due to shifting regularization and representational effects in pretrained LLMs.

Model-Internal Representation as a Guide

Since internal feature-space alignment drives effective transfer, tools that measure or optimize cross-lingual representation proximity may enable principled selection or even automated discovery of strong source languages for future multilingual ICL scenarios.

Evaluating Generative Robustness

Language confusion, rather than just task accuracy, should be a focal metric in cross-lingual generative applications deploying ICL—particularly in low-resource/underrepresented language contexts—since user comprehension depends on correct target language rendering, not source language prevalence.

Limitations and Directions for Future Research

While the study's design ensures typological and script diversity, the restriction to 18 languages and sub-4B parameter models imposes limitations on generalizability and scalability. Extending these analyses to more languages and larger LLMs is necessary to fully capture emergent effects in true web-scale or foundation models. Furthermore, the boundary where low-resource source languages become ineffective due to insufficient model competence requires further investigation.

Conclusion

This work fundamentally challenges the applicability of fine-tuning-based heuristics to cross-lingual in-context learning, providing solid evidence that linguistic similarity, resource level, and even English-centric strategies are unreliable or counterproductive in ICL. Instead, model-internal representational alignment and underrepresented source languages—particularly those with non-Latin scripts—emerge as primary drivers of successful transfer. These insights realign both the theoretical understanding and practical guidance for leveraging multilingual LLMs as cross-lingual in-context learners and highlight significant open questions in model design, evaluation, and cross-lingual prompt construction.