Cross-Lingual Transfer Failures

Updated 21 March 2026

Cross-lingual transfer failures are systematic underperformances arising when models trained on one language do not generalize well to another due to misaligned representations and data imbalances.
Empirical evidence shows significant drops in performance on tasks such as classification, generation, and speech processing, especially for low-resource or typologically distant languages.
Diagnostic analyses identify representation misalignment, catastrophic forgetting, and evaluation artifacts as key factors, prompting strategies like robust fine-tuning, adaptive architectures, and tailored evaluation protocols.

Cross-lingual transfer failures refer to the systematic underperformance or breakdown of model generalization when representations, knowledge, or behaviors acquired in one language are applied to another. This phenomenon is pervasive in multilingual LLMs, neural machine translation, continual learning, and downstream applications (classification, generation, paralinguistics) across both text and speech modalities. While shared representation spaces and parameter-efficient architectures have enabled substantial progress in zero-shot and few-shot cross-lingual transfer, a host of empirical studies demonstrate persistent and sometimes severe failures—especially for low-resource, typologically distant, or script-divergent languages. The following sections present a rigorous overview of foundational causes, empirical manifestations, experimental methodologies, and practical implications for research on cross-lingual transfer failures.

1. Formal Characterization and Metrics

Cross-lingual transfer is typically formalized as the ability of a model $f_\theta$ to leverage parameters or internal representations learned from data $D_s$ in a source language $S$ for effective generalization to a target language $T$ with data $D_t$ , often under zero-shot, few-shot, or continual learning constraints (Lauscher et al., 2020, Khelli et al., 29 Apr 2025, Koloski et al., 2023). The core objective is to minimize $E_t(\theta)=\mathbb{E}_{(x,y)\sim D_t}[\ell(f(x;\theta),y)]$ given $\theta$ trained predominantly on $D_s$ .

Failure is quantified via:

Absolute transfer gap: $\Delta^{(T)} = A_{within}^{(T)} - A_{across}^{(T)}$ , where $A_{within}$ is in-language test performance, and $A_{across}$ is cross-lingual (e.g., mixed-language inputs) (Rajaee et al., 2024).
Catastrophic forgetting $(\Delta_{src})$ : $P(\theta_{src},D_{src}) - P(\theta_{src \rightarrow tgt},D_{src})$ after transfer to $T$ (Koloski et al., 2023, Khelli et al., 29 Apr 2025).
Representation discrepancy: similarity metrics (CKA, cosine, Frobenius distance) between hidden-layer embeddings for parallel sentences in $S$ and $T$ (Yang et al., 2022, Ji et al., 2024).
Knowledge transferability: KTS, FRS, and X-FAKT, measuring factual recall and transfer uniformity across languages (Aggarwal et al., 25 Feb 2025).
CLTM (Cross-Lingual Transfer Matrix): normalized transfer scores $T_{j \rightarrow i}$ comparing in-language and cross-language data gains (Buitrago et al., 9 Mar 2026).

Together, these metrics reveal both average effects and language/task-specific vulnerabilities.

2. Empirical Manifestations: Tasks and Benchmarks

Cross-lingual transfer failures are empirically observed across several core domains:

Classification tasks (NLI, QA, POS, NER, paraphrase): Substantial performance drops—often up to 40–55 UAS/F1 points on NER, parsing, or NLI in typologically distant/low-resource languages—compared to losses of 5–15 points in related, resource-rich languages (Lauscher et al., 2020, Rajaee et al., 2024).
Natural language generation: Fine-tuned multilingual models exhibit high cross-lingual similarity in representation (XLRS), but this destroys generation quality and leads to code-switching or accidental language translation errors, which persist even at scale (Li et al., 2023).
Sequence labeling, dependency parsing: Models with strong sequential or syntactic inductive biases (RNNs, transformers w/ absolute position) fail to adapt to different word-orders, with transfer degrading monotonically in word-order distance from English (Ahmad et al., 2018).
Factual recall and knowledge QA: Multilingual LLMs recall facts unevenly—performance is high in “associated” (native) languages, low for the same facts in “non-associated” tongues (e.g., Llama-3-70B: $\mu_{assoc}=2.4\%$ errors, $\mu_{non-assoc}=9.9\%$ ; X-FAKT=$0.848$ for high-resource, $0.336$ for 1B-parameter models) (Aggarwal et al., 25 Feb 2025).
Paralinguistic speech tasks: In speaker verification, negative transfer is the norm except among closely related languages (only $~9\%$ of donor-target pairs yield positive gains), while gender recognition is near-agnostic (Buitrago et al., 9 Mar 2026).

Notably, even tasks considered language-agnostic, such as paralinguistics, manifest complex, language-dependent transfer patterns.

3. Root Causes and Failure Mechanisms

A multi-factorial synthesis emerges from cross-linguistic, architectural, and data-driven analyses:

Representation misalignment and discrepancy: Multilingual encoders (e.g., XLM-R, mBERT) do not yield tightly overlapping representations for translation pairs; offsets, as measured by CKA or cosine distance, highly correlate with transfer failure ( $\rho \approx 0.7-0.9$ ) (Yang et al., 2022).
Inductive bias mismatch: Order-sensitive architectures encode native-language sequential patterns that generalize poorly to divergent word orders. Order-agnostic self-attention with relative position bias ameliorates but does not erase this effect (Ahmad et al., 2018).
Polysemantic/inseparable neurons: Language-specific neurons, as defined by activation probability entropy (LAPE), are polysemantic and entangled with task features. Targeted interventions (e.g., neuron-specific LoRA, masked activations) neither yield >1 point improvements nor reliably improve cross-lingual performance (Mondal et al., 21 Mar 2025).
Catastrophic forgetting: Continual/sequential training degrades prior language task performance; non-Latin scripts are much more vulnerable (F1 drops $\geq$ 15 points after introducing Chinese), reflecting tokenization and parameter allocation biases (Khelli et al., 29 Apr 2025).
Output space separability: Cross-lingual objectives that force explicit alignment (e.g., MT-based continued pretraining, strict parallel sentence alignment) increase latent space separability to the detriment of transfer performance (Ji et al., 2024).
Language silos and data imbalance: Factual knowledge and task-relevant representations cluster in resource-rich, script-dominant languages, leading to inconsistent recall and asymmetric error rates between “associated” and "non-associated" languages (Aggarwal et al., 25 Feb 2025).
Evaluation artifacts and dataset shortcuts: Standard benchmarks often overstate cross-lingual ability by transferring task and surface-level artifacts, rather than genuine cross-linguistic knowledge (relative drops $\delta^{(T)}$ for NLI up to 17%, QA up to 31–49%) (Rajaee et al., 2024).
Translation and annotation noise: Benchmarks relying on human translation for low-resource languages (e.g., XNLI) are disproportionately affected by translation drift; accuracy gaps $\Delta_\ell$ for Swahili/Urdu $>10\%$ , inter-annotator agreements $\kappa < 0.5$ (Agrawal et al., 2024).

4. Diagnostics, Methodologies, and Analysis Frameworks

Failure analysis incorporates both internal (model-centric) and external (data-centric) tools:

Activation and representation probing: Cosine similarity, gradient alignment, mean/singular value statistics, and logit-lens analysis disambiguate shared subspaces from language-specific “drift” in hidden states (Lim et al., 19 May 2025, Veitsman et al., 19 Mar 2026).
Cross-lingual perturbation: Robust training, adversarial or randomized smoothing, and manifold mixup explicitly target embedding misalignment (adversarial noise, cross-attention interpolation), reducing the transfer gap by up to 40% (Huang et al., 2021, Yang et al., 2022).
Gradient similarity and orthogonality: Task and alignment gradients are often orthogonal (near-zero cosine), so improved embedding similarity through explicit alignment can leave downstream performance unchanged or degraded, especially on token-level tasks (Veitsman et al., 19 Mar 2026).
Cross-Lingual Transfer Matrix (CLTM): Systematic quantification of donor-target dynamics for paralinguistic and speech tasks reveals high asymmetry, intra-family clustering, and frequent negative transfer (Buitrago et al., 9 Mar 2026).
Evaluation enhancements: More challenging mixed-language or across-language instances, artifact baselines (shuffled inputs), and control for translation errors yield a truer picture of model transfer ability (Rajaee et al., 2024, Agrawal et al., 2024).

5. Empirical Results and Quantitative Insights

Selected results (see full block for further details):

Model/Task/Lang Group	Δ (Drop)	Error Rate / Score	Notable Pattern/Comment
mBERT NLI (low-resource)	–33%	Within=65.7%; Across=54.5%	Large drop in accuracy (Rajaee et al., 2024)
Llama-3-70B X-FAKT (low-res)	–0.848	$\mu_{assoc}=2.4\%$ , $\mu_{non-assoc}=9.9\%$	Strong asymmetry in factual recall (Aggarwal et al., 25 Feb 2025)
CLTM SV (cross-ling. speech)	–2.02	(e.g., German←Portuguese)	Negative transfer dominates (Buitrago et al., 9 Mar 2026)
LoRA (Catastrophic forgetting)	–15 pts	Post-Chinese on Latin scripts	Script effect, more loss for non-Latin (Khelli et al., 29 Apr 2025)
X-Mixup (CKA, XNLI)	+0.08	From 0.77→0.85 (CKA), +1.8 pts	Transfer gap shrinks up to 40% (Yang et al., 2022)

These results indicate that artifact bias, language resource availability, data quality, and model architecture interact nontrivially, and failure rates can remain high despite advanced pretraining and transfer strategies.

6. Remediation, Limitations, and Recommendations

The literature identifies several approaches and caveats:

Representation-level remedies: Robust fine-tuning (with noise, mixup, syntax supervision) and multi-source or contrastive objectives can mitigate, but not eliminate, transfer failures. For generation, preserving controlled language-specific representation variance is essential (Li et al., 2023, Ahmad et al., 2021).
Adapter and parameter allocation schemes: Non-shared or partially-shared adapters, especially for script-diverse or typologically distant languages, minimize knowledge loss in continual settings (Khelli et al., 29 Apr 2025).
Architectural adjustments: Order-agnostic models (self-attention with relative bias), syntax-aware architectures (GAT-augmented BERT), and explicit multidomain initialization outperform fixed sequential or monolithic designs (Ahmad et al., 2018, Ahmad et al., 2021, Edmiston et al., 2022).
Dataset audit and challenging evaluation: Across-language evaluation protocols, detailed translation-quality measurement ( $\Delta_\ell$ ), and artifact baselining are critical. Explicitly report performance under both human and synthetic translation (Rajaee et al., 2024, Agrawal et al., 2024).
Continual learning scheduling: Sequence introduction of high-transfer or script-compatible languages before vulnerable ones reduces catastrophic forgetting (Khelli et al., 29 Apr 2025).
Task-specific objectives: Match the alignment granularity (sentence/token) and the mixture of alignment and task objectives to the downstream use (Veitsman et al., 19 Mar 2026).
Research limitations: Persistent dependency on high-quality parallel data, inability to close performance gaps for distant languages, and incomplete recovery from domain mismatch without targeted joint-pretraining remain bottlenecks (Edmiston et al., 2022, Chen et al., 2020).

7. Future Directions and Open Challenges

Continued progress on robust cross-lingual transfer requires:

Dynamic alignment and regularization: Adaptive control of language invariance (e.g., explicit $P_{shared}$ subspace regularization), steering, and meta-learning for optimal knowledge sharing (Lim et al., 19 May 2025).
Mitigating data imbalance: Synthetic data augmentation for low-resource scripts/languages and self-supervised or mining-based methods for synonym/parallel pairs.
Evaluation standards: Community-wide adoption of artifact-aware benchmarks (e.g., mixed-language, X-FAKT) and reporting of per-language, per-script breakdowns (Aggarwal et al., 25 Feb 2025, Rajaee et al., 2024).
Phonetic and paralinguistic modeling: In speech, intra-family transfer optimization and embedding geometry adaptation to reduce speaker and language-induced manifold shifts (Buitrago et al., 9 Mar 2026).
Unifying multilingual and monolingual objectives: Curriculum and architectural innovations for convergent representations that retain cross-language flexibility while maximizing transfer utility across a broad range of tasks and resource regimes.

In summary, cross-lingual transfer failures are the consequence of intertwined architectural, representational, data, and evaluation artifacts. Achieving robust multilingual generalization—especially for low-resource, script-diverse tongues—requires explicit attention to representation alignment, careful objective selection, rigorous evaluation protocols, and adaptive, language-aware model design.