Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer

Abstract: Cross-lingual transfer is a model's ability to generalize capabilities from well-represented source languages to under-represented target languages. Existing measures of a model's transfer strength conflate improvements in transfer with general improvements to accuracy in the source language. We advocate for an alternate metric that reliably captures transfer strength called Hardness Adjusted Transfer (HAT) Score, and use it to derive multiple insights on factors influencing transfer strength. Our analysis across twenty diverse LLMs and three popular mainstream multilingual benchmarks argues that 1) transfer in small models is not broken, 2) we are making slower than expected progress in cross-lingual transfer with model size, and 3) we have made clear progress over time.

• The paper introduces the Hardness Adjusted Transfer (HAT) Score to decouple true cross-lingual transfer from inflated source language performance.
• Empirical analysis across 20 LLMs and three multilingual benchmarks shows that raw target accuracy gains largely reflect improvements in source language proficiency.
• The study reveals that scaling models and refining pretraining recipes yield modest actual transfer progress, highlighting the need for more precise evaluation metrics.

Isolating True Cross-Lingual Transfer Progress in Multilingual LLMs

Motivation: Challenges in Measuring Cross-Lingual Transfer

The paper "Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer" (2606.21954) interrogates the extent to which progress on multilingual evaluation benchmarks translates to genuine advances in cross-lingual transfer (XLT)—the ability of an LLM to generalize from a high-resource source language to underrepresented target languages. Standard XLT evaluation practices rely on metrics such as raw target language accuracy or the source–target performance gap. However, the authors empirically show that these metrics are strongly confounded by improvements in source language accuracy: across 20 LLMs and three major multilingual benchmarks, target language performance is predictably and tightly coupled to source performance, with an $R^2 > 0.94$ in linear fits. This predictable coupling obscures the signal of true cross-lingual transfer ability, as improvements in source language accuracy automatically induce increases in target performance—thus rendering existing progress benchmarks ambiguous.

HAT: A Principled Metric for Cross-Lingual Transfer

To resolve this fundamental confound, the authors propose the Hardness Adjusted Transfer (HAT) Score, a metric rooted in unsupervised domain adaptation. The HAT formulation explicitly discounts improvements to target accuracy that derive solely from higher source accuracy, calibrating the measure of transfer to reflect only the surplus in target performance over the linear regression baseline defined by source ability. Formally, HAT represents the normalized expected target accuracy $E[\text{Te} | S = s,\ M]$ under a uniform distribution over source pass rates $s$, bounded to $[0,1]$ for comparability.

The HAT score is distinct from earlier approaches:

• Absolute target accuracy and XLT gap are both functions of the distribution of source accuracies, causing spurious apparent progress purely from source-LLM improvements.
• Metrics like the "transfer score" (target conditioned on correct source prediction) remain heavily source-dependent, and, as shown by the authors' counterexamples, may vary widely even under perfect transfer, depending only on the source pass rate distribution.
• HAT is independent of these confounders, robustly quantifying transfer strength irrespective of monotonically increasing source language accuracy.

Empirical estimation of HAT requires repeated model evaluations to approximate per-example pass rates, followed by linear regression enforced to pass through $(0,0)$.

Experimental Landscape

The evaluation spans 20 state-of-the-art LLMs (Gemini, Gemma, Qwen, GPT-OSS, Claude) across three challenging multilingual datasets:

• ECLeKTic: factual closed-book QA spanning 12 languages, focused on knowledge transfer with language-varied sources.
• MGSMv2: mathematical problem solving in 11 languages; recent curation excludes known artifacts present in the original MGSM.
• MMLU-ProX-Lite: multiple-choice reasoning and conceptual knowledge over 29 languages.

Accuracy is evaluated with LLM-as-judge ensembles for open-ended tasks; bootstrap confidence intervals quantify uncertainty of all reported results.

Empirical Insights: XLT Progress is Nontrivial

HAT Versus Traditional Metrics

Analysis demonstrates the systematic inflation of traditional metrics:

• Large-scale source accuracy improvements in models (e.g., GPT-OSS 120B vs. 20B) manifest substantial apparent XLT progress in target accuracy, but are not matched by improvements in HAT—indicating that actual transfer ability remains essentially unchanged.
• Scaling up Gemma-3 from 1B to 27B parameters, the XLT gap metric misleadingly suggests transfer has worsened; HAT instead reveals a modest, positive trend.

Model Scaling, Architecture, and Recipe Contributions

• Small Model Transfer: Even the smallest evaluated models (1B–4B) achieve HAT in the 50-80% range on some tasks, indicating nontrivial cross-lingual generalization at small scale—contradicting the notion that transfer is 'broken' in these models.
• Scaling Trends: Model size enables statistically significant but much slower XLT progress than previously assumed. Across families and comparable scales, pretraining recipes and architectural differences matter more than parameter count in maximizing HAT.
• Temporal Advances: A break-point analysis (time-based or version-based) reveals post-2025 models achieve higher HAT on MGSMv2 and MMLU-ProX-Lite, indicating definite but task-dependent progress; ECLeKTic, focused on factual recall, shows little improvement.

The Effect of "Thinking Time" and Script

• Thinking Budget: Increasing the inference "thinking budget" (depth of reasoning) improves HAT, especially for non-English prompts, with longer thinking traces empirically observed in target languages. Nevertheless, extra thinking alone does not guarantee transfer parity: Gemini-2.5-Flash remains below Gemini-3-Flash in HAT despite doubling the average thinking trace.
• Script Similarity: Cross-script transfer (Latin to Cyrillic, etc.) is weaker on ECLeKTic (factual transfer), but script has negligible effect on reasoning-based tasks. This points toward different mechanisms for parametric knowledge transfer versus abstract reasoning.

Near-Perfect Transfer in Reasoning

MGSMv2 and MMLU-ProX-Lite are nearly saturated in HAT among 2025+ models—with median HAT $>0.93$ on the latter—indicating that fundamentally harder, compositional, or context-rich benchmarks are needed for further discrimination of XLT ability.

Implications, Limitations, and Future Directions

By disentangling intrinsic cross-lingual transfer from raw accuracy improvements, HAT enables rigorous comparison across models, architectures, and training strategies. This allows researchers to:

• Accurately audit progress in transfer (as opposed to general improvements in language modeling).
• Benchmark pretraining and finetuning strategies more robustly.
• Inform resource allocation for extending LLM reach to truly underrepresented languages.

However, two key limitations surface:

• Contamination Sensitivity: Neither HAT nor other black-box metrics can distinguish algorithmic progress from data leakage or contamination, especially for near-perfect transfer. Progress claimed by HAT in resolved benchmarks requires independent control (e.g., data curation or adversarial evaluation).
• Computational Cost: Estimating HAT entails significant evaluation overhead, requiring repeated promptings and regression fitting for each benchmark-task-model triple.

Looking forward, improvements in transfer evaluation will likely involve both more sophisticated, multi-step, or tool-augmented benchmarks and mechanistic interpretability approaches. The nearly saturated reasoning benchmarks (MGSMv2, MMLU-ProX-Lite) prompt the development of next-generation, harder tasks, while transfer in factual recall remains challenging. Further, combining HAT-like behavioral measures with model-internal metrics (neuron overlap, subnetwork similarity) would provide granular insight into the mechanisms underlying successful cross-lingual transfer.

Conclusion

This study establishes the Hardness Adjusted Transfer (HAT) Score as a more principled, robust, and interpretable measure of cross-lingual transfer capability in multilingual LLMs, disentangling true transfer progress from confounds introduced by general source-language improvements. The HAT metric reframes much of the perceived rapid progress in cross-lingual transfer as a slower, more nuanced phenomenon, highlighting the ongoing need for both challenging benchmarks and more precise evaluation methodologies to foster equitable multilingual language technology.