Cross-lingual Language Models (XLMs)

Updated 1 March 2026

Cross-lingual Language Models (XLMs) are neural architectures pretrained on multilingual text using shared subword vocabularies and transformer encoders to enable robust knowledge transfer.
They employ objectives like Multilingual Masked LM, Translation LM, and supervised MT to align language representations, yielding significant improvements in zero-shot and few-shot performance.
Leveraging parameter sharing, XLMs provide a unified framework for tasks such as NLU, machine translation, and sequence labeling across over 100 languages, especially benefiting low-resource scenarios.

Cross-lingual LLMs (XLMs) are neural architectures pretrained on unlabeled text in multiple languages with the explicit goal of learning representations that support accurate and robust transfer of linguistic knowledge and task information across languages. XLMs provide a unified framework to encode and process text spanning dozens to hundreds of languages via parameter sharing, shared subword vocabularies, and cross-lingual pretraining objectives. This paradigm has yielded unprecedented zero-shot and few-shot performance in NLU, sequence labeling, and machine translation, especially for low-resource languages.

1. Core Principles and Pretraining Objectives

The core mechanism of XLMs is parameter sharing: all languages are encoded with the same neural architecture and set of weights—most commonly deep Transformer encoders (Lample et al., 2019, Goyal et al., 2021). Input texts are tokenized via language-agnostic subword segmenters (e.g., SentencePiece) to maximize lexical overlap and anchor multilingual representations in a joint vocabulary (Goyal et al., 2021).

Principal Pretraining Objectives

Multilingual Masked Language Modeling (MLM): Randomly mask 15% of tokens and require the model to predict them from context. This induces shared latent structure across languages encountered in training, especially when subword units are shared (Lample et al., 2019, Goyal et al., 2021).
Translation Language Modeling (TLM): Concatenate aligned parallel sentences from different languages, mask tokens jointly, and permit cross-attention between source and target side. TLM directly aligns latent space such that context from either language aids masked token prediction (Lample et al., 2019).
Span Corruption (UL2, mT5-style): Mask contiguous spans; the objective mimics both MLM and sequence-to-sequence behaviors crucial for text generation and translation (Schioppa et al., 2023).
Supervised MT Cross-Entropy: When parallel data are available, standard source-to-target cross-entropy allows the model to learn explicit mappings between languages, further tightening multilingual alignment (Schioppa et al., 2023).

Hybrid approaches dynamically tune the mixture of MLM and supervised MT objectives (e.g., via multi-armed-bandit curriculum) to maximize cross-lingual generalization (Schioppa et al., 2023).

2. Model Architectures and Scaling

The architectural backbone of contemporary XLMs is the Transformer encoder, often in extremely large configurations:

Model	Layers	Hidden Size	Attention Heads	Params	Input Tokenization	Pretraining Data
XLM-R-base	12	768	12	270M	SentencePiece (250k)	CC100, 100 languages
XLM-R-XL	36	2560	32	~3.5B	SentencePiece (250k)	CC100, 100 languages
XLM-R-XXL	48	4096	32	~10.7B	SentencePiece (250k)	CC100, 100 languages
mT5-XL	24	2048	variable	3.8B	SentencePiece	mC4, 101 languages

Scaling model size substantially improves both high- and low-resource language performance, with larger models achieving +2.2% accuracy in zero-shot XNLI compared to smaller predecessors and closing the gap to monolingual baselines despite covering nearly 100 languages (Goyal et al., 2021). Encoder-only architectures (XLM-R, mBERT) excel at representation learning and NLU, while encoder–decoder models (mT5) produce more universal semantic embedding spaces, aiding cross-lingual generation and translation (Wen-Yi et al., 2023).

3. Mechanisms of Cross-Lingual Alignment

XLMs force all languages through a shared parameterization of the encoder; this architectural constraint naturally induces alignment of representations, especially for words and contexts playing analogous roles across languages. Empirical analyses show that:

After mean-centering, representations for 88 languages in XLM-R occupy nearly identical affine subspaces, with language differences encoded along a handful of orthogonal “language-sensitive” axes; most variability is “language-neutral” and encodes token position or part-of-speech (Chang et al., 2022).
Token and subword embedding layers reflect the model's cross-lingual geometry. In XLM-R, embeddings cluster perfectly by script, enabling 99.2% linear separability between scripts; in mT5, token embeddings intermix across languages, and nearest neighbors often correspond to translations, indicating a more collapsed and semantically aligned space (Wen-Yi et al., 2023).

Objective-Driven Alignment

MLM alone, via parameter sharing and a shared vocabulary, is sufficient for strong cross-lingual transfer on structurally similar languages, as it promotes shared composition structures and order-invariant meaning representations (Chai et al., 2022).
TLM and Supervised MT further reinforce cross-language alignment, making zero-shot transfer robust even on more distant language pairs and for token-level tasks (Lample et al., 2019, Schioppa et al., 2023, Chi et al., 2021).
Explicit Token-Level Alignment: Augmenting pretraining by self-labeled word alignments via optimal transport and denoising pointer networks further improves parameter-level mapping across languages and yields strong gains for structured prediction and QA (Chi et al., 2021).

4. Empirical Performance and Transferability

XLMs consistently set state-of-the-art results in zero-shot and few-shot cross-lingual transfer settings on NLU, sequence labelling, and NLG tasks:

On XNLI (cross-lingual NLI), XLM-R-XXL achieves 83.1% average zero-shot accuracy, outperforming previous models by +2.2%; multilingual fine-tuning pushes this to 86% (Goyal et al., 2021).
On cross-lingual QA (MLQA, XQuAD), large XLM variants offer >4 point improvements in F1 over earlier models and match or surpass monolingual RoBERTa on English (Goyal et al., 2021).
On entity-centric sequence labeling and slot-filling, intermediate code-switching training (EntityCS) using only English Wikipedia and Wikidata yields +2.8 F1 on NER and a 48% relative increase in fact retrieval accuracy, particularly benefiting non-Latin scripts (Whitehouse et al., 2022).
Language/parameter-specific distillation (BiStil) compresses large XLMs into fast, bilingual models while maintaining <2 point transfer loss and outperforming subspace-minimized multilingual distilled models (Ansell et al., 2023).

Notably, performance drop in zero-shot transfer varies by downstream task. Semantic textual similarity and NLI transfer robustly, sentiment analysis somewhat less, and complex MRC tasks the least—a complexity-dependent decay is reliably observed (Choi et al., 2021). Discourse-level phenomena (ordering, co-reference, argument structure) suffer the largest cross-lingual degradation, highlighting persistent limitations (Kurfalı et al., 2021).

5. Extensions and Model Adaptation

Advanced XLMs exploit several techniques to bridge gaps between pretraining and end-task demands or improve transfer properties:

Meta-Pretraining: A monolingual phase first optimizes generalization, followed by multilingual MLM to promote cross-language alignment, explicitly disentangling the learning of generalization and transfer capabilities (Chi et al., 2021).
Domain Adaptation: Mutual-information–based decomposition (UFD) splits model representations into domain-invariant and domain-specific components, retaining state-of-the-art transfer after unsupervised adaptation to new input domains purely from raw source-language text (Li et al., 2020).
Objective Bridging: For extractive tasks (e.g., QA/slot labeling), span-level masking aligned with cross-lingual links (CLISM) and contrastive regularization (CACR) address "objective gaps" between local (MLM) and global (span reasoning) objectives, improving both zero- and few-shot performance with orders of magnitude less pretraining data (Chen et al., 2022).
Sub-Network Similarity for Source Selection: Analysis of parameter-sensitivity (Fisher Information) allows prediction of which source languages are optimal for zero-shot transfer to a given target based solely on a small raw corpus; Jaccard similarity of activated sub-networks reliably predicts transfer gains (Yun et al., 2023).

6. Challenges, Analyses, and Design Guidelines

Emergence and behavior of cross-lingual capabilities within XLMs follow nontrivial dynamics:

Monolingual linguistic skills are rapidly acquired in upper layers early in pretraining, but cross-lingual alignment emerges later and exhibits more variance across tasks and language pairs (Blevins et al., 2022).
During pretraining, best-performing layers for transfer shift downward; final layer performance often degrades due to overspecialization, while robust cross-lingual information persists in mid-layers.
Cross-lingual transfer is weakly correlated with language relatedness and data size, but significant asymmetries and “forgetting” effects remain unexplained (Blevins et al., 2022, Kurfalı et al., 2021).
The most critical factor for transfer is not surface order or word co-occurrence, but recursive, order-invariant composition structures—shared composition enables accurate cross-lingual bootstrapping in MLM-trained models; disrupting composition sharply impairs transfer (Chai et al., 2022).

Practical design guidelines include shared subword vocabularies, dynamic data sampling to upweight low-resource languages, judicious use of cross-lingual alignment objectives (TLM, DWA), and task-specific pretraining adaptation. For model selection, internal representation statistics (e.g., mean [CLS] vectors on pivot or target-language text) significantly improve cross-lingual performance over held-out evaluation in the source language (Chen et al., 2020). Layer and architecture selection should reflect task characteristics: encoder-only XLMs for discrimination and clustering, encoder–decoder XLMs (e.g., mT5) for universal subword alignment and transfer in generative settings (Wen-Yi et al., 2023).

7. Ongoing Directions and Open Issues

Although scaling up parameter count and multilingual depth yields strong empirical improvements, several critical directions are under active pursuit:

Data saturation: Gains from model scaling may plateau absent more diverse or higher-quality pretraining corpora, suggesting that simply increasing depth/width yields diminishing returns without novel data or objectives (Goyal et al., 2021).
Compression and Adaptation: Efficient model distillation (BiStil) and entity-level code-switching (EntityCS) target practical deployment and fine-grained transfer, essential for real-world adoption (Ansell et al., 2023, Whitehouse et al., 2022).
Fine-grained Alignment: Tasks such as fact retrieval and WSD particularly benefit from explicit code-switched data and entity-centric masking; future models may incorporate morphological adaptation and multiword expression coverage (Whitehouse et al., 2022).
Bridging Pretrain–Finetune Gaps: Task-adaptive span objectives (CLISM), improved contrastive regularization, and careful construction of few-shot recipes remain crucial for high performance with minimal supervision (Chen et al., 2022).
Cross-lingual Instruction Tuning: Systematic exploitation of instruction data, scaling laws, and resource allocation for semantic alignment continue to close the gap between English-dominant LMs and their non-English performance (Zhu et al., 2023).
Discourse and Beyond: Persistent gaps in cross-lingual discourse phenomena suggest new objectives and alignment techniques remain necessary (Kurfalı et al., 2021).

Cross-lingual LLMs now anchor the modern landscape of multilingual NLP, offering robust, extensible architectures for zero-shot and pivot learning across typologically diverse languages. Methodological advances in pretraining, alignment, tuning, and efficient adaptation promise continued expansion of XLMs' utility—simultaneously deepening linguistic theory and practical global deployment.